[
  "Lecture 03\nShallow Networks\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "\u2022 Univariate regression problem (one output, real value)\n\u2022 Fully connected network\nRecap: Regression\n2",
  "Recap: 1D Linear regression loss function\n<latexit sha1_base64=\"/hMh896NPSehdG/Bg07J5FjBuSo=\">AW9HiclZhbc9Q2FI\nA3lLY0LTS07z0xdMHdpCJsvQywszkBAgJDQJuUKc7Mhe2Ssiy4vyQaP/0nfOn3t/+lLf0uPbO8Kn6M8dGfCivN9uh1JtdeIkWLy39M3Pto+sf/Lpjc9mP/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVb",
  "/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hAZz40j1wuSkTh2vn/kuFkRDUrxqF+dlGvVXTfy4nEZVEdjCFb3XC+Ww+wSgtLVNe5f6vAPJw9cd\n9ZSGZRBuVT91BT6Vd3ItM5gbmFpcan+OLTQbwsLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+Oyz",
  "sLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+OyzpFlXMHIkMniFP4U7lTRz+sUbIo01MDM2L5KMNM\nB23sqMiD345LoZIi58pvOgoK6eSxo/PtDEXK/VxeQoH5qYCxOv6IpczPYVmXcUv/DiKmBqW7vLqdlW6Hg+FKvlZUa9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0q",
  "9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0qfPjBU4fUdiREnDZbCjYcM7rijStch5CTjraW6JBIZF83LFWiAVLGXWUHVAc546jAc9TWAUYKnxtAY7CVPVpF7Ox3kalZmO4R5SpkJedwFT9pnUM+oaqpASqvod63d\nsvWbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK",
  "WbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK4KhVBNuZWGnvdvhMdwXtznMB56XqrJUn/OUMZ0QE4fpbMOXzr4ST21nkpz2tcFPnZGsF\njdKiwNm2lNOoFZtbGKmnWukEmzBaE0vuiaejQWlSeiO0EdwIeuSIUKPtDu1SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4",
  "SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4E6sWLJVzf0dKxF\nG9sHanXDgpCMSnyS3T8Rai6deoIHmwcobFCQLcL30wotMhB0JV1QMvwDfdeywby0ST9Zo6+jLMi5eTih/YzRGpdXxZToW9W3Quq1EL3usHltBaU4eZwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/G",
  "Zwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/GywjtcjJBZ1JGoLHnasbUliWfqDtqb9cORlesnP5KtHVpcuylJu+0o7bFvWIE/GzDMtoN4hGLOhK1Y6QesSy9Adt2fO4YZuFxbWb\nkrQ7yaPVtrhTE23/YHfEc6Yfk8wTbRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu",
  "bRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu2cawumFRN+wqk8kImU0Ii89ZhGfdhLAYUjG0iqcsSZDYhEg\neRziPI5rHBEuJTcIrklhWhGwp24ZKR3FX0gEsjVFvY0tnMAIZK9RhG8RyRndeZt15Cu1iRXfxnq3jvSs6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4",
  "6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4YekHpgaEHlKaGkl8EXvDaUPLrxAvODT2ndN/QfUoLQwtK9wzdozQwNKD0maHPKPUN9SldMXSF0txQ8kQKdwRDdykdGTqi9NDQ0rfGPqG0heGvqD0raFvKX1v6HtKnxj6hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8M",
  "hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8MJY83cGM0VFK6ZugapcJQ8vNC14Z+orSyNCI0peGvqT0naHvKH1u6HNKQ0PJuwF4OjF0h1LzFqjMKN02d\nJvSM0P7O8F+HQZPdvG3DQNbFIaGxpTum4o+aUAjxKGnpLnyUC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfz",
  "UC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfzC308VtYWth/sNj/ZfHh9sOFx8vtG\n9obvW973/Xu9vq9X3uPey96W729nt/7d+b6zM2ZW/Pn83/M/zn/V6Nem2nrfN3rfOb/g9NB/xE</latexit>\nL[\u03c6] =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\nLoss function:\n\u201cLeast squares loss function\u201d\n3",
  "Recap: 1D Linear regression training\nThis technique is known as gradient descent\n4",
  "Shallow neural networks\n\u2022 1D regression model is obviously limited\n\u2022 Want to be able to describe input/output that are not lines\n\u2022 Want multiple inputs\n\u2022 Want multiple outputs\n\u2022 Shallow neural networks \n\u2022 Flexible enough to describe arbitrarily complex input/output mappings\n\u2022 Can have as many inputs as we want\n\u2022 Can have as many outputs as we want\n5",
  "This lecture we\u2019ll cover\u2026\n\u2022 Example network, 1 input, 1 output\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n6",
  "1D Linear Regression\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sq",
  "PjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiS",
  "k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4",
  "mc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8j",
  "l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEq",
  "Jal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "qFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5",
  "Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFo",
  "6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FBy",
  "8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATid",
  "GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t",
  "FK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\n<latexi\nt sha1_base64=\n\"ktj2B8/mNFNy1\nioPnb0sW2wRwtY\n=\">AWq3iclZj\nZcts2FECZrqm7O\ne",
  "e64=\n\"ktj2B8/mNFNy1\nioPnb0sW2wRwtY\n=\">AWq3iclZj\nZcts2FECZrqm7O\ne3UL3h1JNOp0\n1UqZMuL51J7Dib\nnVqOLdux5WhACq\nQgyDNxZbC0Wf0\na/rafkT/phckJY\nT3wg/VjCPknkMs\nFwAJ0UukyPJu9\n98b7z73vsfHj\nzo5WP/n0s89Xb\n31xmMVF6vOBH8s\n4PfZYxqVQfJCLX\nPLjJOUs8iQ/8s4\n3NT+65GkmYnWQ\nzxJ+Fr",
  "1xmMVF6vOBH8s\n4PfZYxqVQfJCLX\nPLjJOUs8iQ/8s4\n3NT+65GkmYnWQ\nzxJ+FrFQiUD4LI\nfQaPXHmfvt7+4w\n8uJpGcxPp3eGXp\nBMxNlwuKLjUByV\n3fkPdaE3n45W17\nudbvVxaHXFNa\nd5tMf3fpqPBzHf\nhFxlfuSZdlpr5v\nkZyVLc+FLPl8ZF\nhlPmH/OQn4KRcU\ninp2V1cjm7m2Ij\nN0gTuFP5W4Vf\nuKkVZNos8MCOW\nTzLMdNDG",
  "lPmH/OQn4KRcU\ninp2V1cjm7m2Ij\nN0gTuFP5W4Vf\nuKkVZNos8MCOW\nTzLMdNDGTos8+O\n2sFCopcq78uqGg\nkG4euzpN7lik3M\n/lDArMTwX01fUn\nLGV+DslcGSp+5\ncdRxNS4HG5s7c3\nLocdDoUp+UVSJn\nc/bzlblcCheZ2w\n8PVjWInIeiTecV\nFIpupJrB7Oy5J\n3wg4GgMQHU5A\nrHgGder8eIHbQx\nQWkgRc1sDVob7\nYk6qVjkPIS",
  "pupJrB7Oy5J\n3wg4GgMQHU5A\nrHgGder8eIHbQx\nQWkgRc1sDVob7\nYk6qVjkPISct7Y\nRoUEgkn7asTWLB\nVEYtZR8U173tas\nDzFGYBugpfHM3B\nfsLUfHFdzqd5G\npWZjuEWUqZCXjU\nBQ/aZ1CNqG6qQE\ni71W9Yf2HrB1Hm\nTuDipuprqCLIO0\nraTpzQvatx2qgi\nyYBGbauKIEvC\nth+ziEGWm/IBh\ny5OmJXhcKqIAuz\nn8Zeu+1ER/D",
  "zQvatx2qgi\nyYBGbauKIEvC\nth+ziEGWm/IBh\ny5OmJXhcKqIAuz\nn8Zeu+1ER/DanC\nawX9reVknSf8lQ\nRnQAdp/+Fkz5vK\n1vxkvbXSTnsvJ\n1gU/dCUxW+xKWh\nvWwFo3AqJrYnJp\nVrpBJswWhNL5qm\n7o3FpUnoj1AHcC\nbrkiFCt7S7lQlW\nLI6PLwDQ0LyU\n/vdn7m07Oyq7eN\n/odkEyrKisRWkQ\n7/j4rG8KDB6wsi\nePJiSYPAtXk",
  "wDQ0LyU\n/vdn7m07Oyq7eN\n/odkEyrKisRWkQ\n7/j4rG8KDB6wsi\nePJiSYPAtXkxR\nLu72jqWIoXto5U\ncwcFoZgU+Qxtf\nxGq9jVBHc2jlB\nfIaDrhW8mFJrkI\nGjLOqBl+IZHpmU\nB+WiQfj1GX8Zk\nXJy80PrGSKVrm+\nLqdAPq/YNVWqh\nfd/gcnkVlOHhcM\nmvudxDGfXqfHpx\nocYsRcmc6imdvh\npmOWwx2+6vprwu\nWq2QX2w37UG/Y",
  "OHhcM\nmvudxDGfXqfHpx\nocYsRcmc6imdvh\npmOWwx2+6vprwu\nWq2QX2w37UG/YH\nYK3+cXo208HyG\nxqCNRXBGsdYli\nWVpD+paLte3e1Z\nuv/qeLO3Q4tpNS\neptem3Le41PeA\nXO5be7hCPWNSRq\nK6mh9QjlqU9qM\nuex3bKCyu3ZSk\n3kUerbFXZpo+Q\ncHE54zfUyK5Vgf\n+2I5rENYzKmYW8\nU4iES6xAWo6Jt\nwf+xsi/g4dG26\nhAW",
  "+Q\ncHE54zfUyK5Vgf\n+2I5rENYzKmYW8\nU4iES6xAWo6Jt\nwf+xsi/g4dG26\nhAW+5loazqApTG\nXeAh1CIv1Fm6bT\nQyrOxZ1x64ymUy\nQWYew+JhFeNR1C\nIshFUOreM6SBIl\n1iORxgvM4oXlM\nsJTYJDwjiWVGyJ\nKyLah0ErclHcDS\nFLU2tTQGPZCxQg\n02QSxndOVl1pWn\n0CpWdBUPbA0Prm\nk4Z6hCHcDSLtl\nj7nDXusk8nGI4Z\ntm",
  "Qg\n02QSxndOVl1pWn\n0CpWdBUPbA0Prm\nk4Z6hCHcDSLtl\nj7nDXusk8nGI4Z\ntmSnAhkJTSBfez\n0qbM4/XlBSU5yX\njAzdEbplaFXlB4\nZekRpaij5ReAFL\nwlv0684NLQS0\noPDT2ktDC0oHRg\n6IDSwNCA0keGPq\nLUN9SndNPQTUpz\nQ8mJFJ4Ih5QOj\nF0QumxoceUvjT0\nJaVPDH1C6YmhJ\n5S+MfQNpQ8MfUA\npM5RumXoFqXcU\nPLqw",
  "j\nF0QumxoceUvjT0\nJaVPDH1C6YmhJ\n5S+MfQNpQ8MfUA\npM5RumXoFqXcU\nPLqwAs2DN2g1DO\nU/PaDvWZon9LE0\nITSh4Y+pHRsKPl\nVDM8zQ8nxBh6M\nhkpKnxr6lFJhKP\nn95gXPDX1OaWRo\nROkzQ59R+trQ15\nQ+NvQxpaGh5N0A\nnE4M3afUvAUqM0\nr3DN2j9MLQC/t\n7Ab6cRs+2MHdNB\nbuUxobGlG4bSn4\npwFHC0HNyngxUc\n1db",
  "0\nr3DN2j9MLQC/t\n7Ab6cRs+2MHdNB\nbuUxobGlG4bSn4\npwFHC0HNyngxUc\n1dbvG0i97VALbm\nFNRlfXE1yHqglt\n7Dm7rS4mtyfAr\nXkE9L1rcPlixRI\nKdzpR6vrPfwWlh\nYOf+r0func27u3\nfn+jeUN70/na+c\nb5zuk5vzr3nSdO\n3xk4vOn85fzt\n/P2t21/bWTtWG\ntvnOjueZLp/VZ4\n/8B0OTgog=</l\natexit>y = f[x,",
  "vOn85fzt\n/P2t21/bWTtWG\ntvnOjueZLp/VZ4\n/8B0OTgog=</l\natexit>y = f[x, \u03c6]\n= \u03c60 + \u03c61x\nExample shallow network\n7",
  "Example shallow network\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr",
  "jR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj",
  "3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh",
  "c\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8ja",
  "1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqw",
  "al/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "FJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5t",
  "am36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFob",
  "hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByI",
  "NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidG",
  "A2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nz",
  "K\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\n8",
  "Example shallow network\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr",
  "jR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj",
  "3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh",
  "c\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8ja",
  "1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqw",
  "al/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "FJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5t",
  "am36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFob",
  "hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByI",
  "NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidG",
  "A2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nz",
  "K\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\nActivation function\n9",
  "Example shallow network\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr",
  "jR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj",
  "3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh",
  "c\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8ja",
  "1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqw",
  "al/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "FJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5t",
  "am36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFob",
  "hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByI",
  "NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidG",
  "A2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nz",
  "K\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\n<latexit s\nha1_base64=\"y0cLbs\nhKv4qJCV4w9psY+07H\nPg=\">AW2niclZhJc\n9xEFICVsAWzOVD4wkW",
  "4=\"y0cLbs\nhKv4qJCV4w9psY+07H\nPg=\">AW2niclZhJc\n9xEFICVsAWzOVD4wkWF\nKxRFwdSYCsBqhI7zm\nYHj5exnViOq6VpaTput\nWQt9tiquXCjuPKT+BP\n8Ba7wA3gtaj9oHpi\npR+32fendrc1PpciL\nfv+vGzfePOt9+59e7\nCe+9/8OFHi7c/3s+TM\ngv4MEhkh36LOdSKD4s\nRCH5YZpxFvuSH/ina5\nofnPMsF4naKy5TfhyzS\nIlQ",
  "TM\ngv4MEhkh36LOdSKD4s\nRCH5YZpxFvuSH/ina5\nofnPMsF4naKy5TfhyzS\nIlQBKyA0MniSy/2k0n\nFpkdXx+7PbvPXDt8czg\nI+j4SqAmgin7p9wvXO\nyvZyL1yf+q7ngfHecS\nL+Jnb97gatXpvYeFkcb\nnf69c/lxZW2sKy0/4G\nJ7c/HXmjJChjropAsjw\n/WumnxXHFskIEk8Xv\nDLnKQtOWcSPoKhYzPj\nqk7E1L0DkZEbJhn8U4\nVbR",
  "Asjw\n/WumnxXHFskIEk8Xv\nDLnKQtOWcSPoKhYzPj\nqk7E1L0DkZEbJhn8U4\nVbR18/o2Jxnl/GPpgxK\n8Y5ZjpoY0dlEf54XAm\nVlgVXQdNQWEq3SFydVX\nckMh4U8hIKLMgE9NUN\nxixjQG5X/AUvwiSOGa\nQGm91fXtatWnlkDk9D\n9Np1mvHZ3J64zVJ3vz\nWkTBY3HFSW1oiu5Ru\nDRtKp4L+phIDgA0eMEJ\nAqmtfJ0fvzQXUEU1p0\nEXD",
  "3vz\nWkTBY3HFSW1oiu5Ru\nDRtKp4L+phIDgA0eMEJ\nAqmtfJ0fvzQXUEU1p0\nEXDWLyANjZ0qVgWPIC\ncd7QXRoJBKPulYa8SC\nqYw7yi4ornvH1YAXGcw\nCdBUOHM3BbsrUdHZew\nSdFle5juEWMqYiXjcB\nQw6Y1CPqGqUEk4NOt\nYv2Nph6rRNXJLWXc10B\nFl7WdcpMpoXNeo6dQR\nZsAijrlVHkCXhKjFiMY\nMst+UTGHDs6ohdFQqr\ngiz",
  "0B\nFl7WdcpMpoXNeo6dQR\nZsAijrlVHkCXhKjFiMY\nMst+UTGHDs6ohdFQqr\ngizMQZb43bZTHcFrc5L\nCful6xVJ/zlDGdEB2\nH36KJgKeFdfS+a2O0vO\ne3rAp+4Y5is7iksi5p\nhzRqBUbWxKTXrXCGTZ\ngtCWXLRNXVvLCpPRXeA\nOoA3XZkJFb6mfV2XYM\nnqsPc1DUrJT/6pvcdn\nxXfb1t9H8km1BRXqa\n2inT4f1Q0gvsSXl8QwZ\nOXS",
  "XYM\nnqsPc1DUrJT/6pvcdn\nxXfb1t9H8km1BRXqa\n2inT4f1Q0gvsSXl8QwZ\nOXSDR5EKgnL5FwfUdT\nxzK8sHWknjsoCMWkKC7\nR9heR6p5TR3Bnkxj1F\nQK6XjgyodAkh2FX1gEt\nwxHusJYFKBs0YA5\nnkZcbJxQ+tZ4jUur4sZ\nkLfrLoXVKmF7nWDy/l\nZUIabwzm/5nQfZdRv8u\nknpRqxDCVzoqd08tL\nC9hit1fT3lTtFpw9\no24N+we",
  "l\nZUIabwzm/5nQfZdRv8u\nknpRqxDCVzoqd08tL\nC9hit1fT3lTtFpw9\no24N+weyUQcDPTjbwf\nETEo5EdcEjbUuSxL\ne1DXfLm+3rNq4+VXZG\nlHFtduSlJv20u7bXGv6\nQE/27T0dpN4xKORHW\n1PaQesSztQV32PG7aRm\nFx7aYk9c7yaLUt7txE\nyz/cG/OC6cekRI70Y18\nivSaExYKhVMYh4hs\nQlhMS67FvyNlV0BN4+u\n1YSwOMhFV9",
  "cG/OC6cekRI70Y18\nivSaExYKhVMYh4hs\nQlhMS67FvyNlV0BN4+u\n1YSwOMhFV9MBLI24xE\nNoQlhstnDXbGNY3bSom\n3aVyXSMzCaExUcsxqN\nuQliMqBhZxVOWpkhsQi\nSPY5zHMc1jiqXUJuEZ\nS0zQpaUbUFl46Qr6QC\nWJqi1iaUx6IFMFGqwDW\nI5pysvt648hVaxoqt4\naGt4eE3DBUMV6gCWtsg\nec70t6ybzcYr16oly\nalAVkoTOM",
  "pysvt648hVaxoqt4\naGt4eE3DBUMV6gCWtsg\nec70t6ybzcYr16oly\nalAVkoTOMDOgDqzpz8/\nrMiTnB9eGnpJ6YWhF5\nQeGHpAaWYoeSPwx1Dy\nduJH54bek7pvqH7lJa\nGlpQODR1SGhoaUvrQ0I\neUBoYGlK4ZukZpYSh5\nIoU7gqF7lI4NHVN6aOg\nhpc8NfU7pY0MfU/rC0\nBeUXhl6Rel9Q+9Tygxl\nlK4buk4pN5R8OvDVU\nNXKfUNJe",
  "pc8NfU7pY0MfU/rC0\nBeUXhl6Rel9Q+9Tygxl\nlK4buk4pN5R8OvDVU\nNXKfUNJe9+sNcMHVCaG\npS+sDQB5SODCVvxXA\n/M5Q83sCN0VBJ6RNDn1\nAqDCXvb374zNBnlMaG\nxpQ+NfQpa8MfUXpI0M\nfURoZSr4NwNOJobuUm\nq9AVU7ptqHblJ4Zemb/\nLsDn0+jbFuaWqWCL0s\nTQhNINQ8mbAjxKGHpKn\nidD1V7VZl+byHUtVHN\nuYW3GZ2",
  "sDn0+jbFuaWqWCL0s\nTQhNINQ8mbAjxKGHpKn\nidD1V7VZl+byHUtVHN\nuYW3GZ2eTnIdqzi2svT\nrNzibXp1DN+Zh0fX1/\n/iEFUgpX+pPF5RX8FZY\nW9r/trXzfu7t9d/nea\nvuF9pbzmfO586Wz4vzg\n3HMeOwNn6ATOn87fzj\n/Ov0ve0q9Lvy393qg3b\n7TnfOJ0fkt/Acq/Gu\n</latexit>\na[z] = ReLU[z] =\n(\n0\nz < 0\nz\nz \u2265 0 .",
  "y393qg3b\n7TnfOJ0fkt/Acq/Gu\n</latexit>\na[z] = ReLU[z] =\n(\n0\nz < 0\nz\nz \u2265 0 .\nRectified Linear Unit\n(one type of activation function)\nActivation function\n10",
  "Example shallow network\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr",
  "jR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj",
  "3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh",
  "c\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8ja",
  "1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqw",
  "al/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "FJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5t",
  "am36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFob",
  "hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByI",
  "NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidG",
  "A2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nz",
  "K\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\n<latexit s\nha1_base64=\"y0cLbs\nhKv4qJCV4w9psY+07H\nPg=\">AW2niclZhJc\n9xEFICVsAWzOVD4wkW",
  "4=\"y0cLbs\nhKv4qJCV4w9psY+07H\nPg=\">AW2niclZhJc\n9xEFICVsAWzOVD4wkWF\nKxRFwdSYCsBqhI7zm\nYHj5exnViOq6VpaTput\nWQt9tiquXCjuPKT+BP\n8Ba7wA3gtaj9oHpi\npR+32fendrc1PpciL\nfv+vGzfePOt9+59e7\nCe+9/8OFHi7c/3s+TM\ngv4MEhkh36LOdSKD4s\nRCH5YZpxFvuSH/ina5\nofnPMsF4naKy5TfhyzS\nIlQ",
  "TM\ngv4MEhkh36LOdSKD4s\nRCH5YZpxFvuSH/ina5\nofnPMsF4naKy5TfhyzS\nIlQBKyA0MniSy/2k0n\nFpkdXx+7PbvPXDt8czg\nI+j4SqAmgin7p9wvXO\nyvZyL1yf+q7ngfHecS\nL+Jnb97gatXpvYeFkcb\nnf69c/lxZW2sKy0/4G\nJ7c/HXmjJChjropAsjw\n/WumnxXHFskIEk8Xv\nDLnKQtOWcSPoKhYzPj\nqk7E1L0DkZEbJhn8U4\nVbR",
  "Asjw\n/WumnxXHFskIEk8Xv\nDLnKQtOWcSPoKhYzPj\nqk7E1L0DkZEbJhn8U4\nVbR18/o2Jxnl/GPpgxK\n8Y5ZjpoY0dlEf54XAm\nVlgVXQdNQWEq3SFydVX\nckMh4U8hIKLMgE9NUN\nxixjQG5X/AUvwiSOGa\nQGm91fXtatWnlkDk9D\n9Np1mvHZ3J64zVJ3vz\nWkTBY3HFSW1oiu5Ru\nDRtKp4L+phIDgA0eMEJ\nAqmtfJ0fvzQXUEU1p0\nEXD",
  "3vz\nWkTBY3HFSW1oiu5Ru\nDRtKp4L+phIDgA0eMEJ\nAqmtfJ0fvzQXUEU1p0\nEXDWLyANjZ0qVgWPIC\ncd7QXRoJBKPulYa8SC\nqYw7yi4ornvH1YAXGcw\nCdBUOHM3BbsrUdHZew\nSdFle5juEWMqYiXjcB\nQw6Y1CPqGqUEk4NOt\nYv2Nph6rRNXJLWXc10B\nFl7WdcpMpoXNeo6dQR\nZsAijrlVHkCXhKjFiMY\nMst+UTGHDs6ohdFQqr\ngiz",
  "0B\nFl7WdcpMpoXNeo6dQR\nZsAijrlVHkCXhKjFiMY\nMst+UTGHDs6ohdFQqr\ngizMQZb43bZTHcFrc5L\nCful6xVJ/zlDGdEB2\nH36KJgKeFdfS+a2O0vO\ne3rAp+4Y5is7iksi5p\nhzRqBUbWxKTXrXCGTZ\ngtCWXLRNXVvLCpPRXeA\nOoA3XZkJFb6mfV2XYM\nnqsPc1DUrJT/6pvcdn\nxXfb1t9H8km1BRXqa\n2inT4f1Q0gvsSXl8QwZ\nOXS",
  "XYM\nnqsPc1DUrJT/6pvcdn\nxXfb1t9H8km1BRXqa\n2inT4f1Q0gvsSXl8QwZ\nOXSDR5EKgnL5FwfUdT\nxzK8sHWknjsoCMWkKC7\nR9heR6p5TR3Bnkxj1F\nQK6XjgyodAkh2FX1gEt\nwxHusJYFKBs0YA5\nnkZcbJxQ+tZ4jUur4sZ\nkLfrLoXVKmF7nWDy/l\nZUIabwzm/5nQfZdRv8u\nknpRqxDCVzoqd08tL\nC9hit1fT3lTtFpw9\no24N+we",
  "l\nZUIabwzm/5nQfZdRv8u\nknpRqxDCVzoqd08tL\nC9hit1fT3lTtFpw9\no24N+weyUQcDPTjbwf\nETEo5EdcEjbUuSxL\ne1DXfLm+3rNq4+VXZG\nlHFtduSlJv20u7bXGv6\nQE/27T0dpN4xKORHW\n1PaQesSztQV32PG7aRm\nFx7aYk9c7yaLUt7txE\nyz/cG/OC6cekRI70Y18\nivSaExYKhVMYh4hs\nQlhMS67FvyNlV0BN4+u\n1YSwOMhFV9",
  "cG/OC6cekRI70Y18\nivSaExYKhVMYh4hs\nQlhMS67FvyNlV0BN4+u\n1YSwOMhFV9MBLI24xE\nNoQlhstnDXbGNY3bSom\n3aVyXSMzCaExUcsxqN\nuQliMqBhZxVOWpkhsQi\nSPY5zHMc1jiqXUJuEZ\nS0zQpaUbUFl46Qr6QC\nWJqi1iaUx6IFMFGqwDW\nI5pysvt648hVaxoqt4\naGt4eE3DBUMV6gCWtsg\nec70t6ybzcYr16oly\nalAVkoTOM",
  "pysvt648hVaxoqt4\naGt4eE3DBUMV6gCWtsg\nec70t6ybzcYr16oly\nalAVkoTOMDOgDqzpz8/\nrMiTnB9eGnpJ6YWhF5\nQeGHpAaWYoeSPwx1Dy\nduJH54bek7pvqH7lJa\nGlpQODR1SGhoaUvrQ0I\neUBoYGlK4ZukZpYSh5\nIoU7gqF7lI4NHVN6aOg\nhpc8NfU7pY0MfU/rC0\nBeUXhl6Rel9Q+9Tygxl\nlK4buk4pN5R8OvDVU\nNXKfUNJe",
  "pc8NfU7pY0MfU/rC0\nBeUXhl6Rel9Q+9Tygxl\nlK4buk4pN5R8OvDVU\nNXKfUNJe9+sNcMHVCaG\npS+sDQB5SODCVvxXA\n/M5Q83sCN0VBJ6RNDn1\nAqDCXvb374zNBnlMaG\nxpQ+NfQpa8MfUXpI0M\nfURoZSr4NwNOJobuUm\nq9AVU7ptqHblJ4Zemb/\nLsDn0+jbFuaWqWCL0s\nTQhNINQ8mbAjxKGHpKn\nidD1V7VZl+byHUtVHN\nuYW3GZ2",
  "sDn0+jbFuaWqWCL0s\nTQhNINQ8mbAjxKGHpKn\nidD1V7VZl+byHUtVHN\nuYW3GZ2eTnIdqzi2svT\nrNzibXp1DN+Zh0fX1/\n/iEFUgpX+pPF5RX8FZY\nW9r/trXzfu7t9d/nea\nvuF9pbzmfO586Wz4vzg\n3HMeOwNn6ATOn87fzj\n/Ov0ve0q9Lvy393qg3b\n7TnfOJ0fkt/Acq/Gu\n</latexit>\na[z] = ReLU[z] =\n(\n0\nz < 0\nz\nz \u2265 0 .",
  "y393qg3b\n7TnfOJ0fkt/Acq/Gu\n</latexit>\na[z] = ReLU[z] =\n(\n0\nz < 0\nz\nz \u2265 0 .\nRectified Linear Unit\n(particular kind of activation function)\nActivation function\n11\nInactive\nregion\nActive\nregion",
  "Example shallow network\n<latexit\n sha1_base64=\"\nNTIf4aNnvrpa8D\nwPTFP8Igh9dpg=\"\n>AXIXiclZhb\n9s2FICd7talu6Q\nblpe9CAtaDGsW2G\nl3wYCbdL0lnRJ\nmsbpQYlUzIbil\nJ0SZwK/jXDfszeh\nr0N+zM7lGQzOod\n5mIHU7Pk+8XJIS\nrS8RIos73b/mbn2\n3vsfPjR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr",
  "jR9Y9nb3\nzy6Wefz938Yj+L\ni9Tne34s4/TQYxm\nXQvG9XOSHyYpZ\n5En+YF3sqr5wRl\nPMxGr3fwi4cRC5\nUIhM9yCPXn/ri4\nfd9xIy8elcH4aL\nToekEyFMeuO3v7v\ngulfte541SFXqO\nx8ZGbD3nO+mWvO\n74zLfGo+OJu0z\nd5Uvucu1W6l2q3r\n2k3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj",
  "3tXqbH9uobvU\nrT4OLfSawkKn+W\nz1b341cAexX0Rc5\nb5kWXbU6yb5cn\nSXPiSj2fdIuMJ8\n09YyI+gqFjEs+Oy\nyufYuQWRgRPEKf\nyp3Kmil68oWZRl\nF5EHZsTyYaZDtr\nYUZEHPx+XQiVFz\npVfNxQU0sljR0+\nOMxAp93N5AQXmpw\nL6vhDljI/hymc\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh",
  "c\ndRU/9+MoYmpQui\ntr2+PS9XgoVMlPi\n2o6x+O2s1Y5HIp\nXGSvPdqe1iJxH4\nh0nlVSKruQKgYfj\nsuRL4RIGgMQS5\nyAWPEM6tT58QKn\nhygsXwm4rNcFLEj\nn5ZhUrXIeQk5a2\nmuiQSGRfNSyVok\nFUxm1lB1QHOeWow\nHPU5gF6Cp8cTQH\nOwlT48l1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8ja",
  "1OR/laV\nRmOoZbSJkKedUED\nNlnUo+obahCSrj\nUb1m/YeslUydN4\nuKk6mqI8jaTdtO\nntK8qEHbqSLIgk\nUYtq0qgiwJN5sB\nixhkuSn3YcCRoyN\n2VSisCrIwt9LYa\n7ed6Ahem6ME9kv\nbWytJ+s8YyogOwO\n7T34Ipn7f1Xhq\nO5PknFW+LvCRM4\nTJal/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqw",
  "al/C0rAe1qQRG\nFUTG1OzyhUyabY\nglMbnbVP3xqLyR\nLQHqAN40xWpUME\nlbEqwZLVYXcRhp\noWkh9v/QDHx2X\nXb1t9D8km1BRVi\nS2inT4f1Q0gMcbX\nl8QwZMXSzR5EKg\nmL5Zwf0dTx1K8s\nHWkmjsoCMWkyC/Q\n9hehal9TRXBn4w\nj1FQK6XvhmQqFJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v",
  "FJ\nDoK2rANahm94UFs\nWkI8G6dj9GWcF\nSknNz+0niFS6fq\n2mAr9sGrfUKUW2v\ncNLqdXQRkeDmf8\nis9lFGvzqcXF2\nrAUpTMkZ7S0Rs3y\n2GL2XZ/NeV10Wq\nF/HS9aQ/6BbNT+\nD4/7a/j+QiJR2J\n6oKTkbUuSxLe1\nDXdLle7lm5/uY7\nsrRDi2s3Jam36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5t",
  "am36aX\ndtrhX9ICfblh6u\n0E8YlFHorqaHlK\nPWJb2oC57Hjdso7\nC4dlOSeid5tNoW\nd2qi5R/s6qOoPi\nbFcqCPfbF06xAWc\nyrmVjGOeIjEOoT\nFqGhb8H+s7Ah4e\nLStOoTFrUy0NR3A\n0oBLPIQ6hMV6C7\nfNJobVDYu6YVeZ\nTIbIrENYfMIiPOo\n6hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFob",
  "hMWQiqFVPGFJg\nsQ6RPI4xHkc0jw\nmWEpsEp6RxDIjZE\nnZFlQ6jNuSDmBp\nhFobWRqDHshYoQ\nabIJYzuvIy68pT\naBUruor3bA3vXdF\nwzlCFOoClTbLH\nHfTusk8nGI4Ztm\nSnAhkJTSBW9jZos\n7k9OcFJTnJecGF\noReUnht6TumBoQ\neUpoaSXwRe8NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByI",
  "NJQ8\nuvEC84MPaN039B\n9SgtDC0r3DN2jN\nDA0oPSxoY8p9Q31\nKV01dJXS3FByIo\nUngqG7lA4NHVJ6\naOghpa8MfUXpU0O\nfUvra0NeUvjP0H\naUPDX1IKTOUbp\nm6Bql3FDy6sALVg\nxdodQzlPz2g71m\n6BaliaEJpY8MfU\nTpwFDyqxieZ4aS4\nw08GA2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidG",
  "A2VlD4z9Bm\nlwlDy+80LXhj6g\ntLI0IjS54Y+p/St\noW8pfWLoE0pDQ8\nm7ATidGLpDqXkL\nVGaUbhu6Tempoaf\n29wJ8Oo2ebWFum\ngo2KY0NjSldN5T\n8UoCjhKEn5DwZqO\nauNnbRO5rgZpy\nC2syPrma5DxQU2\n5hzd1pcjW5PwVqy\noek62v70xcpkFK\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nz",
  "K\n40/fnFnr4LSwt7\nC8v9X5curd9b+HB\nSvOG9nrn6843nW\n87vc5PnQedp52t\nzl7Hn7kx05v5Zeb\nX+d/n/5z/a/7vW\nr0201zZaf1mf/\n3P5u9C70=</late\nxit>y = f[x, \u03c6]\n= \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x]\nThis model has 10 parameters:\n\u2022 Represents a family of functions\n\u2022 Parameters determine a particular function\n\u2022 Given the parameters, we can perform inference (evaluate the equation)\n\u2022 Given training dataset \n\u2022 Define loss function             (least squares)\n\u2022 Change parameters to minimize loss function\n<latexit sha1_ba",
  "we can perform inference (evaluate the equation)\n\u2022 Given training dataset \n\u2022 Define loss function             (least squares)\n\u2022 Change parameters to minimize loss function\n<latexit sha1_base64=\"y6CteYpwmjXgAuxju3n\nLWkvzGug=\">AW83iclZhbT9xGFICXpJeUJi1pV76YhVFqoU7ZL08lIpgZAbpEBgYQlaOwdeyeMx8Y\newxJrf0nfqr72B/Wh/6VnbO8OPmd46ErEk/N9nsuZsT2n0qR6273n7kbNz/6+JNPb302/ntO198uXD3q\n/08KbKA94NEJtmhz3I",
  "N9nsuZsT2n0qR6273n7kbNz/6+JNPb302/ntO198uXD3q\n/08KbKA94NEJtmhz3IuheJ9LbTkh2nGWexLfuCfrhl+cM6zXCRqT1+m/DhmkRKhCJiG0MnCxcAP05H4bV\nAO4HBSdif360JvWliZFh5AQY+4ZgC7V8o9W165El+5En9wJf6gNxlM5k8WlrL3ern0UKvKSx1mt/2yd1v\nhoNhEhQxVzqQLM+Pet1UH5cs0yKQfDI/KHKesuCURfwIio",
  "n0UKvKSx1mt/2yd1v\nhoNhEhQxVzqQLM+Pet1UH5cs0yKQfDI/KHKesuCURfwIiorFPD8uqwxNvHsQGXphksGf0l4VvXpGyeI8v4\nx9MGOmRzlmJuhiR4UOfz0uhUoLzVQNxQW0tOJZ9LtDUXGAy0vocCTEBfvWDEMhZomJT5geIXQRLHTA3\nLwer6zqQc+DwSquRnRTVBk0nbWa8cDsXrjNUXe7NahOax+MBJZViKrlG4NGkLPlytIyB4ADEMic",
  "wSquRnRTVBk0nbWa8cDsXrjNUXe7NahOax+MBJZViKrlG4NGkLPlytIyB4ADEMicgUTyHO\nk1+/NDrIQoLUgIG7idj6FzovZ6QqpXmEeSkpb0lGhRSycta41YMJVxS9kFxfPueQZwncEsQFfhwNEc7KZ\nMTabnaT7WVzmJoZbyJiKeNUEDlg0oyobahCSjg1aFm/Y+s1U6dN4pK06mpmIsjay9qOzmhe1LDtVBFk\nwSKM2lYVQZaE28eQxQy3JRP",
  "aFm/Y+s1U6dN4pK06mpmIsjay9qOzmhe1LDtVBFk\nwSKM2lYVQZaE28eQxQy3JRPYMCxZyJuVSisCrIwt7PEb7edmghem+MUrpe2t16S9J8zlBETgKvPHAVTAW\n/ra8nM9qbJOa98U+BjbwST1T6FZVE9rGkjMKomNqFmlStk0mxBKEsu2qbpjUPlqWgP0ATwRVdkQoVXtPtV\nCZasCQ/uw1CzQvKjH5d/4uPjsmsuG/MPySZUlBepqyIT/h8VDe",
  "wRVdkQoVXtPtV\nCZasCQ/uw1CzQvKjH5d/4uPjsmsuG/MPySZUlBepqyIT/h8VDeGBhdcXRPDkJRJNHgSqyUsk3N/R1LEML\n2wTqeYOCkIxKfQluvxFpNrnVBHc2SRGfYWAqReOTCg0yWHYlk3AyHCER69jAQVokE9xkAmeZFxcvND6xk\nilW5ui5kwD6v2DVUaoX3f4HJ2FpTh4XDOrzndRxn163z6SaGLEPJHJspHb8b5BouMdfVX015XRaET/",
  "VUaoX3f4HJ2FpTh4XDOrzndRxn163z6SaGLEPJHJspHb8b5BouMdfVX015XRaET/ba\nNqDfsHsFEHAz0428HxExKORHXBXsdZlySWoz2oa7Zcr/as3Hj3A1nakcN1m5LU2/TSbTvca3rAzYdvd\n0kHrGoI1FdTQ+pRyxHe1CXO4+brlE4XLcpSb3TPDpthzsz0fIP98xW1GyTEjk0275EDuoQFjUVtVNMYh4h\nsQ5hMS7aFvwfK7sCHh5tqw5hcT",
  "z0fIP98xW1GyTEjk0275EDuoQFjUVtVNMYh4h\nsQ5hMS7aFvwfK7sCHh5tqw5hcTsXbc0EsDTkEg+hDmGxvoTbZhPD6qZD3XSrTKYjZNYhLD5jMR51HcJiRM\nXIKZ6yNEViHSJ5HOE8jmgeUylLgnPSOqYEbKkXAsqGyVtyQSwNEatjR2NQ9kolCDTRDLOV15uXPlKbS\nKFV3FfVfD/Wsa1gxVaAJY2iLXmDfYcl5kPk4xbLNcSU4FslKawG3sbF",
  "5uXPlKbS\nKFV3FfVfD/Wsa1gxVaAJY2iLXmDfYcl5kPk4xbLNcSU4FslKawG3sbFNnuvzw5Ls5Pzw0tJLSi8svaD0w\nNIDSjNLyRuBH762lLyd+OG5peU7lu6T2lhaUFp39I+paGlIaVPLX1KaWBpQOmapWuUakvJjhSeCJbuUTq\nydETpoaWHlL6x9A2lzy19TulbS9S+sHSD5Q+tvQxpcxSRum6peuUckvJpwM/XLV0lVLfUvLuB9eapduU\npa",
  "19TulbS9S+sHSD5Q+tvQxpcxSRum6peuUckvJpwM/XLV0lVLfUvLuB9eapduU\npamlD6x9AmlQ0vJWzE8zywl2xt4MFoqKX1h6QtKhaXk/c0PX1n6itLY0pjSl5a+pPS9pe8pfWbpM0ojS8\nm3AdidWLpLqf0KVOaU7li6Q+mZpWfu7wJ8No2+a2Fu2Qq2KE0sTSjdsJS8KcBWwtJTsp8MVXNXm35tIve1\nUM24gzUZn5Nch6qGXew5u40PZvcn0",
  "0sTSjdsJS8KcBWwtJTsp8MVXNXm35tIve1\nUM24gzUZn5Nch6qGXew5u40PZvcn0I14yPS9fX92YcUSCnc6U8Wlnr4Kywt7K8s935efrjzcOnRavOF9\nlbn2853ne87vc4vnUed53tTr8TdP6duzl3e+7OYrH4x+Kfi3/V6o25pyvO63f4t/AWsI+3o=</latex\nit>\u03c6 = {\u03c60, \u03c61, \u03c62, \u03c63, \u271310, \u271311, \u271320, \u271321, \u271330, \u271331}\n<latexit sha1_base64=\"aYKJls3",
  "tex\nit>\u03c6 = {\u03c60, \u03c61, \u03c62, \u03c63, \u271310, \u271311, \u271320, \u271321, \u271330, \u271331}\n<latexit sha1_base64=\"aYKJls3gxo4H7A6/dQhUn1+H0GU=\">ACFH\nicbVDLSsNAFJ3UV62vqEs3g0UQlJIUTdC0Y3uKtgHNDFMpN26OTBzEQsIR/hxl9x40IRty7c+TdO0gjaemCGc8+9l3vcSNGhTSML60N7+wuFRerq\nysrq1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvn",
  "60N7+wuFRerq\nysrq1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHuKBhcCPHEbF9NAioRzGSnL0AyuxfCSHrpfcp05C08OfcJyHVvafmeltcpU6et\nWoGTngLDELUgUFmo7+afVDHPskJghIXqmEUk7QVxSzEhasWJBIoRHaEB6igbIJ8JO8qNSuKeUPvRCrl4gYa7+7kiQL8TYd1VltrGYzmXif7leL1TO6\nJnG/yB9vEN+gC",
  "O8qNSuKeUPvRCrl4gYa7+7kiQL8TYd1VltrGYzmXif7leL1TO6\nJnG/yB9vEN+gCgCA=</latexit>FBFEsS4MkgL2ZQhjBzCPYpJ1iysSIc6p2hXiIOMJS+VhRJpjTJ8+S9lHNPK7Vr+vVxnlhRxnsgF2wD0xwAhrgEjRBC2DwAJ7AC3jVHrVn7U17n5SWtK\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"dfENBjZWeBDHyQgIldnakspAK9o=\">\nAC",
  "K\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"dfENBjZWeBDHyQgIldnakspAK9o=\">\nACBnicbVDLSsNAFJ34rPUVdSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc+g3u/BsnbRbaemCYwzn3cu89Qc\naZAtv+NpaWV1bX1isb1c2t7Z1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSMZHewyijfoIHKYsYwaClv",
  "/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSMZHewyijfoIHKYsYwaClvnl063Ea\ngesFgodqlOjPy2LmSTaIwe+bNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5YpmAzxgLqapjihyh9Pz5hYJ1oJrU\nhI/VKwpurvjFOVLGirkwxGreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4WibHEBHRyVR2",
  "kwxGreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4WibHEBHRyVR2CM3/yIumc1Z3zeuOu\nUWtelXFU0CE6RqfIQReoiW5QC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</latexit>L [\u03c6]\n12",
  "Example shallow network\n<latexit\n sha1_base64=\"\ncYzac95S8rmoV6\nyZxLSHjL3VtpU=\n\">AXDniclZhb\nb9s2FICd7tZ1t3\nTD8jJgEBYUK9bO\nsNPu8jKgTZreki\n5pc23jNKBkSmZDU\nYpEJU4F/4dhP2Z\nvw173F/ZP9rhDS\nTajc5iHGWjNnO8\nTL4ekRMtPpch1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/",
  "V3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/zj\nFcP3TnmWi0Rt6/\nOUH8YsUiIUAdMQ\nOpr/fyXQToSR2\nVvcqsu9CeD2E/G\nJZscDPSIawah3s\nS75c3+6k/Gh428\nROWlrx0Qb5D5T\nst+Y6Ru0fzi71ur\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcV",
  "vCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcVinh+\nWVfIm3g2IDL0wy\neCf0l4VvXhFyeI\n8P49MGOmRzlmJu\nhiB4UOfz4shUoL\nzVQNxQW0tOJZ2\nbCG4qMB1qeQ4EF\nmYC+esGIZSzQMF\n/XBoqfBUkcMzUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7U",
  "zyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7URcDwQG\nILicgUTyHOk1+/\nNDrIwprVQIu65U\nxAOPFhFStNI8gJ\ny3tFdGgkEo+blk\nrxIKpjFvKFied\n8MzgOsMZgG6Cl8\nczcFWytRkep3mY5\n3FZW5iuIWMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswq",
  "cCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswqhtVRFkSb\nizDFnMIMtN+QgG\nHsm4laFwqogC3M\nzS/x26mJ4LU5T\nmG/tL3VkqT/lKG\nMmADsPvMtmAp4W\n19JZrY3Tc5p5Zs\nCH3sjmKz2JSyL6\nmFNG4FRNbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDH",
  "cIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDHh2XPbBvzH8\nkmVJQXqasiE/4f\nFQ3hWYbXF0Tw5C\nUSTR4EqslLJNzf\n0dSxDC9sE6nmDg\npCMSn0Odr+IlLt\na6oI7mwSo75CwNQ\nL30woNMlh2JZNw\nMjwDU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6",
  "0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6tf59JNCDVmGk\njk2Uzp+Pcg1bDHX\n7q+mvC46rYifrD\nXtQb9gdog4CdH\na3g+ImJR6K64B\njkrEsSy9Ee1DVb\nrhd7Vq69/o4s7c\njhuk1J6m16bYd\n7iU94Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\nc",
  "E4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\ncmNfIgd1CIuai\ntopJjGPkFiHsBg\nXbQv+xsqWgIdH2\n6pDWNzMRVszASw\nNucRDqENYrLdw2\n2xiWF13qOtulcl0\nhMw6hMVHLMajrk\nNYjKgYOcVjlqZI\nrEMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI",
  "HNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI\nVWsaKreMfV8M4lD\nWuGKjQBLG2QPeY\nNpybzMcphmOWK\n8mpQFZKE7iJnU3\nqTE9/fliSk5wfn\nlt6TumZpWeU7lm\n6R2lmKflF4IcvL\nCW/Tvzw1NJTSnc\nt3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9ael",
  "SgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9aelLSh\n9b+pjSV5a+ovSt\npW8pvW/pfUqZpY\nzSVUtXKeWklcH\nfrhs6TKlvqXktx\n/sNUs3KU0tTSl9Y\nOkDSoeWkl/F8Dy\nzlBxv4MFoqaT0i\naVPKBWkt9vfvj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1eg",
  "xpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1egM+m0\nXctzA1bwQaliaUJ\npWuWkl8KcJSw9J\nicJ0PV3NWmb5vI\nfS1UM+5gTcanV5\nOch2rGHay5O02v\nJvenUM34iHR9dX\nf2IgVSCnf6o/nF\nPn4LSwu7S93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c",
  "z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c+KtWr8w1\n3zRaX0W/v4PMz4\nHJg=</latexit\n>y = \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x].\n13",
  "Example shallow network\nPiecewise linear functions with three joints\n<latexit\n sha1_base64=\"\ncYzac95S8rmoV6\nyZxLSHjL3VtpU=\n\">AXDniclZhb\nb9s2FICd7tZ1t3\nTD8jJgEBYUK9bO\nsNPu8jKgTZreki\n5pc23jNKBkSmZDU\nYpEJU4F/4dhP2Z\nvw173F/ZP9rhDS\nTajc5iHGWjNnO8\nTL4ekRMtPpch1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P",
  "1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/zj\nFcP3TnmWi0Rt6/\nOUH8YsUiIUAdMQ\nOpr/fyXQToSR2\nVvcqsu9CeD2E/G\nJZscDPSIawah3s\nS75c3+6k/Gh428\nROWlrx0Qb5D5T\nst+Y6Ru0fzi71ur\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXD",
  "r\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcVinh+\nWVfIm3g2IDL0wy\neCf0l4VvXhFyeI\n8P49MGOmRzlmJu\nhiB4UOfz4shUoL\nzVQNxQW0tOJZ2\nbCG4qMB1qeQ4EF\nmYC+esGIZSzQMF\n/XBoqfBUkcMzUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA",
  "zUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7URcDwQG\nILicgUTyHOk1+/\nNDrIwprVQIu65U\nxAOPFhFStNI8gJ\ny3tFdGgkEo+blk\nrxIKpjFvKFied\n8MzgOsMZgG6Cl8\nczcFWytRkep3mY5\n3FZW5iuIWMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204",
  "WMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswqhtVRFkSb\nizDFnMIMtN+QgG\nHsm4laFwqogC3M\nzS/x26mJ4LU5T\nmG/tL3VkqT/lKG\nMmADsPvMtmAp4W\n19JZrY3Tc5p5Zs\nCH3sjmKz2JSyL6\nmFNG4FRNbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJ",
  "NbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDHh2XPbBvzH8\nkmVJQXqasiE/4f\nFQ3hWYbXF0Tw5C\nUSTR4EqslLJNzf\n0dSxDC9sE6nmDg\npCMSn0Odr+IlLt\na6oI7mwSo75CwNQ\nL30woNMlh2JZNw\nMjwDU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF",
  "DU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6tf59JNCDVmGk\njk2Uzp+Pcg1bDHX\n7q+mvC46rYifrD\nXtQb9gdog4CdH\na3g+ImJR6K64B\njkrEsSy9Ee1DVb\nrhd7Vq69/o4s7c\njhuk1J6m16bYd\n7iU94Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmK",
  "4Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\ncmNfIgd1CIuai\ntopJjGPkFiHsBg\nXbQv+xsqWgIdH2\n6pDWNzMRVszASw\nNucRDqENYrLdw2\n2xiWF13qOtulcl0\nhMw6hMVHLMajrk\nNYjKgYOcVjlqZI\nrEMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOa",
  "EMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI\nVWsaKreMfV8M4lD\nWuGKjQBLG2QPeY\nNpybzMcphmOWK\n8mpQFZKE7iJnU3\nqTE9/fliSk5wfn\nlt6TumZpWeU7lm\n6R2lmKflF4IcvL\nCW/Tvzw1NJTSnc\nt3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1",
  "t3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9aelLSh\n9b+pjSV5a+ovSt\npW8pvW/pfUqZpY\nzSVUtXKeWklcH\nfrhs6TKlvqXktx\n/sNUs3KU0tTSl9Y\nOkDSoeWkl/F8Dy\nzlBxv4MFoqaT0i\naVPKBWkt9vfvj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npP",
  "vj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1egM+m0\nXctzA1bwQaliaUJ\npWuWkl8KcJSw9J\nicJ0PV3NWmb5vI\nfS1UM+5gTcanV5\nOch2rGHay5O02v\nJvenUM34iHR9dX\nf2IgVSCnf6o/nF\nPn4LSwu7S93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht",
  "93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c+KtWr8w1\n3zRaX0W/v4PMz4\nHJg=</latexit\n>y = \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x].\n14",
  "Hidden units\n<latexit sha1\n_base64=\"4mwMW+AxXHfhTyI/k\naVAw9s5G24=\">AWsniclZhb\nb9s2FIDV7tZ1t3TD8rIXYUGBY\nesMu+26vQxok6a3pIvTxEmaODU\nomZLZUJSiS2JX8D/Zr9nr9gf2\nb3YoyWZ1DvMwA4np83i5ZDUzU\nukyPJu9r1z/48KOP7nx6c3\nPv/iy69Wbn19kMVF6vOBH8s4P\nfJYxqVQfJCLXPKjJOUs8iQ/9M\n42ND+84GkmYrW",
  "v/iy69Wbn19kMVF6vOBH8s4P\nfJYxqVQfJCLXPKjJOUs8iQ/9M\n42ND+84GkmYrWfzxJ+GrFQiUD\n4LIfQaOXBzP3dHSYTMSq785/qQ\nm8+0f+aX3f1r7uLX/f0r3vz0c\npat9OtPi4t9JrCmtN8+qNb346H\n49gvIq5yX7IsO+l1k/y0ZGkuf\nMnN4dFxhPmn7GQn0BRsYhnp2U\n1wLl7GyJjN4hT+FO5W0XfP6Jk\nUZbNIg/MiOWTDMdtLGTIg",
  "mn7GQn0BRsYhnp2U\n1wLl7GyJjN4hT+FO5W0XfP6Jk\nUZbNIg/MiOWTDMdtLGTIg9+O\ny2FSoqcK79uKCikm8euzpY7Fin\n3czmDAvNTAX1/QlLmZ9DTm8O\nFb/04yhialwO1zd35+XQ46FQJT\n8vqvzO521ns3I4FK8y1p/vL2s\nROY/EO04qRdyRUCD+dlyTthB\nwPBAYgOJyBWPIM6dX68wO0hCu\ntJAgbuxVPoXOC+mpOqVc5DyEl\nLOyYaF",
  "yTthB\nwPBAYgOJyBWPIM6dX68wO0hCu\ntJAgbuxVPoXOC+mpOqVc5DyEl\nLOyYaFBLJpy1rg1gwlVFL2QPFd\nW+7GvA8hVmArsIXR3OwlzA1Xx\nyX82meRmWmY7iFlKmQV03AkH0m\n9YjahiqkhEP9lvUHtl4xdYkL\nk6qrqY6gqz9tO3kKc2LGredKoI\nsWIRh26oiyJKw+8csYpDlpjyC\nAUeujthVobAqyMLsp7HXbjvRE\nbw2pwnsl7a3WZL",
  "IRh26oiyJKw+8csYpDlpjyC\nAUeujthVobAqyMLsp7HXbjvRE\nbw2pwnsl7a3WZL0XzCUER2A3ae\n/BVM+b+sb8dJ2F8m5qHxd4FN3\nApPVPoSlYT2sRSMwqiY2p2aVK2\nTSbEojS/bpu6NReWJaA9QB/C\nmK1Khgve0O1UJlqwOD+/AUNC8\npOfO7/w6WnZ1dtG/yPZhIqyIr\nFVpMP/o6IxXG/w+oInrxYosm\nDQDV5sYTzO5o6luKFrSPV3EF",
  "1dtG/yPZhIqyIr\nFVpMP/o6IxXG/w+oInrxYosm\nDQDV5sYTzO5o6luKFrSPV3EFBK\nCZFPkPbX4SqfUwVwZ2NI9RXCO\nh64ZsJhSY5CNqyDmgZvuHKaVlA\nPhqkX4/Rl3FWpJyc/NB6hkil6\n9NiKvTFqn1ClVponze4XB4FZbg\n4XPArDvdQRr06n15cqDFLUTKn\nekqnb4ZDlvMtvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6kh\nUF9yq",
  "FLUTKn\nekqnb4ZDlvMtvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6kh\nUF9yqWOuSxLK0B3Utl+v7PSu3\n3vxIlnZoce2mJPU2vbTbFveKHv\nDzbUtvt4lHLOpIVFfTQ+oRy9I\ne1GXP47ZtFBbXbkpS7yKPVtviL\nk20/IP9Cc+Zvk2K5Vjf9sVyWI\newmFMxt4pxEMk1iEsRkXbgt9\nY2RNw8WhbdQiL/Uy0NR3A0phLP\nIQ6hMV6C7fNJo",
  "wmFMxt4pxEMk1iEsRkXbgt9\nY2RNw8WhbdQiL/Uy0NR3A0phLP\nIQ6hMV6C7fNJobVbYu6bVeZTC\nbIrENYfMoiPOo6hMWQiqFVPGNJ\ngsQ6RPI4wXmc0DwmWEpsEp6Rx\nDIjZEnZFlQ6iduSDmBpilqbWhq\nDHshYoQabIJYzuvIy68pTaBUr\nuoHtoYHVzScM1ShDmBph+wxd\n7hj3WQeTjHcZtmSnAhkJTSBfez\n0qbO4+/OCktzJecHM0Bml",
  "ScM1ShDmBph+wxd\n7hj3WQeTjHcZtmSnAhkJTSBfez\n0qbO4+/OCktzJecHM0Bml4Ze\nUnpo6CGlqaHkicALXhlKnk684M\nLQC0oPD2gtDC0oHRg6IDSwNC\nA0ieGPqHUN9SndMPQDUpzQ8kdK\nVwRDN2ndGLohNIjQ48ofW3oa0\nqfGfqM0mNDjyl9Z+g7Sh8Z+oh\nSZijdNPQTUq5oeTVgResG7pOq\nWcoefaDvWZon9LE0ITSx4Y+pn\nRsKHk",
  "h8Z+oh\nSZijdNPQTUq5oeTVgResG7pOq\nWcoefaDvWZon9LE0ITSx4Y+pn\nRsKHkqhuZoeT2Bi6MhkpKnxv6\nnFJhKHl+84KXhr6kNDI0ovSFo\nS8ofWvoW0qfGvqU0tBQ8m4A7k4\nM3aPUvAUqM0p3Dd2l9NzQc/t7\nAb6cRs+2MHdMBTuUxobGlG4ZS\np4U4FbC0DNyPxmo5qy2eNtEzmu\nBWnILazK+OJrkPFBLbmHN2Wlx\nNDk/BWrJ6Trmw",
  "U4FbC0DNyPxmo5qy2eNtEzmu\nBWnILazK+OJrkPFBLbmHN2Wlx\nNDk/BWrJ6TrmwfLFymQUjTj1\nbWevgtLC0c3O30HnTu795fe7j\nevKG94XznfO/84PScX52HzjOn\n7wc3/nT+cv52/ln9f7q8Spb9W\nv1+rXmG+c1mdV/ge0FuRK</l\natexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\nBreak down into two parts:\nwhere:\nHidden units\n<latexit sha1_base64=\"OexYJKlfph/YpHs\n7sx68K",
  "+ \u03c62h2 + \u03c63h3\nBreak down into two parts:\nwhere:\nHidden units\n<latexit sha1_base64=\"OexYJKlfph/YpHs\n7sx68KlEAH9c=\">AXC3iclZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84kEHKDFAgskL\nCEkb2yV0GWjS3DEs/+hE5/TN86fe2P6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6dufb\nOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSKT7MBnOZdC8Z4WvKDNOMs9i",
  "lv6dufb\nOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSKT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41ku\nErWrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlWx82NdDrhmEl8be970r+54dOT1+7NQY\ndldYblVYflShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4xtfDvqDJChirnQgWZ4fdpdSfVS\nyTItA8vFsv8h5yoITFvFDKCoW8/yo",
  "1joNJ+t4xtfDvqDJChirnQgWZ4fdpdSfVS\nyTItA8vFsv8h5yoITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJYvz/CL2wYyZHuaYmaCL\nHRY6/OWoFCotNFdB3VFYSE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D5I4ZmpQ9lfWt\nsdl3+eRUCU/LaqFG4/bzlrlcCheZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcgFjkBie\nI",
  "/LaqFG4/bzlrlcCheZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcgFjkBie\nI5tGny4deF1HYqBJwW+MPhjPx6RpXkEOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg0M2obqpASqgYt6zdsPWfqpElcklZDzUwE\nWbtZ29EZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOPZM",
  "WfqpElcklZDzUwE\nWbtZ29EZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOPZMxK0KhVBNuZWlvjtvlMTwXtzl\nML10vbWSpL+M4YyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7Asqc16QRm1cTG1KxyhU\nyaLQhlyXnbNKNxqDwV7QmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98dFQumcvG/EeyCQ3\nlRepqyITfoqEBPMjw/oIX",
  "8J2uyrBljXh/m2YalZIfvjD4o98dFQumcvG/EeyCQ3\nlRepqyITfoqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu04VwYN\nYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo3z\ne4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLRq36u4RJzXf3VktdFpxXx0/WmPxgXrE",
  "NaCMjwczvgV1X2Ub/Op58UasAylMyRWdLRq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz\n0eB2vR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR05XLcpSbvNKN2w71iBPx0wzHaDeIR\nizoStdWMkHrEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnETNMSmRA3PsS2S/DmFRU1E7x\nSTmERLrEBbjom3B31jZEfDwaFt1CItbuWhrJoClAZ",
  "NMSmRA3PsS2S/DmFRU1E7x\nSTmERLrEBbjom3B31jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGFY3HOqGW2UyHSKzDm\nHxEYvxrOsQFiMqRk7xhKUpEusQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmgKUR6m3k6Ax\nGIBOFOmyCWM7pzsudO0+hXazoLu65Ou5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W",
  "XazoLu65Ou5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaWUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2\nqM0tDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YOKT2w9IDSF5a+oPSxpY8pfWnpS0rfWP\nqG0vuW3qeUWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx9Q",
  "pS0rfWP\nqG0vuW3qeUWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx9QOnAUvKrGJ5nlpL\njDTwYLZWUPrH0CaXCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiNLybsBOJ1YukOpfQtU5\npRuW7pN6amlp+73Any6jL5rY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42edtE7muhmnIHaz\nI+qU1yHqopd7Dm7jSpTe5PoZryIR",
  "3lPxSgKOEpSfkPBmq5q42edtE7muhmnIHaz\nI+qU1yHqopd7Dm7jSpTe5PoZryIRn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG+leUN\n7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuznw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8C\njQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit\n sha1_base64=\"\ncYz",
  "it>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit\n sha1_base64=\"\ncYzac95S8rmoV6\nyZxLSHjL3VtpU=\n\">AXDniclZhb\nb9s2FICd7tZ1t3\nTD8jJgEBYUK9bO\nsNPu8jKgTZreki\n5pc23jNKBkSmZDU\nYpEJU4F/4dhP2Z\nvw173F/ZP9rhDS\nTajc5iHGWjNnO8\nTL4ekRMtPpch1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neu",
  "DS\nTajc5iHGWjNnO8\nTL4ekRMtPpch1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/zj\nFcP3TnmWi0Rt6/\nOUH8YsUiIUAdMQ\nOpr/fyXQToSR2\nVvcqsu9CeD2E/G\nJZscDPSIawah3s\nS75c3+6k/Gh428\nROWlrx0Qb5D5T\nst+Y6Ru0fzi71ur\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4",
  "8\nROWlrx0Qb5D5T\nst+Y6Ru0fzi71ur\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcVinh+\nWVfIm3g2IDL0wy\neCf0l4VvXhFyeI\n8P49MGOmRzlmJu\nhiB4UOfz4shUoL\nzVQNxQW0tOJZ2\nbCG4qMB1qeQ4EF\nmYC+esGIZSzQMF\n/XBoqfBUkcMzUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl",
  "4EF\nmYC+esGIZSzQMF\n/XBoqfBUkcMzUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7URcDwQG\nILicgUTyHOk1+/\nNDrIwprVQIu65U\nxAOPFhFStNI8gJ\ny3tFdGgkEo+blk\nrxIKpjFvKFied\n8MzgOsMZgG6Cl8\nczcFWytRkep3mY5\n3FZW5iuIWMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2Xj",
  "G6Cl8\nczcFWytRkep3mY5\n3FZW5iuIWMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswqhtVRFkSb\nizDFnMIMtN+QgG\nHsm4laFwqogC3M\nzS/x26mJ4LU5T\nmG/tL3VkqT/lKG\nMmADsPvMtmAp4W\n19JZrY3Tc5p5Zs\nCH3sjmKz2JSyL6\nmFNG4FRNbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4Am",
  "Tc5p5Zs\nCH3sjmKz2JSyL6\nmFNG4FRNbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDHh2XPbBvzH8\nkmVJQXqasiE/4f\nFQ3hWYbXF0Tw5C\nUSTR4EqslLJNzf\n0dSxDC9sE6nmDg\npCMSn0Odr+IlLt\na6oI7mwSo75CwNQ\nL30woNMlh2JZNw\nMjwDU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3",
  "7mwSo75CwNQ\nL30woNMlh2JZNw\nMjwDU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6tf59JNCDVmGk\njk2Uzp+Pcg1bDHX\n7q+mvC46rYifrD\nXtQb9gdog4CdH\na3g+ImJR6K64B\njkrEsSy9Ee1DVb\nrhd7Vq69/o4s7c\njhuk1J6m16bYd\n7iU94Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53H",
  "7Vq69/o4s7c\njhuk1J6m16bYd\n7iU94Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\ncmNfIgd1CIuai\ntopJjGPkFiHsBg\nXbQv+xsqWgIdH2\n6pDWNzMRVszASw\nNucRDqENYrLdw2\n2xiWF13qOtulcl0\nhMw6hMVHLMajrk\nNYjKgYOcVjlqZI\nrEMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZa",
  "Mw6hMVHLMajrk\nNYjKgYOcVjlqZI\nrEMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI\nVWsaKreMfV8M4lD\nWuGKjQBLG2QPeY\nNpybzMcphmOWK\n8mpQFZKE7iJnU3\nqTE9/fliSk5wfn\nlt6TumZpWeU7lm\n6R2lmKflF4IcvL\nCW/Tvzw1NJTSnc\nt3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0o",
  "6R2lmKflF4IcvL\nCW/Tvzw1NJTSnc\nt3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9aelLSh\n9b+pjSV5a+ovSt\npW8pvW/pfUqZpY\nzSVUtXKeWklcH\nfrhs6TKlvqXktx\n/sNUs3KU0tTSl9Y\nOkDSoeWkl/F8Dy\nzlBxv4MFoqaT0i\naVPKBWkt9vfvj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nR",
  "8Dy\nzlBxv4MFoqaT0i\naVPKBWkt9vfvj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1egM+m0\nXctzA1bwQaliaUJ\npWuWkl8KcJSw9J\nicJ0PV3NWmb5vI\nfS1UM+5gTcanV5\nOch2rGHay5O02v\nJvenUM34iHR9dX\nf2IgVSCnf6o/nF\nPn4LSwu7S93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/",
  "HR9dX\nf2IgVSCnf6o/nF\nPn4LSwu7S93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c+KtWr8w1\n3zRaX0W/v4PMz4\nHJg=</latexit\n>y = \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x].\n15",
  "1. compute three \nlinear functions\nLinear\nFunctions\n16",
  "<latexit sha1_base64=\"6yInH/eKr6BVR1p\n+osf4KbjHrPs=\">AXDHiclZhbU9w2FICX9JbSG2mnvLQPnjLJdNqUYUl6elMAiE3SIHA\nglLGNkrexVk2dgyLPHsX+j0x/St09f+h/6RPvfI9q7wOeIhO5OsON+n25Fsa+2nUuR6aenfm\nWvPve+x9c/3D2o48/+fSzuRuf7+VJkQW8FyQyQ58lnMpFO9poSU/SDPOYl/yf9k1fD9M\n57lIlG7+iLlR",
  "+fSzuRuf7+VJkQW8FyQyQ58lnMpFO9poSU/SDPOYl/yf9k1fD9M\n57lIlG7+iLlRzGLlAhFwDSEjuf+GB6X3bF361evH/vJqGTjw74ecs0gvDT2vemf3XHoyOv3\n5+FCsvuCsutCsuXKtxV7jTqnDHVLh9PLewtLhUfTxa6DaFhU7z2Tq+8eWgP0iCIuZKB5Ll+\nWF3KdVHJcu0CQfz/aLnKcsOGERP4SiYjHPj8oqdWPvJkQGXphk8E9pr4perl",
  "Ll+\nWF3KdVHJcu0CQfz/aLnKcsOGERP4SiYjHPj8oqdWPvJkQGXphk8E9pr4perlGyOM8vYh/Mm\nOlhjpkJuthocNfjkqh0kJzFdQdhYX0dOKZdfAGIuOBlhdQYEmYKxeMGQZCzSs1mxf8fMgi\nWOmBmV/ZW17XPZ9HglV8tOiWrnxuO2sVQ6H4lXGypPdaStC81i84aSRSjGNXCHwaFyWfDFax\nEBwAGKRE5AonkObJj9+6HURhZ0qAZf1zuiD8",
  "tC81i84aSRSjGNXCHwaFyWfDFax\nEBwAGKRE5AonkObJj9+6HURhZ0qAZf1zuiD8XxMmlaR5CTlvaSaFBIJR+1rFViwVLGLWUHF\nM+76RnAdQarAEOFL47WYCdlajyp/lIZ3GZmxjuIWMq4lUXMOWASTOjtqEKaFq0LJ+w9Zzp\nk6axCVpNdTMRJC1m7UdndG8qEHbqSLIgk0Yta0qgiwJ95UBixlkuSkfw4Rjz0TcqlBYFWRjb\nmWJ3+47NRG8N",
  "G8qEHbqSLIgk0Yta0qgiwJ95UBixlkuSkfw4Rjz0TcqlBYFWRjb\nmWJ3+47NRG8N0cpXC9tb60k6T9jKCMmAFef+RZMBbytryZT25sk56zyTYGPvCEsVrsKy6J6W\npNOYFZNbEzNKlfIpNmCUJact0zGofKU9GeoAngi67IhAovaberEmxZE+7fhqlmheSHPyz+y\nEdH5ZK5bMx/JvQUF6kroZM+C0aGsCTDO8viODFSyRaPAhUi5dIuL+jpWMZ3",
  "yz+y\nEdH5ZK5bMx/JvQUF6kroZM+C0aGsCTDO8viODFSyRaPAhUi5dIuL+jpWMZ3tgmUq0dFIRiU\nugLdPmLSLXrVBE82CRGY4WAaRe+mVBokcOwLZuAkeEbnsmODRSgSQb1HAOZ5EXGyc0P7WeIV\nLq5LWbCPKzaN1RphPZ9g8tpLSjDw+GMX1HdRxn163z6SaEGLEPJHJklHb3q5xouMdfVXy15X\nXRaET9db/qDcHqFEHAT4/X8XpExKORG3B",
  "6SaEGLEPJHJklHb3q5xouMdfVXy15X\nXRaET9db/qDcHqFEHAT4/X8XpExKORG3BIcjZliSWoz9oa7pdL4+sXH/1HdnakcN1m5K02\n4zSbTvcK0bATzco90gHrGoI1FbzQipRyxHf9CWO48brlk4XLcpSbuTPDpthzs10fYPd81R1\nByTEjkwx75E9usQFjUVtVNMYh4hsQ5hMS7aFvyNlR0BD4+2VYewuJWLtmYCWBpwiadQh7BYX\n8Jts4lhdc",
  "VtVNMYh4hsQ5hMS7aFvyNlR0BD4+2VYewuJWLtmYCWBpwiadQh7BYX\n8Jts4lhdcOhbrhVJtMhMusQFh+xGM+6DmExomLkFE9YmiKxDpE8DnEehzSPKZSl4RXJHWsC\nNlSrg2VDZO2ZAJYGqHeRo7OYAQyUajDJojlnO683LnzFNrFiu7inqvj3hUda4YaNAEsbZJrz\nOtvOi8yH6cYjlmuJKcCWSlN4BZ2tqgzOf35YUlOcn54YekFpeWnlO6b+",
  "EsbZJrz\nOtvOi8yH6cYjlmuJKcCWSlN4BZ2tqgzOf35YUlOcn54YekFpeWnlO6b+k+pZml5BeBHz63l\nPw68cMzS8o3bN0j9LC0oLSnqU9SkNLQ0ofWvqQ0sDSgNJVS1cp1ZaSEyk8ESzdpXRo6ZDSA\n0sPKH1h6QtKH1v6mNKXlr6k9I2lbyi9b+l9SpmljNI1S9co5ZaSVwd+uGLpCqW+peS3H1xrl\nm5RmlqaUvrA0geUDiwlv4rheWYpOd7A",
  "NI1S9co5ZaSVwd+uGLpCqW+peS3H1xrl\nm5RmlqaUvrA0geUDiwlv4rheWYpOd7Ag9FSekTS59QKiwlv9/8JmlzyiNLY0pfWrpU0pfW\n/qa0keWPqI0spS8G4DTiaU7lNq3QGVO6bal25SeWnrqfi/Ap8vouzbmpm1gk9LE0oTSdUvJL\nwU4Slh6Qs6ToWruapO3TeS+Fqopd7Am45PaJOehmnIHa+5Ok9rk/hSqKR+Soa/tTV+kQErhT\nn8t9DF",
  "apO3TeS+Fqopd7Am45PaJOehmnIHa+5Ok9rk/hSqKR+Soa/tTV+kQErhT\nn8t9DFb2FpYW95sfvT4t3tuwv3Vpo3tNc7X3W+6Xzb6XZ+7tzrPO5sdXqdoPfzNczN2duz\nf8+/+f8X/N/1+q1mabOF53WZ/6f/wH3WALD</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x],\n2. Pass through ReLU \nfunctions (creates \nhidden units)\nLinear\nFunctions\nAfter\nActivation\n17",
  "2. Weight the hidden \nunits\nAfter\nActivation\nWeight the\nHidden units\n18",
  "4. Sum the weighted \nhidden units to create \noutput\n<latexit sha1_base64=\"4mwMW+AxXHfhT\nyI/kaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3p\nIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzUukyPJu9\nr1z/48KOP7nx6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42\nND+84Gk",
  "x6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42\nND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/f0r3\nvz0cpat9OtPi4t9JrCmtN8+qNb346H49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhPmn7GQ\nn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiOWTDMdtLGTIg9",
  "mn7GQ\nn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2FSoqcK7\n9uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8OFb/04yhialwO1zd35+XQ46FQJT8\nvqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyRUCD+dlyTthBwPBAYgOJyBWPIM6dX68w\nO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYaFBLJpy1",
  "BwPBAYgOJyBWPIM6dX68w\nO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYaFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsI\nXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9YjahiqkhEP9lvUHtl4xdYkLk6qrqY6\ngqz9tO3kKc2LGredKoIsWIRh26oiyJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREb\nw2pwnsl7a3WZL0XzCUER2A3",
  "Kw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREb\nw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2\np2aVK2TSbEojS/bpu6NReWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1\ndtG/yPZhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkP\nbX4Sqf",
  "rFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkP\nbX4SqfUwVwZ2NI9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil6\n9NiKvTFqn1ClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+\nLVivk51tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3",
  "vurKa+\nLVivk51tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2m\nJPU2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/I\nP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A0phL\nPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo",
  "2RNw8WhbdQiL/Uy0NR3A0phL\nPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGNJgsQ6RPI4wXmc0DwmW\nEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvIy68pTaBUruoHtoYHVzS\ncM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJecHM0Bml4ZeUnpo6\nCGlqaHkicALXhlKnk684M",
  "mSnAhkJTSBfez0qbO4+/OCktzJecHM0Bml4ZeUnpo6\nCGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieGPqHUN9SndMPQDUpzQ8\nkdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh8Z+ohSZijdNPQTUq5oeTV\ngResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84\nKXhr",
  "coefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84\nKXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6\ncRs+2MHdMBTuUxobGlG4ZSp4U4FbC0DNyPxmo5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2\nWlxNDk/BWrJ6TrmwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevK",
  "FBLbmHN2\nWlxNDk/BWrJ6TrmwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevKG94XznfO/84PS\ncX52HzjOn7wc3/nT+cv52/ln9f7q8Spb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\nWeight the\nhidden units\nSum the weighted \nhidden units\n19",
  "<latexit\n sha1_base64=\"\ncYzac95S8rmoV6\nyZxLSHjL3VtpU=\n\">AXDniclZhb\nb9s2FICd7tZ1t3\nTD8jJgEBYUK9bO\nsNPu8jKgTZreki\n5pc23jNKBkSmZDU\nYpEJU4F/4dhP2Z\nvw173F/ZP9rhDS\nTajc5iHGWjNnO8\nTL4ekRMtPpch1r\n/fP3JV3n3v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/zj\nF",
  "v/Q+\nufnjto48/+fSz+\neuf7+ZJkQV8J0h\nku37LOdSKL6jh\nZ8P804i3J9/zj\nFcP3TnmWi0Rt6/\nOUH8YsUiIUAdMQ\nOpr/fyXQToSR2\nVvcqsu9CeD2E/G\nJZscDPSIawah3s\nS75c3+6k/Gh428\nROWlrx0Qb5D5T\nst+Y6Ru0fzi71ur\n/p4tNBvCoud5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcVinh+",
  "d5rN\n5dP3L4WCYBEXMl\nQ4ky/ODfi/VhyX\nLtAgkn1wbFDlPW\nXDMIn4ARcVinh+\nWVfIm3g2IDL0wy\neCf0l4VvXhFyeI\n8P49MGOmRzlmJu\nhiB4UOfz4shUoL\nzVQNxQW0tOJZ2\nbCG4qMB1qeQ4EF\nmYC+esGIZSzQMF\n/XBoqfBUkcMzUs\nB8urzyflwOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7URcDw",
  "wOeRUC\nU/Kaq5m0zazmrl\ncCheZiw/2Z7VIj\nSPxVtOKqkU8klA\no8mZcm7URcDwQG\nILicgUTyHOk1+/\nNDrIwprVQIu65U\nxAOPFhFStNI8gJ\ny3tFdGgkEo+blk\nrxIKpjFvKFied\n8MzgOsMZgG6Cl8\nczcFWytRkep3mY5\n3FZW5iuIWMqYhX\nTcCQAybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswqhtVR",
  "ybNiNqGKq\nSES4OW9Su2XjB1\n3CQuSauZiaCrO\n2s7eiM5kUN204V\nQRYswqhtVRFkSb\nizDFnMIMtN+QgG\nHsm4laFwqogC3M\nzS/x26mJ4LU5T\nmG/tL3VkqT/lKG\nMmADsPvMtmAp4W\n19JZrY3Tc5p5Zs\nCH3sjmKz2JSyL6\nmFNG4FRNbEJNat\ncIZNmC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDHh2XP",
  "mC0JZctY2T\nW8cKk9Fe4AmgDd\ndkQkVXtBuVyVYsi\nY8uA1DzQrJD7v\n/sDHh2XPbBvzH8\nkmVJQXqasiE/4f\nFQ3hWYbXF0Tw5C\nUSTR4EqslLJNzf\n0dSxDC9sE6nmDg\npCMSn0Odr+IlLt\na6oI7mwSo75CwNQ\nL30woNMlh2JZNw\nMjwDU9lxwIK0C\nDeoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6tf59",
  "eoyBTPIi4+Tmh\n9YzRCrd3BYzYR5\nW7RuqNEL7vsHl7\nCow8PhlF9yuY8\ny6tf59JNCDVmGk\njk2Uzp+Pcg1bDHX\n7q+mvC46rYifrD\nXtQb9gdog4CdH\na3g+ImJR6K64B\njkrEsSy9Ee1DVb\nrhd7Vq69/o4s7c\njhuk1J6m16bYd\n7iU94Cfrjt6uE4\n9Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\ncmNfI",
  "Y1JGorqaH1COW\noz2oy53HdcoHK7\nblKTeaR6dtsOdm\nWj5h9vmKGqOSYk\ncmNfIgd1CIuai\ntopJjGPkFiHsBg\nXbQv+xsqWgIdH2\n6pDWNzMRVszASw\nNucRDqENYrLdw2\n2xiWF13qOtulcl0\nhMw6hMVHLMajrk\nNYjKgYOcVjlqZI\nrEMkjyOcxHNY4\nql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI\nVWs",
  "ql1CXhGUkdM0KW\nlGtBZaOkLZkAls\naotbGjMeiBTBRq\nsAliOacrL3euPI\nVWsaKreMfV8M4lD\nWuGKjQBLG2QPeY\nNpybzMcphmOWK\n8mpQFZKE7iJnU3\nqTE9/fliSk5wfn\nlt6TumZpWeU7lm\n6R2lmKflF4IcvL\nCW/Tvzw1NJTSnc\nt3aW0sLSgdMfSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9aelLSh",
  "fSH\nUpDS0NKH1r6kNLA\n0oDSFUtXKNWkh\nMpPBEs3aZ0ZOmI\n0n1L9yl9aelLSh\n9b+pjSV5a+ovSt\npW8pvW/pfUqZpY\nzSVUtXKeWklcH\nfrhs6TKlvqXktx\n/sNUs3KU0tTSl9Y\nOkDSoeWkl/F8Dy\nzlBxv4MFoqaT0i\naVPKBWkt9vfvj\nM0meUxpbGlD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1egM+m0",
  "lD619\nCmlbyx9Q+kjSx9\nRGlK3g3A6cTSL\nUrtW6Ayp/S5pc8\npPbH0xP1egM+m0\nXctzA1bwQaliaUJ\npWuWkl8KcJSw9J\nicJ0PV3NWmb5vI\nfS1UM+5gTcanV5\nOch2rGHay5O02v\nJvenUM34iHR9dX\nf2IgVSCnf6o/nF\nPn4LSwu7S93+j9\n27z+8u3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c+KtW",
  "3ltu3tBe7\nXzV+aZzs9Pv/NS\n513nc2ezsdILOv\n3Nfz307d3Pht4U\n/Fv5c+KtWr8w1\n3zRaX0W/v4PMz4\nHJg=</latexit\n>y = \u03c60 + \u03c61a[\u271310 + \u271311x] + \u03c62a[\u271320 + \u271321x] + \u03c63a[\u271330 + \u271331x].\nExample shallow network = piecewise linear functions\n1 \u201cjoint\u201d per ReLU function\nExample: 3 different shallow networks\n20",
  "Activation pattern = which hidden units are activated\nShaded region:\n\u2022\nUnit 1 active\n\u2022\nUnit 2 inactive\n\u2022\nUnit 3 active\n21",
  "Depicting neural networks\n<latexit sha1_base64=\"4mwMW+AxXHfh\nTyI/kaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxo\nk6a3pIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzU\nukyPJu9r1z/48KOP7nx6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKj\nJOUs8iQ/9M42ND+84GkmYrWfzxJ",
  "y69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKj\nJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f\n+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb346H49gvIq5yX7IsO+l1k/y0\nZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiO\nWTDMdtLGTIg9+Oy2FSo",
  "BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiO\nWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8O\nFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyR\nUCD+dlyTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYa\nFBLJpy1rg1gwlV",
  "OJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYa\nFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03\nAkH0m9YjahiqkhEP9lvUHtl4xdYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiy\nJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae\n/BVM+",
  "pDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae\n/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NR\neWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZhIqyIrFVpMP\n/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI",
  "o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI\n9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1\nClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZo",
  "Vivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU\n2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/\nIP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A\n0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiq",
  "bdQiL/Uy0NR3A\n0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGNJgsQ6RPI4wX\nmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvIy68pTaBUr\nuoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD",
  "TSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieG\nPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh\n8Z+ohSZijdNPQTUq5oeTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZ\noeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0o",
  "WZon9LE0ITSx4Y+pnRsKHkqhuZ\noeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k\n4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMBTuUxobGlG4ZSp4U4FbC0DNyPxmo\n5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWev\ngtLC0c3O30HnTu795fe7jevKG94Xzn",
  "N2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWev\ngtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8\nSpb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_bas\ne64=\"OexYJKlfph/YpHs7sx68KlEAH9c=\"\n>AXC3iclZhbU9w2FICX9JbSG2mnvOTFU\n5pOp0ZlqSXl84kEHKD",
  "x68KlEAH9c=\"\n>AXC3iclZhbU9w2FICX9JbSG2mnvOTFU\n5pOp0ZlqSXl84kEHKDFAgskLCEkb2yV0G\nWjS3DEs/+hE5/TN86fe2P6A/pe49s7wqfI\nx6yM8mK8326Hcm21n4qRa6Xlv6dufbOu+\n+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSKT\n7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41\nkuErWrL1J+FLNIiVAETEPoeO6P4XHZHXvf\n/Or1",
  "Z4WvKDNOMs9iXf909WDd8/41\nkuErWrL1J+FLNIiVAETEPoeO6P4XHZHXvf\n/Or1Yz8ZlWx82NdDrhmEl8be970r+54dO\nT1+7NQYdldYblVYflShTvuCndaFe6YCsd\nzC0uLS9XHo4VuU1joNJ+t4xtfDvqDJChir\nnQgWZ4fdpdSfVSyTItA8vFsv8h5yoITFvF\nDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6",
  "ITFvF\nDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFd\nB3VFYSE8nlkGbyAyHmh5AQUWZALG6gVD\nlrFAw2LN9hU/D5I4ZmpQ9lfWtsdl3+eRUC\nU/LaqFG4/bzlrlcCheZaw82Z2IjSPxRtO\nGqkU08gVAo/GZckXo0UMBAcgFjkBieI5tG\nny4deF1HYqBJwW+MPhjPx6RpXkEOWl\npL4kGhVTyUct",
  "kXo0UMBAcgFjkBieI5tG\nny4deF1HYqBJwW+MPhjPx6RpXkEOWl\npL4kGhVTyUctaJRYsZdxSdkDxvFueAVxns\nAowVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJ\nedQFTDpg0M2obqpASqgYt6zdsPWfqpElck\nlZDzUwEWbtZ29EZzYsatJ0qgizYhFHbqiL\nIknBbGbCYQZab8jFMOPZMxK0KhVBNuZW\nlvjtvlMTwXtzlML10vbWSpL+M4YyYgJ",
  "IknBbGbCYQZab8jFMOPZMxK0KhVBNuZW\nlvjtvlMTwXtzlML10vbWSpL+M4YyYgJw9Z\nlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7As\nqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqD\nwV7QmaAL7oikyo8J2uyrBljXh/m2YalZI\nfvjD4o98dFQumcvG/EeyCQ3lRepqyITfo\nqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luG\nNbSLV2kFBKCaFvkCX",
  "Q3lRepqyITfo\nqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luG\nNbSLV2kFBKCaFvkCXv4hUu04VwYNYjRWC\nJh24ZsJhRY5DNuyCRgZvuGR7NhAZpkUM8\nxkEleZJzc/NB+hkilm9tiJszDqn1DlUZo3\nze4nNaCMjwczvgV1X2Ub/Op58UasAylM\nyRWdLRq36u4RJzXf3VktdFpxXx0/WmPxgX\nrE4RBPz0eB2vR0Qs6kjUFpyBnG1JYjn6g7\nam2",
  "u4RJzXf3VktdFpxXx0/WmPxgX\nrE4RBPz0eB2vR0Qs6kjUFpyBnG1JYjn6g7\nam2/XyMr1V9+RrR05XLcpSbvNKN2w71i\nBPx0wzHaDeIRizoStdWMkHrEcvQHbnzuO\nGahcN1m5K0O8mj03a4UxNt/3DXnETNMSm\nRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B3\n1jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/U\nl3DabGFY3HOqGW2UyHSKzDmH",
  "m3B3\n1jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/U\nl3DabGFY3HOqGW2UyHSKzDmHxEYvxrOsQF\niMqRk7xhKUpEusQyeMQ53FI85hiKXVJeEV\nSx4qQLeXaUNkwaUsmgKUR6m3k6AxGIBOF\nOmyCWM7pzsudO0+hXazoLu65Ou5d0bFmqE\nETwNImuca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7l\nOaWUp+E",
  "zIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7l\nOaWUp+Efjhc0vJrxM/PLP0jNI9S/coLSwt\nKO1Z2qM0tDSk9KGlDykNLA0oXbV0lVJtK\nTmRwhPB0l1Kh5YOKT2w9IDSF5a+oPSxpY8\npfWnpS0rfWPqG0vuW3qeUWcoXbN0jVJuK\nXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx\n9QOnAUvKrGJ5nlpLjDTwYLZWUPrH",
  "uK\nXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx\n9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaXC\nUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZO\nkjSiNLybsBOJ1YukOpfQtU5pRuW7pN6aml\np+73Any6jL5rY27aBjYpTSxNKF23lPxSgK\nOEpSfkPBmq5q42edtE7muhmnIHazI+qU1y\nHqopd7Dm7jSpTe5PoZryIRn62t70RQqkF\nO70x3MLXfwWl",
  "E7muhmnIHazI+qU1y\nHqopd7Dm7jSpTe5PoZryIRn62t70RQqkF\nO70x3MLXfwWlhb2lhe7Py3e3b67cG+leUN\n7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/w\nPvt8CjQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nEach parameter multiplies its source and adds to its target\n22",
  "Depicting neural networks\nUsually don\u2019t show the bias terms\n<latexit sha1_base64=\"4mwMW+AxXHfh\nTyI/kaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxo\nk6a3pIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzU\nukyPJu9r1z/48KOP7nx6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKj\nJOUs8iQ/9M42ND+84Gk",
  "x6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKj\nJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f\n+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb346H49gvIq5yX7IsO+l1k/y0\nZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiO\nWTDMdtLGTIg",
  "Pmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiO\nWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8O\nFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyR\nUCD+dlyTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYa\nFBLJpy",
  "hBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYa\nFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03\nAkH0m9YjahiqkhEP9lvUHtl4xdYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiy\nJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3",
  "JKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae\n/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NR\neWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZhIqyIrFVpMP\n/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4Sqf",
  "rFVpMP\n/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI\n9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1\nClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu",
  "tvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU\n2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/\nIP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A\n0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPO",
  "Y2RNw8WhbdQiL/Uy0NR3A\n0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGNJgsQ6RPI4wX\nmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvIy68pTaBUr\nuoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684",
  "tmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieG\nPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh\n8Z+ohSZijdNPQTUq5oeTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZ\noeT2Bi6MhkpKnxv6nFJhKHl+84KXh",
  "WcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZ\noeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k\n4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMBTuUxobGlG4ZSp4U4FbC0DNyPxmo\n5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWev\ngtLC0c3O30HnTu795fe7je",
  "kPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWev\ngtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8\nSpb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_bas\ne64=\"OexYJKlfph/YpHs7sx68KlEAH9c=\"\n>AXC3iclZhbU9w2FICX9JbSG2mnvOTFU\n5pOp0ZlqSX",
  "h/YpHs7sx68KlEAH9c=\"\n>AXC3iclZhbU9w2FICX9JbSG2mnvOTFU\n5pOp0ZlqSXl84kEHKDFAgskLCEkb2yV0G\nWjS3DEs/+hE5/TN86fe2P6A/pe49s7wqfI\nx6yM8mK8326Hcm21n4qRa6Xlv6dufbOu+\n+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSKT\n7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41\nkuErWrL1J+FLNIiVAETEPoeO6P4XHZH",
  "MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41\nkuErWrL1J+FLNIiVAETEPoeO6P4XHZHXvf\n/Or1Yz8ZlWx82NdDrhmEl8be970r+54dO\nT1+7NQYdldYblVYflShTvuCndaFe6YCsd\nzC0uLS9XHo4VuU1joNJ+t4xtfDvqDJChir\nnQgWZ4fdpdSfVSyTItA8vFsv8h5yoITFvF\nDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaY",
  "Fsv8h5yoITFvF\nDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFd\nB3VFYSE8nlkGbyAyHmh5AQUWZALG6gVD\nlrFAw2LN9hU/D5I4ZmpQ9lfWtsdl3+eRUC\nU/LaqFG4/bzlrlcCheZaw82Z2IjSPxRtO\nGqkU08gVAo/GZckXo0UMBAcgFjkBieI5tG\nny4deF1HYqBJwW+MPhjPx6RpXkEOWl\npL4k",
  "gVAo/GZckXo0UMBAcgFjkBieI5tG\nny4deF1HYqBJwW+MPhjPx6RpXkEOWl\npL4kGhVTyUctaJRYsZdxSdkDxvFueAVxns\nAowVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJ\nedQFTDpg0M2obqpASqgYt6zdsPWfqpElck\nlZDzUwEWbtZ29EZzYsatJ0qgizYhFHbqiL\nIknBbGbCYQZab8jFMOPZMxK0KhVBNuZW\nlvjtvlMTwXtzlML10vbWSpL",
  "hFHbqiL\nIknBbGbCYQZab8jFMOPZMxK0KhVBNuZW\nlvjtvlMTwXtzlML10vbWSpL+M4YyYgJw9Z\nlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7As\nqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqD\nwV7QmaAL7oikyo8J2uyrBljXh/m2YalZI\nfvjD4o98dFQumcvG/EeyCQ3lRepqyITfo\nqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luG\nNbSLV2kFB",
  "cvG/EeyCQ3lRepqyITfo\nqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luG\nNbSLV2kFBKCaFvkCXv4hUu04VwYNYjRWC\nJh24ZsJhRY5DNuyCRgZvuGR7NhAZpkUM8\nxkEleZJzc/NB+hkilm9tiJszDqn1DlUZo3\nze4nNaCMjwczvgV1X2Ub/Op58UasAylM\nyRWdLRq36u4RJzXf3VktdFpxXx0/WmPxgX\nrE4RBPz0eB2vR0Qs6kjUFpyBnG1JYj",
  "RWdLRq36u4RJzXf3VktdFpxXx0/WmPxgX\nrE4RBPz0eB2vR0Qs6kjUFpyBnG1JYjn6g7\nam2/XyMr1V9+RrR05XLcpSbvNKN2w71i\nBPx0wzHaDeIRizoStdWMkHrEcvQHbnzuO\nGahcN1m5K0O8mj03a4UxNt/3DXnETNMSm\nRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B3\n1jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/U\nl3DabGFY3HOqGW2U",
  "RLrEBbjom3B3\n1jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/U\nl3DabGFY3HOqGW2UyHSKzDmHxEYvxrOsQF\niMqRk7xhKUpEusQyeMQ53FI85hiKXVJeEV\nSx4qQLeXaUNkwaUsmgKUR6m3k6AxGIBOF\nOmyCWM7pzsudO0+hXazoLu65Ou5d0bFmqE\nETwNImuca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7l",
  "muca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7l\nOaWUp+Efjhc0vJrxM/PLP0jNI9S/coLSwt\nKO1Z2qM0tDSk9KGlDykNLA0oXbV0lVJtK\nTmRwhPB0l1Kh5YOKT2w9IDSF5a+oPSxpY8\npfWnpS0rfWPqG0vuW3qeUWcoXbN0jVJuK\nXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx\n9QOnAUvKrGJ5nlpLjDTw",
  "oXbN0jVJuK\nXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx\n9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaXC\nUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZO\nkjSiNLybsBOJ1YukOpfQtU5pRuW7pN6aml\np+73Any6jL5rY27aBjYpTSxNKF23lPxSgK\nOEpSfkPBmq5q42edtE7muhmnIHazI+qU1y\nHqopd7Dm7jSpTe5PoZryIRn62t70RQqkF\nO70x",
  "q5q42edtE7muhmnIHazI+qU1y\nHqopd7Dm7jSpTe5PoZryIRn62t70RQqkF\nO70x3MLXfwWlhb2lhe7Py3e3b67cG+leUN\n7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/w\nPvt8CjQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n23",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 output\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n24",
  "With 3 hidden units:\n<latexit sha1_ba\nse64=\"GTMey3nznlpmn4PVpDS172U4Y8\ns=\">AWqHiclZhb9s2FIDV7tZ1t3TD\n8rIXYUGBoWuDeOguLwPapOkt6eI0cZI2\nTg1KpmQ2FKVIVOJU8J/Yr9nr9i/2b3Yo\nyWZ1DvMwA62Z83i5ZCUaAWZFIVeW/v3\n2vUPvzo409ufHrzs8+/+PKrpVtfHxRp\nmYd8EKYyzY8CVnApFB9oSU/ynLOkDy\nw+B0w/Dc54XIlX",
  "rzs8+/+PKrpVtfHxRp\nmYd8EKYyzY8CVnApFB9oSU/ynLOkDy\nw+B0w/Dc54XIlX7+jLjJwmLlYhEyDSE\nRkt3J6NqPN/94dJkE4rNjse6gnXDKJr\nM/9Hf/FXbzY9GS2trK2u1R+fFnptYcV\nrP/3RrW/Hw3EalglXOpSsKI57a5k+qVi\nuRSj57OawLHjGwlMW82MoKpbw4qSqhzX\nzb0Nk7EdpDv+U9uvo+1dULCmKyQAM2F\n6UmBmgi52XOr",
  "wlMW82MoKpbw4qSqhzX\nzb0Nk7EdpDv+U9uvo+1dULCmKyQAM2F\n6UmBmgi52XOrot5NKqKzUXIVNQ1EpfZ3\n6Jkf+WOQ81PISCizMBfTVDycsZ6GTN4\ncKn4RpknC1Lgarm/uzqphwGOhKn5W1lm\ndzbrOZu1wKF5lrD/bX9QiNE/EO04qRV\nTyRUCj2dVxVfjVQwEByBWOQGp4gXUafI\nTRH4PUVhFEnDVLIwhGC9npGqleQw56Wi\nviQaFTPJ",
  "VfjVQwEByBWOQGp4gXUafI\nTRH4PUVhFEnDVLIwhGC9npGqleQw56Wi\nviQaFTPJpx9ogFkxl0lH2QPH9274BXO\ncwC9BV+OJoDvYypmbz6zSf6jypChPDLe\nRMxbxuAoYcMmlG1DVUKSVcGnasP7D1kq\nnTNnFpVnc1NxFk7edR+c0L2rcdeoIsm\nARxl2rjiBLwp4fs4RBltvyCAac+CbiVo\nXCqiALs5+nQbftzETw2pxmsF+63mZF0n\n/OUE",
  "jiBLwp4fs4RBltvyCAac+CbiVo\nXCqiALs5+nQbftzETw2pxmsF+63mZF0n\n/OUEZMAHaf+RZMhbyrb6QL258n57z2TY\nFP/QlMVvcSlsfNsOaNwKja2Iyada6QSb\nMFoTy96JqmNw6VZ6I7QBPAm67MhYre0+\n7WJViyJjy8C0PNS8mP763+zKcn1ZrZNu\nY/k2oqCgzV0Um/D8qGsNTBq8viODJS\nyWaPAjUk5dKuL+jqWM5XtgmUs8dFIRiU",
  "Y/k2oqCgzV0Um/D8qGsNTBq8viODJS\nyWaPAjUk5dKuL+jqWM5XtgmUs8dFIRiU\nuhLtP1FrLrX1BHc2TRBfYWAqRe+mVBok\nqOoK5uAkeEbnpeOBRSiQYbNGEOZFmXOy\nc0PrWeI1Lq5LebCPKy6N1RphO59g8vFV\nVCGh8M5v+LyAGU0aPIZpKUasxwlc2qmd\nPpmWGjYq7dX095U3RaMT/batuDfsHsl\nGHIz0ZbeD5iYlFHorgOKsSxL0R7U",
  "qmd\nPpmWGjYq7dX095U3RaMT/batuDfsHsl\nGHIz0ZbeD5iYlFHorgOKsSxL0R7Ut\nViu7/es2npzhyzt2OG6TUnqbXvpth3uF\nT3gZ9uO3m4Tj1jUkaiutofUI5ajPajL\nncdt1ygcrtuUpN5Hp2w12YaPlH+Yk\nao5JqRybY18qh0Ii5qK2imCY+R2ISw\nmJRdC/7Gyp6Ah0fXakJY7Beiq5kAlsZc\n4iE0ISw2W7hrtjGsbjvUbfKZDZBZhPC",
  "mJRdC/7Gyp6Ah0fXakJY7Beiq5kAlsZc\n4iE0ISw2W7hrtjGsbjvUbfKZDZBZhPC\n4hOW4FE3ISzGVIyd4inLMiQ2IZLHCc7j\nhOYxw1LmkvCMZI4ZIUvKtaDySdqVTABL\nU9Ta1NEY9ECmCjXYBrFc0JVXOFeQqtY\n0VU8cDU8uKJhzVCFJoClHbLH/OGOc5MF\nOMVwzHIlORPIymgC+9jpU2d+guipzk\ngujS0ktKLy9oPTQ0kNKc0vJL4Igem",
  "5MF\nOMVwzHIlORPIymgC+9jpU2d+guipzk\ngujS0ktKLy9oPTQ0kNKc0vJL4Igemk\np+XUSROeWnlN6YOkBpaWlJaUDSweURpZ\nGlD629DGloaUhpRuWblCqLSUnUngiWLp\nP6cTSCaVHlh5R+srSV5Q+tfQpa8tfU3\npO0vfUfrQ0oeUMksZpZuWblLKLSWvDoJ\no3dJ1SgNLyW8/2GuW9inNLM0ofWTpI0r\nHlpJfxfA8s5Qcb+DBaKmk9Jmlz",
  "vDoJ\no3dJ1SgNLyW8/2GuW9inNLM0ofWTpI0r\nHlpJfxfA8s5Qcb+DBaKmk9JmlzygVlpL\nfb0H0wtIXlCaWJpQ+t/Q5pW8tfUvpE0u\nfUBpbSt4NwOnE0j1K7VugqB019JdSs8\nsPXO/F+CLaQxcC3PHVrBDaWpSumWpeS\nXAhwlLD0l58lItXe1+dsmcl+L1I7WJ\nvx+dUk5FacAdr707zq8n9KVILPiFd3z\nxYvEiBlMKdfrS0sNvYWnh4KfV",
  "L1I7WJ\nvx+dUk5FacAdr707zq8n9KVILPiFd3z\nxYvEiBlMKdfrS0sNvYWnh4KfV3i+r93\nfvrzxYb9/Q3vC+873fvB63q/eA+p1/\ncGXuj96f3l/e39s3xnub98uPyqUa9fa6\n/5xut8loP/AJCt4Gc=</latexit>hd = a[\u2713d0 + \u2713d1x]\n<latexit sha1_base64=\"VOjV5fHfEnHrCM\nyzG9o2bUpFioE=\">AWpHiclZhb9s2FIDV7tZ1t3TD8rIXY",
  "4=\"VOjV5fHfEnHrCM\nyzG9o2bUpFioE=\">AWpHiclZhb9s2FIDV7tZ1t3TD8rIXYUGHYmsNe+guLwXapOkt6eI\n0cZI2TgxKpmQ2FKXoktgV/Bf2a/a6/Y/9mx1KslmdwzMQGP2fJ94OSQlWl4iRZ3u/9eu/7\nBhx9/MmNT29+9vkX361cuvrgywuUp8P/FjG6ZHMi6F4oNc5JIfJSlnkSf5oXe2ofnhBU\n8zEav9fJbwk4iFSgTCZzmERit3Zu4",
  "G6ZHMi6F4oNc5JIfJSlnkSf5oXe2ofnhBU\n8zEav9fJbwk4iFSgTCZzmERit3Zu4Dd5hMxKjszn8aZkU0KscPevPT8vG8Do/nE/1ntLW7\nXSrj0sLvaw5jSf/ujWt+PhOPaLiKvclyzLjnvdJD8pWZoLX/L5zWGR8YT5Zyzkx1BULOLZS\nVkNae7ehsjYDeIU/qncraLvX1GyKMtmkQdmxPJhpkO2thxkQe/n5RCJUXOlV83FBTSzWNX\n58cdi5T",
  "U/qncraLvX1GyKMtmkQdmxPJhpkO2thxkQe/n5RCJUXOlV83FBTSzWNX\n58cdi5T7uZxBgfmpgL6/oSlzM8hizeHil/6cRQxNS6H65u783Lo8VCokp8XVUbn87azWTk\ncilcZ68/3l7WInEfiHSeVIqu5AqBh/Oy5J2wg4HgAESHExArnkGdOj9e4PYQhRUkAQP34il\n0LnBfzUnVKuch5KSlvSEaFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsI",
  "AQP34il\n0LnBfzUnVKuch5KSlvSEaFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1X1yX82\nmeRmWmY7iFlKmQV03AkH0m9YjahiqkhEv9lvUHtl4xdYkLk6qrqY6gqz9tO3kKc2LGredK\noIsWIRh26oiyJKw38csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUE\nR2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3A",
  "Lsp7HXbjvREbw2pwnsl7a3WZL0XzCUE\nR2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVvoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6N\nReWJaA9QB/CmK1Khgve0u1UJlqwOD+/CUNC8uN7nV/49KTs6m2j/5BsQkVZkdgq0uH/UdE\nYnjB4fUET14s0eRBoJq8WML9HU0dS/HC1pFq7qAgFJMin6HtL0LVvqaK4M7GEeorBHS98M2\nEQpMcBG1ZB",
  "q8WML9HU0dS/HC1pFq7qAgFJMin6HtL0LVvqaK4M7GEeorBHS98M2\nEQpMcBG1ZB7QM3/CstCwgHw3Sr8foyzgrUk5ufmg9Q6TS9W0xFfph1b6hSi207xtcLq+CMj\nwcLvgVl3so16dTy8u1JilKJlTPaXT02GWwxaz7f5qyui1Qr5+VbTHvQLZqfwfX4+2sLzE\nRKLOhLVBYcTa12SWJb2oK7lcn2/Z+XW6Y9kaYcW125KUm/TS7tca/oAT/ftv",
  "LzE\nRKLOhLVBYcTa12SWJb2oK7lcn2/Z+XW6Y9kaYcW125KUm/TS7tca/oAT/ftvR2m3jEo5Ed\nTU9pB6xLO1BXfY8btGYXHtpiT1LvJotS3u0kTLP9if8JzpY1Isx/rYF8thHcJiTsXcKsYR\nD5FYh7AYFW0L/o+VPQEPj7ZVh7DYz0Rb0wEsjbnEQ6hDWKy3cNtsYljdtqjbdpXJZILMOoT\nFpyzCo65DWAypGFrFM5YkSKxDJI8TnMcJzWOCp",
  "y3cNtsYljdtqjbdpXJZILMOoT\nFpyzCo65DWAypGFrFM5YkSKxDJI8TnMcJzWOCpcQm4RlJLDNClpRtQaWTuC3pAJamqLWpTH\nogYwVarAJYjmjKy+zrjyFVrGiq3hga3hwRcM5QxXqAJZ2yB5zhzvWTebhFMxy5bkRCAroQ\nnsY6dPncXpzwtKcpLzgpmhM0ovDb2k9NDQ0pTQ8kvAi94ZSj5deIF4ZeUHpg6AGlhaEFp\nQNDB5QGhgaUPjH0Ca",
  "0ovDb2k9NDQ0pTQ8kvAi94ZSj5deIF4ZeUHpg6AGlhaEFp\nQNDB5QGhgaUPjH0CaW+oT6lG4ZuUJobSk6k8EQwdJ/SiaETSo8MPaL0taGvKX1m6DNK3xj6h\ntJ3hr6j9JGhjyhlhjJKNw3dpJQbSl4deMG6oeuUeoaS36w1wztU5oYmlD62NDHlI4NJb+K\n4XlmKDnewIPRUEnpc0OfUyoMJb/fvOCloS8pjQyNKH1h6AtK3xr6ltKnhj6lNDSUv",
  "XlmKDnewIPRUEnpc0OfUyoMJb/fvOCloS8pjQyNKH1h6AtK3xr6ltKnhj6lNDSUvBuA04m\nhe5Sat0BlRumuobuUnht6bn8vwJfT6NkW5o6pYIfS2NCY0i1DyS8FOEoYekbOk4Fq7mqLt03\nkvhaoJbewJuOLq0nOA7XkFtbcnRZXk/tToJZ8Qrq+ebB8kQIphTv9aGWth9/C0sLBz53er5\n37u/fXHq43b2hvON853zt3nJ7zm/PQeb0nYHjO38",
  "phTv9aGWth9/C0sLBz53er5\n37u/fXHq43b2hvON853zt3nJ7zm/PQeb0nYHjO386fzl/O/+s/rC6vbq3OqjV69ea75xW\np/V0/8AZeTfEg=</latexit>\ny = \u03c60 +\nD\nX\nd=1\n\u03c6dhd\n<latexit sha1_base64=\"4mwMW+AxXHfh\nTyI/kaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxo\nk6a3pIvTxEmaODUomZLZUJSiS2",
  "hb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxo\nk6a3pIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzU\nukyPJu9r1z/48KOP7nx6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKj\nJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f\n+aX3f1r7uLX/f0r3vz0cpa",
  "xJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f\n+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb346H49gvIq5yX7IsO+l1k/y0\nZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiO\nWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8O\nFb/04yhialwO1zd3",
  "SoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8O\nFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyR\nUCD+dlyTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYa\nFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03\nAkH0m9Yj",
  "lVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03\nAkH0m9YjahiqkhEP9lvUHtl4xdYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiy\nJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae\n/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NR\ne",
  "M+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NR\neWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZhIqyIrFVpMP\n/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI\n9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKv",
  "NI\n9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1\nClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk5\n1tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU\n2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7",
  "Zoce2mJPU\n2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/\nIP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A\n0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGNJgsQ6RPI4wX\nmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHsh",
  "iqFVPGNJgsQ6RPI4wX\nmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvIy68pTaBUr\nuoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieG\nPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48",
  "PD2gtDC0oHRg6IDSwNCA0ieG\nPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh\n8Z+ohSZijdNPQTUq5oeTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZ\noeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k\n4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs",
  "0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k\n4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMBTuUxobGlG4ZSp4U4FbC0DNyPxmo\n5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWev\ngtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8\nSpb9Wv1+rXmG+c1mdV/ge0FuR",
  "znfO/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8\nSpb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_bas\ne64=\"OexYJKlfph/YpHs7sx68KlEAH9c=\"\n>AXC3iclZhbU9w2FICX9JbSG2mnvOTFU\n5pOp0ZlqSXl84kEHKDFAgskLCEkb2yV0G\nWjS3DEs/+hE5/TN86fe2P6A/pe49s7wqfI\nx6yM8mK8326",
  "KDFAgskLCEkb2yV0G\nWjS3DEs/+hE5/TN86fe2P6A/pe49s7wqfI\nx6yM8mK8326Hcm21n4qRa6Xlv6dufbOu+\n+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSKT\n7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41\nkuErWrL1J+FLNIiVAETEPoeO6P4XHZHXvf\n/Or1Yz8ZlWx82NdDrhmEl8be970r+54dO\nT1+7NQYdldYblVYflShTvuCndaFe6YCs",
  "r1Yz8ZlWx82NdDrhmEl8be970r+54dO\nT1+7NQYdldYblVYflShTvuCndaFe6YCsd\nzC0uLS9XHo4VuU1joNJ+t4xtfDvqDJChir\nnQgWZ4fdpdSfVSyTItA8vFsv8h5yoITFvF\nDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFd\nB3VFYSE8nlkGbyAyHmh5AQUWZALG6gVD\nlrFAw2LN9hU/D5I4Z",
  "Y6/OWoFCotNFd\nB3VFYSE8nlkGbyAyHmh5AQUWZALG6gVD\nlrFAw2LN9hU/D5I4ZmpQ9lfWtsdl3+eRUC\nU/LaqFG4/bzlrlcCheZaw82Z2IjSPxRtO\nGqkU08gVAo/GZckXo0UMBAcgFjkBieI5tG\nny4deF1HYqBJwW+MPhjPx6RpXkEOWl\npL4kGhVTyUctaJRYsZdxSdkDxvFueAVxns\nAowVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJ\nedQF",
  "ctaJRYsZdxSdkDxvFueAVxns\nAowVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJ\nedQFTDpg0M2obqpASqgYt6zdsPWfqpElck\nlZDzUwEWbtZ29EZzYsatJ0qgizYhFHbqiL\nIknBbGbCYQZab8jFMOPZMxK0KhVBNuZW\nlvjtvlMTwXtzlML10vbWSpL+M4YyYgJw9Z\nlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7As\nqc16QRm1cTG1KxyhUyaLQhl",
  "gJw9Z\nlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7As\nqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqD\nwV7QmaAL7oikyo8J2uyrBljXh/m2YalZI\nfvjD4o98dFQumcvG/EeyCQ3lRepqyITfo\nqEBPMjw/oIXrxEosWDQLV4iYT7O1o6luG\nNbSLV2kFBKCaFvkCXv4hUu04VwYNYjRWC\nJh24ZsJhRY5DNuyCRgZvuGR7NhAZpkUM8\nxkEleZJzc/N",
  "CXv4hUu04VwYNYjRWC\nJh24ZsJhRY5DNuyCRgZvuGR7NhAZpkUM8\nxkEleZJzc/NB+hkilm9tiJszDqn1DlUZo3\nze4nNaCMjwczvgV1X2Ub/Op58UasAylM\nyRWdLRq36u4RJzXf3VktdFpxXx0/WmPxgX\nrE4RBPz0eB2vR0Qs6kjUFpyBnG1JYjn6g7\nam2/XyMr1V9+RrR05XLcpSbvNKN2w71i\nBPx0wzHaDeIRizoStdWMkHrEcvQHbnzu",
  "m2/XyMr1V9+RrR05XLcpSbvNKN2w71i\nBPx0wzHaDeIRizoStdWMkHrEcvQHbnzuO\nGahcN1m5K0O8mj03a4UxNt/3DXnETNMSm\nRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B3\n1jZEfDwaFt1CItbuWhrJoClAZd4CnUIi/U\nl3DabGFY3HOqGW2UyHSKzDmHxEYvxrOsQF\niMqRk7xhKUpEusQyeMQ53FI85hiKXVJeEV\nSx4qQLeXaUNkwaUs",
  "mHxEYvxrOsQF\niMqRk7xhKUpEusQyeMQ53FI85hiKXVJeEV\nSx4qQLeXaUNkwaUsmgKUR6m3k6AxGIBOF\nOmyCWM7pzsudO0+hXazoLu65Ou5d0bFmqE\nETwNImuca8/qbzIvNxiuGY5UpyKpCV0gRu\nYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7l\nOaWUp+Efjhc0vJrxM/PLP0jNI9S/coLSwt\nKO1Z2qM0tDSk9KGlDykNLA0oXbV0lVJtK",
  "+Efjhc0vJrxM/PLP0jNI9S/coLSwt\nKO1Z2qM0tDSk9KGlDykNLA0oXbV0lVJtK\nTmRwhPB0l1Kh5YOKT2w9IDSF5a+oPSxpY8\npfWnpS0rfWPqG0vuW3qeUWcoXbN0jVJuK\nXl14Icrlq5Q6ltKfvBtWbpFqWpSmlDyx\n9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaXC\nUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZO\nkjSiNLybsBOJ1YukOpfQt",
  "rH0CaXC\nUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZO\nkjSiNLybsBOJ1YukOpfQtU5pRuW7pN6aml\np+73Any6jL5rY27aBjYpTSxNKF23lPxSgK\nOEpSfkPBmq5q42edtE7muhmnIHazI+qU1y\nHqopd7Dm7jSpTe5PoZryIRn62t70RQqkF\nO70x3MLXfwWlhb2lhe7Py3e3b67cG+leUN\n7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c",
  "Wlhb2lhe7Py3e3b67cG+leUN\n7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/w\nPvt8CjQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nWith D hidden units:\n25",
  "With enough hidden units\u2026\n\u2026 we can describe any 1D function to arbitrary accuracy\n26",
  "Universal approximation theorem\n\u201ca formal proof that, with enough hidden units, a shallow \nneural network can describe any continuous function in        to \narbitrary precision\u201d\n<latexit sha1_base64=\"K4U1ZHkglIR02KHQbhLTcSGE2I=\">AWiHiclZhb9\ns2FIDVXbvs1m5YXvYiLCgwDJ3hDN26vbVJ01vSxbk4SRsnASVTMhuKUiQqcSr4f+x1+1f7NzuUZLM6h3mYgdTs+T7xckhKtIJMikL3+/e+uDjz7+5NPbny19/sWX3195\n+43B0Va5iEfhqlM86OAFVwKxYdaMm",
  "tIJMikL3+/e+uDjz7+5NPbny19/sWX3195\n+43B0Va5iEfhqlM86OAFVwKxYdaMmPspyzJD8MDhfN/zwkueFSNW+vs74ScJiJSIRMg2h01HC9CQIqt3ZafVkdnZnpd/r1x+fFlbworXfgZnd78bj8ZpWCZc6VCyojhe\n7Wf6pGK5FqHks6VRWfCMhecs5sdQVCzhxUlVd3vm34PI2I/SHP6U9uvo+1dULCmK6yQA03SzwMwEXey41NHvJ5VQWam5",
  "VCzhxUlVd3vm34PI2I/SHP6U9uvo+1dULCmK6yQA03SzwMwEXey41NHvJ5VQWam5CpuGolL6OvVNDvyxyHmo5TUWJgL6KsfTljOQg2\nZWhopfhWmScLUuBqtbezMqlHAY6EqflHWZvNus5G7XAo3mSsvdhf1CI0T8Q7TiqpFVPJDQKPZ1XFe3EPA8EBiB4nIFW8gDrKY78VURhlUjAwIN0Cp2L/N0ZqVpHkNOt\nobokEhk3zasdaJBVOZdJQ9UHz",
  "FW8gDrKY78VURhlUjAwIN0Cp2L/N0ZqVpHkNOt\nobokEhk3zasdaJBVOZdJQ9UHz/nm8A1znMAnQVvjiag72Mqdn8Os2nOk+qwsRwCzlTMa+bgCGHTJoRdQ1VSgmXh3rT2ztMnXeJi7N6q7mJoKs/bzr6JzmRY27Th1BFizCu\nGvVEWRJ2NjljDIcls+gwEnvom4VaGwKsjCHORp0G07MxG8NqcZ7Jeut1GR9F8ylBETgN1nvgVTIe/q6+nC9ufJ",
  "vom4VaGwKsjCHORp0G07MxG8NqcZ7Jeut1GR9F8ylBETgN1nvgVTIe/q6+nC9ufJuax9U+BTfwKT1b2E5XEzrHkjMKo2NqNmnStk0mxBKE+vu\nqbpjUPlmegO0ATwpitzoaL3tPt1CZasCY/uw1DzUvLjn3u/8ulJ1TfbxvxDsgkVFWXmqsiE/0dFY3iK4PUFETx5qUSTB4F68lIJ93c0dSzHC9tE6rmDglBMCn2Ntr+IVfea\nOoI7myaorxAw9c",
  "FETx5qUSTB4F68lIJ93c0dSzHC9tE6rmDglBMCn2Ntr+IVfea\nOoI7myaorxAw9cI3EwpNchR1ZRMwMnzD89CxgEI0yLAZYyjTosw5ufmh9QyRWje3xVyYh1X3hiqN0L1vcLm4CsrwcLjkN1weoIwGT6DtFRjlqNkTs2UTk9HhYt5tr9ZQ\n3RacV84vNtj3oF8xOGYb84mwTz0dMLOpIVBcQJx1SWI52oO6Fsv1/Z5Vm6c/kaUdO1y3KUm9bS/dt",
  "F8xOGYb84mwTz0dMLOpIVBcQJx1SWI52oO6Fsv1/Z5Vm6c/kaUdO1y3KUm9bS/dtsO9oQf8YsvR2y3iEYs6EtXV9pB6xHK0B3W587jlGoXDdZuS1DvPo9\nN2uAsTLf9of8I1M8ekVI7NsS+VoyaERU1F7RThMdIbEJYTMquBf/Hyp6Ah0fXakJYHBSiq5kAlsZc4iE0ISw2W7hrtjGsbjnULbfKZDZBZhPC4jOW4FE3ISzGVIyd4jnLM\niQ2IZLH",
  "sZc4iE0ISw2W7hrtjGsbjnULbfKZDZBZhPC4jOW4FE3ISzGVIyd4jnLM\niQ2IZLHCc7jhOYxw1LmkvCMZI4ZIUvKtaDySdqVTABLU9Ta1NEY9ECmCjXYBrFc0JVXOFeQqtY0VU8dDU8vKFhzVCFJoClbLH/NG2c5MFOMVwzHIlORPIymgCB9gZUGd+\n+guipzkguja0mtKry9ovTQ0kNKc0vJL4Ig2rWU/DoJoktLyk9sPSA0tLSktKhpUNKI0sj",
  "zkguja0mtKry9ovTQ0kNKc0vJL4Ig2rWU/DoJoktLyk9sPSA0tLSktKhpUNKI0sjSp9a+pTS0NKQ0nVL1ynVlpITKTwRLN2ndGLphNIjS48ofW3pa0qfW/qc0je\nWvqH0naXvKH1s6WNKmaWM0g1LNyjlpJXB0G0ZukapYGl5Lcf7DVLB5RmlmaUPrH0CaVjS8mvYnieWUqON/BgtFRS+sLSF5QKS8nvtyB6ZekrShNLE0pfWvqS0reWvqX0ma\nq",
  "S8mvYnieWUqON/BgtFRS+sLSF5QKS8nvtyB6ZekrShNLE0pfWvqS0reWvqX0ma\nqRASuFOf3ZnZRW/haWFg196q7/1Huw8WHm01r6hve197/3g/eiteg+9R95zb+ANvdDLvb+8v71/lpeW+8sPl/9o1A9utd863U+y2v/ATPZ1Fc=</latexit>XPKI0tJe8G4HRi6R6l9i1QVC6Y+kOpReWXrjfC/DFNAauhbltK9imNLU0pXTUvJLAY4Slp6T8",
  "G4HRi6R6l9i1QVC6Y+kOpReWXrjfC/DFNAauhbltK9imNLU0pXTUvJLAY4Slp6T82Sk2rva/G0Tua9FasEdrM34/GqS80gtuIO1d6f51eT+FKkFn5CubxwsX\nRD\n27",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 ouput\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n28",
  "Two outputs \n\u2022 1 input, 4 hidden units, 2 outputs\n<latexit sha1_base64=\"R\n8vqOAXY+V+odbvKA1u5ta3TLRI=\">AXWniclZhbU9w\n2FICX9JaQXkhvPTFUyadTpsyLKGXl84kEHKDFAjXBN\nG9speBVk2tgxLPtDO9Ppb+mR7V3F54iH7ky4nyfbke\nyrXWQSVHopaW/Z258OFH39y89bs7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiF",
  "7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiFTt6auMnyQsViISIdMQOp37d3ha9cfeD3/6SZCOKjY\n+9vWQawbRpbH3szf9qz8enXi+SlWZBDz3fH8Wai6bmh6p\nutypuyset9d9X6n6n1n1RV31ZVO1RVTdfZ0bmFpcan+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgf",
  "an+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgfQ1GxhBcnVb0QY+8uRAZelObwT2mv\njr5fo2JUVwlAZgJ08MCMxN0seNSR3+cVEJlpeYqbDqK\nSunp1DOr6g1EzkMtr6DAwlzAWL1wyHIWalj7WV/xyzBN\nEqYGlb+6vjOu/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iID",
  "Ou/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iIDgAscgJSBUvoE2Tn\nyDy+ojCvpeAq2Zn+GC8HJOmleYx5KSjvSYaFDLJRx1rj\nViwlElH2QXF8+56BnCdwyrAUOGLozXYzZgaT+pPtJ5U\nhUmhnvImYp53QVMOWTSzKhrqFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasAS",
  "qFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasASB\nluy6cw4cQzEbcqFYF2ZjbeRp0+85MBO/NUQbXS9dbr\n0j6LxjKiAnA1We+BVMh7+pr6dT2Jsm5qH1T4CNvCIvVrc\nLyuJnWpBOYVRsbU7POFTJptiCUp5d04zGofJMdCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+",
  "dCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+Hw0N4LmI9xdE8OKlEi0eBOrFSyXc39\nHSsRxvbBOp1w4KQjEp9BW6/EWsunXqCB5smqCxQsC0C9\n9MKLTIUdSVTcDI8A1PeMcGCtEkw2aOoUyLMufk5of2M0\nRq3dwWc2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS9",
  "2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS94UnVbMzfa/mBcsDp\nlGPLz0w28HjGxqCNRW3CkcrYlieXoD9qabtf3R1ZtvPm\nJbO3Y4bpNSdptR+m2He41I+Dnm47RbhKPWNSRqK12hNQ\njlqM/aMudx03XLByu25Sk3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do",
  "3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do2s\n1ISxuF6KrmQCWBlziKTQhLDaXcNdsY1jdKibpXJbIj\nMJoTFJyzBs25CWIypGDvFM5ZlSGxCJI9DnMchzWOGpcw\nl4RXJHCtCtpRrQ+XDtCuZAJZGqLeRozMYgUwV6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4G",
  "6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4GsjCZwGzvb1Jmc/oKoIie5ILqy9IrS0sv\nKT209JDS3FLyiyCIXlpKfp0E0YWlF5QeWHpAaWlpSem+\npfuURpZGlD629DGloaUhpWuWrlGqLSUnUngiWLpH6dDS\nIaVHlh5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLy",
  "h5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLyW8/uNYs3aY0szSj9JGlj\nygdWEp+FcPzFJyvIEHo6WS0meWPqNUWEp+vwXRC0tfU\nJpYmlD63NLnlL619C2lTyx9QmlsKXk3AKcTS3cptW+Bq\noLSHUt3KD239Nz9XoBPlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJzi",
  "PlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJziM15Q7W3\np0mtcn9KVJTPiRDXz+YvkiBlMKd/nRuoY/fwtLCwfJi/\n7fFlZ2VhQer7Rvam73vet/3fuz1e7/3HvSe9rZ7+71w5\ntHM25liRn/7z/yN+Vvztxv1xkxb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 +",
  "xb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nh4 = a[\u271340 + \u271341x]\n<latexit sha1_base64=\"Pl76mATi6i7yjrijGrQ\n4B2+SCU=\">AXIXiclZhZb9tGEIDl9EqdHk6L+qUvRI0URZsakuIeaFEgseNcdmo5PhPLEZbUktp4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+",
  "p4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+9/4H1z9cvPHRx598unTzs8M0\nyhOPH3iRjJjl6VcCsUPMpFJfhwnIWu5Efu2YbmRxc8SUWk9rNJzE9DFijhC49lEBos/TkZdJyvf+vHIzE\noOu3pd3WrMx3B/lmV2925t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2ar",
  "t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2arv8OLTRqRsrfrTG9z8YtgfRl4ecpV5kqXpSacdZ6cFSzLhST5d7Ocpj5l3xgJ+Ak3FQp6eFmU9p84ti\nAwdP0rgT2VOGX1zj4KFaToJXTBDlo1SzHTQxk7yzP/5tBAqzjOuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7L",
  "OuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7LfouD4Qq+HleTud02nQ2S4dD8ypj/fH+PIvIeChec5KkVHSKwQeTIu\nCrwarGAgOQKxyAiLFU8ip6+P6TgdRWL4SMHA3GsPgfOfZlKRWGQ+gJg3tBdGgEUs+blgbxIKpDBvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTUL",
  "BvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTULmUsKvXsH7H1jOmzurCRXE51ERHk\nLWfNJ0soXVRw6ZTRpAFizBoWmUEWRIuNkMWMqhy3R7AYeOjthVobAqyMLsJZHb7DvWEbw2xzGcL01vsyDl\nv2CoIjoAZ5/+Fkx5vKlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei",
  "KlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei+Y\nB6gA+6fJEKP8N7XbZgiWrw/3bcKhJLvnJ96s/8PFp0danjf5HqgmJ0jy2JdLht0g0hNsbXl8QwZMXSTR5EC\ngnL5JwfUdTxK8sHWknDtoCMWkyCbo9BeBau5TRvBgoxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH",
  "oxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH6ZtW8oEotNK8bXM73gjbcHC74Fbu7qKJuVU83ytWQJaiYz2l45f9NINTzHb2\nl1NeNa1WwM+36v5gXDA7uefx8EWno+AWNSRKBc8GVlzSWJZ+oNc8+X65siKrZfkqUdWFy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tN",
  "Fy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tNoWd26i5e/vj3jG9GNSJIf6sS+S/SqExY\nyKmVWMQh4gsQphMcybFmxjZU/AzaNpVSEs9lLR1HQAS0Mu8SFUISxWp3DTrGNY3bao23aVyXiEzCqExYcsx\nEdhbAYUDGwimcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhH",
  "mcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhHcRySlde\nal15Cq1iRVfxga3jgys6zhKqANY2iHnmNPfsZ5kLi4xPGbZihwLZMW0gD3s9Kgze/pz/YI8ybn+xNAJpZe\nGXlJ6ZOgRpYmh5BeB6z8zlPw6cf0LQy8oPT0kNLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLn",
  "NLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLnhj6n9JGhjyh9YegLSl8b+prSe4beo5QZyijdNHSTUm4oeXg+uGrlPqGkp+8G5Z\nmiP0tjQmNL7ht6ndGgo+VUM9zNDyeMN3BgNlZQ+NvQxpcJQ8vN9Z8a+pTS0NCQ0ieGPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DE",
  "GPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DEJdiNDI0o3TKU/FKARwlDz8jzpK/q9rsbRO\n5rvlqzi2srvhsb1JzX825hdVXp9ne5PrkqzkfkaFvHs5fpEBJ4Uo/WFrp4LewtHYXe38uLq2u7Zyd71+Q3\nu9WXrq9Y3rU7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ<",
  "U7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ</latexit\n>y1 = \u03c610 + \u03c611h1 + \u03c612h2 + \u03c613h3 + \u03c614h4\ny2 = \u03c620 + \u03c621h1 + \u03c622h2 + \u03c623h3 + \u03c624h4\n29",
  "Two outputs \n\u2022 1 input, 4 hidden units, 2 outputs\n<latexit sha1_base64=\"R\n8vqOAXY+V+odbvKA1u5ta3TLRI=\">AXWniclZhbU9w\n2FICX9JaQXkhvPTFUyadTpsyLKGXl84kEHKDFAjXBN\nG9speBVk2tgxLPtDO9Ppb+mR7V3F54iH7ky4nyfbke\nyrXWQSVHopaW/Z258OFH39y89bs7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiF",
  "7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiFTt6auMnyQsViISIdMQOp37d3ha9cfeD3/6SZCOKjY\n+9vWQawbRpbH3szf9qz8enXi+SlWZBDz3fH8Wai6bmh6p\nutypuyset9d9X6n6n1n1RV31ZVO1RVTdfZ0bmFpcan+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgf",
  "an+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgfQ1GxhBcnVb0QY+8uRAZelObwT2mv\njr5fo2JUVwlAZgJ08MCMxN0seNSR3+cVEJlpeYqbDqK\nSunp1DOr6g1EzkMtr6DAwlzAWL1wyHIWalj7WV/xyzBN\nEqYGlb+6vjOu/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iID",
  "Ou/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iIDgAscgJSBUvoE2Tn\nyDy+ojCvpeAq2Zn+GC8HJOmleYx5KSjvSYaFDLJRx1rj\nViwlElH2QXF8+56BnCdwyrAUOGLozXYzZgaT+pPtJ5U\nhUmhnvImYp53QVMOWTSzKhrqFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasAS",
  "qFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasASB\nluy6cw4cQzEbcqFYF2ZjbeRp0+85MBO/NUQbXS9dbr\n0j6LxjKiAnA1We+BVMh7+pr6dT2Jsm5qH1T4CNvCIvVrc\nLyuJnWpBOYVRsbU7POFTJptiCUp5d04zGofJMdCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+",
  "dCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+Hw0N4LmI9xdE8OKlEi0eBOrFSyXc39\nHSsRxvbBOp1w4KQjEp9BW6/EWsunXqCB5smqCxQsC0C9\n9MKLTIUdSVTcDI8A1PeMcGCtEkw2aOoUyLMufk5of2M0\nRq3dwWc2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS9",
  "2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS94UnVbMzfa/mBcsDp\nlGPLz0w28HjGxqCNRW3CkcrYlieXoD9qabtf3R1ZtvPm\nJbO3Y4bpNSdptR+m2He41I+Dnm47RbhKPWNSRqK12hNQ\njlqM/aMudx03XLByu25Sk3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do",
  "3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do2s\n1ISxuF6KrmQCWBlziKTQhLDaXcNdsY1jdKibpXJbIj\nMJoTFJyzBs25CWIypGDvFM5ZlSGxCJI9DnMchzWOGpcw\nl4RXJHCtCtpRrQ+XDtCuZAJZGqLeRozMYgUwV6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4G",
  "6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4GsjCZwGzvb1Jmc/oKoIie5ILqy9IrS0sv\nKT209JDS3FLyiyCIXlpKfp0E0YWlF5QeWHpAaWlpSem+\npfuURpZGlD629DGloaUhpWuWrlGqLSUnUngiWLpH6dDS\nIaVHlh5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLy",
  "h5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLyW8/uNYs3aY0szSj9JGlj\nygdWEp+FcPzFJyvIEHo6WS0meWPqNUWEp+vwXRC0tfU\nJpYmlD63NLnlL619C2lTyx9QmlsKXk3AKcTS3cptW+Bq\noLSHUt3KD239Nz9XoBPlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJzi",
  "PlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJziM15Q7W3\np0mtcn9KVJTPiRDXz+YvkiBlMKd/nRuoY/fwtLCwfJi/\n7fFlZ2VhQer7Rvam73vet/3fuz1e7/3HvSe9rZ7+71w5\ntHM25liRn/7z/yN+Vvztxv1xkxb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 +",
  "xb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nh4 = a[\u271340 + \u271341x]\n<latexit sha1_base64=\"Pl76mATi6i7yjrijGrQ\n4B2+SCU=\">AXIXiclZhZb9tGEIDl9EqdHk6L+qUvRI0URZsakuIeaFEgseNcdmo5PhPLEZbUktp4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+",
  "p4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+9/4H1z9cvPHRx598unTzs8M0\nyhOPH3iRjJjl6VcCsUPMpFJfhwnIWu5Efu2YbmRxc8SUWk9rNJzE9DFijhC49lEBos/TkZdJyvf+vHIzE\noOu3pd3WrMx3B/lmV2925t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2ar",
  "t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2arv8OLTRqRsrfrTG9z8YtgfRl4ecpV5kqXpSacdZ6cFSzLhST5d7Ocpj5l3xgJ+Ak3FQp6eFmU9p84ti\nAwdP0rgT2VOGX1zj4KFaToJXTBDlo1SzHTQxk7yzP/5tBAqzjOuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7L",
  "OuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7LfouD4Qq+HleTud02nQ2S4dD8ypj/fH+PIvIeChec5KkVHSKwQeTIu\nCrwarGAgOQKxyAiLFU8ip6+P6TgdRWL4SMHA3GsPgfOfZlKRWGQ+gJg3tBdGgEUs+blgbxIKpDBvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTUL",
  "BvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTULmUsKvXsH7H1jOmzurCRXE51ERHk\nLWfNJ0soXVRw6ZTRpAFizBoWmUEWRIuNkMWMqhy3R7AYeOjthVobAqyMLsJZHb7DvWEbw2xzGcL01vsyDl\nv2CoIjoAZ5/+Fkx5vKlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei",
  "KlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei+Y\nB6gA+6fJEKP8N7XbZgiWrw/3bcKhJLvnJ96s/8PFp0danjf5HqgmJ0jy2JdLht0g0hNsbXl8QwZMXSTR5EC\ngnL5JwfUdTxK8sHWknDtoCMWkyCbo9BeBau5TRvBgoxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH",
  "oxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH6ZtW8oEotNK8bXM73gjbcHC74Fbu7qKJuVU83ytWQJaiYz2l45f9NINTzHb2\nl1NeNa1WwM+36v5gXDA7uefx8EWno+AWNSRKBc8GVlzSWJZ+oNc8+X65siKrZfkqUdWFy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tN",
  "Fy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tNoWd26i5e/vj3jG9GNSJIf6sS+S/SqExY\nyKmVWMQh4gsQphMcybFmxjZU/AzaNpVSEs9lLR1HQAS0Mu8SFUISxWp3DTrGNY3bao23aVyXiEzCqExYcsx\nEdhbAYUDGwimcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhH",
  "mcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhHcRySlde\nal15Cq1iRVfxga3jgys6zhKqANY2iHnmNPfsZ5kLi4xPGbZihwLZMW0gD3s9Kgze/pz/YI8ybn+xNAJpZe\nGXlJ6ZOgRpYmh5BeB6z8zlPw6cf0LQy8oPT0kNLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLn",
  "NLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLnhj6n9JGhjyh9YegLSl8b+prSe4beo5QZyijdNHSTUm4oeXg+uGrlPqGkp+8G5Z\nmiP0tjQmNL7ht6ndGgo+VUM9zNDyeMN3BgNlZQ+NvQxpcJQ8vN9Z8a+pTS0NCQ0ieGPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DE",
  "GPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DEJdiNDI0o3TKU/FKARwlDz8jzpK/q9rsbRO\n5rvlqzi2srvhsb1JzX825hdVXp9ne5PrkqzkfkaFvHs5fpEBJ4Uo/WFrp4LewtHYXe38uLq2u7Zyd71+Q3\nu9WXrq9Y3rU7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ<",
  "U7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ</latexit\n>y1 = \u03c610 + \u03c611h1 + \u03c612h2 + \u03c613h3 + \u03c614h4\ny2 = \u03c620 + \u03c621h1 + \u03c622h2 + \u03c623h3 + \u03c624h4\n30",
  "Two outputs \n\u2022 1 input, 4 hidden units, 2 outputs\n<latexit sha1_base64=\"R\n8vqOAXY+V+odbvKA1u5ta3TLRI=\">AXWniclZhbU9w\n2FICX9JaQXkhvPTFUyadTpsyLKGXl84kEHKDFAjXBN\nG9speBVk2tgxLPtDO9Ppb+mR7V3F54iH7ky4nyfbke\nyrXWQSVHopaW/Z258OFH39y89bs7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiF",
  "7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiFTt6auMnyQsViISIdMQOp37d3ha9cfeD3/6SZCOKjY\n+9vWQawbRpbH3szf9qz8enXi+SlWZBDz3fH8Wai6bmh6p\nutypuyset9d9X6n6n1n1RV31ZVO1RVTdfZ0bmFpcan+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgf",
  "an+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgfQ1GxhBcnVb0QY+8uRAZelObwT2mv\njr5fo2JUVwlAZgJ08MCMxN0seNSR3+cVEJlpeYqbDqK\nSunp1DOr6g1EzkMtr6DAwlzAWL1wyHIWalj7WV/xyzBN\nEqYGlb+6vjOu/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iID",
  "Ou/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iIDgAscgJSBUvoE2Tn\nyDy+ojCvpeAq2Zn+GC8HJOmleYx5KSjvSYaFDLJRx1rj\nViwlElH2QXF8+56BnCdwyrAUOGLozXYzZgaT+pPtJ5U\nhUmhnvImYp53QVMOWTSzKhrqFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasAS",
  "qFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasASB\nluy6cw4cQzEbcqFYF2ZjbeRp0+85MBO/NUQbXS9dbr\n0j6LxjKiAnA1We+BVMh7+pr6dT2Jsm5qH1T4CNvCIvVrc\nLyuJnWpBOYVRsbU7POFTJptiCUp5d04zGofJMdCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+",
  "dCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+Hw0N4LmI9xdE8OKlEi0eBOrFSyXc39\nHSsRxvbBOp1w4KQjEp9BW6/EWsunXqCB5smqCxQsC0C9\n9MKLTIUdSVTcDI8A1PeMcGCtEkw2aOoUyLMufk5of2M0\nRq3dwWc2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS9",
  "2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS94UnVbMzfa/mBcsDp\nlGPLz0w28HjGxqCNRW3CkcrYlieXoD9qabtf3R1ZtvPm\nJbO3Y4bpNSdptR+m2He41I+Dnm47RbhKPWNSRqK12hNQ\njlqM/aMudx03XLByu25Sk3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do",
  "3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do2s\n1ISxuF6KrmQCWBlziKTQhLDaXcNdsY1jdKibpXJbIj\nMJoTFJyzBs25CWIypGDvFM5ZlSGxCJI9DnMchzWOGpcw\nl4RXJHCtCtpRrQ+XDtCuZAJZGqLeRozMYgUwV6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4G",
  "6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4GsjCZwGzvb1Jmc/oKoIie5ILqy9IrS0sv\nKT209JDS3FLyiyCIXlpKfp0E0YWlF5QeWHpAaWlpSem+\npfuURpZGlD629DGloaUhpWuWrlGqLSUnUngiWLpH6dDS\nIaVHlh5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLy",
  "h5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLyW8/uNYs3aY0szSj9JGlj\nygdWEp+FcPzFJyvIEHo6WS0meWPqNUWEp+vwXRC0tfU\nJpYmlD63NLnlL619C2lTyx9QmlsKXk3AKcTS3cptW+Bq\noLSHUt3KD239Nz9XoBPlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJzi",
  "PlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJziM15Q7W3\np0mtcn9KVJTPiRDXz+YvkiBlMKd/nRuoY/fwtLCwfJi/\n7fFlZ2VhQer7Rvam73vet/3fuz1e7/3HvSe9rZ7+71w5\ntHM25liRn/7z/yN+Vvztxv1xkxb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 +",
  "xb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nh4 = a[\u271340 + \u271341x]\n<latexit sha1_base64=\"Pl76mATi6i7yjrijGrQ\n4B2+SCU=\">AXIXiclZhZb9tGEIDl9EqdHk6L+qUvRI0URZsakuIeaFEgseNcdmo5PhPLEZbUktp4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+",
  "p4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+9/4H1z9cvPHRx598unTzs8M0\nyhOPH3iRjJjl6VcCsUPMpFJfhwnIWu5Efu2YbmRxc8SUWk9rNJzE9DFijhC49lEBos/TkZdJyvf+vHIzE\noOu3pd3WrMx3B/lmV2925t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2ar",
  "t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2arv8OLTRqRsrfrTG9z8YtgfRl4ecpV5kqXpSacdZ6cFSzLhST5d7Ocpj5l3xgJ+Ak3FQp6eFmU9p84ti\nAwdP0rgT2VOGX1zj4KFaToJXTBDlo1SzHTQxk7yzP/5tBAqzjOuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7L",
  "OuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7LfouD4Qq+HleTud02nQ2S4dD8ypj/fH+PIvIeChec5KkVHSKwQeTIu\nCrwarGAgOQKxyAiLFU8ip6+P6TgdRWL4SMHA3GsPgfOfZlKRWGQ+gJg3tBdGgEUs+blgbxIKpDBvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTUL",
  "BvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTULmUsKvXsH7H1jOmzurCRXE51ERHk\nLWfNJ0soXVRw6ZTRpAFizBoWmUEWRIuNkMWMqhy3R7AYeOjthVobAqyMLsJZHb7DvWEbw2xzGcL01vsyDl\nv2CoIjoAZ5/+Fkx5vKlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei",
  "KlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei+Y\nB6gA+6fJEKP8N7XbZgiWrw/3bcKhJLvnJ96s/8PFp0danjf5HqgmJ0jy2JdLht0g0hNsbXl8QwZMXSTR5EC\ngnL5JwfUdTxK8sHWknDtoCMWkyCbo9BeBau5TRvBgoxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH",
  "oxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH6ZtW8oEotNK8bXM73gjbcHC74Fbu7qKJuVU83ytWQJaiYz2l45f9NINTzHb2\nl1NeNa1WwM+36v5gXDA7uefx8EWno+AWNSRKBc8GVlzSWJZ+oNc8+X65siKrZfkqUdWFy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tN",
  "Fy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tNoWd26i5e/vj3jG9GNSJIf6sS+S/SqExY\nyKmVWMQh4gsQphMcybFmxjZU/AzaNpVSEs9lLR1HQAS0Mu8SFUISxWp3DTrGNY3bao23aVyXiEzCqExYcsx\nEdhbAYUDGwimcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhH",
  "mcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhHcRySlde\nal15Cq1iRVfxga3jgys6zhKqANY2iHnmNPfsZ5kLi4xPGbZihwLZMW0gD3s9Kgze/pz/YI8ybn+xNAJpZe\nGXlJ6ZOgRpYmh5BeB6z8zlPw6cf0LQy8oPT0kNLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLn",
  "NLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLnhj6n9JGhjyh9YegLSl8b+prSe4beo5QZyijdNHSTUm4oeXg+uGrlPqGkp+8G5Z\nmiP0tjQmNL7ht6ndGgo+VUM9zNDyeMN3BgNlZQ+NvQxpcJQ8vN9Z8a+pTS0NCQ0ieGPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DE",
  "GPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DEJdiNDI0o3TKU/FKARwlDz8jzpK/q9rsbRO\n5rvlqzi2srvhsb1JzX825hdVXp9ne5PrkqzkfkaFvHs5fpEBJ4Uo/WFrp4LewtHYXe38uLq2u7Zyd71+Q3\nu9WXrq9Y3rU7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ<",
  "U7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ</latexit\n>y1 = \u03c610 + \u03c611h1 + \u03c612h2 + \u03c613h3 + \u03c614h4\ny2 = \u03c620 + \u03c621h1 + \u03c622h2 + \u03c623h3 + \u03c624h4\n31",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 ouput\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n32",
  "Two inputs\n\u2022 2 inputs, 3 hidden units, 1 output\n<latexit sh\na1_base64=\"s0UuQl8oja\nlO49HAr3NW0dwfYA=\">A\nAXU3iclZhb9s2FICdrtu6\nbN3SDWsf9iIs6DBsXWAn3\neWlQJs0vSVdnHvaKDUomZL\nVUJQiUYlTwT9zGPZb9rJD\nSTajc5iHBWhNn+/TIXlIXS\nwvFVGut1/5m58dPjTz6\n9dn851/c/vKrhTtfH+RJk\nfl8309Ekh15LOciknxfRU\nrwozTjLPYEP/R",
  "dPjTz6\n9dn851/c/vKrhTtfH+RJk\nfl8309Ekh15LOciknxfRU\nrwozTjLPYEP/RO1zQ/POdZ\nHiVyT12m/CRmoYyCyGcKQ\noOFv0eDsjdxfnjkuLGXjEs\n2OXbViCsG4e7E+dmZfetN\nxoPele/L8H35xHFlIovY45\nnjuvOQbNmebLmVbLlONvt\n6Ta4Ve6Vq6Vdq6VOtdgY\nbG71K3+HNroNY3FTvPXH9\nz5dugOE7+IuVS+YHl+3Oum\n6qRk",
  "6Vq6Vdq6VOtdgY\nbG71K3+HNroNY3FTvPXH9\nz5dugOE7+IuVS+YHl+3Oum\n6qRkmYp8wSfzbpHzlPmnL\nOTH0JQs5vlJWa3AxLkPkaE\nTJBn8k8qpolePKFmc5ex\nB2bM1CjHTAdt7LhQwR8nZS\nTQnHp1x0FhXBU4ujldIZ\nRxn0lLqHB/CyCsTr+iGXMV\n7Do867kF34Sx0wOS3d1fX\ntSuh4PI1nys6LaAJNJ21mv\nHA7N64zVl3uzLJHicf",
  "7Do867kF34Sx0wOS3d1fX\ntSuh4PI1nys6LaAJNJ21mv\nHA7N64zVl3uzLJHicfSBk\nySVopNcI/BwUpZ8KVzCIOI\nAoiVOQCJ5Djl1fbzA6SEK\nG14ALud4oKxMyGpeIh1K\nSlvSUaNFLBxy1rjViwlHF\nL2QXFce47GnCVwSrAUOGDo\nzXYTZmcTI9TfKyuMx1DP\neQMRnyqguYs+EnlHbkIUQ\ncKjfsv7E1g6Tp03hkrQa\nqYjyNrL2o7KaF",
  "yuMx1DP\neQMRnyqguYs+EnlHbkIUQ\ncKjfsv7E1g6Tp03hkrQa\nqYjyNrL2o7KaF3ksO1UEWT\nBJgzbVhVBloDL05DFDKrc\ntAcw4djREbsaSaxGZGP2s8\nRr953qCN6b4xTOl7a3XpL\nynzNUER2As09/Rkz6vK2vJ\nTPbmRbnvPJ1g4+dESxW+x\nCWhfW0p3ArJrYhJpVrZBJ\nqwWhLlom3o0FpWnUXuCO\noBPuiKLZHBFe1C1YMvqsPs\nApoV",
  "p3ArJrYhJpVrZBJ\nqwWhLlom3o0FpWnUXuCO\noBPuiKLZHBFe1C1YMvqsPs\nApoVgh/svQrH5+UX3a\n6P9INSFRXqS2RDr8PxIN4Y\naI9xdE8OIlAi0eBKrFSwR\nc39HSsQxvbB2p1g4akWQiU\npfo9I9C2T6miuDBJjEaKw\nR0XvhkUSLHARtWQe0DJ9w\na7dsIB9N0q/n6IskLzJOL\nn5oP0Ok0vVlMYv0zap9QRV\naF83uJgdBW24OZzaw73",
  "7dsIB9N0q/n6IskLzJOL\nn5oP0Ok0vVlMYv0zap9QRV\naF83uJgdBW24OZzaw73\nUEW9up5eUsghy1Ax3pJx+\n/cXMEpZjv7qyWvm1Yr5Gc\nbTX8wLlidwvf52WADr0dIL\nOoIlAuepay5BLEs/UGu2X\na9OrJy491PZGuHFtduCpK3\nGaXdtrjXjICfbVpGu0k8Y\nlFHoFzNCKlHLEt/kMtex03\nbLCyu3RQk7SOVtvizky0\n/YM9/SyqH5MSMd",
  "u0k8Y\nlFHoFzNCKlHLEt/kMtex03\nbLCyu3RQk7SOVtvizky0\n/YM9/SyqH5MSMdSPfYlw6x\nAWFRWVUxiHiKxDmExLto\nWfMfKbgQ3j7ZVh7DYz6O2\npgNYGnKBp1CHsFifwm2ziW\nF106Ju2lUm0hEy6xAWn7M\nYz7oOYTGkYmgVT1maIrEOk\nTqOcB1HtI4plKbhFckta\nwI2VK2DZWNkrakA1gao97G\nls5gBCKRqMmiOWc7rzcu\nvMk2sWS",
  "I4plKbhFckta\nwI2VK2DZWNkrakA1gao97G\nls5gBCKRqMmiOWc7rzcu\nvMk2sWS7uJ9W8f713SsGEq\noA1jaIueY425ZTzIPlxge\ns2xFTiNkpbSAfez0qTN9+v\nOCkjzJecGloZeUXh6Qem\nhoYeUZoaSXwResGMo+XiB\neGnlN6YOgBpYWhBaX7hu\n5TGhgaUPrM0GeU+ob6lK4Z\nukapMpQ8kcIdwdA9SkeGj\nig9MvSI0jeGvqH0haEvKH1",
  "hgaUPrM0GeU+ob6lK4Z\nukapMpQ8kcIdwdA9SkeGj\nig9MvSI0jeGvqH0haEvKH1\nr6FtKPxj6gdInhj6hlBnK\nKF03dJ1Sbih5deAFq4auUu\noZSn7wblmaJ/S1NCU0qe\nGPqV0aCj5VQz3M0PJ4w3cG\nA0VlL409CWlkaHk95sXvD\nb0NaWxoTGlrwx9Rel7Q9T\n+tzQ5SGhpJ3A/B0Yugup\neYtUJlTum3oNqVnhp7Z3wv\nw2TJ6to25ZRJsU",
  "l7Q9T\n+tzQ5SGhpJ3A/B0Yugup\neYtUJlTum3oNqVnhp7Z3wv\nw2TJ6to25ZRJsUZoYmlC6\nYSj5pQCPEoaekufJQDZXte\nnbJnJdC+SMW1hT8enRpOa\nBnHELa65O06PJ9SmQMz4iQ\n18/mL1IgZLClX6wsNjDb2\nFp42B5qfb0sPth4uPV5s3\ntLc63W+7/zY6XV+7zuv\nOj0O/sdf+7RnD8n5uK7f93\n96Nezdr9cZc8w3ndbf\nvdv/Ab4sGx",
  "/zY6XV+7zuv\nOj0O/sdf+7RnD8n5uK7f93\n96Nezdr9cZc8w3ndbf\nvdv/Ab4sGxg=</latexit>h1 = a[\u271310 + \u271311x1 + \u271312x2]\nh2 = a[\u271320 + \u271321x1 + \u271322x2]\nh3 = a[\u271330 + \u271331x1 + \u271332x2]\n<latexit sha1_base64=\"4mwMW+AxXHfhT\nyI/kaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3\npIvTxEmaODUomZLZUJSiS2JX8",
  "s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3\npIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzUukyPJu\n9r1z/48KOP7nx6c3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9\nM42ND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/\nf0r3vz0cpat9O",
  "GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/\nf0r3vz0cpat9OtPi4t9JrCmtN8+qNb346H49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhP\nmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2F\nSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8OFb/04yhialwO1zd35+XQ",
  "cK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DTm8OFb/04yhialwO1zd35+XQ\n46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyRUCD+dlyTthBwPBAYgOJyBWP\nIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYaFBLJpy1rg1gwlVFL2QPFdW+7Gv\nA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9Yjahiq",
  "2QPFdW+7Gv\nA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9YjahiqkhEP9lvUHtl4xdY\nkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiyJKw+8csYpDlpjyCAUeujthVobAqyML\nsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSl\nYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NReWJaA",
  "sb8dJ2F8m5qHxd4FN3ApPVPoSl\nYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NReWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8\npOfO7/w6WnZ1dtG/yPZhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrS\nPV3EFBKCZFPkPbX4SqfUwVwZ2NI9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3\nFWpJyc/NB6hkil69NiKvTFq",
  "RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3\nFWpJyc/NB6hkil69NiKvTFqn1ClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekq\nnb4ZDlvMtvurKa+LVivk51tNe9AvmJ3C9/n5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7\nPSu3vxIlnZoce2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXb\nkpS7yKP",
  "e2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXb\nkpS7yKPVtviLk20/IP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8W\nhbdQiL/Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPG\nNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQa",
  "PG\nNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvI\ny68pTaBUruoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OC\nktzJecHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0\nieGPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3",
  "tDC0oHRg6IDSwNCA0\nieGPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7S\nh8Z+ohSZijdNPQTUq5oeTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoe\nT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aP\nUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MH",
  "FoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aP\nUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMBTuUxobGlG4ZSp4U4FbC0DNyPxmo5qy2eNtE\nzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWevgtLC0c3O30H\nnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8Spb9Wv1+rXm\nG+c1mdV/ge0FuRK</l",
  "/84PScX52HzjOn7wc3/nT+cv52/ln9f7q8Spb9Wv1+rXm\nG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n33",
  "34\nLinear\nFunctions",
  "35\nLinear\nFunctions\nAfter\nActivation",
  "36\nAfter\nActivation\nWeight the\nHidden units",
  "37\nWeight the\nhidden units\nSum the weighted \nhidden units",
  "Convex polygonal \nregions\n38\nA region of \u211d! is convex if we can \ndraw a straight line between any \ntwo points on the boundary of the \nregion without intersecting the \nboundary in another place.",
  "Fitting a dataset where:\neach sample has 2 inputs and 1 output \n<latexit sha1_base64=\"bhUr\nd52vj8mM9aesL3p+GlHvyac=\">AWjHiclZhb9s2FIDV7tZ1W\n5duWF72IiwoMAydkQzdBRgGtEnTW9IlaeIkTZwGlEzJbChKkSjH\nruB/stftP+3f7FCSzeoc5mEGUrPn+8TLISnRCjIpCr26+u+Nmx\n98+NHn9z69PZn39x58ulu18dFmZh7wfpjLNjwNWcCkU72uhJ\nT/Ocs6SQPKj4GLD8",
  "8+NHn9z69PZn39x58ulu18dFmZh7wfpjLNjwNWcCkU72uhJ\nT/Ocs6SQPKj4GLD8KMxzwuRqgM9zfhZwmIlIhEyDaHzpaVBEqST\naqfUWanv+7Pp+dLKam+1/vi0sNYWVrz2s3t+95vhYJiGZcKVDiU\nritO1UyfVSzXIpR8dntQFjxj4QWL+SkUFUt4cVbVXZ/59yAy9K\nM0hz+l/Tr6/hUVS4pimgRgJkyPCsxM0MVOSx39dlYJBcPiKmwai\nk",
  "XZ/59yAy9K\nM0hz+l/Tr6/hUVS4pimgRgJkyPCsxM0MVOSx39dlYJBcPiKmwai\nkrp69Q3efCHIuehlMosDAX0Fc/HLGchRqydXug+FWYJglTw2qw\nvrk3qwYBj4Wq+GVZ2426zqbtcOheJ2x/vxgUYvQPBHvOKmkVk\nwl1wg8nlUV78U9DAQHIHqcgFTxAuo0+Qkifw1RWCkScNUsgEYr\n2akaqV5DnpaCdEg0Im+aRjbRALpjLpKPug+P493wC",
  "o0+Qkifw1RWCkScNUsgEYr\n2akaqV5DnpaCdEg0Im+aRjbRALpjLpKPug+P493wCuc5gF6Cp8\ncTQH+xlTs/l1mk90nlSFieEWcqZiXjcBQw6ZNCPqGqUEi4NO9a\nf2HrF1EWbuDSru5qbCLIO8q6jc5oXNew6dQRZsAjrlVHkCVhXw\n9ZwiDLbfkcBpz4JuJWhcKqIAtzN0+DbtuZieC1Oclgv3S9zYqkf\n8xQRkwAdp/5FkyFvKtvpAvbnydnX",
  "JWhcKqIAtzN0+DbtuZieC1Oclgv3S9zYqkf\n8xQRkwAdp/5FkyFvKtvpAvbnydnXPumwCf+CarewnL42ZY80Zg\nVG1sRs06V8ik2YJQnl51TdMbh8oz0R2gCeBNV+ZCRe9p9+sSLF\nkTHtyHoeal5Kc/9n7mk7Nq1Wwb8w/JlRUlJmrIhP+HxUN4UmC1\nxdE8OSlEk0eBOrJSyXc39HUsRwvbBOp5w4KQjEp9BRtfxGr7jV1\nBHc2TVBfIWDqhW8",
  "8OSlEk0eBOrJSyXc39HUsRwvbBOp5w4KQjEp9BRtfxGr7jV1\nBHc2TVBfIWDqhW8mFJrkKOrKJmBk+IZnomMBhWiQYTPGUKZFmXN\ny80PrGSK1bm6LuTAPq+4NVRqhe9/gcnEVlOHhMObXB6gjAZNPo\nO0VEOWo2ROzJRO3gwKDVvMtfvrKW+KTivml1te9AvmJ0yDPnl+\nRaej5hY1JGoLjiEOuSxHK0B3Utluv7Pau23vxAlnbscN2mJPW2\nvX",
  "vmJ0yDPnl+\nRaej5hY1JGoLjiEOuSxHK0B3Utluv7Pau23vxAlnbscN2mJPW2\nvXTbDveaHvDLbUdvt4lHLOpIVFfbQ+oRy9Ee1OXO47ZrFA7XbU\npS7zyPTtvhLky0/KODEdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxC\nWExKbsW/B8r+wIeHl2rCWFxtxBdzQSwNOQSD6EJYbHZwl2zjWF1\n26Fu1UmsxEymxAWn7IEj7oJYTGmYuwUL1iWIbEJk",
  "QSwNOQSD6EJYbHZwl2zjWF1\n26Fu1UmsxEymxAWn7IEj7oJYTGmYuwUL1iWIbEJkTyOcB5HNI8\nZljKXhGckc8wIWVKuBZWP0q5kAliaoNYmjsagBzJVqME2iOWCr\nzCufIUWsWKruK+q+H+NQ1rhio0ASztkD3mD3acmyzAKYZjlivJm\nUBWRhO4i51d6sxPf0FUkZNcE0tnVJ6ZekVpUeWHlGaW0p+EQTR\nK0vJr5MgGls6pvTQ0kNKS0tLSvu",
  "Pf0FUkZNcE0tnVJ6ZekVpUeWHlGaW0p+EQTR\nK0vJr5MgGls6pvTQ0kNKS0tLSvuW9imNLI0ofWLpE0pDS0NKNy\nzdoFRbSk6k8ESw9IDSkaUjSo8tPab0taWvKX1m6TNKTyw9ofSdp\ne8ofWTpI0qZpYzSTUs3KeWklcHQbRu6TqlgaXktx/sNUt3Kc0s\nzSh9bOljSoeWkl/F8DyzlBxv4MFoqaT0uaXPKRWkt9vQfTS0pe\nUJpYmlL6w9AWlb",
  "Sh9bOljSoeWkl/F8DyzlBxv4MFoqaT0uaXPKRWkt9vQfTS0pe\nUJpYmlL6w9AWlby19S+lTS59SGltK3g3A6cTSfUrtW6CqoHTP0j\n1KLy29dL8X4ItpDFwLc8dWsENpamlK6Zal5JcCHCUsvSDnyUi1d\n7X52yZyX4vUgjtYm/H51STnkVpwB2vTvOryf0pUgs+Il3fPFy\n8SIGUwp3+fGlDb+FpYXDn3prv/Qe7D1YebjevqG95X3rfed976\n1",
  "Ugs+Il3fPFy\n8SIGUwp3+fGlDb+FpYXDn3prv/Qe7D1YebjevqG95X3rfed976\n15v3oPvWfertf3Qm/s/eX97f2zfGf5wfLvy3806s0b7TVfe53P8\npP/AYn1U4=</latexit>Output, y\n<latexit sha1_base64=\"Nhy2+PWdsa8Gas+5nVN6kmQqXI=\"\n>AWj3iclZhb9s2FIDVXbvulq5YXvYiLCgwDJ2RFN26p6FNmrZp0sVprm2cGpRMyWwo",
  "Wj3iclZhb9s2FIDVXbvulq5YXvYiLCgwDJ2RFN26p6FNmrZp0sVprm2cGpRMyWwoSpGoxK7g37LX7Sft3+xQks3qHOZhBlKz5\n/vEyEp0QoyKQq9uvrvjY8+/uTz6/+cWtL7/6+ptvl25/d1SkZR7ywzCVaX4SsIJLofihFlrykyznLAkPw7ONw/vuR5IVJ1oKc\nZP0tYrEQkQqYhNFy6M0iCdFJtqazU9/zZFjdnw2XVlZ7q/XHp4W1trDitZ/+",
  "oKc\nZP0tYrEQkQqYhNFy6M0iCdFJtqazU9/zZFjdnw2XVlZ7q/XHp4W1trDitZ/+8Pb3o8EoDcuEKx1KVhSna6uZPqtYrkUo+ezWoCx4\nxsJzFvNTKCqW8OKsqns/8+9CZORHaQ5/Svt19MrKpYUxTQJwEyYHheYmaCLnZY6+v2sEmZoXIVNQ1EpfZ36JhX+SOQ81HIKBRbmAv\nrqh2OWs1BDwm4NFL8K0yRhalQN1jf3ZtUg4LFQFb8o6+TNZl1",
  "SOQ81HIKBRbmAv\nrqh2OWs1BDwm4NFL8K0yRhalQN1jf3ZtUg4LFQFb8o6+TNZl1ns3Y4FK8z1rcOFrUIzRPxnpNKasVUco3A41lV8V7cw0BwAKLHCUgV\nL6BOk58g8tcQhcUiAVfNOhiA8WpGqlax5CTjvaGaFDIJ90rA1iwVQmHWUfFN+/6xvAdQ6zAF2FL47mYD9jaja/TvOJzpOqMDHcQs\n5UzOsmYMghk2ZEXUOVUsKlYcf6E1uvmDpvE5d",
  "L47mYD9jaja/TvOJzpOqMDHcQs\n5UzOsmYMghk2ZEXUOVUsKlYcf6E1uvmDpvE5dmdVdzE0HWQd51dE7zokZdp4gCxZh3LXqCLIkbO0RSxhkuS0PYcCJbyJuVSisCrI\nw+3kadNvOTASvzUkG+6XrbVYk/ZcMZcQEYPeZb8FUyLv6Rrqw/XlyLmvfFPjEH8NkdS9hedwMa94IjKqNzahZ5wqZNFsQytOrml64\n1B5JroDNAG86cpcqOgD7V5dgiV",
  "kdS9hedwMa94IjKqNzahZ5wqZNFsQytOrml64\n1B5JroDNAG86cpcqOgD7V5dgiVrwoN7MNS8lPz0l96vfHJWrZptY/4h2YSKijJzVWTC/6OiETxM8PqCJ68VKLJg0A9eamE+zuaOpb\njhW0i9dxBQSgmhZ6i7S9i1b2mjuDOpgnqKwRMvfDNhEKTHEVd2QSMDN/wWHQsoBANMmzGMq0KHNObn5oPUOk1s1tMRfmYdW9oUoj\ndO8bXC6ugjI8HC7",
  "MDN/wWHQsoBANMmzGMq0KHNObn5oPUOk1s1tMRfmYdW9oUoj\ndO8bXC6ugjI8HC75NZcHKNBk8gLdWI5SiZEzOlk7eDQsMWc+3+esqbotOK+cV2x70C2anDEN+MdzG8xETizoS1QXnEGdkliO9q\nCuxXL9sGfV9tufydKOHa7blKTetpdu2+Fe0wN+sePo7Q7xiEUdiepqe0g9Yjnag7rcedxjcLhuk1J6p3n0Wk73IWJln90MOamWNS\nKkfm2Jf",
  "7xiEUdiepqe0g9Yjnag7rcedxjcLhuk1J6p3n0Wk73IWJln90MOamWNS\nKkfm2JfKQRPCoqaidopwmMkNiEsJmXgv9jZV/Aw6NrNSEs9gvR1UwASyMu8RCaEBabLdw12xhWdxzqjltlMhsjswlh8RlL8KibEB\nZjKsZO8ZxlGRKbEMnjGOdxTPOYSlzSXhGMseMkCXlWlD5O1KJoClCWpt4mgMeiBThRpsg1gu6MornCtPoVWs6Co+dDV8eE3Dmq",
  "seMkCXlWlD5O1KJoClCWpt4mgMeiBThRpsg1gu6MornCtPoVWs6Co+dDV8eE3DmqE\nKTQBLu2SP+YNd5yYLcIrhmOVKciaQldE9rHTp8789BdEFTnJBdHU0imlV5ZeUXps6TGluaXkF0EQvbKU/DoJoktLyk9svSI0tLSk\ntJDSw8pjSyNKH1q6VNKQ0tDSjcs3aBUW0pOpPBEsPSA0rGlY0pPLD2h9LWlryl9bulzSt9Y+obS95a+p/SxpY8pZY",
  "cs3aBUW0pOpPBEsPSA0rGlY0pPLD2h9LWlryl9bulzSt9Y+obS95a+p/SxpY8pZYySjct3aSUW0p\neHQTRuqXrlAaWkt9+sNcs7VOaWZpR+sTSJ5SOLCW/iuF5Zik53sCD0VJ6ZalW5QKS8nvtyB6aelLShNLE0pfWPqC0neWvqP0maXP\nKI0tJe8G4HRi6T6l9i1QVC6Z+kepReWXrjfC/DFNAauhblrK9ilNLU0pXTbUvJLAY4Slp6T82Sk2rv",
  "6l9i1QVC6Z+kepReWXrjfC/DFNAauhblrK9ilNLU0pXTbUvJLAY4Slp6T82Sk2rva/G0Tua9FasEdrM34/GqS80\ngtuIO1d6f51eT+FKkFH5Oubx4tXqRASuFOP1xaWcNvYWnh6H5v7bfeg70HK4/W2ze0N70fvB+9n7w176H3yHvu9b1DL/Sm3l/e394/\ny7eXHy7/sfyoUT+60V5zx+t8lrf+A3ki1nM=</latexit>Input, x2\n<latexit sha1_b",
  "7eXHy7/sfyoUT+60V5zx+t8lrf+A3ki1nM=</latexit>Input, x2\n<latexit sha1_base64=\"y5iCZag0tm0U1J5r3+/H8z+KE1Q=\"\n>AWj3iclZhb9s2FIDV7tZ1t3TF8rIXYUGBYeiMeOjWPQ1t0rRNky5Oc23j1KBkSmZDUYpEJU4F/5a9bj9p/2aHkmxW5zAPM5CaP\nd8nXg5JiVaQSVHo1dV/b9z86ONPv3s1ue3v/jyq6+/Wbrz7WGRlnID8",
  "zAPM5CaP\nd8nXg5JiVaQSVHo1dV/b9z86ONPv3s1ue3v/jyq6+/Wbrz7WGRlnID8JUpvlxwAouheIHWmjJj7OcsySQ/Cg4Wzf86ILnhUjVvr7\nK+GnCYiUiETINodHS3WESpNqU2Wlvu/PpqOqPxstraz2VuPTwv9trDitZ/B6M534+E4DcuEKx1KVhQn/dVMn1Ys1yKUfHZ7WBY8\nY+EZi/kJFBVLeHFa1b2f+fcgMvajNIc/pf06+uEVFUuK4i",
  "Mn1Ys1yKUfHZ7WBY8\nY+EZi/kJFBVLeHFa1b2f+fcgMvajNIc/pf06+uEVFUuK4ioJwEyYnhSYmaCLnZQ6+v20EmZoXIVNQ1EpfZ36JhX+WOQ81PIKCizMBf\nTVDycsZ6GhN0eKn4ZpknC1Lgarm3szqphwGOhKn5e1smbzbrORu1wKF5nrG3uL2oRmifiPSeV1Iqp5BqBx7Oq4r24h4HgAESPE5Aq\nXkCdJj9B5PcRhcUiAVfNOhiC8WpGqlax5",
  "eV1Iqp5BqBx7Oq4r24h4HgAESPE5Aq\nXkCdJj9B5PcRhcUiAVfNOhiC8WpGqlax5CTjvaGaFDIJ92rHViwVQmHWUPFN+/5xvAdQ6zAF2FL47mYC9jaja/TvOpzpOqMDHcQs\n5UzOsmYMghk2ZEXUOVUsKlYcf6E1uvmDprE5dmdVdzE0HWft51dE7zosZdp4gCxZh3LXqCLIkbO0xSxhkuS2PYMCJbyJuVSisCrI\nwB3kadNvOTASvzWkG+6XrbV",
  "gCxZh3LXqCLIkbO0xSxhkuS2PYMCJbyJuVSisCrI\nwB3kadNvOTASvzWkG+6XrbVQk/RcMZcQEYPeZb8FUyLv6erqw/XlyLmrfFPjUn8BkdS9hedwMa94IjKqNzahZ5wqZNFsQytPLrml64\n1B5JroDNAG86cpcqOgD7X5dgiVrwsP7MNS8lPzk596vfHparZptY/4h2YSKijJzVWTC/6OiMTxM8PqCJ68VKLJg0A9eamE+zuaOpb\njhW0i9dxBQ",
  "ZptY/4h2YSKijJzVWTC/6OiMTxM8PqCJ68VKLJg0A9eamE+zuaOpb\njhW0i9dxBQSgmhb5C21/EqntNHcGdTRPUVwiYeuGbCYUmOYq6sgkYGb7hsehYQCEaZNiMZRpUeac3PzQeoZIrZvbYi7Mw6p7Q5VG\n6N43uFxcBWV4OFzway4PUEaDJp9BWqoxy1Eyp2ZKp2+HhYt5tr9ZQ3RacV8/Otj3oF8xOGYb8fLSF5yMmFnUkqgvOIc6JLEc7U\nFd",
  "2ZKp2+HhYt5tr9ZQ3RacV8/Otj3oF8xOGYb8fLSF5yMmFnUkqgvOIc6JLEc7U\nFdi+X6Yc+qrbc/kaUdO1y3KUm9bS/dtsO9pgf8fNvR23iEYs6EtXV9pB6xHK0B3W587jtGoXDdZuS1DvPo9N2uAsTLf9of8I1M8ek\nVI7NsS+VwyaERU1F7RThMdIbEJYTMquBf/Hyp6Ah0fXakJYHBSiq5kAlsZc4iE0ISw2W7hrtjGsbjvUbfKZDZBZhPC4j",
  "MquBf/Hyp6Ah0fXakJYHBSiq5kAlsZc4iE0ISw2W7hrtjGsbjvUbfKZDZBZhPC4jOW4FE3IS\nzGVIyd4hnLMiQ2IZLHCc7jhOYxw1LmkvCMZI4ZIUvKtaDySdqVTABLU9Ta1NEY9ECmCjXYBrFc0JVXOFeQqtY0V84Gr4JqGNUM\nVmgCWdsge84c7zk0W4BTDMcuV5EwgK6MJHGBnQJ356S+IKnKSC6IrS68ovbT0ktIjS48ozS0lvwiC6JWl5Nd",
  "DMcuV5EwgK6MJHGBnQJ356S+IKnKSC6IrS68ovbT0ktIjS48ozS0lvwiC6JWl5NdJEF1YekHpoaWHlJaWl\npQeWHpAaWRpROlTS59SGloaUrpu6Tql2lJyIoUngqX7lE4snVB6bOkxpa8tfU3pc0ufU/rG0jeUvrf0PaWPLX1MKbOUbph6Qal3FL\ny6iCI1ixdozSwlPz2g71m6YDSzNKM0ieWPqF0bCn5VQzPM0vJ8QYejJZKSjct3aRUWEp+vw",
  "xdozSwlPz2g71m6YDSzNKM0ieWPqF0bCn5VQzPM0vJ8QYejJZKSjct3aRUWEp+vwXRS0tfUpYmlD6wtIXlL6z9B2lzyx9\nRmlsKXk3AKcTS/cotW+BqoLSXUt3KT239Nz9XoAvpjFwLcwdW8EOpamlKaVblpJfCnCUsPSMnCcj1d7V5m+byH0tUgvuYG3G51eTnE\ndqwR2svTvNryb3p0gt+IR0feNw8SIFUgp3+tHSh+/haWFw196/d96D3YfrD",
  "eTnE\ndqwR2svTvNryb3p0gt+IR0feNw8SIFUgp3+tHSh+/haWFw196/d96D3YfrDxa9/Q3vK+937wfvT63kPvkfcG3gHXuhdeX95f3v/\nLN9Zfrj8x/KjRr15o73mrtf5LG/+B2MK1nI=</latexit>Input, x1\n39",
  "\u2022 For the 2D case, what if there were two outputs?\n\u2022 If this is one of the outputs, what would the other one look like?\nQuestion:\n40",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 ouput\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n41",
  "Arbitrary inputs, hidden units, outputs\n\u2022 \ud835\udc37! inputs, D hidden units, and \ud835\udc37\" Outputs\n<latexit sha1_base64=\n\"eu7R9HxZr4T2SlEuSDpbPsp4w0=\">AWx3iclZ\njLbtw2FECV9JWmL6dFvelGqBGgaFPDLtLHJkBix0k\ncO7UdP2PLGVAaSsOYomSJscRtOgn9WuK7to/6aVG\nM4zupRcdwBnmniM+LkmJozCXotRLS3/fuPne+x98\n+NGtj29/8uln38xd+fLgzKriojvR5nMiqOQlVwKx\nfe10JIf",
  "tRLS3/fuPne+x98\n+NGtj29/8uln38xd+fLgzKriojvR5nMiqOQlVwKx\nfe10JIf5QVnaSj5YXi2avjhBS9Kkak9fZXz05QlSs\nQiYhpCg7mN0aAeNv4DP0jDbFyzJpA81ieBHnHNAC0\n1/g9+UFbpoBYPlpvX9WMoNM2Mi2Y8EhkpE+Hcwt\nLC0utR+fFpa7woLXfbYHd74eBsMsqlKudCRZWZ4sL\n+X6tGaFpHkze2gKnOojOW8BMoKpby8rRu",
  "7woLXfbYHd74eBsMsqlKudCRZWZ4sL\n+X6tGaFpHkze2gKnOojOW8BMoKpby8rRuR934dy\nEy9OsgD+l/Tb67hU1S8vyKg3BTJkelZiZoIudVDr\n+7bQWKq80V9GkobiSvs58k0J/KAoeaXkFBRYVAvr\nqRyNWsEhDom8Hil9GWZoyNayDlbWdpg5CnghV8/Oq\nTXrT9J21uFQvM5YWd+b1SI0T8VbTipFVPJNQJPm\nrmi8kiBoIDEIucgEzxEuo",
  "Oq\nTXrT9J21uFQvM5YWd+b1SI0T8VbTipFVPJNQJPm\nrmi8kiBoIDEIucgEzxEuo0+QljfxlRWGQScD1ZNw\nEYLxtStdI8gZz0tGOiQSGXfNyzVokFU5n2lF1QfP+\nubwDXBcwCdBW+OJqD3ZypZnqd5mNdpHVpYriFgqmE\nt03AkCMmzYj6hqkhEujnvU7tl4ydYlLsvbrhYmg\nqy9ou/oguZFDftOG0EWLMKkb7URZEm4JQxZyiDLX\nXkA059E",
  "l4ydYlLsvbrhYmg\nqy9ou/oguZFDftOG0EWLMKkb7URZEm4JQxZyiDLX\nXkA059E3GrQmFVkIW5XWRhv+3cRPDaHOewX/reWk\n3Sf8FQRkwAdp/5FkxFvK+vZjPbnybnovVNgY/9EUx\nW/xJWJNhTRuBUXWxhptrpBJswWhIrvsm6Y3DpXn\noj9AE8CbriqEit/R7rUlWLImHNyDoRaV5Cc/Lv7Mx\n6f1ktk25h+STaiorHJXRSb8PyoawkMIry+",
  "it/R7rUlWLImHNyDoRaV5Cc/Lv7Mx\n6f1ktk25h+STaiorHJXRSb8PyoawkMIry+I4MnLJ\no8CLSTl0m4v6OpYwVe2CbSzh0UhGJS6Cu0/UWi+te\n0EdzZLEV9hYCpF76ZUGiS47gvm4CR4Rsep4FKFB\nRpMxRjIrq4KTmx9azxBpdXNbLIR5WPVvqNI/fsG\nl7OroAwPhwt+zeUhymg4yWeYVWrICpTMsZnS8eug1\nLDFXLu/nfJ0Wkl/Hyjaw/",
  "l7OroAwPhwt+zeUhymg4yWeYVWrICpTMsZnS8eug1\nLDFXLu/nfJ0Wkl/Hyjaw/6BbNTRE/H2zg+UiIR\n2J6oLzi7MuSxHe1DXbLm+27N64/X3ZGknDtdtSlJ\nv10u37XCv6QE/3T0dpN4xKORHV1PaQesRztQV3u\nPG6RuFw3aYk9U7z6LQd7sxEyz/eMydSc0zK5NAc+\nzIZTEJY1FTUTjFLeYLESQiLadW34P9Y2RXw8Ohbkx\nAWt0vR10wA",
  "dSc0zK5NAc+\nzIZTEJY1FTUTjFLeYLESQiLadW34P9Y2RXw8Ohbkx\nAWt0vR10wAS0Mu8RAmISxOtnDf7GJY3XSom26VyXy\nEzEkIi09Zikc9CWExoWLiFM9YniNxEiJ5HOE8jmg\necyzlLgnPSO6YEbKkXAuqGV9yQSwNEatjR2NQ9k\nplCDXRDLJV15pXPlKbSKFV3F+6G969pWDNUoQlga\nYvsMT/Ycm6yEKcYjlmuJOcCWTlN4DZ2tqkzPf2",
  "lKbSKFV3F+6G969pWDNUoQlga\nYvsMT/Ycm6yEKcYjlmuJOcCWTlN4DZ2tqkzPf2FcU\n1OcmF8ZekVpZeWXlJ6aOkhpYWl5BdBGL+0lPw6CeM\nLSy8oPbD0gNLK0orSfUv3KY0tjSl9YukTSiNLI0pX\nLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSVpa8ofWbpM\n0qPLT2m9K2lbyl9ZOkjSpmljNI1S9co5ZaSVwdhv\nGLpCqWhpeS3H+w1S7cpzS",
  "M\n0qPLT2m9K2lbyl9ZOkjSpmljNI1S9co5ZaSVwdhv\nGLpCqWhpeS3H+w1S7cpzS3NKX1s6WNKh5aSX8XwPL\nOUHG/gwWipHTd0nVKhaXk91sYv7D0BaWpSmlzy1\n9TukbS9Q+tTSp5QmlpJ3A3A6sXSXUvsWqC4p3bF0\nh9JzS8/d7wX4bBpD18LcshVsUZpZmlG6YSn5pQBHC\nUvPyHkyVt1dbfq2idzXYjXjDtZlfHo1yXmsZtzBur\nvT9Gpy",
  "pZmlG6YSn5pQBHC\nUvPyHkyVt1dbfq2idzXYjXjDtZlfHo1yXmsZtzBur\nvT9Gpyf4rVjI9I19cOZi9SIKVwpx/MLSzjt7C0cPD\nT4vIvi/d37i8XOne0N7yvG+9b7zlr1fvYfeM2/b\n2/ci70/vL+8f79/59fls/mJ+PFv3uiu+crfeb/\n+A+HQO5X</latexit>\nhd = a\n\"\n\u2713d0 +\nDi\nX\ni=1\n\u2713dixi\n#\n<latexit sha1_base64=\"iQ/nJ5VzZC6Vgn",
  "exit>\nhd = a\n\"\n\u2713d0 +\nDi\nX\ni=1\n\u2713dixi\n#\n<latexit sha1_base64=\"iQ/nJ5VzZC6Vgn\nR5+vmegEip1K8=\">AWqHiclZhb9s2FIDV7tZ1t3TD8rIXYUGBoesMe+guLwXapOkt6eI0\ncS6NU4OSKJkJRSm6JHYF/4n9mr1u/2L/ZoeybFbnMA8zkJo93ydeDkmJlpdKkRfd7r83bn7w\n4Ucf3Lr09uf7Fl1+t3Pn6IE/KzOcDP5FJduSxnEuh+KAQheRH",
  "dKkRfd7r83bn7w\n4Ucf3Lr09uf7Fl1+t3Pn6IE/KzOcDP5FJduSxnEuh+KAQheRHacZ7El+6J1vaH54ybN\ncJGq/mKb8NGaREqHwWQGh0cr96ejMfegO07EYVWfd2Y/DvIxHVfCwN3tbPZk18WA2hthstL\nW7XTrj0sLvaw5jSf/ujOt8EwSPwy5qrwJcvzk143LU4rlhXCl3x2e1jmPGX+OYv4CRQVi3l\n+WtXDmrl3IRK4YZLBnyrcOvr+FRWL",
  "143LU4rlhXCl3x2e1jmPGX+OYv4CRQVi3l\n+WtXDmrl3IRK4YZLBnyrcOvr+FRWL83wae2DGrBjnmOmgjZ2URfj7aSVUWhZc+fOGwlK6ReL\nqHLmByLhfyCkUmJ8J6Kvrj1nG/AIyeXuo+JWfxDFTQTVc39ydVUOPR0JV/KszqbtZ3N2u\nFQvM5Yf7G/rEUPBbvOKmkVnQl1wg8mlUV70QdDAQHIDqcgETxHOrU+fFCt4corCIJGLiXTK\nBzoft6",
  "BbvOKmkVnQl1wg8mlUV70QdDAQHIDqcgETxHOrU+fFCt4corCIJGLiXTK\nBzoft6RqpWBY8gJy3tDdGgkEo+aVkbxIKpjFvKHiue9fVgBcZzAJ0Fb4moO9lKnZ4rqCT4\nosrnIdwy1kTEW8bgKG7DOpR9Q2VCklXOq3rD+w9Zqp8yZxSVp3NdMRZO1nbafIaF5U0HbqC\nLJgEUZtq4gS8KeD1jMIMtNeQDjl0dsatCYVWQhdnPEq/dqojeG1OUtg",
  "F5U0HbqC\nLJgEUZtq4gS8KeD1jMIMtNeQDjl0dsatCYVWQhdnPEq/dqojeG1OUtgvbW+zIum/ZCgjO\ngC7T38Lpnze1jeSpe0uknNZ+7rAJ+4YJqt9Ccui+bAWjcComtiMmnWukEmzBaEsuWqbujcWl\naeiPUAdwJuzIQK39Pu1yVYsjo8vA9DzUrJT37q/MInp1VXbxv9D8kmVJSXqa0iHf4fFQXw\nlMHrCyJ48hKJg8C9eQlEu7vaOpYhe2jtR",
  "Inp1VXbxv9D8kmVJSXqa0iHf4fFQXw\nlMHrCyJ48hKJg8C9eQlEu7vaOpYhe2jtRzBwWhmBTFG1/Ean2NXUEdzaJUV8hoOuFbyYU\nmuQwbMs6oGX4huelZQH5aJD+fIy+TPIy4+Tmh9YzRGpd3xYzoR9W7Ruq1EL7vsHl8iow8Ph\nkl9zuYcy6s3z6SWlCliGkjnRUzp5O8wL2GK23V9P+bxotSJ+sdW0B/2C2Sl9n1+MtvB8RMSi\njkR1wQHFWp",
  "GkjnRUzp5O8wL2GK23V9P+bxotSJ+sdW0B/2C2Sl9n1+MtvB8RMSi\njkR1wQHFWpcklqU9qGu5XN/vWbX19h5Z2pHFtZuS1Nv0m5b3Gt6wC+2Lb3dJh6xqCNRXU0\nPqUcsS3tQlz2P27ZRWFy7KUm9izxabYu7NHyD/fHvGD6mJTIQB/7Ejmch7BYULGwiknMIyT\nOQ1iMy7YF/8fKnoCHR9uah7DYz0Vb0wEsBVziIcxDWJxv4bZxLC6bVG37Sq",
  "MIyT\nOQ1iMy7YF/8fKnoCHR9uah7DYz0Vb0wEsBVziIcxDWJxv4bZxLC6bVG37SqT6RiZ8xAWn7E\nYj3oewmJExcgqnrM0ReI8RPI4xnkc0zymWEptEp6R1DIjZEnZFlQ2TtqSDmBpglqbWBqDHs\nhEoQabIJZzuvJy68pTaBUruoHtoYH1zRcMFShDmBph+wxd7hj3WQeTjEcs2xJTgWyUprAPn\nb61Fmc/rywIic5L5waOqX0ytArSg8NPaQ0M",
  "wxd7hj3WQeTjEcs2xJTgWyUprAPn\nb61Fmc/rywIic5L5waOqX0ytArSg8NPaQ0M5T8IvDC14aSXydeGnoJaUHh5QWhpaUjowdE\nBpaGhI6VNDn1LqG+pTumHoBqWFoeRECk8EQ/cpHRs6pvTI0CNKjw09pvS5oc8pfWPoG0rfG\nfqO0seGPqaUGco3TR0k1JuKHl14IXrhq5T6hlKfvBXjO0T2lqaErpE0OfUBoYSn4Vw/PMU\nHK8gQejoZLSF4",
  "JuKHl14IXrhq5T6hlKfvBXjO0T2lqaErpE0OfUBoYSn4Vw/PMU\nHK8gQejoZLSF4a+oFQYSn6/eErQ19RGhsaU/rS0JeUnhl6RukzQ59RGhlK3g3A6cTQPUrNW\n6Aqp3TX0F1KLwy9sL8X4Mtp9GwLc8dUsENpYmhC6Zah5JcCHCUMPSfnyVA1d7XF2yZyXwvVk\nltYk/HF1STnoVpyC2vuTouryf0pVEs+Jl3fPFi+SIGUwp1+tLWw29haeHg50",
  "vVk\nltYk/HF1STnoVpyC2vuTouryf0pVEs+Jl3fPFi+SIGUwp1+tLWw29haeHg507v186D3Qdr\nj9abN7S3nO+c750fnJ7zm/PIe70nYHjO386fzl/O/+s3lvtrx6uHs/Vmzea75xWp9V7z8z\noeDX</latexit>\nyj = \u03c6j0 +\nD\nX\nd=1\n\u03c6jdhd\n\u2022 e.g. Three inputs, three hidden units, two outputs\n42",
  "Question:\n\u2022 How many parameters does this model have?\n43",
  "How many \nhidden units?\n44",
  "Output with boundaries and in 3D\n45",
  "How would you draw and write this neural network?\n46",
  "Inputs\nOutput\nNeurons\n\u201cneural network\u201d\n<latexit sha1_base64=\"4mwMW+AxXHfhTyI/kaVAw9s5G24=\">AWsniclZhb9s2F\nIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3pIvTxEmaODUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzUukyPJu9r1z/48KOP7nx6c3Pv/iy69Wbn19kMV\nF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42ND+84GkmYrW",
  "Pv/iy69Wbn19kMV\nF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiUD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb346H\n49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2F",
  "Qn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX\n1/QlLmZ9DTm8OFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyRUCD+dlyTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5\nDyElLOyYaFBLJpy1rg1gwl",
  "gOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5\nDyElLOyYaFBLJpy1rg1gwlVFL2QPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9YjahiqkhEP9lvUHtl4xdYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh2\n6oiyJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b",
  "pDlpjyCAUeujthVobAqyMLsp7HXbjvREbw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2p2aVK2TSbEojS/bpu6NReW\nJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI9R",
  "IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2NI9RXCOh6\n4ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1ClVponze4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk51tNe9AvmJ3C9/\nn5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2",
  "k51tNe9AvmJ3C9/\nn5aAvPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFfTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/IP9Cc+Zvk2K5Vjf9sV\nyWIewmFMxt4pxEMk1iEsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGN",
  "L/Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hMWQiqFVPGNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZF\nlQ6iduSDmBpilqbWhqDHshYoQabIJYzuvIy68pTaBUruoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJecHM0Bml4ZeUnpo6CGlqaHkicALXh\nlKnk684MLQC0oPD2gtDC0",
  "0qbO4+/OCktzJecHM0Bml4ZeUnpo6CGlqaHkicALXh\nlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieGPqHUN9SndMPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh8Z+ohSZijdNPQTUq5oeTVgResG7pOqWco\nefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8of",
  "0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84KXhr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMB\nTuUxobGlG4ZSp4U4FbC0DNyPxmo5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6TrmwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX5",
  "BWrJ6TrmwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjO\nn7wc3/nT+cv52/ln9f7q8Spb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_base64=\"s0UuQl8ojalO49HAr3NW0dwfYA=\">AXU3iclZhb9s2FICdrtu6bN3SDWsf9i\nIs6DBsXWAn3eWlQJs0vSVdnHvaKDUom",
  ">AXU3iclZhb9s2FICdrtu6bN3SDWsf9i\nIs6DBsXWAn3eWlQJs0vSVdnHvaKDUomZLVUJQiUYlTwT9zGPZb9rJDSTajc5iHBWhNn+/TIXlIXSwvFVGut1/5m58dPjTz69dn851/c/vKrhTtfH+RJkfl8309Ekh15LOciknxfRUrwozTjLPYEP/RO1zQ/POdZHiVyT12m/C\nRmoYyCyGcKQoOFv0eDsjdxfnjkuLGXjEs2OXbViCsG4e7E+dmZfet",
  "HiVyT12m/C\nRmoYyCyGcKQoOFv0eDsjdxfnjkuLGXjEs2OXbViCsG4e7E+dmZfetNxoPele/L8H35xHFlIovY45njuvOQbNmebLmVbLlONvt6Ta4Ve6Vq6Vdq6VOtdgYbG71K3+HNroNY3FTvPXH9z5dugOE7+IuVS+YHl+3Oum6qRkmYp8wS\nfzbpHzlPmnLOTH0JQs5vlJWa3AxLkPkaETJBn8k8qpolePKFmc5exB2bM1CjHTAdt7LhQwR8nZ",
  "LOTH0JQs5vlJWa3AxLkPkaETJBn8k8qpolePKFmc5exB2bM1CjHTAdt7LhQwR8nZSTQnHp1x0FhXBU4ujldIZRxn0lLqHB/CyCsTr+iGXMV7Do867kF34Sx0wOS3d1fXtSuh4PI1nys6LaAJNJ21mvHA7N64zVl3\nuzLJHicfSBkySVopNcI/BwUpZ8KVzCIOIAoiVOQCJ5Djl1fbzA6SEKG14ALud4oKxMyGpeIh1KSlvSUaNFLBxy1rjViwlHF",
  "OIAoiVOQCJ5Djl1fbzA6SEKG14ALud4oKxMyGpeIh1KSlvSUaNFLBxy1rjViwlHFL2QXFce47GnCVwSrAUOGDozXYTZmcTI9TfKyuMx1DPeQMRnyqguYs+EnlHbkIUQcKjfsv7E1g\n6Tp03hkrQaqYjyNrL2o7KaF3ksO1UEWTBJgzbVhVBloDL05DFDKrctAcw4djREbsaSaxGZGP2s8Rr953qCN6b4xTOl7a3XpLynzNUER2As09/Rkz6vK2vJ",
  "Acw4djREbsaSaxGZGP2s8Rr953qCN6b4xTOl7a3XpLynzNUER2As09/Rkz6vK2vJTPbmRbnvPJ1g4+dESxW+xCWhfW0p3ArJrYhJpVrZBJqwWhLlom3\no0FpWnUXuCOoBPuiKLZHBFe1C1YMvqsPsApoVgh/svQrH5+UX3a6P9INSFRXqS2RDr8PxIN4YaI9xdE8OIlAi0eBKrFSwRc39HSsQxvbB2p1g4akWQiUpfo9I9C2T6miuDBJjEaKwR0X",
  "xdE8OIlAi0eBKrFSwRc39HSsQxvbB2p1g4akWQiUpfo9I9C2T6miuDBJjEaKwR0XvhkUSLHARtWQe0DJ9wa7dsIB9N0q\n/n6IskLzJOLn5oP0Ok0vVlMYv0zap9QRVaF83uJgdBW24OZzaw73UEW9up5eUsghy1Ax3pJx+/cXMEpZjv7qyWvm1Yr5GcbTX8wLlidwvf52WADr0dILOoIlAuepay5BLEs/UGu2Xa9OrJy491PZGuHFtduCpK3GaXd",
  "Llidwvf52WADr0dILOoIlAuepay5BLEs/UGu2Xa9OrJy491PZGuHFtduCpK3GaXdtrjXjI\nCfbVpGu0k8YlFHoFzNCKlHLEt/kMtex03bLCyu3RQk7SOVtvizky0/YM9/SyqH5MSMdSPfYlw6xAWFRWVUxiHiKxDmExLtoWfMfKbgQ3j7ZVh7DYz6O2pgNYGnKBp1CHsFifwm2ziWF106Ju2lUm0hEy6xAWn7MYz7oOYTGkYm\ngVT1maIrEOkTqO",
  "nKBp1CHsFifwm2ziWF106Ju2lUm0hEy6xAWn7MYz7oOYTGkYm\ngVT1maIrEOkTqOcB1HtI4plKbhFcktawI2VK2DZWNkrakA1gao97Gls5gBCKRqMmiOWc7rzcuvMk2sWS7uJ9W8f713SsGEqoA1jaIueY425ZTzIPlxges2xFTiNkpbSAfez0qTN9+vOCkjzJecGloZeUXh6QemhoYeUZoaSXw\nResGMo+XiBeGnlN6YOgBpYWhBaX7hu5TGhga",
  "zJecGloZeUXh6QemhoYeUZoaSXw\nResGMo+XiBeGnlN6YOgBpYWhBaX7hu5TGhgaUPrM0GeU+ob6lK4ZukapMpQ8kcIdwdA9SkeGjig9MvSI0jeGvqH0haEvKH1r6FtKPxj6gdInhj6hlBnKF03dJ1Sbih5deAFq4auUuoZSn7wblmaJ/S1NCU0qeGPqV0aCj5VQ\nz3M0PJ4w3cGA0VlL409CWlkaHk95sXvDb0NaWxoTGlrwx9Rel7Q9T+tzQ5S",
  "j5VQ\nz3M0PJ4w3cGA0VlL409CWlkaHk95sXvDb0NaWxoTGlrwx9Rel7Q9T+tzQ5SGhpJ3A/B0YugupeYtUJlTum3oNqVnhp7Z3wvw2TJ6to25ZRJsUZoYmlC6YSj5pQCPEoaekufJQDZXtenbJnJdC+SMW1hT8enRpOaBnHELa65O06\nPJ9SmQMz4iQ18/mL1IgZLClX6wsNjDb2Fp42B5qfb0sPth4uPV5s3tLc63W+7/zY6XV+7zuvOj0O/sdf",
  "1IgZLClX6wsNjDb2Fp42B5qfb0sPth4uPV5s3tLc63W+7/zY6XV+7zuvOj0O/sdf+7RnD8n5uK7f9396Nezdr9cZc8w3ndbfvdv/Ab4sGxg=</latexit>h1 = a[\u271310 + \u271311x1 + \u271312x2]\nh2 = a[\u271320 + \u271321x1 + \u271322x2]\nh3 = a[\u271330 + \u271331x1 + \u271332x2]\nHow would you draw and write this neural network?\n47",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 ouput\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n49",
  "Number of output regions\n\u2022 In general, each output consists of mult-dimensional convex polytopes\n\u2022 With two inputs, and three hidden units, we saw there were seven \npolygons for each output:\n50\nPolytope -- Wikipedia\nIn elementary geometry, a polytope \nis a geometric object with flat sides \n(faces). Polytopes are the \ngeneralization of three-dimensional \npolyhedra to any number of \ndimensions. Polytopes may exist in \nany general number of dimensions \nn as an n-dimensional polytope or \nn-polytope.",
  "Example with \ud835\udc37 = \ud835\udc37! \u00e0 2\"! regions\n1 input (1-dimension) with 1 \nhidden unit creates two \nregions (one joint)\n2 input (2-dimensions) with 2 \nhidden units creates four \nregions (two lines)\n3 inputs (3-dimensions) with 3 \nhidden units creates eight \nregions (three planes)\n51\n\ud835\udc37\" : # of inputs\n\ud835\udc37  : # of hidden units\n\ud835\udc37# : # of outputs",
  "\u2022 Number of regions created by D > \ud835\udc37! hyper-planes in \ud835\udc37! dimensions \nwas proved by Zaslavsky (1975) to be:\n\u2022 How big is this?  It\u2019s greater than 2#!but less than 2#.\nNumber of regions:\n<latexit sha1\n_base64=\"K7k+Z+5xCN14scF9l\nTipkuFEJZg=\">AWmHiclZhb\nb9s2FIDVXbvulm5YHrY9CAsKD\nENnOEN3eRnQ5tI0TbpcnaSNU4O\nSKZkJRSkSlTgV/LJfs9ft3+zf\n7FCWzeoc5mEGUrPn+8TLISnRCj\nIpCt3t/nvnXfe/+D+9+dO/\njTz7",
  "fs9ft3+zf\n7FCWzeoc5mEGUrPn+8TLISnRCj\nIpCt3t/nvnXfe/+D+9+dO/\njTz797POF+18cFWmZh7wXpjLNT\nwJWcCkU72mhJT/Jcs6SQPLj4G\nLV8OMrnhciVYf6JuNnCYuViET\nINIQGC9/2izIZVOe/dyevq7WBm\nPQDodKkWptU5PBwlK3060/Pi\n0sN4Ulr/nsDu5/NewP07BMuNKh\nZEVxutzN9FnFci1CySf3+mXBM\nxZesJifQlGxh",
  "sN4Ulr/nsDu5/NewP07BMuNKh\nZEVxutzN9FnFci1CySf3+mXBM\nxZesJifQlGxhBdnVT2Mif8AIkM\n/SnP4U9qvo29fUbGkKG6SAMyE\n6VGBmQm62Gmpo9/OKqGyUnMVT\nhuKSunr1Dc58Yci56GWN1BgYS6\ngr34YjkLNWTuXl/x6zBNEqaG\nVX9lfW9S9QMeC1Xxy7LO4mTSdt\nZrh0PxNmNl83Bei9A8EW84qaR\nWTCW3CDyeVBXvxB0MBAc",
  "QMeC1Xxy7LO4mTSdt\nZrh0PxNmNl83Bei9A8EW84qaR\nWTCW3CDyeVBXvxB0MBAcgOpyAV\nPEC6jT5CSJ/GVFYNRIw8CAdQ+\ncif39Cqlax5CTlvaKaFDIJB+\n3rFViwVQmLeUAFN9/4BvAdQ6zA\nF2FL47m4CBjajK7TvOxzpOqMD\nHcQs5UzOsmYMghk2ZEbUOVUsKl\nYcv6A1v7TF0iUuzuqu5iSDrM\nG87Oqd5UcO2U0eQBYswblt1BFk\nS9",
  "EbUOVUsKl\nYcv6A1v7TF0iUuzuqu5iSDrM\nG87Oqd5UcO2U0eQBYswblt1BFk\nS9viQJQy3JQHMODENxG3KhRW\nBVmYu3katNvOTASvzXEG+6Xtr\nVck/VcMZcQEYPeZb8FUyNv6ajq\n3/VlyrmrfFPjYH8FktS9heTwd\n1qwRGFUTm1CzhUyabYglKfXbd\nP0xqHyTLQHaAJ405W5UNFb2sO\n6BEvWhPsPYah5Kfnpj52f+fis6\nptY/4h2YSKi",
  "P0xqHyTLQHaAJ405W5UNFb2sO\n6BEvWhPsPYah5Kfnpj52f+fis6\nptY/4h2YSKijJzVWTC/6OiIT\nxV8PqCJ68VKLJg0A9eamE+zu\naOpbjhW0i9dxBQSgmhb5B21/Eq\nn1NHcGdTRPUVwiYeuGbCYUmOY\nrasgkYGb7h+ehYQCEaZDgdYyjT\nosw5ufmh9QyRWje3xVyYh1X7h\niqN0L5vcDm/CsrwcLjit1weoIw\nG03wGamGLEfJHJspHb/u",
  "QyRWje3xVyYh1X7h\niqN0L5vcDm/CsrwcLjit1weoIw\nG03wGamGLEfJHJspHb/uFxq2\nmGv31M+LTqtmF9uNe1Bv2B2y\njDkl4MtPB8xsagjUV1wIHWJYn\nlaA/qmi/Xt3tWb3+gSzt2OG6\nTUnqbXrpth3uLT3gl9uO3m4Tj1\njUkaiupofUI5ajPajLncdt1yg\ncrtuUpN5ZHp2w52baPlHhyOum\nTkmpXJojn2p7E9DWNRU1E4xTX\niMxGkI",
  "dt1yg\ncrtuUpN5ZHp2w52baPlHhyOum\nTkmpXJojn2p7E9DWNRU1E4xTX\niMxGkIi0nZtuD/WDkQ8PBoW9M\nQFncL0dZMAEtDLvEQpiEsTrdw2\n2xiWN12qNtulclshMxpCIsbLM\nGjnoawGFMxdoXLMuQOA2RPI5w\nHkc0jxmWMpeEZyRzAhZUq4Fl\nY/StmQCWBqj1saOxqAHMlWowSa\nI5YKuvMK58hRaxYqu4p6r4d4t\nDWuGKjQBLO2QPeb",
  "mQCWBqj1saOxqAHMlWowSa\nI5YKuvMK58hRaxYqu4p6r4d4t\nDWuGKjQBLO2QPeb3d5ybLMAph\nmOWK8mZQFZGE7iLnV3qzE5/QVS\nRk1wQ3Vh6Q+m1pdeUHlt6TGlu\nKflFET7lpJfJ0F0ZekVpUeWHl\nFaWlpS2rO0R2lkaUTpU0ufUhp\naGlK6aukqpdpSciKFJ4Klh5SOL\nB1RemLpCaUvLX1J6TNLn1H6yt\nJXlL6x9A2lTyx9QimzlFG6b",
  "SciKFJ4Klh5SOL\nB1RemLpCaUvLX1J6TNLn1H6yt\nJXlL6x9A2lTyx9QimzlFG6buk\n6pdxS8uogiFYsXaE0sJT89oO9Z\nukupZmlGaVrlq5ROrSU/CqG5\nml5HgD0ZLJaWblm5SKiwlv9+C\n6IWlLyhNLE0ofW7pc0rPLT2nd\nMPSDUpjS8m7ATidWHpAqX0LVBW\nU7lm6R+mlpZfu9wJ8Po2Ba2Hu\n2Ap2KE0tTSndspT8UoCjhKUX5\nDwZqea",
  "0LVBW\nU7lm6R+mlpZfu9wJ8Po2Ba2Hu\n2Ap2KE0tTSndspT8UoCjhKUX5\nDwZqeauNnvbRO5rkZpzB2syPru\na5DxSc+5gzd1pdjW5P0Vqzkek\n6+tH8xcpkFK40w8WlpbxW1haOP\nqps/xL59Heo6XHK80b2rveN95\n3vfesver9h75u16PS/0/vT+\n8v72/ln8evHx4sbi5lR9505zZ\nde67O4/x83Ndq2</latexit>",
  "75u16PS/0/vT+\n8v72/ln8evHx4sbi5lR9505zZ\nde67O4/x83Ndq2</latexit> Di\nX\nj=0\n\u2713D\nj\n\u25c6\nBinomial coefficients!\n52\n= \n!!\n#! !$# !\n\ud835\udc37\" : # of inputs\n\ud835\udc37  : # of hidden units\n\ud835\udc37# : # of outputs",
  "Number of output regions\n\u2022 In general, each output consists of D dimensional convex polytopes\n\u2022 How many?\nHighlighted point = 500 hidden units or 51,001 parameters\n53\n\ud835\udc37\" : # of inputs\n\ud835\udc37  : # of hidden units\n\ud835\udc37# : # of outputs",
  "Shallow neural networks\n\u2022 Example network, 1 input, 1 ouput\n\u2022 Universal approximation theorem\n\u2022 More than one output\n\u2022 More than one input\n\u2022 General case\n\u2022 Number of regions\n\u2022 Terminology\n54",
  "Nomenclature\n55",
  "Nomenclature\n\u2022\nY-offsets = biases\n\u2022\nSlopes = weights\n\u2022\nEverything in one layer connected to everything in the next = fully connected network (multi-layer \nperceptron) \n\u2022\nNo loops = feedforward network\n\u2022\nValues after ReLU (activation functions) = activations\n\u2022\nValues before ReLU = pre-activations\n\u2022\nOne hidden layer = shallow neural network\n\u2022\nMore than one hidden layer = deep neural network\n\u2022\nNumber of hidden units \u2248 capacity\n56",
  "Other activation functions\n57",
  "We have built a model that can:\n\u2022 take an arbitrary number of inputs\n\u2022 output an arbitrary number of outputs\n\u2022 model a function of arbitrary complexity between the two\nRegression\n<latexit sha1_base64=\n\"eu7R9HxZr4T2SlEuSDpbPsp4w0=\">AWx3iclZ\njLbtw2FECV9JWmL6dFvelGqBGgaFPDLtLHJkBix0k\ncO7UdP2PLGVAaSsOYomSJscRtOgn9WuK7to/6aVG\nM4zupRcdwBnmniM+LkmJozCXotRLS3/fuPne+x98\n+NGtj29/8uln38xd+fLgzKriojvR5nMiqOQlVwK",
  "LkmJozCXotRLS3/fuPne+x98\n+NGtj29/8uln38xd+fLgzKriojvR5nMiqOQlVwKx\nfe10JIf5QVnaSj5YXi2avjhBS9Kkak9fZXz05QlSs\nQiYhpCg7mN0aAeNv4DP0jDbFyzJpA81ieBHnHNAC0\n1/g9+UFbpoBYPlpvX9WMoNM2Mi2Y8EhkpE+Hcwt\nLC0utR+fFpa7woLXfbYHd74eBsMsqlKudCRZWZ4sL\n+X6tGaFpHkze2gKnOojOW8BM",
  "0utR+fFpa7woLXfbYHd74eBsMsqlKudCRZWZ4sL\n+X6tGaFpHkze2gKnOojOW8BMoKpby8rRuR934dy\nEy9OsgD+l/Tb67hU1S8vyKg3BTJkelZiZoIudVDr\n+7bQWKq80V9GkobiSvs58k0J/KAoeaXkFBRYVAvr\nqRyNWsEhDom8Hil9GWZoyNayDlbWdpg5CnghV8/Oq\nTXrT9J21uFQvM5YWd+b1SI0T8VbTipFVPJNQJPm\nrmi8kiBoIDEI",
  "g5CnghV8/Oq\nTXrT9J21uFQvM5YWd+b1SI0T8VbTipFVPJNQJPm\nrmi8kiBoIDEIucgEzxEuo0+QljfxlRWGQScD1ZNw\nEYLxtStdI8gZz0tGOiQSGXfNyzVokFU5n2lF1QfP+\nubwDXBcwCdBW+OJqD3ZypZnqd5mNdpHVpYriFgqmE\nt03AkCMmzYj6hqkhEujnvU7tl4ydYlLsvbrhYmg\nqy9ou/oguZFDftOG0EWLMKkb7URZEm4JQxZyiDL",
  "hEujnvU7tl4ydYlLsvbrhYmg\nqy9ou/oguZFDftOG0EWLMKkb7URZEm4JQxZyiDLX\nXkA059E3GrQmFVkIW5XWRhv+3cRPDaHOewX/reWk\n3Sf8FQRkwAdp/5FkxFvK+vZjPbnybnovVNgY/9EUx\nW/xJWJNhTRuBUXWxhptrpBJswWhIrvsm6Y3DpXn\noj9AE8CbriqEit/R7rUlWLImHNyDoRaV5Cc/Lv7Mx\n6f1ktk25h+STaiorHJXRSb8Py",
  "AE8CbriqEit/R7rUlWLImHNyDoRaV5Cc/Lv7Mx\n6f1ktk25h+STaiorHJXRSb8PyoawkMIry+I4MnLJ\no8CLSTl0m4v6OpYwVe2CbSzh0UhGJS6Cu0/UWi+te\n0EdzZLEV9hYCpF76ZUGiS47gvm4CR4Rsep4FKFB\nRpMxRjIrq4KTmx9azxBpdXNbLIR5WPVvqNI/fsG\nl7OroAwPhwt+zeUhymg4yWeYVWrICpTMsZnS8eug1\nLDFXLu/nfJ0W",
  "VvqNI/fsG\nl7OroAwPhwt+zeUhymg4yWeYVWrICpTMsZnS8eug1\nLDFXLu/nfJ0Wkl/Hyjaw/6BbNTRE/H2zg+UiIR\n2J6oLzi7MuSxHe1DXbLm+27N64/X3ZGknDtdtSlJ\nv10u37XCv6QE/3T0dpN4xKORHV1PaQesRztQV3u\nPG6RuFw3aYk9U7z6LQd7sxEyz/eMydSc0zK5NAc+\nzIZTEJY1FTUTjFLeYLESQiLadW34P9Y2RXw8Ohbkx\nA",
  "sxEyz/eMydSc0zK5NAc+\nzIZTEJY1FTUTjFLeYLESQiLadW34P9Y2RXw8Ohbkx\nAWt0vR10wAS0Mu8RAmISxOtnDf7GJY3XSom26VyXy\nEzEkIi09Zikc9CWExoWLiFM9YniNxEiJ5HOE8jmg\necyzlLgnPSO6YEbKkXAuqGV9yQSwNEatjR2NQ9k\nplCDXRDLJV15pXPlKbSKFV3F+6G969pWDNUoQlga\nYvsMT/Ycm6yEKcYjlmuJOcCWTlN4D",
  "DLJV15pXPlKbSKFV3F+6G969pWDNUoQlga\nYvsMT/Ycm6yEKcYjlmuJOcCWTlN4DZ2tqkzPf2FcU\n1OcmF8ZekVpZeWXlJ6aOkhpYWl5BdBGL+0lPw6CeM\nLSy8oPbD0gNLK0orSfUv3KY0tjSl9YukTSiNLI0pX\nLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSVpa8ofWbpM\n0qPLT2m9K2lbyl9ZOkjSpmljNI1S9co5ZaSVwdhv\nGLpCqWhpeS3H",
  "Vpa8ofWbpM\n0qPLT2m9K2lbyl9ZOkjSpmljNI1S9co5ZaSVwdhv\nGLpCqWhpeS3H+w1S7cpzS3NKX1s6WNKh5aSX8XwPL\nOUHG/gwWipHTd0nVKhaXk91sYv7D0BaWpSmlzy1\n9TukbS9Q+tTSp5QmlpJ3A3A6sXSXUvsWqC4p3bF0\nh9JzS8/d7wX4bBpD18LcshVsUZpZmlG6YSn5pQBHC\nUvPyHkyVt1dbfq2idzXYjXjDtZlfHo1yXmsZtzB",
  "8LcshVsUZpZmlG6YSn5pQBHC\nUvPyHkyVt1dbfq2idzXYjXjDtZlfHo1yXmsZtzBur\nvT9Gpyf4rVjI9I19cOZi9SIKVwpx/MLSzjt7C0cPD\nT4vIvi/d37i8XOne0N7yvG+9b7zlr1fvYfeM2/b\n2/ci70/vL+8f79/59fls/mJ+PFv3uiu+crfeb/\n+A+HQO5X</latexit>\nhd = a\n\"\n\u2713d0 +\nDi\nX\ni=1\n\u2713dixi\n#\n<latexit sha1_base64=\"iQ/nJ5",
  "QO5X</latexit>\nhd = a\n\"\n\u2713d0 +\nDi\nX\ni=1\n\u2713dixi\n#\n<latexit sha1_base64=\"iQ/nJ5VzZC6Vgn\nR5+vmegEip1K8=\">AWqHiclZhb9s2FIDV7tZ1t3TD8rIXYUGBoesMe+guLwXapOkt6eI0\ncS6NU4OSKJkJRSm6JHYF/4n9mr1u/2L/ZoeybFbnMA8zkJo93ydeDkmJlpdKkRfd7r83bn7w\n4Ucf3Lr09uf7Fl1+t3Pn6IE/KzOcDP5FJduSxnEu",
  "ydeDkmJlpdKkRfd7r83bn7w\n4Ucf3Lr09uf7Fl1+t3Pn6IE/KzOcDP5FJduSxnEuh+KAQheRHacZ7El+6J1vaH54ybN\ncJGq/mKb8NGaREqHwWQGh0cr96ejMfegO07EYVWfd2Y/DvIxHVfCwN3tbPZk18WA2hthstL\nW7XTrj0sLvaw5jSf/ujOt8EwSPwy5qrwJcvzk143LU4rlhXCl3x2e1jmPGX+OYv4CRQVi3l\n+WtXDmrl3IRK4YZLBnyr",
  "5qrwJcvzk143LU4rlhXCl3x2e1jmPGX+OYv4CRQVi3l\n+WtXDmrl3IRK4YZLBnyrcOvr+FRWL83wae2DGrBjnmOmgjZ2URfj7aSVUWhZc+fOGwlK6ReL\nqHLmByLhfyCkUmJ8J6Kvrj1nG/AIyeXuo+JWfxDFTQTVc39ydVUOPR0JV/KszqbtZ3N2u\nFQvM5Yf7G/rEUPBbvOKmkVnQl1wg8mlUV70QdDAQHIDqcgETxHOrU+fFCt4corCIJGLiX",
  "Yf7G/rEUPBbvOKmkVnQl1wg8mlUV70QdDAQHIDqcgETxHOrU+fFCt4corCIJGLiXTK\nBzoft6RqpWBY8gJy3tDdGgkEo+aVkbxIKpjFvKHiue9fVgBcZzAJ0Fb4moO9lKnZ4rqCT4\nosrnIdwy1kTEW8bgKG7DOpR9Q2VCklXOq3rD+w9Zqp8yZxSVp3NdMRZO1nbafIaF5U0HbqC\nLJgEUZtq4gS8KeD1jMIMtNeQDjl0dsatCYVWQhdnPEq/dq",
  "ZO1nbafIaF5U0HbqC\nLJgEUZtq4gS8KeD1jMIMtNeQDjl0dsatCYVWQhdnPEq/dqojeG1OUtgvbW+zIum/ZCgjO\ngC7T38Lpnze1jeSpe0uknNZ+7rAJ+4YJqt9Ccui+bAWjcComtiMmnWukEmzBaEsuWqbujcWl\naeiPUAdwJuzIQK39Pu1yVYsjo8vA9DzUrJT37q/MInp1VXbxv9D8kmVJSXqa0iHf4fFQXw\nlMHrCyJ48hKJg8C9eQlEu7va",
  "UrJT37q/MInp1VXbxv9D8kmVJSXqa0iHf4fFQXw\nlMHrCyJ48hKJg8C9eQlEu7vaOpYhe2jtRzBwWhmBTFG1/Ean2NXUEdzaJUV8hoOuFbyYU\nmuQwbMs6oGX4huelZQH5aJD+fIy+TPIy4+Tmh9YzRGpd3xYzoR9W7Ruq1EL7vsHl8iow8Ph\nkl9zuYcy6s3z6SWlCliGkjnRUzp5O8wL2GK23V9P+bxotSJ+sdW0B/2C2Sl9n1+MtvB8RMSi\nj",
  "3z6SWlCliGkjnRUzp5O8wL2GK23V9P+bxotSJ+sdW0B/2C2Sl9n1+MtvB8RMSi\njkR1wQHFWpcklqU9qGu5XN/vWbX19h5Z2pHFtZuS1Nv0m5b3Gt6wC+2Lb3dJh6xqCNRXU0\nPqUcsS3tQlz2P27ZRWFy7KUm9izxabYu7NHyD/fHvGD6mJTIQB/7Ejmch7BYULGwiknMIyT\nOQ1iMy7YF/8fKnoCHR9uah7DYz0Vb0wEsBVziIcxDWJxv4bZxL",
  "BYULGwiknMIyT\nOQ1iMy7YF/8fKnoCHR9uah7DYz0Vb0wEsBVziIcxDWJxv4bZxLC6bVG37SqT6RiZ8xAWn7E\nYj3oewmJExcgqnrM0ReI8RPI4xnkc0zymWEptEp6R1DIjZEnZFlQ2TtqSDmBpglqbWBqDHs\nhEoQabIJZzuvJy68pTaBUruoHtoYH1zRcMFShDmBph+wxd7hj3WQeTjEcs2xJTgWyUprAPn\nb61Fmc/rywIic5L5waOqX0ytAr",
  "FShDmBph+wxd7hj3WQeTjEcs2xJTgWyUprAPn\nb61Fmc/rywIic5L5waOqX0ytArSg8NPaQ0M5T8IvDC14aSXydeGnoJaUHh5QWhpaUjowdE\nBpaGhI6VNDn1LqG+pTumHoBqWFoeRECk8EQ/cpHRs6pvTI0CNKjw09pvS5oc8pfWPoG0rfG\nfqO0seGPqaUGco3TR0k1JuKHl14IXrhq5T6hlKfvBXjO0T2lqaErpE0OfUBoYSn4Vw/PMU\nHK8g",
  "Gco3TR0k1JuKHl14IXrhq5T6hlKfvBXjO0T2lqaErpE0OfUBoYSn4Vw/PMU\nHK8gQejoZLSF4a+oFQYSn6/eErQ19RGhsaU/rS0JeUnhl6RukzQ59RGhlK3g3A6cTQPUrNW\n6Aqp3TX0F1KLwy9sL8X4Mtp9GwLc8dUsENpYmhC6Zah5JcCHCUMPSfnyVA1d7XF2yZyXwvVk\nltYk/HF1STnoVpyC2vuTouryf0pVEs+Jl3fPFi+SIGUwp1+tLWw",
  "7XF2yZyXwvVk\nltYk/HF1STnoVpyC2vuTouryf0pVEs+Jl3fPFi+SIGUwp1+tLWw29haeHg507v186D3Qdr\nj9abN7S3nO+c750fnJ7zm/PIe70nYHjO386fzl/O/+s3lvtrx6uHs/Vmzea75xWp9V7z8z\noeDX</latexit>\nyj = \u03c6j0 +\nD\nX\nd=1\n\u03c6jdhd\n58",
  "Next time:\n\u2022 What happens if we feed one neural network into another neural \nnetwork?\n59\nFeedback?\nLink",
  "Lecture 04\nDeep Networks\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Project Proposal Template\nLet me know if you want help finding teammates.\nhttps://dl4ds.github.io/sp2024/project/#proposal",
  "Deep neural networks\n\u2022 Composing two networks\n\u2022 Combining the two networks into one\n\u2022 Hyperparameters\n\u2022 Notation change and general case\n\u2022 Shallow vs. deep networks",
  "Composing two networks.\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41ku",
  "1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCF",
  "vqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1",
  "rlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVB",
  "lZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\ns",
  "BljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4R",
  "MjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZ",
  "NMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx",
  "hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSml",
  "pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5P",
  "SxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base6",
  "=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"4mwMW+AxXHfhTyI/\nkaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3pIvTxEmaOD\nUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzUukyPJu9r1z/48KOP7nx6c\n3Pv/iy69Wbn19kMVF6vOBH8s4",
  "DvMwA4np83i5ZDUzUukyPJu9r1z/48KOP7nx6c\n3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiU\nD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb34\n6H49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4h",
  "6H49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP\n6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DT\nm8OFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyRUCD+dly\nTthBwPBAYgOJyBWPIM6dX68wO0hCutJ",
  "4FK8y1p/vL2sROY/EO04qRdyRUCD+dly\nTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYaFBLJpy1rg1gwlVFL2\nQPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9YjahiqkhEP9lvUHtl4x\ndYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiyJKw+8csYpDlpjyCAUeujthVobAqyML",
  "qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiyJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HX\nbjvREbw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2\np2aVK2TSbEojS/bpu6NReWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZ\nhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV",
  "wOD+/AUNC8pOfO7/w6WnZ1dtG/yPZ\nhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2N\nI9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1ClVponze\n4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk51tNe9AvmJ3C9/n5aA",
  "PArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk51tNe9AvmJ3C9/n5aAv\nPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFf\nTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/IP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iE\nsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A0phLPIQ6h",
  "Vjf9sVyWIewmFMxt4pxEMk1iE\nsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hM\nWQiqFVPGNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuv\nIy68pTaBUruoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nH",
  "toYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieGPqHUN9SndM\nPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh8Z+ohSZijdNPQTUq5oe\nTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkq",
  "Sh8Z+ohSZijdNPQTUq5oe\nTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84KXh\nr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMB\nTuUxobGlG4ZSp4U4FbC0DNyPxmo5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6Trm\nwfLFym",
  "C0DNyPxmo5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6Trm\nwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52\n/ln9f7q8Spb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_base64\n=\"/+I4vBnqKIqWuIs3M1Saj+zUNbk=\">AXKHic\nlZhb",
  "\u03c63h3\n<latexit sha1_base64\n=\"/+I4vBnqKIqWuIs3M1Saj+zUNbk=\">AXKHic\nlZhbU9w2FICXlN6I+mUl754yqTtCnDkrTNS2c\nSCLlBCoRrgkje2WvgiwbW4Ylnv1Dnf6YvnXy2l\n/SI9u7is8RD92ZMX5Pt2OZFvrIJOi0EtLb2fe/\n+Dz/6+Nons59+9vkX85dv7FfpGUe8r0wlWl+G\nLCS6H4nhZa8sMs5ywJD8ITlcNPzjneSFStasvM\n36c",
  "X85dv7FfpGUe8r0wlWl+G\nLCS6H4nhZa8sMs5ywJD8ITlcNPzjneSFStasvM\n36csFiJSIRMQ+hk7q/h9ydVf+x97vnJ0E6qtj4\nyNdDrpmJL429nz7Z398ez5KlVlEvDc8/1ZU3v5\nitrL3drL7tq3r6h9u1v7tql9MrewtLhUfzxa6Le\nFhV72Tq5/vXAH6RhmXClQ8mK4qi/lOnjiuVahJ\nKPZ/2y4BkLT1nMj6CoWMKL46pO69i7CZGBF6U5",
  "6RhmXClQ8mK4qi/lOnjiuVahJ\nKPZ/2y4BkLT1nMj6CoWMKL46pO69i7CZGBF6U5/F\nPaq6Pv1qhYUhSXSQBmwvSwMwEXeyo1NHd40qor\nNRchU1HUSk9nXpmjbyByHmo5SUWJgLGKsXDlnOQ\ng0rOesrfhGmScLUoPJX1rbHlR/wWKiKn5X1qo7H\nXWetdjgUrzJWnuxOWxGaJ+INJ43UimnkCoH46r\ni/EiBoIDEIucgFTxAto0+Qkir48o7GI",
  "gUrzJWnuxOWxGaJ+INJ43UimnkCoH46r\ni/EiBoIDEIucgFTxAto0+Qkir48o7GIJuGq2hg/\nG8zFpWmkeQ0462kuiQSGTfNSxVokFS5l0lB1QPO\n+mZwDXOawCDBW+OFqDnYyp8aSe5iOdJ1VhYriHnK\nmY13AlEMmzYy6hiqlhKphx/oDW8+ZOm0Tl2b1U\nHMTQdZu3nV0TvOiBl2njiALNmHcteoIsiTcwYsY\nZDltnwCE048E3GrQmFVkI25",
  "HMTQdZu3nV0TvOiBl2njiALNmHcteoIsiTcwYsY\nZDltnwCE048E3GrQmFVkI25ladBt+/MRPDeHGVw\nvXS9tYqk/5yhjJgAXH3mWzAV8q6+mk5tb5Kc89o\n3BT7yhrBY3Sosj5tpTqBWbWxMTXrXCGTZgtCeXr\nRNc1oHCrPRHeCJoAvujIXKnpHu1WXYMuasH8Lp\nqXkh/9vPgLHx1XS+ayMf+RbEJDRZm5GjLh/9HQAJ\n5yeH9BC9eKtHiQa",
  "uasH8Lp\nqXkh/9vPgLHx1XS+ayMf+RbEJDRZm5GjLh/9HQAJ\n5yeH9BC9eKtHiQaBevFTC/R0tHcvxjaReu2gI\nBSTQl+iy1/EqlunjuDBpgkaKwRMu/DNhEKLHEVd2\nQSMDN/wvHZsoBNMmzmGMq0KHNObn5oP0Ok1s1t\nMRfmYdW9oUojdO8bXE5rQRkeDuf8iuoBymjQ5DN\nISzVgOUrmyCzp6JVfaLjEXFd/veRN0WnF/Gy97Q/\nGBatThiE",
  "uf8iuoBymjQ5DN\nISzVgOUrmyCzp6JVfaLjEXFd/veRN0WnF/Gy97Q/\nGBatThiE/O1nH6xETizoStQUHJGdbkliO/qCt6X\nZ9d2TV+qsfydaOHa7blKTdpRu2+FeMQJ+tuEY7Q\nbxiEUdidpqR0g9Yjn6g7bcedxwzcLhuk1J2p3k0\nWk73KmJtn+0a46i5piUyoE59qXSb0JY1FTUTjFN\neIzEJoTFpOxa8DdWdgQ8PLpWE8LiViG6mglgacAl",
  "piUyoE59qXSb0JY1FTUTjFN\neIzEJoTFpOxa8DdWdgQ8PLpWE8LiViG6mglgacAl\nnkITwmJzCXfNobVDYe64VaZzIbIbEJYfMQSPOs\nmhMWYirFTPGVZhsQmRPI4xHkc0jxmWMpcEl6RzLE\niZEu5NlQ+TLuSCWBphHobOTqDEchUoQ7bIJYLuv\nMK585TaBcruov3XB3vXdGxZqhBE8DSJrnGPH/TeZ\nEFOMVwzHIlORPIymgCt7CzRZ3J6S+I",
  "Bcruov3XB3vXdGxZqhBE8DSJrnGPH/TeZ\nEFOMVwzHIlORPIymgCt7CzRZ3J6S+IKnKSC6JLS\ny8pvbD0gtIDSw8ozS0lvwiC6Lml5NdJEJ1bek7p\nvqX7lJaWlpTuWbpHaWRpROlDSx9SGloaUrpq6Sql\n2lJyIoUngqW7lA4tHVJ6aOkhpS8sfUHpY0sfU/r\nS0peUvrH0DaX3Lb1PKbOUbpm6Rql3FLy6iCIVix\ndoTSwlPz2g2vN0i1KM0sz",
  "/r\nS0peUvrH0DaX3Lb1PKbOUbpm6Rql3FLy6iCIVix\ndoTSwlPz2g2vN0i1KM0szSh9Y+oDSgaXkVzE8zy\nwlxt4MFoqKX1i6RNKhaXk91sQPbP0GaWJpQmlT\ny19SulrS19T+sjSR5TGlpJ3A3A6sXSHUvsWqCo3\nbZ0m9IzS8/c7wX4dBkD18bctA1sUpamlK6bin5\npQBHCUtPyXkyUu1dbfK2idzXIjXlDtZmfFKb5DxS\nU+5g7d1pUpvcny",
  "mlK6bin5\npQBHCUtPyXkyUu1dbfK2idzXIjXlDtZmfFKb5DxS\nU+5g7d1pUpvcnyI15UMy9LX96YsUSCnc6U/mFvr\n4LSwt7C8v9n9dvLN9Z+HeSvuG9lrvm963vR96/d5\n+baet81et85v/9Dzr3C9Y=</latexit>vXu9x72t3l4vnLkxc3fm/szK/J/zf8/M/+2Ud\nh0\n1 = a[\u27130\n10 + \u27130\n11y]\nh0\n2 = a[\u27130\n20 + \u27130\n21y]\nh0\n3 = a[\u27130\n30 +",
  "J/zf8/M/+2Ud\nh0\n1 = a[\u27130\n10 + \u27130\n11y]\nh0\n2 = a[\u27130\n20 + \u27130\n21y]\nh0\n3 = a[\u27130\n30 + \u27130\n31y]\n<latexit sha1_base64=\"qvab5eVFtFKvws6yR\n9bTatgWmaY=\">AWuniclZhbU9w2FICdXtP0RtopL3xlMmk06Y7kKSXh7aTQMgNUpbAglLGNkre\nxVk2dgyLPHsv+mv6Wv70n/TI9u7is8RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nl",
  "+mv6Wv70n/TI9u7is8RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlWl+ELCS6H4QAst+UGWc5YEku8HJ2uG75/xvBCp2tUXGT9KWKxEJEKmIXS8\n8PvFTf83f5iNxc3jan6fVtamY7r/7Pft+vft+e/79S/70yPF5aWe8v1x6eFlbaw5LWf/vH1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzm",
  "8v1x6eFlbaw5LWf/vH1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgMjIj9Ic/pT26+jbR1QsKY\nqLJAzYXpcYGaCLnZY6uiXo0qorNRchU1DUSl9nfoma/5I5DzU8gIKLMwF9NUPxyxnoYbcXhsqfh6m\nScLUqBqurm9Pq2HAY6EqflrWeZ5Ou8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EB",
  "Pq2HAY6EqflrWeZ5Ou8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX0EU1pUEDxIJ9C5yH8+JVUrzWPISUd7STQoZJPOtYasWAqk46yA4rv3/AN4Dq\nHWYCuwhdHc7CTMTWdHaf5ROdJVZgYbiFnKuZ1EzDkEkzoq6hSinh0LBj/YGt50ydtIlLs7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5Yw",
  "inh0LBj/YGt50ydtIlLs7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsCrIwuznadBtOzMRvDYnGeyXr\ndekfSfMZQRE4DdZ74FUyHv6mvp3PZnyTmrfVPgE38Mk9U9hOVxM6xZIzCqNjalZp0rZNJsQShPz7u\nm6Y1D5ZnoDtAE8KYrc6Git7RbdQmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9",
  "oDtAE8KYrc6Git7RbdQmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealEkweBevJSCed3NHUsxwvbROq5g4JQTAp9gba/iFX3mDqCO5smqK8QMPXCNxMKTXIUdW\nUTMDJ8wxXUsYBCNMiwGWMo06LMOTn5ofUMkVo3p8VcmItV94QqjdA9b3A5PwrKcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4p",
  "ItV94QqjdA9b3A5PwrKcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+enxBp6PmFjUkaguGVx1iWJ5WgP6\npov17d7Vm28+o4s7djhuk1J6m176bYd7iU94Kebjt5uEo9Y1JGoraH1COWoz2oy53HTdcoHK7blK\nTeWR6dtsOdm2j5R7tjrpm5TUrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulY",
  "dtsOdm2j5R7tjrpm5TUrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIPoQlhsdnCXbONYXToW6VSazMTKbEBYfsQSPuglhMaZi7BRPWJYhsQmRPI5xHsc0jxmWM\npeEZyRzAhZUq4FlY/TrmQCWJqg1iaOxqAHMlWowTaI5YKuvMK58hRaxYqu4oGr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCP",
  "5YKuvMK58hRaxYqu4oGr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgurD0gtJzS8p3bd0n9LcUvJETPLSVP\nJ0F0ZukZpXuW7lFaWlpSOrB0QGlkaUTpQ0sfUhpaGlK6ZukapdpSckcKVwRLdykdWzqm9MDSA0pfWP\nqC0seWPqb0paUvKX1j6RtK71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0sz",
  "seWPqb0paUvKX1j6RtK71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG65ml5PYGLoyWSkqfWPqEUmEpeX4LomeWPqM0sTSh9KmlTyl9belrSh9Z+ojS2FLybgDu\nTizdodS+BaoKSrct3ab01NJT93sBPp/GwLUwt2wFW5SmlqaUblhKnhTgVsLSE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0",
  "5SmlqaUblhKnhTgVsLSE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZBSONMfLyt4LewtLB3u7fyU+/u9t2le6v\nexit>tG9qr3tfeN963or3s3fPe+z1vYEXen96f3l/e/8s/roYLIrFk0Z950p7zJde57Oo/wPjzuXS</lat\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\nNetwork 1:\nNetwork 2:",
  "Composing two networks: Example\nAssume:\n\u2022 ReLU Activation\n\u2022 Slopes and Intercepts \nas shown\n\u2022 3 hidden units in each\nExample: Pick parameters so that \n\ud835\udc65 \u2208 [\u22121,1] maps to\n\ud835\udc66 \u2208 [\u22121,1] with alternating slope",
  "Composing two networks: Example\nAssume:\n\u2022 ReLU Activation\n\u2022 Slopes and Intercepts \nas shown\n\u2022 3 hidden units in each\nExample: Pick parameters so that \n\ud835\udc65 \u2208 [\u22121,1] maps to\n\ud835\udc66 \u2208 [\u22121,1] with alternating slope\nLet\u2019s see what happens \nwhen we map\n \ud835\udc65 \u2192 \ud835\udc66 \u2192 \ud835\udc66\u2032",
  "",
  "",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0\n0.0\n-1.0\n0.0\n-1.0\n1.0\n1.0",
  "\u201cFolding analogy\u201d",
  "Comparing to shallow with six hidden units\n\u2022 20 parameters\n\u2022 (at least) 9 regions\n\u2022 19 parameters\n\u2022 Max 7 regions",
  "Composing networks in 2D",
  "Deep neural networks\n\u2022 Composing two networks\n\u2022 Combining the two networks into one\n\u2022 Hyperparameters\n\u2022 Notation change and general case\n\u2022 Shallow vs. deep networks",
  "Combine two networks into one\n<latexit sha1_base64=\"C\nOhQq39LxN7pgvr5YZY9diqSzqQ=\">AYzniclZhbT9xG\nFIA39JbSW9KqElJfrK0VZOivaSXl0oJhNwgZQkskGCyG\nnvHXofx2PgCSyr/1J/St96Wv7M3rG692Jz5ltFCTY4X\nyfZ+bMxZ61E4sgzdrtv64svfPue+9/cPXD5Y8+/uTz65\nd/wgjfLE5QM3ElFy5LCUi0DyQRZkgh/FCWehI/ihc7qh\n+OE5T9Ig",
  "5Y8+/uTz65\nd/wgjfLE5QM3ElFy5LCUi0DyQRZkgh/FCWehI/ihc7qh\n+OE5T9IgkvZcxPQubLwAtclkFoeH1pd/ztsOiU1je/2\nmc5G1l26ESTgpXHdjbmGVOwXVo3Lf1vp7w8eSvdjsfBsG\niXN2mwU47VHwPpKtI1kZ4ivfLEsmUk89DhiWXbyqNrkr\nDqjpG+9Vt9qv7hjSoTtPoLkyjuzCN7pvT6P3fbPSa/eq9\nIQ2q0zR6C9PoLUyjh9P",
  "qjpG+9Vt9qv7hjSoTtPoLkyjuzCN7pvT6P3fbPSa/eq9\nIQ2q0zR6C9PoLUyjh9PQWQyvrbX2tWPRQudurDaqn/6w\n+tfjuxR5OYhl5krWJoed9pxdlKwJAtcwctlO095zNxT5v\nNjKEoW8vSkqBZ/ad2AyMjyogR+ZWZV0devKFiYpehA2b\nIsnGKmQqa2HGeb+cFIGM84xLd9qQlwsriy1k6xRkHA3\nE5dQYG4SQF8td8wS5maw35ZtyS/cKAy",
  "Geb+cFIGM84xLd9qQlwsriy1k6xRkHA3\nE5dQYG4SQF8td8wS5maw35ZtyS/cKAyZHBX2+uZuWdgO9\nwNZcJgptfKsulsVg6H4iJj/dH+vJYg42HwipNKkVsk\nDgflkUfM1fwyDgAI1TkAkeQp1qvFxPKuDKNxrBOBiuvJs\nMJ6WpGqZcR/GpKE9JxoUYsEnDWuDWDCVYUPZA8WyblgK8\nCyBWYCuwgdHc7AXM1nOrsv4JEvCIlUx3ELCpM+rJiB",
  "DWuDWDCVYUPZA8WyblgK8\nCyBWYCuwgdHc7AXM1nOrsv4JEvCIlUx3ELCpM+rJiBlw\nmVUdOQuRBwqduwfsPWUyZP64GL4qriYogaz9pOlCx0W\nOmk4VQRYsQr9pVRFkCXgyjFjIYJTr8hASDi0VMauBxGpA\nFmY/iZxm27GK4LU5iWG/NL3Ngz/OUMjogKw+9RnwKTLm\n/pGNLet2eCcV74q8Ik1hslqXsISf5rWrBHIqo6V1KzGCp\nl0tCURBd",
  "+9RnwKTLm\n/pGNLet2eCcV74q8Ik1hslqXsISf5rWrBHIqo6V1KzGCp\nl0tCURBdNU/XGoPI4aCaoAnjT5Ukgvde0W1UJlqwK27c\ng1SQX/PiHtR/5KRoq2j/pDRhIrSPDZVpMJvUdEIziJ4\nfUET14k0ORBoJq8SMD9HU0dS/DCVpFq7qAQSCaC7BJt/\n8CXzWuqCO5sFK+QkDVC58skGiSPa8pq4CS4RNOVYF5K\nIk3WmOrojSPOHk5ofWM0Qq",
  "XzWuqCO5sFK+QkDVC58skGiSPa8pq4CS4RNOVYF5K\nIk3WmOrojSPOHk5ofWM0QqXd0Wk0A9rJo3VKGE5n2Di/l\nVUIaHwzlfcLmDRtSZjqcT5XLEjSYEzWlkxd2msEWM+3+\nasqnRaPl87Otuj3oF8xO7r8bLiF58MnFnUEqguOsca6B\nLEM7UFd8+X6es+KrRfk6XtG1yzKUi9dS/NtsFd0AN+tm\n3o7TbxiEUdgeqe0g9Yhnag7rM47htysLg",
  "KrRfk6XtG1yzKUi9dS/NtsFd0AN+tm\n3o7TbxiEUdgeqe0g9Yhnag7rM47htysLgmk1B6p2No9E\n2uHMTLX9vXx1Q1TEpEiN17IuEPQ1hMaNiZhSjkPtInIaw\nGOZNC/7Hyl4AD4+mNQ1hsZ8GTU0FsDTiAqcwDWFxuoWbZ\nh3D6rZB3TarTMRjZE5DWHzAQpz1NIRFn4q+UTxlcYzEaY\niM4xiP45iOY4yl2CThGYkNM0KWlGlBJeOoKakAliao",
  "pz1NIRFn4q+UTxlcYzEaY\niM4xiP45iOY4yl2CThGYkNM0KWlGlBJeOoKakAliaotYm\nhMeiBiCRqsA5iOaUrLzWuPIlWsaSreGBqeLCg4YyhClUA\nSztkj1n2jnGTOXiI4ZhlGuQ4QFZMB7CPnT51Zqc/xyvIS\nc7xLjW9pPRC0wtKDzU9pDTRlHwjcLynmpJvJ453ruk5pQ\neaHlCa5pTOtB0QKmnqUfpfU3vU+pq6lK6oekGpZm5EQ\nKTwRN9",
  "pJvJ453ruk5pQ\neaHlCa5pTOtB0QKmnqUfpfU3vU+pq6lK6oekGpZm5EQ\nKTwRN9ykdazqm9EjTI0qfafqM0oeaPqT0uabPKX2l6StK\n72p6l1KmKaN0U9NSrm5NWB461ruk6poyn57gd7TdM+p\nbGmMaX3NL1H6UhT8q0YnmeakuMNPBg1FZQ+0vQRpYGm5P\nub4z3R9AmloaYhpY81fUzpS01fUvpA0weU+pqSdwNwOtF\n0j1L9FqhIKd3VdJf",
  "P\nub4z3R9AmloaYhpY81fUzpS01fUvpA0weU+pqSdwNwOtF\n0j1L9FqhIKd3VdJfSM03PzO8F+HwaHdPC3NEV7FAaRpR\nuqUp+aYARwlNT8l50pP1XW32tonc1zw5wZWj/jsajLmn\npxzA6vTrOryf3Jk3M+Jl3fPJi/SIEhTv98NpqB7+FpY\nWD7lrnp7Xbu7dX76zXb2ivtr5qfd36rtVp/dy603rY6rc\nV6WZ</latexit>GLXfpz6W/l/5Z+",
  "u7dX76zXb2ivtr5qfd36rtVp/dy603rY6rc\nV6WZ</latexit>GLXfpz6W/l/5Z+nelv3K+Uq78PlWXrtTXfNFq/Kz8R/W\nh0\n1 =\na[\u27130\n10 + \u27130\n11y]\n=\na[\u27130\n10 + \u27130\n11\u03c60 + \u27130\n11\u03c61h1 + \u27130\n11\u03c62h2 + \u27130\n11\u03c63h3]\nh0\n2 =\na[\u27130\n20 + \u27130\n21y]\n=\na[\u27130\n20 + \u27130\n21\u03c60 + \u27130\n21\u03c61h1 + \u27130\n21\u03c62h2 + \u27130\n21\u03c63h3]\nh0\n3 =\na[\u27130\n30 + \u27130\n31y]\n=\na[\u27130\n30 + \u27130\n31\u03c60 +",
  "+ \u27130\n21\u03c61h1 + \u27130\n21\u03c62h2 + \u27130\n21\u03c63h3]\nh0\n3 =\na[\u27130\n30 + \u27130\n31y]\n=\na[\u27130\n30 + \u27130\n31\u03c60 + \u27130\n31\u03c61h1 + \u27130\n31\u03c62h2 + \u27130\n31\u03c63h3]\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n",
  "skLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS",
  "x82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWt",
  "otNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2ob",
  "xSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqD",
  "TA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9ti",
  "wYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K",
  "RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6A",
  "FiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5Y",
  "PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp",
  "+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tW",
  "67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"4mwMW+AxXHfhTyI/\nkaVAw9s5G24=\">AWsniclZhb9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3pIvTxEmaOD\nUomZLZUJSiS2J",
  "b9s2FIDV7tZ1t3TD8rIXYUGBYesMu+26vQxok6a3pIvTxEmaOD\nUomZLZUJSiS2JX8D/Zr9nr9gf2b3YoyWZ1DvMwA4np83i5ZDUzUukyPJu9r1z/48KOP7nx6c\n3Pv/iy69Wbn19kMVF6vOBH8s4PfJYxqVQfJCLXPKjJOUs8iQ/9M42ND+84GkmYrWfzxJ+GrFQiU\nD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/f0r3vz0cpat9",
  "+GrFQiU\nD4LIfQaOXBzP3dHSYTMSq785/qQm8+0f+aX3f1r7uLX/f0r3vz0cpat9OtPi4t9JrCmtN8+qNb34\n6H49gvIq5yX7IsO+l1k/y0ZGkufMnN4dFxhPmn7GQn0BRsYhnp2U1wLl7GyJjN4hT+FO5W0XfP\n6JkUZbNIg/MiOWTDMdtLGTIg9+Oy2FSoqcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DT\nm8OFb/04yhialwO1zd35+",
  "qcK79uKCikm8euzpY7Fin3czmDAvNTAX1/QlLmZ9DT\nm8OFb/04yhialwO1zd35+XQ46FQJT8vqvzO521ns3I4FK8y1p/vL2sROY/EO04qRdyRUCD+dly\nTthBwPBAYgOJyBWPIM6dX68wO0hCutJAgbuxVPoXOC+mpOqVc5DyElLOyYaFBLJpy1rg1gwlVFL2\nQPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9Yjahi",
  "L2\nQPFdW+7GvA8hVmArsIXR3OwlzA1XxyX82meRmWmY7iFlKmQV03AkH0m9YjahiqkhEP9lvUHtl4x\ndYkLk6qrqY6gqz9tO3kKc2LGredKoIsWIRh26oiyJKw+8csYpDlpjyCAUeujthVobAqyMLsp7HX\nbjvREbw2pwnsl7a3WZL0XzCUER2A3ae/BVM+b+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2\np2aVK2TSbEojS/bpu6NReWJa",
  "+sb8dJ2F8m5qHxd4FN3ApPVPoSlYT2sRSMwqiY2\np2aVK2TSbEojS/bpu6NReWJaA9QB/CmK1Khgve0O1UJlqwOD+/AUNC8pOfO7/w6WnZ1dtG/yPZ\nhIqyIrFVpMP/o6IxXG/w+oInrxYosmDQDV5sYTzO5o6luKFrSPV3EFBKCZFPkPbX4SqfUwVwZ2N\nI9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFq",
  "9RXCOh64ZsJhSY5CNqyDmgZvuHKaVlAPhqkX4/Rl3FWpJyc/NB6hkil69NiKvTFqn1ClVponze\n4XB4FZbg4XPArDvdQRr06n15cqDFLUTKnekqnb4ZDlvMtvurKa+LVivk51tNe9AvmJ3C9/n5aAv\nPR0gs6khUF9yqWOuSxLK0B3Utl+v7PSu3vxIlnZoce2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFf\nTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKP",
  "e2mJPU2vbTbFveKHvDzbUtvt4lHLOpIVFf\nTQ+oRy9Ie1GXP47ZtFBbXbkpS7yKPVtviLk20/IP9Cc+Zvk2K5Vjf9sVyWIewmFMxt4pxEMk1iE\nsRkXbgt9Y2RNw8WhbdQiL/Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6bVeZTCbIrENYfMoiPOo6hM\nWQiqFVPGNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQa",
  "VPGNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBpilqbWhqDHshYoQabIJYzuv\nIy68pTaBUruoHtoYHVzScM1ShDmBph+wxd7hj3WQeTjHcZtmSnAhkJTSBfez0qbO4+/OCktzJec\nHM0Bml4ZeUnpo6CGlqaHkicALXhlKnk684MLQC0oPD2gtDC0oHRg6IDSwNCA0ieGPqHUN9SndM\nPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3",
  "tDC0oHRg6IDSwNCA0ieGPqHUN9SndM\nPQDUpzQ8kdKVwRDN2ndGLohNIjQ48ofW3oa0qfGfqM0mNDjyl9Z+g7Sh8Z+ohSZijdNPQTUq5oe\nTVgResG7pOqWcoefaDvWZon9LE0ITSx4Y+pnRsKHkqhuZoeT2Bi6MhkpKnxv6nFJhKHl+84KXh\nr6kNDI0ovSFoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHd",
  "FoS8ofWvoW0qfGvqU0tBQ8m4A7k4M3aPUvAUqM0p3Dd2l9NzQc/t7Ab6cRs+2MHdMB\nTuUxobGlG4ZSp4U4FbC0DNyPxmo5qy2eNtEzmuBWnILazK+OJrkPFBLbmHN2WlxNDk/BWrJ6Trm\nwfLFymQUjTj1bWevgtLC0c3O30HnTu795fe7jevKG94XznfO/84PScX52HzjOn7wc3/nT+cv52\n/ln9f7q8Spb9Wv1+rXmG+c1mdV/ge0FuRK</la",
  "84PScX52HzjOn7wc3/nT+cv52\n/ln9f7q8Spb9Wv1+rXmG+c1mdV/ge0FuRK</latexit>y = \u03c60 + \u03c61h1 + \u03c62h2 + \u03c63h3\n<latexit sha1_base64\n=\"/+I4vBnqKIqWuIs3M1Saj+zUNbk=\">AXKHic\nlZhbU9w2FICXlN6I+mUl754yqTtCnDkrTNS2c\nSCLlBCoRrgkje2WvgiwbW4Ylnv1Dnf6YvnXy2l\n/SI9u7is8RD92ZMX5Pt2OZFvrIJOi0EtLb",
  "gkje2WvgiwbW4Ylnv1Dnf6YvnXy2l\n/SI9u7is8RD92ZMX5Pt2OZFvrIJOi0EtLb2fe/\n+Dz/6+Nons59+9vkX85dv7FfpGUe8r0wlWl+G\nLCS6H4nhZa8sMs5ywJD8ITlcNPzjneSFStasvM\n36csFiJSIRMQ+hk7q/h9ydVf+x97vnJ0E6qtj4\nyNdDrpmJL429nz7Z398ez5KlVlEvDc8/1ZU3v5\nitrL3drL7tq3r6h9u1v7tql9MrewtLhU",
  "JL429nz7Z398ez5KlVlEvDc8/1ZU3v5\nitrL3drL7tq3r6h9u1v7tql9MrewtLhUfzxa6Le\nFhV72Tq5/vXAH6RhmXClQ8mK4qi/lOnjiuVahJ\nKPZ/2y4BkLT1nMj6CoWMKL46pO69i7CZGBF6U5/F\nPaq6Pv1qhYUhSXSQBmwvSwMwEXeyo1NHd40qor\nNRchU1HUSk9nXpmjbyByHmo5SUWJgLGKsXDlnOQ\ng0rOesrfhGmScLUoPJX1rbHlR",
  "RchU1HUSk9nXpmjbyByHmo5SUWJgLGKsXDlnOQ\ng0rOesrfhGmScLUoPJX1rbHlR/wWKiKn5X1qo7H\nXWetdjgUrzJWnuxOWxGaJ+INJ43UimnkCoH46r\ni/EiBoIDEIucgFTxAto0+Qkir48o7GIJuGq2hg/\nG8zFpWmkeQ0462kuiQSGTfNSxVokFS5l0lB1QPO\n+mZwDXOawCDBW+OFqDnYyp8aSe5iOdJ1VhYriHnK\nmY13AlEMmzYy6hiql",
  "B1QPO\n+mZwDXOawCDBW+OFqDnYyp8aSe5iOdJ1VhYriHnK\nmY13AlEMmzYy6hiqlhKphx/oDW8+ZOm0Tl2b1U\nHMTQdZu3nV0TvOiBl2njiALNmHcteoIsiTcwYsY\nZDltnwCE048E3GrQmFVkI25ladBt+/MRPDeHGVw\nvXS9tYqk/5yhjJgAXH3mWzAV8q6+mk5tb5Kc89o\n3BT7yhrBY3Sosj5tpTqBWbWxMTXrXCGTZgtCeXr\nRNc1oHCrPR",
  "6+mk5tb5Kc89o\n3BT7yhrBY3Sosj5tpTqBWbWxMTXrXCGTZgtCeXr\nRNc1oHCrPRHeCJoAvujIXKnpHu1WXYMuasH8Lp\nqXkh/9vPgLHx1XS+ayMf+RbEJDRZm5GjLh/9HQAJ\n5yeH9BC9eKtHiQaBevFTC/R0tHcvxjaReu2gI\nBSTQl+iy1/EqlunjuDBpgkaKwRMu/DNhEKLHEVd2\nQSMDN/wvHZsoBNMmzmGMq0KHNObn5oP0Ok1s1t\nMRfm",
  "gkaKwRMu/DNhEKLHEVd2\nQSMDN/wvHZsoBNMmzmGMq0KHNObn5oP0Ok1s1t\nMRfmYdW9oUojdO8bXE5rQRkeDuf8iuoBymjQ5DN\nISzVgOUrmyCzp6JVfaLjEXFd/veRN0WnF/Gy97Q/\nGBatThiE/O1nH6xETizoStQUHJGdbkliO/qCt6X\nZ9d2TV+qsfydaOHa7blKTdpRu2+FeMQJ+tuEY7Q\nbxiEUdidpqR0g9Yjn6g7bcedxwzcLhuk1J2",
  "daOHa7blKTdpRu2+FeMQJ+tuEY7Q\nbxiEUdidpqR0g9Yjn6g7bcedxwzcLhuk1J2p3k0\nWk73KmJtn+0a46i5piUyoE59qXSb0JY1FTUTjFN\neIzEJoTFpOxa8DdWdgQ8PLpWE8LiViG6mglgacAl\nnkITwmJzCXfNobVDYe64VaZzIbIbEJYfMQSPOs\nmhMWYirFTPGVZhsQmRPI4xHkc0jxmWMpcEl6RzLE\niZEu5NlQ+TLuSCWBphHobOTqDE",
  "WYirFTPGVZhsQmRPI4xHkc0jxmWMpcEl6RzLE\niZEu5NlQ+TLuSCWBphHobOTqDEchUoQ7bIJYLuv\nMK585TaBcruov3XB3vXdGxZqhBE8DSJrnGPH/TeZ\nEFOMVwzHIlORPIymgCt7CzRZ3J6S+IKnKSC6JLS\ny8pvbD0gtIDSw8ozS0lvwiC6Lml5NdJEJ1bek7p\nvqX7lJaWlpTuWbpHaWRpROlDSx9SGloaUrpq6Sql\n2lJyIoUngqW7lA4t",
  "1bek7p\nvqX7lJaWlpTuWbpHaWRpROlDSx9SGloaUrpq6Sql\n2lJyIoUngqW7lA4tHVJ6aOkhpS8sfUHpY0sfU/r\nS0peUvrH0DaX3Lb1PKbOUbpm6Rql3FLy6iCIVix\ndoTSwlPz2g2vN0i1KM0szSh9Y+oDSgaXkVzE8zy\nwlxt4MFoqKX1i6RNKhaXk91sQPbP0GaWJpQmlT\ny19SulrS19T+sjSR5TGlpJ3A3A6sXSHUvsWqCo3\nbZ0m9IzS8",
  "QPbP0GaWJpQmlT\ny19SulrS19T+sjSR5TGlpJ3A3A6sXSHUvsWqCo3\nbZ0m9IzS8/c7wX4dBkD18bctA1sUpamlK6bin5\npQBHCUtPyXkyUu1dbfK2idzXIjXlDtZmfFKb5DxS\nU+5g7d1pUpvcnyI15UMy9LX96YsUSCnc6U/mFvr\n4LSwt7C8v9n9dvLN9Z+HeSvuG9lrvm963vR96/d5\n+baet81et85v/9Dzr3C9Y=</latexit>vXu9x72t",
  "Z+HeSvuG9lrvm963vR96/d5\n+baet81et85v/9Dzr3C9Y=</latexit>vXu9x72t3l4vnLkxc3fm/szK/J/zf8/M/+2Ud\nh0\n1 = a[\u27130\n10 + \u27130\n11y]\nh0\n2 = a[\u27130\n20 + \u27130\n21y]\nh0\n3 = a[\u27130\n30 + \u27130\n31y]\n<latexit sha1_base64=\"qvab5eVFtFKvws6yR\n9bTatgWmaY=\">AWuniclZhbU9w2FICdXtP0RtopL3xlMmk06Y7kKSXh7aTQMgNUpbAglLG",
  "WmaY=\">AWuniclZhbU9w2FICdXtP0RtopL3xlMmk06Y7kKSXh7aTQMgNUpbAglLGNkre\nxVk2dgyLPHsv+mv6Wv70n/TI9u7is8RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlWl+ELCS6H4QAst+UGWc5YEku8HJ2uG75/xvBCp2tUXGT9KWKxEJEKmIXS8\n8PvFTf83f5iNxc3jan6fVtamY7r/7Pft+vf",
  "75/xvBCp2tUXGT9KWKxEJEKmIXS8\n8PvFTf83f5iNxc3jan6fVtamY7r/7Pft+vft+e/79S/70yPF5aWe8v1x6eFlbaw5LWf/vH1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgMjIj9Ic/pT26+jbR1QsKY\nqLJAzYXpcYGaCLnZY6uiXo0qorNRchU1DUSl9nfoma/5I5DzU8gIKLMwF9NUPxyxnoYbcXhs",
  "cYGaCLnZY6uiXo0qorNRchU1DUSl9nfoma/5I5DzU8gIKLMwF9NUPxyxnoYbcXhsqfh6m\nScLUqBqurm9Pq2HAY6EqflrWeZ5Ou8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX0EU1pUEDxIJ9C5yH8+JVUrzWPISUd7STQoZJPOtYasWAqk46yA4rv3/AN4Dq\nHWYCuwhdHc7CTMTWdHaf5ROdJVZgYbiF",
  "7STQoZJPOtYasWAqk46yA4rv3/AN4Dq\nHWYCuwhdHc7CTMTWdHaf5ROdJVZgYbiFnKuZ1EzDkEkzoq6hSinh0LBj/YGt50ydtIlLs7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsCrIwuznadBtOzMRvDYnGeyXr\ndekfSfMZQRE4DdZ74FUyHv6mvp3PZnyTmrfVPgE38Mk9U9hOVxM6xZIzCqNjalZp0rZNJ",
  "fMZQRE4DdZ74FUyHv6mvp3PZnyTmrfVPgE38Mk9U9hOVxM6xZIzCqNjalZp0rZNJsQShPz7u\nm6Y1D5ZnoDtAE8KYrc6Git7RbdQmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealEkweBevJSCed3NHUsxwvbROq5g4JQTAp9gba/iFX3mDqCO5smqK8QMPXCNxMKTXIUdW\nUTMDJ8wxXUsYBCNMiwGWMo06LM",
  "Ap9gba/iFX3mDqCO5smqK8QMPXCNxMKTXIUdW\nUTMDJ8wxXUsYBCNMiwGWMo06LMOTn5ofUMkVo3p8VcmItV94QqjdA9b3A5PwrKcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+enxBp6PmFjUkaguGVx1iWJ5WgP6\npov17d7Vm28+o4s7djhuk1J6m176bYd7iU94Kebjt5uEo9Y1JGoraH1COWoz2oy53",
  "ov17d7Vm28+o4s7djhuk1J6m176bYd7iU94Kebjt5uEo9Y1JGoraH1COWoz2oy53HTdcoHK7blK\nTeWR6dtsOdm2j5R7tjrpm5TUrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIPoQlhsdnCXbONYXToW6VSazMTKbEBYfsQSPuglhMaZi7BRPWJYhsQmRPI5xHsc0jxmWM\npeEZyRzAhZUq4FlY/TrmQCWJq",
  "fsQSPuglhMaZi7BRPWJYhsQmRPI5xHsc0jxmWM\npeEZyRzAhZUq4FlY/TrmQCWJqg1iaOxqAHMlWowTaI5YKuvMK58hRaxYqu4oGr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgurD0gtJzS8p3bd0n9LcUvJETPLSVP\nJ0F0ZukZpXuW7lFaWlpSOrB0QGlkaUTpQ0sfUhpaGlK6ZukapdpSckcKVwRLdykdW",
  "0F0ZukZpXuW7lFaWlpSOrB0QGlkaUTpQ0sfUhpaGlK6ZukapdpSckcKVwRLdykdWzqm9MDSA0pfWP\nqC0seWPqb0paUvKX1j6RtK71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG65ml5PYGLoyWSkqfWPqEUmEpeX4LomeWPqM0sTSh9KmlTyl9belrSh9Z+ojS2FLybgDu\nTizdodS+BaoKSrct3ab01NJT",
  "meWPqM0sTSh9KmlTyl9belrSh9Z+ojS2FLybgDu\nTizdodS+BaoKSrct3ab01NJT93sBPp/GwLUwt2wFW5SmlqaUblhKnhTgVsLSE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZBSONMfLyt4LewtLB3u7fyU+/u9t2le6v\nexit>tG9qr3tfeN963or3s3fPe+z1vYEXen96f3l/e/8s/roYLIrFk0Z950p",
  "e6v\nexit>tG9qr3tfeN963or3s3fPe+z1vYEXen96f3l/e/8s/roYLIrFk0Z950p7zJde57Oo/wPjzuXS</lat\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\nNetwork 1:\nNetwork 2:\nHidden units of second network in terms of first:\n\ud835\udf03 : theta\n\ud835\udf19 : phi",
  "Create new variables: \ud835\udf13 (psi)\n<latexit sha1_base64=\"C\nOhQq39LxN7pgvr5YZY9diqSzqQ=\">AYzniclZhbT9xG\nFIA39JbSW9KqElJfrK0VZOivaSXl0oJhNwgZQkskGCyG\nnvHXofx2PgCSyr/1J/St96Wv7M3rG692Jz5ltFCTY4X\nyfZ+bMxZ61E4sgzdrtv64svfPue+9/cPXD5Y8+/uTz65\nd/wgjfLE5QM3ElFy5LCUi0DyQRZkgh/FCWehI/ihc7qh\n+OE5",
  "cPXD5Y8+/uTz65\nd/wgjfLE5QM3ElFy5LCUi0DyQRZkgh/FCWehI/ihc7qh\n+OE5T9IgkvZcxPQubLwAtclkFoeH1pd/ztsOiU1je/2\nmc5G1l26ESTgpXHdjbmGVOwXVo3Lf1vp7w8eSvdjsfBsG\niXN2mwU47VHwPpKtI1kZ4ivfLEsmUk89DhiWXbyqNrkr\nDqjpG+9Vt9qv7hjSoTtPoLkyjuzCN7pvT6P3fbPSa/eq9\nIQ2q0zR6C9PoLUy",
  "kr\nDqjpG+9Vt9qv7hjSoTtPoLkyjuzCN7pvT6P3fbPSa/eq9\nIQ2q0zR6C9PoLUyjh9PQWQyvrbX2tWPRQudurDaqn/6w\n+tfjuxR5OYhl5krWJoed9pxdlKwJAtcwctlO095zNxT5v\nNjKEoW8vSkqBZ/ad2AyMjyogR+ZWZV0devKFiYpehA2b\nIsnGKmQqa2HGeb+cFIGM84xLd9qQlwsriy1k6xRkHA3\nE5dQYG4SQF8td8wS5maw35ZtyS/",
  "qa2HGeb+cFIGM84xLd9qQlwsriy1k6xRkHA3\nE5dQYG4SQF8td8wS5maw35ZtyS/cKAyZHBX2+uZuWdgO9\nwNZcJgptfKsulsVg6H4iJj/dH+vJYg42HwipNKkVsk\nDgflkUfM1fwyDgAI1TkAkeQp1qvFxPKuDKNxrBOBiuvJs\nMJ6WpGqZcR/GpKE9JxoUYsEnDWuDWDCVYUPZA8WyblgK8\nCyBWYCuwgdHc7AXM1nOrsv4JEvCIlUx3ELCpM+",
  "YsEnDWuDWDCVYUPZA8WyblgK8\nCyBWYCuwgdHc7AXM1nOrsv4JEvCIlUx3ELCpM+rJiBlw\nmVUdOQuRBwqduwfsPWUyZP64GL4qriYogaz9pOlCx0W\nOmk4VQRYsQr9pVRFkCXgyjFjIYJTr8hASDi0VMauBxGpA\nFmY/iZxm27GK4LU5iWG/NL3Ngz/OUMjogKw+9RnwKTLm\n/pGNLet2eCcV74q8Ik1hslqXsISf5rWrBHIqo6V1KzGCp\nl0tC",
  "ogKw+9RnwKTLm\n/pGNLet2eCcV74q8Ik1hslqXsISf5rWrBHIqo6V1KzGCp\nl0tCURBdNU/XGoPI4aCaoAnjT5Ukgvde0W1UJlqwK27c\ng1SQX/PiHtR/5KRoq2j/pDRhIrSPDZVpMJvUdEIziJ4\nfUET14k0ORBoJq8SMD9HU0dS/DCVpFq7qAQSCaC7BJt/\n8CXzWuqCO5sFK+QkDVC58skGiSPa8pq4CS4RNOVYF5K\nIk3WmOrojSPOHk5ofW",
  "/\n8CXzWuqCO5sFK+QkDVC58skGiSPa8pq4CS4RNOVYF5K\nIk3WmOrojSPOHk5ofWM0QqXd0Wk0A9rJo3VKGE5n2Di/l\nVUIaHwzlfcLmDRtSZjqcT5XLEjSYEzWlkxd2msEWM+3+\nasqnRaPl87Otuj3oF8xO7r8bLiF58MnFnUEqguOsca6B\nLEM7UFd8+X6es+KrRfk6XtG1yzKUi9dS/NtsFd0AN+tm\n3o7TbxiEUdgeqe0g9Yhnag7rM47ht",
  "6es+KrRfk6XtG1yzKUi9dS/NtsFd0AN+tm\n3o7TbxiEUdgeqe0g9Yhnag7rM47htysLgmk1B6p2No9E\n2uHMTLX9vXx1Q1TEpEiN17IuEPQ1hMaNiZhSjkPtInIaw\nGOZNC/7Hyl4AD4+mNQ1hsZ8GTU0FsDTiAqcwDWFxuoWbZ\nh3D6rZB3TarTMRjZE5DWHzAQpz1NIRFn4q+UTxlcYzEaY\niM4xiP45iOY4yl2CThGYkNM0KWlGlBJeOoKakA",
  "HzAQpz1NIRFn4q+UTxlcYzEaY\niM4xiP45iOY4yl2CThGYkNM0KWlGlBJeOoKakAliaotYm\nhMeiBiCRqsA5iOaUrLzWuPIlWsaSreGBqeLCg4YyhClUA\nSztkj1n2jnGTOXiI4ZhlGuQ4QFZMB7CPnT51Zqc/xyvIS\nc7xLjW9pPRC0wtKDzU9pDTRlHwjcLynmpJvJ453ruk5pQ\neaHlCa5pTOtB0QKmnqUfpfU3vU+pq6lK6oekGpZm5EQ\nKT",
  "LynmpJvJ453ruk5pQ\neaHlCa5pTOtB0QKmnqUfpfU3vU+pq6lK6oekGpZm5EQ\nKTwRN9ykdazqm9EjTI0qfafqM0oeaPqT0uabPKX2l6StK\n72p6l1KmKaN0U9NSrm5NWB461ruk6poyn57gd7TdM+p\nbGmMaX3NL1H6UhT8q0YnmeakuMNPBg1FZQ+0vQRpYGm5P\nub4z3R9AmloaYhpY81fUzpS01fUvpA0weU+pqSdwNwOtF\n0j1L9FqhIKd3",
  "YGm5P\nub4z3R9AmloaYhpY81fUzpS01fUvpA0weU+pqSdwNwOtF\n0j1L9FqhIKd3VdJfSM03PzO8F+HwaHdPC3NEV7FAaRpR\nuqUp+aYARwlNT8l50pP1XW32tonc1zw5wZWj/jsajLmn\npxzA6vTrOryf3Jk3M+Jl3fPJi/SIEhTv98NpqB7+FpY\nWD7lrnp7Xbu7dX76zXb2ivtr5qfd36rtVp/dy603rY6rc\nV6WZ</latexit>GLXfpz6W/l",
  "p7Xbu7dX76zXb2ivtr5qfd36rtVp/dy603rY6rc\nV6WZ</latexit>GLXfpz6W/l/5Z+nelv3K+Uq78PlWXrtTXfNFq/Kz8R/W\nh0\n1 =\na[\u27130\n10 + \u27130\n11y]\n=\na[\u27130\n10 + \u27130\n11\u03c60 + \u27130\n11\u03c61h1 + \u27130\n11\u03c62h2 + \u27130\n11\u03c63h3]\nh0\n2 =\na[\u27130\n20 + \u27130\n21y]\n=\na[\u27130\n20 + \u27130\n21\u03c60 + \u27130\n21\u03c61h1 + \u27130\n21\u03c62h2 + \u27130\n21\u03c63h3]\nh0\n3 =\na[\u27130\n30 + \u27130\n31y]\n=\na[\u27130\n30 +",
  "+ \u27130\n21\u03c60 + \u27130\n21\u03c61h1 + \u27130\n21\u03c62h2 + \u27130\n21\u03c63h3]\nh0\n3 =\na[\u27130\n30 + \u27130\n31y]\n=\na[\u27130\n30 + \u27130\n31\u03c60 + \u27130\n31\u03c61h1 + \u27130\n31\u03c62h2 + \u27130\n31\u03c63h3]\n<latexit sha1_base64=\"eo6xbZtUDEx9RMe\nyKb8msDGOEPA=\">AXg3iclZhZc9s2EICltElT90ra6fihL5x60na1GNJ6fGSmcSOc9mp7y\nMxHQ9IgRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSL",
  "SOc9mp7y\nMxHQ9IgRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65\n/encZ59/8eVXd+5+vZ/FRerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8\nOIhUoEwmc5hE7udm+NfjwpexPnh0eOG3nxuGSTIzfJBASXJs59p2n3JiOtzb739fe+T7Q3w\neTY8dVsSoij6eO687p3H1b7v6V3H2U",
  "IzfJBASXJs59p2n3JiOtzb739fe+T7Q3w\neTY8dVsSoij6eO687p3H1b7v6V3H2Uu49y96/PbDlHlzJPUC5Byj3YJr75M7C0uJS9efQRq9\npLHSav82Tu98O3WHsFxFXuS9Zlh31lpL8uGRpLnzJ3NukfGE+acs5EfQVCzi2XFZzdjEuQeRo\nRPEKfxTuVNFrx5RsijLiMPzIjlowzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEh\nrMT",
  "rx5RsijLiMPzIjlowzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEh\nrMTwWM1fFHLGV+DotkzlX8wo+jiKlh6S6vbk1K1+OhUCU/K6oFM5m0ndXK4dC8zlh+uTvLInI\neiQ+cJKkUneQagYeTsuSL4SIGgMQi5yAWPEMcur6eIHTQxQ2iARc1gvHBWN7QlKrnIdQk5b2l\nmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcFOwtRke",
  "WN7QlKrnIdQk5b2l\nmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcFOwtRkelzOx3kalZmO4R5SpkJedQGn7D\nOpz6htqEJKONRvWX9ia5up06ZwcVINdURZO2mbSdPaV3UsO1UEWTBIgzbVhVBloTL2ZBFDKr\nctE/ghCNHR+yqUFgVZGFuprHX7jvREbw2xwnsl7a3WpLynzNUER2A3ac/BVM+b+sr8cx2psU5\nr3zd4GNnBJPVPoSlYX1",
  "Ebw2xwnsl7a3WpLynzNUER2A3ac/BVM+b+sr8cx2psU5\nr3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwWhNL5om3o0FpUnon2COoA3XZEKFVzRHlQtWL\nI67D6AU0LyY9+WfyVj4/LJb1t9H+kmpAoKxJbIh3+H4mGcAPF6wsiePJiSYPAtXkxRKu72j\nqWIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC",
  "WIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC8tFJ+vU5\n+jLOipSTix9azxCpdH1ZTIW+WbUvqFIL7esGl7OjoA03h3N+zeEeqhX19OLCzVkKSrmWE/p+\nJ2b5bDFbLu/mvK6abVCfrbW9AfjgtkpfJ+fnazh+QiJR2JcsGzlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpI",
  "csGzlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG47C4trNyXJO62j1ba4\nMxMt/2B3xHOmH5NiOdSPfbF06xAWcyrmVjGOeIjEOoTFqGhb8B0rOwJuHm2rDmFxMxNtTQewN\nOQSn0IdwmK9hdtmE8PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWE\npsEp6Rx",
  "PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWE\npsEp6RxDIjZEnZFlQ6ituSDmBpjHobWzqDEchYoQ6bIJYzuvIy68pTaBUruor3bB3vXdNxzlBC\nHcDSBtljrth3WQeLjE8ZtmKnAhkJbSAm9jZpM706c8LSvIk5wWXhl5SemHoBaUHh5QmhpKf\nhF4wbah5NeJF5wbek7pvqH7lBaGFpTuGbpHaWBoQOkzQ59R6hvqU",
  "aUHh5QmhpKf\nhF4wbah5NeJF5wbek7pvqH7lBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9QelbQ9S+sHQD5Q+MfQJpcxQRumqoauUckPJqwMvWDZ0mVLPUPL\nbD/aoZuUJoYmlD419CmlQ0PJr2K4nxlKHm/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0F\naXvDX1P6XNDn1MaGkreDcDT",
  "m/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0F\naXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6ZahW5SeGXpmfy/AZ9Po2RbmhkmwQWlsaEzpmq\nHklwI8Sh6Sp4nA9Vc1aZvm8h1LVAzbmFNxadHk5oHasYtrLk6TY8m16dAzfiIDH1f/YiBUo\n5m/O35/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>KV/qTOws9/BaWNvb7i73fFh9u",
  "5/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>KV/qTOws9/BaWNvb7i73fFh9uPVx4vNy8ob3d+a7zfenTq/ze+dx50Vns7PX8btR96/u391/\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]\n\ud835\udf03 : theta\n\ud835\udf19 : phi\n\ud835\udf13 : psi",
  "Two-layer network\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuE",
  "D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFS",
  "qDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1H",
  "lcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBN",
  "ZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsW",
  "ljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RB",
  "jwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd",
  "MSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+",
  "XazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlD",
  "fWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5Po",
  "xNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64",
  "</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"qvab5eVFtFKvws6yR\n9bTatgWmaY=\">AWuniclZhbU9w2FICdXtP0RtopL3xlMmk06Y7kKSXh7aTQMgNUpbAglLGNkre\nxVk2dgyLPHsv+mv6Wv70n/TI9u7is8RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlWl+E",
  "0ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlWl+ELCS6H4QAst+UGWc5YEku8HJ2uG75/xvBCp2tUXGT9KWKxEJEKmIXS8\n8PvFTf83f5iNxc3jan6fVtamY7r/7Pft+vft+e/79S/70yPF5aWe8v1x6eFlbaw5LWf/vH1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgMjIj",
  "bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgMjIj9Ic/pT26+jbR1QsKY\nqLJAzYXpcYGaCLnZY6uiXo0qorNRchU1DUSl9nfoma/5I5DzU8gIKLMwF9NUPxyxnoYbcXhsqfh6m\nScLUqBqurm9Pq2HAY6EqflrWeZ5Ou8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX0EU",
  "7XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX0EU1pUEDxIJ9C5yH8+JVUrzWPISUd7STQoZJPOtYasWAqk46yA4rv3/AN4Dq\nHWYCuwhdHc7CTMTWdHaf5ROdJVZgYbiFnKuZ1EzDkEkzoq6hSinh0LBj/YGt50ydtIlLs7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsCrIw",
  "qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsCrIwuznadBtOzMRvDYnGeyXr\ndekfSfMZQRE4DdZ74FUyHv6mvp3PZnyTmrfVPgE38Mk9U9hOVxM6xZIzCqNjalZp0rZNJsQShPz7u\nm6Y1D5ZnoDtAE8KYrc6Git7RbdQmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealEkwe",
  "rAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealEkweBevJSCed3NHUsxwvbROq5g4JQTAp9gba/iFX3mDqCO5smqK8QMPXCNxMKTXIUdW\nUTMDJ8wxXUsYBCNMiwGWMo06LMOTn5ofUMkVo3p8VcmItV94QqjdA9b3A5PwrKcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+enx",
  "E45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+enxBp6PmFjUkaguGVx1iWJ5WgP6\npov17d7Vm28+o4s7djhuk1J6m176bYd7iU94Kebjt5uEo9Y1JGoraH1COWoz2oy53HTdcoHK7blK\nTeWR6dtsOdm2j5R7tjrpm5TUrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIPoQl",
  "yNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIPoQlhsdnCXbONYXToW6VSazMTKbEBYfsQSPuglhMaZi7BRPWJYhsQmRPI5xHsc0jxmWM\npeEZyRzAhZUq4FlY/TrmQCWJqg1iaOxqAHMlWowTaI5YKuvMK58hRaxYqu4oGr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgurD0g",
  "cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgurD0gtJzS8p3bd0n9LcUvJETPLSVP\nJ0F0ZukZpXuW7lFaWlpSOrB0QGlkaUTpQ0sfUhpaGlK6ZukapdpSckcKVwRLdykdWzqm9MDSA0pfWP\nqC0seWPqb0paUvKX1j6RtK71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG65m",
  "t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG65ml5PYGLoyWSkqfWPqEUmEpeX4LomeWPqM0sTSh9KmlTyl9belrSh9Z+ojS2FLybgDu\nTizdodS+BaoKSrct3ab01NJT93sBPp/GwLUwt2wFW5SmlqaUblhKnhTgVsLSE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZBSO",
  "I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZBSONMfLyt4LewtLB3u7fyU+/u9t2le6v\nexit>tG9qr3tfeN963or3s3fPe+z1vYEXen96f3l/e/8s/roYLIrFk0Z950p7zJde57Oo/wPjzuXS</lat\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\n<latexit sha1_base64=\"AYd3CQ6km9QG8r7RX\nMS1GBLgYc",
  "+ \u03c60\n2h0\n2 + \u03c60\n3h0\n3\n<latexit sha1_base64=\"AYd3CQ6km9QG8r7RX\nMS1GBLgYcI=\">AXgniclZhZc9s2EIClNGlT90ra6fihL5x60nbS1GNJ6fHQziR2nMtOfR+J6XhAC\nqQgyDNw5bD0Z/oa/vH+m+6ICnB3IUf6plE0H4fF+ACICl6iRZvrT0b/fGBzdvfjR7Y/nPvn0s8+\n/uHP3y/0sLlKf7/mxjNDj2VcCsX3cpFLfpiknEWe5Afe6YrmB",
  "fjR7Y/nPvn0s8+\n/uHP3y/0sLlKf7/mxjNDj2VcCsX3cpFLfpiknEWe5Afe6YrmB+c8zUSsdvPLhB9HLFQiED7LIXRyt\n3tr9P1J2Zs43/3hRl48LtnkyE0yAbGlifOj07R7k5G2Zt/7+nvfB/o74PJseOqWBWRx1PHded06r5\nO7aDc/Su5+712rj7K3b8+98CWe3Al9wDlHqDcg2nukzsLS4tL1Z9DG72msdBp/jZP7n49dIexX0Rc\n5b5kWXb",
  "98CWe3Al9wDlHqDcg2nukzsLS4tL1Z9DG72msdBp/jZP7n49dIexX0Rc\n5b5kWXbUW0ry45KlufAln8y5RcYT5p+ykB9BU7GIZ8dlNWET5x5Ehk4Qp/BP5U4VvXpEyaIsu4w8MC\nOWjzLMdNDGjo8+O24FCopcq78uqOgkE4eO3r2naFIuZ/LS2gwPxUwVscfsZT5OayROVfxCz+OIqaG\npbu8ujUpXY+HQpX8rKjWy2TSdlYrh0PzOmP5xe4si8",
  "fsZT5OayROVfxCz+OIqaG\npbu8ujUpXY+HQpX8rKjWy2TSdlYrh0PzOmP5xe4si8h5JN5zkqRSdJrB5OypIvhosYCA5ALHICYs\nUzyKnr4wVOD1HYHxJwWS8cF4ztCUmtch5CTVraG6JBI5F83LJWiAVTGbWUHVAc56jAc9TmAUYKnx\nwNAc7CVOT6XE5H+dpVGY6hntImQp51QWcs+kPqO2oQop4VC/Zf2JrW2mTpvCxUk1FRHkLWbtp08p\nXVR",
  "pVGY6hntImQp51QWcs+kPqO2oQop4VC/Zf2JrW2mTpvCxUk1FRHkLWbtp08p\nXVRw7ZTRZAFizBsW1UEWRKuZkMWMahy0z6BE4cHbGrQmFVkIW5mcZeu+9ER/DaHCewX9reaknKf85\nQRXQAdp/+FEz5vK2vxDPbmRbnvPJ1g4+dEUxW+xCWhvVpTuBs2piE2pWtUImrRaE0viberRWFSei\nPYJ6gDedEUqVHBFe1C1YMnqsPsATjUtJD/6afFnP",
  "2pWtUImrRaE0viberRWFSei\nPYJ6gDedEUqVHBFe1C1YMnqsPsATjUtJD/6afFnPj4ul/S20f+RakKirEhsiXT4fyQawv0Try+I4Mm\nLJZo8CFSTF0u4vqOpYyle2DpSzR0hGJS5Jdo+4tQtY+pIniwcYTGCgGdFz6ZUGiSg6At64CW4ROe\nBCwLyEcn6dfn6Ms4K1JOLn5oPUOk0vVlMRX6ZtW+oEotK8bXM6OgjbcHM75NYd7qKJeXU8vLtSQp",
  "6Ms4K1JOLn5oPUOk0vVlMRX6ZtW+oEotK8bXM6OgjbcHM75NYd7qKJeXU8vLtSQpa\niYz2l47dulsMWs+3+asrptUK+dla0x+MC2an8H1+drKG5yMkFnUkygWPXtZckliW/iDXbLleHVm5\n9vY+WdqhxbWbkuRtRm3Le41I+Bn65bRrhOPWNSRKFczQuoRy9If5LXcd12FhbXbkqSd1pHq21xZy\nZa/sHuiOdMPybFcqgf+2Lp1iEs5lTMrWIc8R",
  "If5LXcd12FhbXbkqSd1pHq21xZy\nZa/sHuiOdMPybFcqgf+2Lp1iEs5lTMrWIc8RCJdQiLUdG24DtWdgTcPNpWHcLiZibamg5gacglPoU\n6hMV6C7fNJobVdYu6bleZTEbIrENYfMYifNZ1CIshFUOreMqSBIl1iNRxhOs4onVMsJTYJDwjiWVGy\nJKyLah0FLclHcDSGPU2tnQGI5CxQh02QSxndOVl1pWn0CpWdBXv2Treu6bjnKGEOoClDbLH",
  "FLclHcDSGPU2tnQGI5CxQh02QSxndOVl1pWn0CpWdBXv2Treu6bjnKGEOoClDbLHfDusk\n8XGJ4zLIVORHISmgBN7GzSZ3p058XlORJzgsuDb2k9MLQC0oPD2gNDWU/CLwgm1Dya8TLzg39JzSf\nUP3KS0MLSjdM3SP0sDQgNKnhj6l1DfUp3TF0BVKc0PJEyncEQzdpXRk6IjSQ0MPKX1t6GtKnxv6nNI\n3hr6h9L2h7yl9bOhjSpmhjNJVQ1cp",
  "yncEQzdpXRk6IjSQ0MPKX1t6GtKnxv6nNI\n3hr6h9L2h7yl9bOhjSpmhjNJVQ1cp5YaSVwdesGzoMqWeoeS3H+w1QzcpTQxNKH1i6BNKh4aSX8Vw\nPzOUPN7AjdFQSekLQ19QKgwlv9+84JWhryiNDI0ofWnoS0rfGfqO0meGPqM0NJS8G4CnE0N3KDVvgc\nqM0i1Dtyg9M/TM/l6Az6bRsy3MDZNg9LY0JjSNUPJLwV4lD0lDxPBq5qk3fNpHrW",
  "qM0i1Dtyg9M/TM/l6Az6bRsy3MDZNg9LY0JjSNUPJLwV4lD0lDxPBq5qk3fNpHrWqBm3MKaik+P\n4PnV7n186jzvPOZmev43dl96/u391/5m/O35/vzQ9q9Ua3OearTutv/vf/AH2JKcE=</latexit>JjUP1IxbWHN1mh5Nrk+BmvERGfrq/uxFCpQUrvQndxZ6+C0sbez3F3u/LD7cerjwaLl5Q3u7803n28\nh0\n1 = a[ 10 +  11h1 +  12h2 +",
  "QndxZ6+C0sbez3F3u/LD7cerjwaLl5Q3u7803n28\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h2 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h2 +  32h2 +  33h3]\n\ud835\udf03 : theta\n\ud835\udf19 : phi\n\ud835\udf13 : psi",
  "Two-layer network as one equation\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41",
  "8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZO",
  "fDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4de",
  "zlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0Kh",
  "cklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo",
  "yrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE",
  "aCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoCl",
  "T\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPT",
  "0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npS",
  "Y8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe",
  "pTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_bas",
  "Q=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"qvab5eVFtFKvws6yR\n9bTatgWmaY=\">AWuniclZhbU9w2FICdXtP0RtopL3xlMmk06Y7kKSXh7aTQMgNUpbAglLGNkre\nxVk2dgyLPHsv+mv6Wv70n/TI9u7is8RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlW",
  "RD90ZWO35PutyJPkWZFIUen53yvPve+x98ePWjax9/8u\nlny9c/2KvSMs85IMwlWl+ELCS6H4QAst+UGWc5YEku8HJ2uG75/xvBCp2tUXGT9KWKxEJEKmIXS8\n8PvFTf83f5iNxc3jan6fVtamY7r/7Pft+vft+e/79S/70yPF5aWe8v1x6eFlbaw5LWf/vH1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgM",
  "1r0bDU\nRqWCVc6lKwoDleWM31UsVyLUPLptWFZ8IyFJyzmh1BULOHFUVUPdOrfgMjIj9Ic/pT26+jbR1QsKY\nqLJAzYXpcYGaCLnZY6uiXo0qorNRchU1DUSl9nfoma/5I5DzU8gIKLMwF9NUPxyxnoYbcXhsqfh6m\nScLUqBqurm9Pq2HAY6EqflrWeZ5Ou8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX",
  "8567XAoXmasPtmd1yI0T8QbTiqpFVPJQKPp1XFe3EPA8EBi\nB4nIFW8gDpNfoLIX0EU1pUEDxIJ9C5yH8+JVUrzWPISUd7STQoZJPOtYasWAqk46yA4rv3/AN4Dq\nHWYCuwhdHc7CTMTWdHaf5ROdJVZgYbiFnKuZ1EzDkEkzoq6hSinh0LBj/YGt50ydtIlLs7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsC",
  "Ls7qruYkg\nazfvOjqneVGjrlNHkAWLMO5adQRZEs4CI5YwyHJbPoYBJ76JuFWhsCrIwuznadBtOzMRvDYnGeyXr\ndekfSfMZQRE4DdZ74FUyHv6mvp3PZnyTmrfVPgE38Mk9U9hOVxM6xZIzCqNjalZp0rZNJsQShPz7u\nm6Y1D5ZnoDtAE8KYrc6Git7RbdQmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealE",
  "QmWrAkPb8FQ81Lywx96P/LJUbVsto35R7IJFRVl5qrIhP9HRSO47\nuD1BRE8ealEkweBevJSCed3NHUsxwvbROq5g4JQTAp9gba/iFX3mDqCO5smqK8QMPXCNxMKTXIUdW\nUTMDJ8wxXUsYBCNMiwGWMo06LMOTn5ofUMkVo3p8VcmItV94QqjdA9b3A5PwrKcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+",
  "KcHE45cHqCMBk0\n+g7RUI5ajZE7MlE5eDQsNW8y1+spb4pOK+anG2170C+YnTIM+enxBp6PmFjUkaguGVx1iWJ5WgP6\npov17d7Vm28+o4s7djhuk1J6m176bYd7iU94Kebjt5uEo9Y1JGoraH1COWoz2oy53HTdcoHK7blK\nTeWR6dtsOdm2j5R7tjrpm5TUrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIP",
  "UrlyNz2pXLYhLCoqaidYprwGIlNCItJ2bXgN1Z2BFw8ulYTwmK/EF3N\nBLA04hIPoQlhsdnCXbONYXToW6VSazMTKbEBYfsQSPuglhMaZi7BRPWJYhsQmRPI5xHsc0jxmWM\npeEZyRzAhZUq4FlY/TrmQCWJqg1iaOxqAHMlWowTaI5YKuvMK58hRaxYqu4oGr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgur",
  "Gr4cElDWuGKjQBLG2\nRPeYPt5ybLMAphtsV5IzgayMJrCPnT51Znd/QVSRO7kgurD0gtJzS8p3bd0n9LcUvJETPLSVP\nJ0F0ZukZpXuW7lFaWlpSOrB0QGlkaUTpQ0sfUhpaGlK6ZukapdpSckcKVwRLdykdWzqm9MDSA0pfWP\nqC0seWPqb0paUvKX1j6RtK71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG",
  "K71t6n1JmKaN03dJ1Srml5NVBEK1aukpYCl59oO9Zmf0szSjNIHlj6\ngdGQpeSqG65ml5PYGLoyWSkqfWPqEUmEpeX4LomeWPqM0sTSh9KmlTyl9belrSh9Z+ojS2FLybgDu\nTizdodS+BaoKSrct3ab01NJT93sBPp/GwLUwt2wFW5SmlqaUblhKnhTgVsLSE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZ",
  "SE3I/Gan2rDZ720TOa5\nGacwdrMz47muQ8UnPuYO3ZaXY0OT9Fas7HpOvre/MXKZBSONMfLyt4LewtLB3u7fyU+/u9t2le6v\nexit>tG9qr3tfeN963or3s3fPe+z1vYEXen96f3l/e/8s/roYLIrFk0Z950p7zJde57Oo/wPjzuXS</lat\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\n<late\nxit sha1_\nbase64=\"y\nFXKWFAp5/\nLfGhFiO4Y",
  "+ \u03c60\n2h0\n2 + \u03c60\n3h0\n3\n<late\nxit sha1_\nbase64=\"y\nFXKWFAp5/\nLfGhFiO4Y\niKSGOJI=\n\">AY1Xi\nclZjbts2\nGIDd7NRlp\n3bDgAC7ER\na0HdbO8KH\ndjOgTZq\neki5Jc2wj\nN6BkSmZDU\nYpEJU4F3w\n273SPtOfY\nAu91eYT8l\n2Yz4M0Njo\nDXzf594+E\nlKtLyEs0x\n2On9dmXv\n/Q8+/Ojqx\n/OfPrZ51\n9cu/7lbhb\nnqU93/JjH\n6b5H",
  "E\nlKtLyEs0x\n2On9dmXv\n/Q8+/Ojqx\n/OfPrZ51\n9cu/7lbhb\nnqU93/JjH\n6b5HMsqZo\nDuSU73k5\nSyON0zt\naVnzvhKYZ\ni8W2PEvoI\nCKhYAHziY\nTQ4fW5/bN\nbzs1fHDc\nZsVuHRWdy\nuy51J27kx\neOCTFxOA3\nngJhmDaGf\ni3HbqslYO\nXDmiksz49\nK/uZDzQfg\n/7vYbfU/5\nM72O939D7\nSndTFo7kw\nBWxyCOPpo\n7rzt90R1l",
  "z49\nK/uZDzQfg\n/7vYbfU/5\nM72O939D7\nSndTFo7kw\nBWxyCOPpo\n7rzt90R1l\nCfFp02vfu\n+dFsTOc7U\nLbROzec3i\nWH07vcHr\nvNpyBMxvJ\n/wykbw6kf\n24g/UsOpH\n+5gfTfcSC\nH1xY7U7\n5cXChWxcW\nW/Vn4/D61\n0N3GPt5RI\nX0Ocmyg24\nnkYOCpJL5\nnE7m3Tyjk\nI0jEtIDKA\noS0WxQlHt\ng4tyAyNAJ\n4hT+CemU0\nfNXFC",
  "kYOCpJL5\nnE7m3Tyjk\nI0jEtIDKA\noS0WxQlHt\ng4tyAyNAJ\n4hT+CemU0\nfNXFCTKsr\nPIAzMicpS\nZTAVt7CX\nwc+Dgokl\n1T4VUNBzh\n0ZO2pDOUO\nWUl/yMygQ\nP2XQV8cfk\nZT4ErbdvC\nvoqR9HERH\nDwl1a2Zw\nUrkdDJgp6\nnJdbcDJpO\niulQ6F4kb\nH0dHtWC5M\n0Ym8pqRU\nVCUXCDScF\nAVth20TMA\nqAtSkCsaA\nZ1Kny4wVO\n1",
  "4kb\nH0dHtWC5M\n0Ym8pqRU\nVCUXCDScF\nAVth20TMA\nqAtSkCsaA\nZ1Kny4wVO\n16Bwy+GAi\n2pluGC8mK\nCqhaQh5KS\nhvUIaFBJO\nxw1rGVkwl\nVFD2QLFcW\n4ClCZwix\nAV+GLGnOw\nlRAxmV4n6\nVimUZGpmN\nlCSkRIyZ\ngyD7hakR\nNQ+Scl9vz\nvPWrab0g4\nqhOXJyUXU\n1VxLC206Y\njU5wXMWw6\nZcSwYBGT\nauMGBaHB8\nSQRASyX",
  "Wrab0g4\nqhOXJyUXU\n1VxLC206Y\njU5wXMWw6\nZcSwYBGT\nauMGBaHB8\nSQRASyXJc\nPYcCRoyJ2\nlQlTZWhb\nqSx12w7UR\nFzbY4T2C9\nNb6VA6T8h\nRkZUAHaf+\nmZE+LSpL8\ncz25km56T\n0VYGOnRFM\nVvMSkobVs\nKaNwKjq2A\nSbZa4ME2c\nLQml82jR\nVbywqTVhz\ngCpgbro8Z\nSI4p90pS7\nBkVdi9A0N\nNc04Pfmjf\no+NB0VHbR\nv",
  "jR\nVbywqTVhz\ngCpgbro8Z\nSI4p90pS7\nBkVdi9A0N\nNc04Pfmjf\no+NB0VHbR\nv2HsgkVZX\nliq0iFL1H\nREI4k5vqC\niDl5MTcmD\nwLl5MUc7u\n/G1JHUXNg\nqUs4dFJg\nnMkzY/uzU\nDSvKSNmZ+\nPI6CsEVL3\nwTZgwJjkI\nmrIKBm+4\nXBlWUC+MU\ni/GqP4yx\nPKbr5Ges\nZIqWubosp\nUw+r5g2VK\n6F536B8dh\nWU4eFwQi+\n43DMy6l",
  "/GqP4yx\nPKbr5Ges\nZIqWubosp\nUw+r5g2VK\n6F536B8dh\nWU4eFwQi+\n43DMy6lX5\n9OJcDElqJ\nHOspnT82s\n0kbDHb7i+\nnvCparZAe\nr9btQb9gd\nnLfp8eHq+\nZ8hMjCDjf\nqgtOstS6O\nLEt7UNdsu\nZ7vWbH6+n\nu0tEOLazc\n5qrfupd2\nuBf0gB6vW\nXq7hjxkY\ncbdU9xB6\nyLO1BXfY\n8rtlGYXHt\nJkf1TvNot\nS3uzDSWf7\nCtj",
  "6vW\nXq7hjxkY\ncbdU9xB6\nyLO1BXfY\n8rtlGYXHt\nJkf1TvNot\nS3uzDSWf7\nCtjqLqmBT\nzoTr2xdyt\nQqYosSitY\nhzR0BCrkC\nlGedOCv01\nli8HDo2lV\nIVPcyFhTU\nwFTGlJuDq\nEKmWK1hZt\nmHTPVNYu6\nZlcJT0aGW\nYVM8TGJzF\nFXIVMsRh\naxSOSJIZY\nhVAeR2YeR\nziPiSklNs\nmckcQyI2h\nJ2RZUOoq\nbkgqY0tho\nbWxpDHr",
  "SOSJIZY\nhVAeR2YeR\nziPiSklNs\nmckcQyI2h\nJ2RZUOoq\nbkgqY0tho\nbWxpDHrAY\n2E0WAdNOc\nMrL7OuPG\nsYoFX8Y6t\n4Z0LGpbEq\nFAFTGkd7T\nHXbduMs9\nMRyzbElO\nmGElOIEbp\nrOBnenpzw\nsKdJLzgjN\nNzA91fQU\n0z1N9zBN\nUW/CLzgha\nbo14kXnGh\n6gumupruY\n5prmO5ou\noNpoGmA6S\nNH2Hqa+p\njuqzpMqZ\nSU3QihS",
  "bo14kXnGh\n6gumupruY\n5prmO5ou\noNpoGmA6S\nNH2Hqa+p\njuqzpMqZ\nSU3QihSeC\nptuYjQdY\nbqv6T6mLz\nV9iekTZ9\ng+krTV5i+\n1fQtpg80f\nYAp0ZRguq\nLpCqZU/T\nqwAuWNF3C\n1NMU/faDv\nabpBqaJpg\nmDzV9iOl\nQU/SrGJ5n\nmqLjDTwYN\neWYPtX0Ka\nZMU/T7zQu\nea/oc0jT\nCNnmj7D9\nI2mbzB9rO\nljTEN0bs\nBOJ1ou",
  "eWYPtX0Ka\nZMU/T7zQu\nea/oc0jT\nCNnmj7D9\nI2mbzB9rO\nljTEN0bs\nBOJ1ouoW\npfgtUZJhu\narqJ6bGmx\n/b3AnQ2jZ\n5tYa7rCtY\nxjTWNMV3V\nFP1SgKOEp\nkfoPBmI+q\n42fduE7mu\nBmHELqzM+\nvRrlPBAzb\nmH13Wl6Nb\no/BWLGR6j\nrK7uzFymQ\nUrjTH15b7\nJpvYXFht9\nfu/ti+u3l\n38f5S/Yb2\naub1ret7\n1rd1k+t+6",
  "FymQ\nUrjTH15b7\nJpvYXFht9\nfu/ti+u3l\n38f5S/Yb2\naub1ret7\n1rd1k+t+6\n0nrY3WTsu\nf+3Pu7l/\n5v5d2FuY\nLPy28Hulz\nl2pr/mq1f\ngs/PEfuP6\ntexit>qZQ=</la\ny0 = \u03c60\n0 + \u03c60\n1a [ 10 +  11a[\u271310 + \u271311x] +  12a[\u271320 + \u271321x] +  13a[\u271330 + \u271331x]]\n+ \u03c60\n2a[ 20 +  21a[\u271310 + \u271311x] +  22a[\u271320 + \u271321x] +  23a[\u271330 + \u271331x]]\n+ \u03c60\n3a[ 30 +",
  "\u03c60\n2a[ 20 +  21a[\u271310 + \u271311x] +  22a[\u271320 + \u271321x] +  23a[\u271330 + \u271331x]]\n+ \u03c60\n3a[ 30 +  31a[\u271310 + \u271311x] +  32a[\u271320 + \u271321x] +  33a[\u271330 + \u271331x]]\n<latexit sha1_base64=\"eo6xbZtUDEx9RMeyK\nb8msDGOEPA=\">AXg3iclZhZc9s2EICltElT90ra6fihL5x60na1GNJ6fGSmcSOc9mp7yMxHQ9Ig\nRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEU",
  "GSmcSOc9mp7yMxHQ9Ig\nRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65/encZ59/8eV\nXd+5+vZ/FRerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hE7ud\nm+NfjwpexPnh0eOG3nxuGSTIzfJBASXJs59p2n3JiOtzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V",
  "uGSTIzfJBASXJs59p2n3JiOtzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V3H2Uu49y96/PbDlHlzJPUC5Byj3YJr75M7C0uJS9efQRq9pLHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfGE+acs5EfQVCzi2XFZzdjEuQeRoRPEKfxTuVNFrx5RsijLiMPzIjlow\nwzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyE",
  "VNFrx5RsijLiMPzIjlow\nwzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEhrMTwWM1fFHLGV+DotkzlX8wo+jiKlh6S6v\nbk1K1+OhUCU/K6oFM5m0ndXK4dC8zlh+uTvLInIeiQ+cJKkUneQagYeTsuSL4SIGgMQi5yAWPEMcu\nr6eIHTQxQ2iARc1gvHBWN7QlKrnIdQk5b2lmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nO",
  "1gvHBWN7QlKrnIdQk5b2lmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nOwtRkelzOx3kalZmO4R5SpkJedQGn7DOpz6htqEJKONRvWX9ia5up06ZwcVINdURZO2mbSdPaV3Us\nO1UEWTBIgzbVhVBloTL2ZBFDKrctE/ghCNHR+yqUFgVZGFuprHX7jvREbw2xwnsl7a3WpLynzNUER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoS",
  "7jvREbw2xwnsl7a3WpLynzNUER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwWhNL5om3o0FpUnon2CO\noA3XZEKFVzRHlQtWLI67D6AU0LyY9+WfyVj4/LJb1t9H+kmpAoKxJbIh3+H4mGcAPF6wsiePJiSY\nPAtXkxRKu72jqWIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRH",
  "72jqWIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC\n8tFJ+vU5+jLOipSTix9azxCpdH1ZTIW+WbUvqFIL7esGl7OjoA03h3N+zeEeqhX19OLCzVkKSrmWE\n/p+J2b5bDFbLu/mvK6abVCfrbW9AfjgtkpfJ+fnazh+QiJR2JcsGzlzWXJalP8g1W65XR1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lH",
  "JR2JcsGzlzWXJalP8g1W65XR1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG47C4trNyXJO62j1ba4MxMt/2\nB3xHOmH5NiOdSPfbF06xAWcyrmVjGOeIjEOoTFqGhb8B0rOwJuHm2rDmFxMxNtTQewNOQSn0IdwmK\n9hdtmE8PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp",
  "tmE8PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp6RxDIjZEnZF\nlQ6ituSDmBpjHobWzqDEchYoQ6bIJYzuvIy68pTaBUruor3bB3vXdNxzlBCHcDSBtljrth3WQeLjE\n8ZtmKnAhkJbSAm9jZpM706c8LSvIk5wWXhl5SemHoBaUHh5QmhpKfhF4wbah5NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6",
  "mHoBaUHh5QmhpKfhF4wbah5NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbuUjoydETpoaGHlL4x9A2lLwx9QelbQ9\nS+sHQD5Q+MfQJpcxQRumqoauUckPJqwMvWDZ0mVLPUPLbD/aoZuUJoYmlD419CmlQ0PJr2K4nxlK\nHm/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreD",
  "lK\nHm/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6Z\nahW5SeGXpmfy/AZ9Po2RbmhkmwQWlsaEzpmqHklwI8Sh6Sp4nA9Vc1aZvm8h1LVAzbmFNxadHk5oH\n/ze+dx50Vns7PX8btR96/u391/5m/O35/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/Y",
  "/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/YiBUoKV/qTOws9/BaWNvb7i73fFh9uPVx4vNy8ob3d+a7zfenTq\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]\n\ud835\udf03 : theta\n\ud835\udf19 : phi\n\ud835\udf13 : psi",
  "Remember shallow network with two outputs?\n\u2022 1 input, 4 hidden units, 2 outputs\n<latexit sha1_base64=\"R\n8vqOAXY+V+odbvKA1u5ta3TLRI=\">AXWniclZhbU9w\n2FICX9JaQXkhvPTFUyadTpsyLKGXl84kEHKDFAjXBN\nG9speBVk2tgxLPtDO9Ppb+mR7V3F54iH7ky4nyfbke\nyrXWQSVHopaW/Z258OFH39y89bs7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h",
  "89bs7U8/+/yLuTtfHhR\npmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwLJD4OzNcMPL3h\neiFTt6auMnyQsViISIdMQOp37d3ha9cfeD3/6SZCOKjY\n+9vWQawbRpbH3szf9qz8enXi+SlWZBDz3fH8Wai6bmh6p\nutypuyset9d9X6n6n1n1RV31ZVO1RVTdfZ0bmFpcan+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxae",
  "mFpcan+\neLTQbwsLvfazfXrnm4E/SMy4UqHkhXFcX8p0ycVy7UI\nJR/P+mXBMxaesZgfQ1GxhBcnVb0QY+8uRAZelObwT2mv\njr5fo2JUVwlAZgJ08MCMxN0seNSR3+cVEJlpeYqbDqK\nSunp1DOr6g1EzkMtr6DAwlzAWL1wyHIWalj7WV/xyzBN\nEqYGlb+6vjOu/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC",
  "+6vjOu/IDHQlX8vKz3wXjcdZrh0PxOmP12d60F\naF5It5x0kitmEauEXg8riq+GC9iIDgAscgJSBUvoE2Tn\nyDy+ojCvpeAq2Zn+GC8HJOmleYx5KSjvSYaFDLJRx1rj\nViwlElH2QXF8+56BnCdwyrAUOGLozXYzZgaT+pPtJ5U\nhUmhnvImYp53QVMOWTSzKhrqFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8Jd",
  "zKhrqFJKqBp2rL+w9ZKpszZxa\nVYPNTcRZO3lXUfnNC9q0HXqCLJgE8Zdq4gS8JdasASB\nluy6cw4cQzEbcqFYF2ZjbeRp0+85MBO/NUQbXS9dbr\n0j6LxjKiAnA1We+BVMh7+pr6dT2Jsm5qH1T4CNvCIvVrc\nLyuJnWpBOYVRsbU7POFTJptiCUp5d04zGofJMdCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmr",
  "ofJMdCdoAv\niK3Ohove0e3UJtqwJ+/dgqnkp+fEvi7/y0Um1ZC4b8x\n/JjRUlJmrIRP+Hw0N4LmI9xdE8OKlEi0eBOrFSyXc39\nHSsRxvbBOp1w4KQjEp9BW6/EWsunXqCB5smqCxQsC0C9\n9MKLTIUdSVTcDI8A1PeMcGCtEkw2aOoUyLMufk5of2M0\nRq3dwWc2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMd",
  "dwWc2EeVt0bqjRC97B5bQWlOHhcMGvqR6gjAZNPo\nO0VAOWo2SOzJKO3viFhkvMdfXS94UnVbMzfa/mBcsDp\nlGPLz0w28HjGxqCNRW3CkcrYlieXoD9qabtf3R1ZtvPm\nJbO3Y4bpNSdptR+m2He41I+Dnm47RbhKPWNSRqK12hNQ\njlqM/aMudx03XLByu25Sk3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2Nl",
  "25Sk3UkenbDnZpo+0d75ihqjkm\npHJhjXyr9JoRFTUXtFNOEx0hsQlhMyq4Ff2NlV8Do2s\n1ISxuF6KrmQCWBlziKTQhLDaXcNdsY1jdKibpXJbIj\nMJoTFJyzBs25CWIypGDvFM5ZlSGxCJI9DnMchzWOGpcw\nl4RXJHCtCtpRrQ+XDtCuZAJZGqLeRozMYgUwV6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFe",
  "gUwV6rANYrmg\nO69w7jyFdrGiu3jf1fH+NR1rho0ASxtkWvM87ecF1mA\nUwzHLFeSM4GsjCZwGzvb1Jmc/oKoIie5ILqy9IrS0sv\nKT209JDS3FLyiyCIXlpKfp0E0YWlF5QeWHpAaWlpSem+\npfuURpZGlD629DGloaUhpWuWrlGqLSUnUngiWLpH6dDS\nIaVHlh5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVS",
  "aVHlh5R+srSV5Q+tfQpa8tfU3pO0vfUfrQ0oeUMksZ\npeuWrlPKLSWvDoJo1dJVSgNLyW8/uNYs3aY0szSj9JGlj\nygdWEp+FcPzFJyvIEHo6WS0meWPqNUWEp+vwXRC0tfU\nJpYmlD63NLnlL619C2lTyx9QmlsKXk3AKcTS3cptW+Bq\noLSHUt3KD239Nz9XoBPlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7",
  "9XoBPlzFwbcwt28AWpamlKaUblpJfC\nnCUsPSMnCcj1d7VJm+byH0tUlPuYG3GJ7VJziM15Q7W3\np0mtcn9KVJTPiRDXz+YvkiBlMKd/nRuoY/fwtLCwfJi/\n7fFlZ2VhQer7Rvam73vet/3fuz1e7/3HvSe9rZ7+71w5\ntHM25liRn/7z/yN+Vvztxv1xkxb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 +",
  "xb56te5zP/9X86WR0R</\nlatexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nh4 = a[\u271340 + \u271341x]\n<latexit sha1_base64=\"Pl76mATi6i7yjrijGrQ\n4B2+SCU=\">AXIXiclZhZb9tGEIDl9EqdHk6L+qUvRI0URZsakuIeaFEgseNcdmo5PhPLEZbUktp4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+",
  "p4uaR52\nFI/ZqiP6ZvRd+K/pnOkpQ2nFk/RICt5XwfZ5ezy0N0YynSrN3+d+HaO+9/4H1z9cvPHRx598unTzs8M0\nyhOPH3iRjJjl6VcCsUPMpFJfhwnIWu5Efu2YbmRxc8SUWk9rNJzE9DFijhC49lEBos/TkZdJyvf+vHIzE\noOu3pd3WrMx3B/lmV2925t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2ar",
  "t39Oad+ea3lyb9lWk8tDlidPvL04GXcjrVEZ3nrjbTNxtJu42E3frxIuDpZ\nX2arv8OLTRqRsrfrTG9z8YtgfRl4ecpV5kqXpSacdZ6cFSzLhST5d7Ocpj5l3xgJ+Ak3FQp6eFmU9p84ti\nAwdP0rgT2VOGX1zj4KFaToJXTBDlo1SzHTQxk7yzP/5tBAqzjOuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7L",
  "OuvKojP5dOFjl6cpyhSLiXyQk0mJcIGKvj\njVjCvAymcLGv+KUXhSFTw6K/vrk7LfouD4Qq+HleTud02nQ2S4dD8ypj/fH+PIvIeChec5KkVHSKwQeTIu\nCrwarGAgOQKxyAiLFU8ip6+P6TgdRWL4SMHA3GsPgfOfZlKRWGQ+gJg3tBdGgEUs+blgbxIKpDBvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTUL",
  "BvKHiOc8\nvRgGcJzAIMFb4moO9mKnpbL+Mj7MkLFIdwz0kTAW87AIO2WNSH1HTULmUsKvXsH7H1jOmzurCRXE51ERHk\nLWfNJ0soXVRw6ZTRpAFizBoWmUEWRIuNkMWMqhy3R7AYeOjthVobAqyMLsJZHb7DvWEbw2xzGcL01vsyDl\nv2CoIjoAZ5/+Fkx5vKlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei",
  "KlvRHPbmRXnovR1g4+dEUxWcxeWBNVhzTqBo6pjU2qWtUImrRaEkuiyaerRWFQei+Y\nB6gA+6fJEKP8N7XbZgiWrw/3bcKhJLvnJ96s/8PFp0danjf5HqgmJ0jy2JdLht0g0hNsbXl8QwZMXSTR5EC\ngnL5JwfUdTxK8sHWknDtoCMWkyCbo9BeBau5TRvBgoxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH",
  "oxCNFQI6L3wzodAk+35T1gEtwzfcqC0LyEMH6VXH6\nMkozRNOLn5oPUOk1PVlMRH6ZtW8oEotNK8bXM73gjbcHC74Fbu7qKJuVU83ytWQJaiYz2l45f9NINTzHb2\nl1NeNa1WwM+36v5gXDA7uefx8EWno+AWNSRKBc8GVlzSWJZ+oNc8+X65siKrZfkqUdWFy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tN",
  "Fy7KUnepR2+J\neMQJ+vm0Z7TbxiEUdiXLVI6QesSz9QS57HbdtR2Fx7aYkeWd1tNoWd26i5e/vj3jG9GNSJIf6sS+S/SqExY\nyKmVWMQh4gsQphMcybFmxjZU/AzaNpVSEs9lLR1HQAS0Mu8SFUISxWp3DTrGNY3bao23aVyXiEzCqExYcsx\nEdhbAYUDGwimcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhH",
  "mcsjpFYhUgdR7iOI1rHGEuxTcIzEltmhCwp24JKRlFT0gEsjVFvY0tnMAIZKdRhHcRySlde\nal15Cq1iRVfxga3jgys6zhKqANY2iHnmNPfsZ5kLi4xPGbZihwLZMW0gD3s9Kgze/pz/YI8ybn+xNAJpZe\nGXlJ6ZOgRpYmh5BeB6z8zlPw6cf0LQy8oPT0kNLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLn",
  "NLc0JzSA0MPKPUN9Sl9YOgDSj1DPUo3DN2gNDOUPJHCHc\nHQfUpHho4oPTb0mNLnhj6n9JGhjyh9YegLSl8b+prSe4beo5QZyijdNHSTUm4oeXg+uGrlPqGkp+8G5Z\nmiP0tjQmNL7ht6ndGgo+VUM9zNDyeMN3BgNlZQ+NvQxpcJQ8vN9Z8a+pTS0NCQ0ieGPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DE",
  "GPqH0laGvKH1o6ENK\nA0PJuwF4OjF0j1LzFqhIKd01dJfSc0P7e8F+HwaXdvC3DEJdiNDI0o3TKU/FKARwlDz8jzpK/q9rsbRO\n5rvlqzi2srvhsb1JzX825hdVXp9ne5PrkqzkfkaFvHs5fpEBJ4Uo/WFrp4LewtHYXe38uLq2u7Zyd71+Q3\nu9WXrq9Y3rU7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ<",
  "U7rp9bd1qNWr3XQ8hZuLHQWfln4dfmP5b+W/17+p1KvLdT7fN5qfJb/+x/3OgsJ</latexit\n>y1 = \u03c610 + \u03c611h1 + \u03c612h2 + \u03c613h3 + \u03c614h4\ny2 = \u03c620 + \u03c621h1 + \u03c622h2 + \u03c623h3 + \u03c624h4",
  "Networks as composing functions\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41k",
  "H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOC",
  "DvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF",
  "lrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhV",
  "klZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo",
  "rBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4",
  "CMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClA",
  "NMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTn",
  "+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSm",
  "8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5",
  "TSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nConsider the pre-activations at the second hidden units\nAt",
  "Q=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nConsider the pre-activations at the second hidden units\nAt this point, it\u2019s a one--layer network with three outputs\n<latexit sha1_base64=\"eo6xbZtUDEx9RMeyK\nb8msDGOEPA=\">AXg3iclZhZc9s2EICltElT90ra6fihL5x60na1GNJ6fGSmcSOc9mp7yMxHQ9Ig\nRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65/enc",
  "KR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65/encZ59/8eV\nXd+5+vZ/FRerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hE7ud\nm+NfjwpexPnh0eOG3nxuGSTIzfJBASXJs59p2n3JiOtzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V3H2Uu49y96/PbDlHlzJPUC5",
  "tzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V3H2Uu49y96/PbDlHlzJPUC5Byj3YJr75M7C0uJS9efQRq9pLHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfGE+acs5EfQVCzi2XFZzdjEuQeRoRPEKfxTuVNFrx5RsijLiMPzIjlow\nwzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEhrMTwWM1fFHLGV+DotkzlX8",
  "HbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEhrMTwWM1fFHLGV+DotkzlX8wo+jiKlh6S6v\nbk1K1+OhUCU/K6oFM5m0ndXK4dC8zlh+uTvLInIeiQ+cJKkUneQagYeTsuSL4SIGgMQi5yAWPEMcu\nr6eIHTQxQ2iARc1gvHBWN7QlKrnIdQk5b2lmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nOwtRkelzOx3kalZmO4R5SpkJ",
  "QSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nOwtRkelzOx3kalZmO4R5SpkJedQGn7DOpz6htqEJKONRvWX9ia5up06ZwcVINdURZO2mbSdPaV3Us\nO1UEWTBIgzbVhVBloTL2ZBFDKrctE/ghCNHR+yqUFgVZGFuprHX7jvREbw2xwnsl7a3WpLynzNUER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwW",
  "UER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwWhNL5om3o0FpUnon2CO\noA3XZEKFVzRHlQtWLI67D6AU0LyY9+WfyVj4/LJb1t9H+kmpAoKxJbIh3+H4mGcAPF6wsiePJiSY\nPAtXkxRKu72jqWIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC\n8tFJ+vU5+jLOipSTix",
  "a/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC\n8tFJ+vU5+jLOipSTix9azxCpdH1ZTIW+WbUvqFIL7esGl7OjoA03h3N+zeEeqhX19OLCzVkKSrmWE\n/p+J2b5bDFbLu/mvK6abVCfrbW9AfjgtkpfJ+fnazh+QiJR2JcsGzlzWXJalP8g1W65XR1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG4",
  "R1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG47C4trNyXJO62j1ba4MxMt/2\nB3xHOmH5NiOdSPfbF06xAWcyrmVjGOeIjEOoTFqGhb8B0rOwJuHm2rDmFxMxNtTQewNOQSn0IdwmK\n9hdtmE8PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp6RxDIjZEnZF\nlQ6ituSDmBp",
  "mcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp6RxDIjZEnZF\nlQ6ituSDmBpjHobWzqDEchYoQ6bIJYzuvIy68pTaBUruor3bB3vXdNxzlBCHcDSBtljrth3WQeLjE\n8ZtmKnAhkJbSAm9jZpM706c8LSvIk5wWXhl5SemHoBaUHh5QmhpKfhF4wbah5NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbu",
  "NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbuUjoydETpoaGHlL4x9A2lLwx9QelbQ9\nS+sHQD5Q+MfQJpcxQRumqoauUckPJqwMvWDZ0mVLPUPLbD/aoZuUJoYmlD419CmlQ0PJr2K4nxlK\nHm/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6Z\nahW5S",
  "329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6Z\nahW5SeGXpmfy/AZ9Po2RbmhkmwQWlsaEzpmqHklwI8Sh6Sp4nA9Vc1aZvm8h1LVAzbmFNxadHk5oH\n/ze+dx50Vns7PX8btR96/u391/5m/O35/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/YiBUoKV/qTOws9/BaWNvb7i7",
  "CKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/YiBUoKV/qTOws9/BaWNvb7i73fFh9uPVx4vNy8ob3d+a7zfenTq\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]",
  "Networks as composing functions\n<latexit sha1_base64\n=\"OexYJKlfph/YpHs7sx68KlEAH9c=\">AXC3ic\nlZhbU9w2FICX9JbSG2mnvOTFU5pOp0ZlqSXl84k\nEHKDFAgskLCEkb2yV0GWjS3DEs/+hE5/TN86fe2\nP6A/pe49s7wqfIx6yM8mK8326Hcm21n4qRa6Xlv6\ndufbOu+9/8H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41k",
  "H1D2c/+viTz+bu/H5Xp4UWcB7QSK\nT7MBnOZdC8Z4WvKDNOMs9iXf909WDd8/41kuErW\nrL1J+FLNIiVAETEPoeO6P4XHZHXvf/Or1Yz8ZlW\nx82NdDrhmEl8be970r+54dOT1+7NQYdldYblVYf\nlShTvuCndaFe6YCsdzC0uLS9XHo4VuU1joNJ+t4x\ntfDvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOC",
  "DvqDJChirnQgWZ4fdpdSfVSyTItA8vFsv8h5y\noITFvFDKCoW8/yorDI39m5BZOCFSQb/lPaq6OUaJ\nYvz/CL2wYyZHuaYmaCLHRY6/OWoFCotNFdB3VFYS\nE8nlkGbyAyHmh5AQUWZALG6gVDlrFAw2LN9hU/D\n5I4ZmpQ9lfWtsdl3+eRUCU/LaqFG4/bzlrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF",
  "lrlcChe\nZaw82Z2IjSPxRtOGqkU08gVAo/GZckXo0UMBAcg\nFjkBieI5tGny4deF1HYqBJwW+MPhjPx6RpXkE\nOWlpL4kGhVTyUctaJRYsZdxSdkDxvFueAVxnsAo\nwVPjiaA12UqbGk3qaj3QWl7mJ4R4ypiJedQFTDpg\n0M2obqpASqgYt6zdsPWfqpElcklZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhV",
  "klZDzUwEWbtZ29E\nZzYsatJ0qgizYhFHbqiLIknBbGbCYQZab8jFMOP\nZMxK0KhVBNuZWlvjtvlMTwXtzlML10vbWSpL+M4\nYyYgJw9ZlvwVTA2/pqMrW9SXLOKt8U+MgbwmK1q7\nAsqc16QRm1cTG1KxyhUyaLQhlyXnbNKNxqDwV7Q\nmaAL7oikyo8J2uyrBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo",
  "rBljXh/m2YalZIfvjD4o98d\nFQumcvG/EeyCQ3lRepqyITfoqEBPMjw/oIXrxEo\nsWDQLV4iYT7O1o6luGNbSLV2kFBKCaFvkCXv4hUu\n04VwYNYjRWCJh24ZsJhRY5DNuyCRgZvuGR7NhA\nAZpkUM8xkEleZJzc/NB+hkilm9tiJszDqn1DlUZo\n3ze4nNaCMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4",
  "CMjwczvgV1X2Ub/Op58UasAylMyRWdLR\nq36u4RJzXf3VktdFpxXx0/WmPxgXrE4RBPz0eB2v\nR0Qs6kjUFpyBnG1JYjn6g7am2/XyMr1V9+RrR0\n5XLcpSbvNKN2w71iBPx0wzHaDeIRizoStdWMkHr\nEcvQHbnzuOGahcN1m5K0O8mj03a4UxNt/3DXnET\nNMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClA",
  "NMSmRA3PsS2S/DmFRU1E7xSTmERLrEBbjom3B31\njZEfDwaFt1CItbuWhrJoClAZd4CnUIi/Ul3DabGF\nY3HOqGW2UyHSKzDmHxEYvxrOsQFiMqRk7xhKUpEu\nsQyeMQ53FI85hiKXVJeEVSx4qQLeXaUNkwaUsmg\nKUR6m3k6AxGIBOFOmyCWM7pzsudO0+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTn",
  "+hXazoLu65O\nu5d0bFmqETwNImuca8/qbzIvNxiuGY5UpyKpCV0\ngRuYWeLOpPTnx+W5CTnhxeWXlB6buk5pfuW7lOaW\nUp+Efjhc0vJrxM/PLP0jNI9S/coLSwtKO1Z2qM0\ntDSk9KGlDykNLA0oXbV0lVJtKTmRwhPB0l1Kh5YO\nKT2w9IDSF5a+oPSxpY8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSm",
  "8pfWnpS0rfWPqG0vuW3qeU\nWcoXbN0jVJuKXl14Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nlpLjDTwYLZWUPrH0CaX\nCUvL7zQ+fWfqM0tjSmNKnlj6l9LWlryl9ZOkjSiN\nLybsBOJ1YukOpfQtU5pRuW7pN6amlp+73Any6jL5\nrY27aBjYpTSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5",
  "TSxNKF23lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd7Dm7jSpTe5PoZryIR\nn62t70RQqkFO70x3MLXfwWlhb2lhe7Py3e3b67cG\n+leUN7vXOz81Xn206383PnXudxZ6vT6wSd/2Zuz\nnw9c2v+9/k/5/+a/7tWr80db7otD7z/wPvt8Cj\nQ=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nConsider the pre-activations at the second hidden units\nAt",
  "Q=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\nConsider the pre-activations at the second hidden units\nAt this point, it\u2019s a one--layer network with three outputs\n<latexit sha1_base64=\"eo6xbZtUDEx9RMeyK\nb8msDGOEPA=\">AXg3iclZhZc9s2EICltElT90ra6fihL5x60na1GNJ6fGSmcSOc9mp7yMxHQ9Ig\nRiEKR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65/enc",
  "KR52HI4+hV9bX9Y/0XJCWYu/BDPZMI2u/jAlwAJEUvkSLl5b+7d746Obtz65/encZ59/8eV\nXd+5+vZ/FRerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hE7ud\nm+NfjwpexPnh0eOG3nxuGSTIzfJBASXJs59p2n3JiOtzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V3H2Uu49y96/PbDlHlzJPUC5",
  "tzb739fe+T7Q3weTY8dVsSoij6eO687p3H1\nb7v6V3H2Uu49y96/PbDlHlzJPUC5Byj3YJr75M7C0uJS9efQRq9pLHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfGE+acs5EfQVCzi2XFZzdjEuQeRoRPEKfxTuVNFrx5RsijLiMPzIjlow\nwzHbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEhrMTwWM1fFHLGV+DotkzlX8",
  "HbSxoyIP/jguhUqKnCu/7igopJPHjp5+ZyhS7ufyEhrMTwWM1fFHLGV+DotkzlX8wo+jiKlh6S6v\nbk1K1+OhUCU/K6oFM5m0ndXK4dC8zlh+uTvLInIeiQ+cJKkUneQagYeTsuSL4SIGgMQi5yAWPEMcu\nr6eIHTQxQ2iARc1gvHBWN7QlKrnIdQk5b2lmjQSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nOwtRkelzOx3kalZmO4R5SpkJ",
  "QSCQft6wVYsFURi1lBxTHuedowPMUZgGCh8czcF\nOwtRkelzOx3kalZmO4R5SpkJedQGn7DOpz6htqEJKONRvWX9ia5up06ZwcVINdURZO2mbSdPaV3Us\nO1UEWTBIgzbVhVBloTL2ZBFDKrctE/ghCNHR+yqUFgVZGFuprHX7jvREbw2xwnsl7a3WpLynzNUER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwW",
  "UER2\nA3ac/BVM+b+sr8cx2psU5r3zd4GNnBJPVPoSlYX1a07grJrYhJpVrZBJqwWhNL5om3o0FpUnon2CO\noA3XZEKFVzRHlQtWLI67D6AU0LyY9+WfyVj4/LJb1t9H+kmpAoKxJbIh3+H4mGcAPF6wsiePJiSY\nPAtXkxRKu72jqWIoXto5UcwcNoZgU+SXa/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC\n8tFJ+vU5+jLOipSTix",
  "a/iJU7WOqCB5sHKGxQkDnhU8mFJrkIGjLOqBl+IRHAcsC\n8tFJ+vU5+jLOipSTix9azxCpdH1ZTIW+WbUvqFIL7esGl7OjoA03h3N+zeEeqhX19OLCzVkKSrmWE\n/p+J2b5bDFbLu/mvK6abVCfrbW9AfjgtkpfJ+fnazh+QiJR2JcsGzlzWXJalP8g1W65XR1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG4",
  "R1auvfuZ\nLO3Q4tpNSfI2o7TbFveaEfCzdcto14lHLOpIlKsZIfWIZekPctnruG47C4trNyXJO62j1ba4MxMt/2\nB3xHOmH5NiOdSPfbF06xAWcyrmVjGOeIjEOoTFqGhb8B0rOwJuHm2rDmFxMxNtTQewNOQSn0IdwmK\n9hdtmE8PqukVdt6tMJiNk1iEsPmcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp6RxDIjZEnZF\nlQ6ituSDmBp",
  "mcRPus6hMWQiqFVPGVJgsQ6ROo4wnUc0TomWEpsEp6RxDIjZEnZF\nlQ6ituSDmBpjHobWzqDEchYoQ6bIJYzuvIy68pTaBUruor3bB3vXdNxzlBCHcDSBtljrth3WQeLjE\n8ZtmKnAhkJbSAm9jZpM706c8LSvIk5wWXhl5SemHoBaUHh5QmhpKfhF4wbah5NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbu",
  "NeJF5wbek7pvqH7l\nBaGFpTuGbpHaWBoQOkzQ59R6hvqU7pi6AqluaHkiRTuCIbuUjoydETpoaGHlL4x9A2lLwx9QelbQ9\nS+sHQD5Q+MfQJpcxQRumqoauUckPJqwMvWDZ0mVLPUPLbD/aoZuUJoYmlD419CmlQ0PJr2K4nxlK\nHm/gxmiopPSloS8pFYaS329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6Z\nahW5S",
  "329e8NrQ15RGhkaUvjL0FaXvDX1P6XNDn1MaGkreDcDTiaE7lJq3QGVG6Z\nahW5SeGXpmfy/AZ9Po2RbmhkmwQWlsaEzpmqHklwI8Sh6Sp4nA9Vc1aZvm8h1LVAzbmFNxadHk5oH\n/ze+dx50Vns7PX8btR96/u391/5m/O35/vz+s1Rvd5phvOq2/+Uf/AXCKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/YiBUoKV/qTOws9/BaWNvb7i7",
  "CKek=</latexit>asYtrLk6TY8m16dAzfiIDH1f/YiBUoKV/qTOws9/BaWNvb7i73fFh9uPVx4vNy8ob3d+a7zfenTq\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]",
  "Let\u2019s walk through example activations \nstarting with pre-activations to the 2nd layer.",
  "2nd Layer Pre-activations\nLike a shallow network with three hidden units and three outputs.",
  "2nd Layer Activations\n2nd Layer Pre-activations",
  "2nd Layer Activations\n2nd Layer Weighted Activations",
  "2nd Layer Weighted Activations\nSummed and Offset\n2nd Analogy:\nCreate new functions \nwhich are clipped and \nrecombined.",
  "Deep neural networks\n\u2022 Composing two networks\n\u2022 Combining the two networks into one\n\u2022 Hyperparameters\n\u2022 Notation change and general case\n\u2022 Shallow vs. deep networks",
  "Hyperparameters\n\u2022 K layers = depth of network\n\u2022 \ud835\udc37! hidden units per layer = width of network\n\u2022 These are called hyperparameters \u2013 chosen before training the \nnetwork\n\u2022 Can try retraining with different hyperparameters \u2013 hyperparameter \noptimization or hyperparameter search\n\u2022 This can be either manual or automated (e.g. Hyperparameter Tuning with \nRay Tune)",
  "Deep neural networks\n\u2022 Composing two networks\n\u2022 Combining the two networks into one\n\u2022 Hyperparameters\n\u2022 Notation change and general case\n\u2022 Shallow vs. deep networks",
  "Propose 3 notation changes to be \nable to generalize to arbitrary \ndeep neural networks.",
  "Notation change #1\n<latexit sha\n1_base64=\"OexYJKlfph/Yp\nHs7sx68KlEAH9c=\">AXC\n3iclZhbU9w2FICX9JbSG2mn\nvOTFU5pOp0ZlqSXl84kEHK\nDFAgskLCEkb2yV0GWjS3DEs\n/+hE5/TN86fe2P6A/pe49s7\nwqfIx6yM8mK8326Hcm21n4\nqRa6Xlv6dufbOu+9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd",
  "9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd8/41kuErWrL1J\n+FLNIiVAETEPoeO6P4XHZH\nXvf/Or1Yz8ZlWx82NdDrhmE\nl8be970r+54dOT1+7NQYdl\ndYblVYflShTvuCndaFe6YCs\ndzC0uLS9XHo4VuU1joNJ+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI",
  "+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI39m5BZOCFS\nQb/lPaq6OUaJYvz/CL2wYyZ\nHuaYmaCLHRY6/OWoFCotNF\ndB3VFYSE8nlkGbyAyHmh5A\nQUWZALG6gVDlrFAw2LN9hU/\nD5I4ZmpQ9lfWtsdl3+eRUCU\n/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBi",
  "/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBieI5tGn\ny4deF1HYqBJwW+MPhjPx6\nRpXkEOWlpL4kGhVTyUctaJ\nRYsZdxSdkDxvFueAVxnsAow\nVPjiaA12UqbGk3qaj3QWl7\nmJ4R4ypiJedQFTDpg0M2obq\npASqgYt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab",
  "Yt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab8j\nFMOPZMxK0KhVBNuZWlvjt\nvlMTwXtzlML10vbWSpL+M4Y\nyYgJw9ZlvwVTA2/pqMrW9SX\nLOKt8U+MgbwmK1q7Asqc16\nQRm1cTG1KxyhUyaLQhlyXn\nbNKNxqDwV7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\ny",
  "7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\nyITfoqEBPMjw/oIXrxEosW\nDQLV4iYT7O1o6luGNbSLV2\nkFBKCaFvkCXv4hUu04VwYN\nYjRWCJh24ZsJhRY5DNuyCRg\nZvuGR7NhAZpkUM8xkEleZJ\nzc/NB+hkilm9tiJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzX",
  "iJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzXf3VktdFpxXx0/WmP\nxgXrE4RBPz0eB2vR0Qs6kjU\nFpyBnG1JYjn6g7am2/XyMr\n1V9+RrR05XLcpSbvNKN2w\n71iBPx0wzHaDeIRizoStdWM\nkHrEcvQHbnzuOGahcN1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B",
  "1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B31jZEfDwaFt\n1CItbuWhrJoClAZd4CnUIi/\nUl3DabGFY3HOqGW2UyHSKzD\nmHxEYvxrOsQFiMqRk7xhKUp\nEusQyeMQ53FI85hiKXVJeE\nVSx4qQLeXaUNkwaUsmgKUR6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzI",
  "R6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzIvNxiuGY5U\npyKpCV0gRuYWeLOpPTnx+W\n5CTnhxeWXlB6buk5pfuW7lO\naWUp+Efjhc0vJrxM/PLP0jN\nI9S/coLSwtKO1Z2qM0tDSk9\nKGlDykNLA0oXbV0lVJtKTmR\nwhPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jV",
  "whPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jVJuKXl1\n4Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nl\npLjDTwYLZWUPrH0CaXCUvL7\nzQ+fWfqM0tjSmNKnlj6l9LW\nlryl9ZOkjSiNLybsBOJ1Yuk\nOpfQtU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42",
  "tU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd\n7Dm7jSpTe5PoZryIRn62t70\nRQqkFO70x3MLXfwWlhb2lhe\n7Py3e3b67cG+leUN7vXOz8\n1Xn206383PnXudxZ6vT6wSd\n/2Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 +",
  "Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha\n1_base64=\"eo6xbZtUDEx9R\nMeyKb8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6fGSmcSO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/jAlwAJEUvk",
  "SO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/jAlwAJEUvkSLl5b+7d\n746Obtz65/encZ59/8eVXd\n+5+vZ/FRerzPT+WcXrosYxL\nofheLnLJD5OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0eOG\n3nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6eO687p3H1b7",
  "nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6eO687p3H1b7v6V3H2\nUu49y96/PbDlHlzJPUC5By\nj3YJr75M7C0uJS9efQRq9pL\nHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEuQ\neRoRPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7igopJPHjp5+Z",
  "RPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7igopJPHjp5+Zyh\nS7ufyEhrMTwWM1fFHLGV+Dot\nkzlX8wo+jiKlh6S6vbk1K1+\nOhUCU/K6oFM5m0ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE\nMcur6eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1lBxTHuedowP",
  "eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1lBxTHuedowPMU\nZgGCh8czcFOwtRkelzOx3k\nalZmO4R5SpkJedQGn7DOpz6\nhtqEJKONRvWX9ia5up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\ntE/ghCNHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BVM+b+sr8cx2",
  "NHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BVM+b+sr8cx2p\nsU5r3zd4GNnBJPVPoSlYX1a\n07grJrYhJpVrZBJqwWhNL5\nom3o0FpUnon2COoA3XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3+H4mGcAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7WOqCB5sHK",
  "cAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7WOqCB5sHK\nGxQkDnhU8mFJrkIGjLOqBl+\nIRHAcsC8tFJ+vU5+jLOipST\nix9azxCpdH1ZTIW+WbUvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFbLu/mvK6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65XR1auvfu\nZL",
  "K6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEf\nCzdcto14lHLOpIlKsZIfWIZ\nekPctnruG47C4trNyXJO62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGhb8B0rOwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk1iEsPmcR\nP",
  "OwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk1iEsPmcR\nPus6hMWQiqFVPGVJgsQ6ROo\n4wnUc0TomWEpsEp6RxDIjZE\nnZFlQ6ituSDmBpjHobWzqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth3WQeLjE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5QmhpKfhF4\nwba",
  "jE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5QmhpKfhF4\nwbah5NeJF5wbek7pvqH7lBaG\nFpTuGbpHaWBoQOkzQ59R6hv\nqU7pi6AqluaHkiRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoauUckPJqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/gxmiop\nPSlo",
  "JqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/gxmiop\nPSloS8pFYaS329e8NrQ15RG\nhkaUvjL0FaXvDX1P6XNDn1M\naGkreDcDTiaE7lJq3QGVG6Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc1aZvm8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/qTOw\ns9/Ba",
  "8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/qTOw\ns9/BaWNvb7i73fFh9uPVx4v\nNy8ob3d+a7zfenTq/ze+dx\n50Vns7PX8btR96/u391/5m/\nf/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +",
  "+  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]\n<latexit sh\na1_base64=\"qvab5eVFtFK\nvws6yR9bTatgWmaY=\">A\nAWuniclZhbU9w2FICdXtP0\nRtopL3xlMmk06Y7kKSXh\n7aTQMgNUpbAglLGNkrexV\nk2dgyLPHsv+mv6Wv70n/T\nI9u7is8RD90ZWO35PutyJP\nkWZFIUen53yvPve+x9\n8ePWja",
  "PHsv+mv6Wv70n/T\nI9u7is8RD90ZWO35PutyJP\nkWZFIUen53yvPve+x9\n8ePWjax9/8ulny9c/2KvS\nMs85IMwlWl+ELCS6H4QA\nst+UGWc5YEku8HJ2uG75/x\nvBCp2tUXGT9KWKxEJEKmI\nXS8PvFTf83f5iNxc3jan\n6fVtamY7r/7Pft+vft+e/\n79S/70yPF5aWe8v1x6eFlb\naw5LWf/vH1r0bDURqWCVc\n6lKwoDleWM31UsVyLUPLpt",
  "/70yPF5aWe8v1x6eFlb\naw5LWf/vH1r0bDURqWCVc\n6lKwoDleWM31UsVyLUPLpt\nWFZ8IyFJyzmh1BULOHFUV\nUPdOrfgMjIj9Ic/pT26+jb\nR1QsKYqLJAzYXpcYGaCL\nnZY6uiXo0qorNRchU1DUSl\n9nfoma/5I5DzU8gIKLMwF\n9NUPxyxnoYbcXhsqfh6mSc\nLUqBqurm9Pq2HAY6Eqflr\nWeZ5Ou8567XAoXmasPtmd1\nyI0T8QbTiqpF",
  "fh6mSc\nLUqBqurm9Pq2HAY6Eqflr\nWeZ5Ou8567XAoXmasPtmd1\nyI0T8QbTiqpFVPJQKPp1\nXFe3EPA8EBiB4nIFW8gDpN\nfoLIX0EU1pUEDxIJ9C5y\nH8+JVUrzWPISUd7STQoZJ\nPOtYasWAqk46yA4rv3/AN\n4DqHWYCuwhdHc7CTMTWdHa\nf5ROdJVZgYbiFnKuZ1EzD\nkEkzoq6hSinh0LBj/YGt5\n0ydtIlLs7qruYkgazfvOj\nqneVGj",
  "gYbiFnKuZ1EzD\nkEkzoq6hSinh0LBj/YGt5\n0ydtIlLs7qruYkgazfvOj\nqneVGjrlNHkAWLMO5adQRZ\nEs4CI5YwyHJbPoYBJ76Ju\nFWhsCrIwuznadBtOzMRvDY\nnGeyXrdekfSfMZQRE4Dd\nZ74FUyHv6mvp3PZnyTmrfV\nPgE38Mk9U9hOVxM6xZIzC\nqNjalZp0rZNJsQShPz7um6\nY1D5ZnoDtAE8KYrc6Git7\nRbdQmWrAkPb8FQ81Lyw",
  "qNjalZp0rZNJsQShPz7um6\nY1D5ZnoDtAE8KYrc6Git7\nRbdQmWrAkPb8FQ81Lywx96\nP/LJUbVsto35R7IJFRVl5\nqrIhP9HRSO47uD1BRE8eal\nEkweBevJSCed3NHUsxwvb\nROq5g4JQTAp9gba/iFX3mD\nqCO5smqK8QMPXCNxMKTXI\nUdWUTMDJ8wxXUsYBCNMiwG\nWMo06LMOTn5ofUMkVo3p8\nVcmItV94QqjdA9b3A5PwrK\ncHE45cHq",
  "UsYBCNMiwG\nWMo06LMOTn5ofUMkVo3p8\nVcmItV94QqjdA9b3A5PwrK\ncHE45cHqCMBk0+g7RUI\n5ajZE7MlE5eDQsNW8y1+s\npb4pOK+anG2170C+YnTIM\n+enxBp6PmFjUkaguGVx1i\nWJ5WgP6pov17d7Vm28+o4\ns7djhuk1J6m176bYd7iU94\nKebjt5uEo9Y1JGoraH1C\nOWoz2oy53HTdcoHK7blKTe\nWR6dtsOdm2j5R7tjrpm5T\nUrl",
  "5uEo9Y1JGoraH1C\nOWoz2oy53HTdcoHK7blKTe\nWR6dtsOdm2j5R7tjrpm5T\nUrlyNz2pXLYhLCoqaidYpr\nwGIlNCItJ2bXgN1Z2BFw8\nulYTwmK/EF3NBLA04hIPoQ\nlhsdnCXbONYXToW6VSa\nzMTKbEBYfsQSPuglhMaZi7\nBRPWJYhsQmRPI5xHsc0jx\nmWMpeEZyRzAhZUq4FlY/T\nrmQCWJqg1iaOxqAHMlWow\nTaI5YKuvMK58hRaxYq",
  "x\nmWMpeEZyRzAhZUq4FlY/T\nrmQCWJqg1iaOxqAHMlWow\nTaI5YKuvMK58hRaxYqu4oG\nr4cElDWuGKjQBLG2RPeYP\nt5ybLMAphtsV5IzgayMJr\nCPnT51Znd/QVSRO7kgurD\n0gtJzS8p3bd0n9LcUvJE\nETPLSVPJ0F0ZukZpXuW7l\nFaWlpSOrB0QGlkaUTpQ0sf\nUhpaGlK6ZukapdpSckcKV\nwRLdykdWzqm9MDSA0pfWPq\nC0seWPqb0p",
  "aUTpQ0sf\nUhpaGlK6ZukapdpSckcKV\nwRLdykdWzqm9MDSA0pfWPq\nC0seWPqb0paUvKX1j6RtK\n71t6n1JmKaN03dJ1Srml5N\nVBEK1aukpYCl59oO9Zm\nf0szSjNIHlj6gdGQpeSqG6\n5ml5PYGLoyWSkqfWPqEUm\nEpeX4LomeWPqM0sTSh9Kml\nTyl9belrSh9Z+ojS2FLyb\ngDuTizdodS+BaoKSrct3ab\n01NJT93sBPp/GwLUwt2wF\nW5",
  "elrSh9Z+ojS2FLyb\ngDuTizdodS+BaoKSrct3ab\n01NJT93sBPp/GwLUwt2wF\nW5SmlqaUblhKnhTgVsLSE3\nI/Gan2rDZ720TOa5Gacwd\nrMz47muQ8UnPuYO3ZaXY0O\nT9Fas7HpOvre/MXKZBSON\nMfLyt4LewtLB3u7fyU+/u\n9t2le6vtG9qr3tfeN963\nor3s3fPe+z1vYEXen96f3l\n/e/8s/roYLIrFk0Z950p7\nxit>zJde57Oo/wPj",
  "63\nor3s3fPe+z1vYEXen96f3l\n/e/8s/roYLIrFk0Z950p7\nxit>zJde57Oo/wPjzuXS</late\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3",
  "Notation change #1\n<latexit sha\n1_base64=\"OexYJKlfph/Yp\nHs7sx68KlEAH9c=\">AXC\n3iclZhbU9w2FICX9JbSG2mn\nvOTFU5pOp0ZlqSXl84kEHK\nDFAgskLCEkb2yV0GWjS3DEs\n/+hE5/TN86fe2P6A/pe49s7\nwqfIx6yM8mK8326Hcm21n4\nqRa6Xlv6dufbOu+9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd",
  "9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd8/41kuErWrL1J\n+FLNIiVAETEPoeO6P4XHZH\nXvf/Or1Yz8ZlWx82NdDrhmE\nl8be970r+54dOT1+7NQYdl\ndYblVYflShTvuCndaFe6YCs\ndzC0uLS9XHo4VuU1joNJ+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI",
  "+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI39m5BZOCFS\nQb/lPaq6OUaJYvz/CL2wYyZ\nHuaYmaCLHRY6/OWoFCotNF\ndB3VFYSE8nlkGbyAyHmh5A\nQUWZALG6gVDlrFAw2LN9hU/\nD5I4ZmpQ9lfWtsdl3+eRUCU\n/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBi",
  "/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBieI5tGn\ny4deF1HYqBJwW+MPhjPx6\nRpXkEOWlpL4kGhVTyUctaJ\nRYsZdxSdkDxvFueAVxnsAow\nVPjiaA12UqbGk3qaj3QWl7\nmJ4R4ypiJedQFTDpg0M2obq\npASqgYt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab",
  "Yt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab8j\nFMOPZMxK0KhVBNuZWlvjt\nvlMTwXtzlML10vbWSpL+M4Y\nyYgJw9ZlvwVTA2/pqMrW9SX\nLOKt8U+MgbwmK1q7Asqc16\nQRm1cTG1KxyhUyaLQhlyXn\nbNKNxqDwV7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\ny",
  "7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\nyITfoqEBPMjw/oIXrxEosW\nDQLV4iYT7O1o6luGNbSLV2\nkFBKCaFvkCXv4hUu04VwYN\nYjRWCJh24ZsJhRY5DNuyCRg\nZvuGR7NhAZpkUM8xkEleZJ\nzc/NB+hkilm9tiJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzX",
  "iJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzXf3VktdFpxXx0/WmP\nxgXrE4RBPz0eB2vR0Qs6kjU\nFpyBnG1JYjn6g7am2/XyMr\n1V9+RrR05XLcpSbvNKN2w\n71iBPx0wzHaDeIRizoStdWM\nkHrEcvQHbnzuOGahcN1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B",
  "1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B31jZEfDwaFt\n1CItbuWhrJoClAZd4CnUIi/\nUl3DabGFY3HOqGW2UyHSKzD\nmHxEYvxrOsQFiMqRk7xhKUp\nEusQyeMQ53FI85hiKXVJeE\nVSx4qQLeXaUNkwaUsmgKUR6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzI",
  "R6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzIvNxiuGY5U\npyKpCV0gRuYWeLOpPTnx+W\n5CTnhxeWXlB6buk5pfuW7lO\naWUp+Efjhc0vJrxM/PLP0jN\nI9S/coLSwtKO1Z2qM0tDSk9\nKGlDykNLA0oXbV0lVJtKTmR\nwhPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jV",
  "whPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jVJuKXl1\n4Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nl\npLjDTwYLZWUPrH0CaXCUvL7\nzQ+fWfqM0tjSmNKnlj6l9LW\nlryl9ZOkjSiNLybsBOJ1Yuk\nOpfQtU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42",
  "tU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd\n7Dm7jSpTe5PoZryIRn62t70\nRQqkFO70x3MLXfwWlhb2lhe\n7Py3e3b67cG+leUN7vXOz8\n1Xn206383PnXudxZ6vT6wSd\n/2Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 +",
  "Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"/Ce9HI1M4ktH7enbrhPJjvhwRDE=\">AXWHiclZhbU9w2FMeX3pLQG0mn5aEvn\njKZ6bTpzm6SXl46k0DIDVIgsECSN7Za+CLBtbhiWe/aDt9MP0yOtd4XPEQ3cGLJ/fX0fS0ZEsO8ikKHSv9/",
  "DVIgsECSN7Za+CLBtbhiWe/aDt9MP0yOtd4XPEQ3cGLJ/fX0fS0ZEsO8ikKHSv9/fCRx9/8ulnN27eWvz8iy+/+nrp9p39Ii3zkA/CVKb5YcAKLoXiAy205IdZzlkSH4QnK4ZfnDO80Kkak9fZv\nw4YbESkQiZBtPJ0r+eH/BYqCpImM7FeLojU6q/sTzfVO4Pys8AOBzNbyi+9PzkyAdV34QeWziSx7pI+TM1yOuGfjrTcDN7O5+6+5Bb9L2/PN",
  "4Pys8AOBzNbyi+9PzkyAdV34QeWziSx7pI+TM1yOuGfjrTcDN7O5+6+5Bb9L2/PN1TvpQbe7j6s0DuLnqYeznIh7p45OlV63V/8Wug3hZVO89s+uf3t0B+mYZl\nwpUPJiuKo38v0cVyLULJ4t+WfCMhacs5kdQVCzhxXFVT8PEuwuWoRelOfwp7dXWqzUqlhTFZRKAEjo6KjAzRhc7KnX0x3ElVFZqrsJpQ1EpPZ16Zk69och5qOUlFiYC+irF45",
  "TFZRKAEjo6KjAzRhc7KnX0x3ElVFZqrsJpQ1EpPZ16Zk69och5qOUlFiYC+irF45YzkINM7/oK34RpknCIEL+6vrOpGoCzM/KO\ngsmk7ZmvdaYgF6nWH2xN/ciNE/EB06c1BLj5BoBjydVxbtxFwPBAYguJyBVvACfJj6Qcn1EIesl4Mom5esJca0jyEmLdlbIoNCJvm4pVojKpjKpCXZBYn3fUM4JCIoVfnY8jRHOxmTE1m9TQf6zypCmPDLeRM",
  "bIoNCJvm4pVojKpjKpCXZBYn3fUM4JCIoVfnY8jRHOxmTE1m9TQf6zypCmPDLeRMxbxuAoYcMm\nlG1FaoUkqoGrZUf2HVa6ZOm8ClWd3V3FiQai9va3RO46KGbU1tQSpIwritqi1IJWGPGrKEQZSb8gkMOPGMxS0VCksFScztPA3abWfGgnNznMF6aevWKxL+c4YiYgyw+sxVMBXytnwtnau9WXDOa70p8LE3gslqV2F5PB3WrBE\nYVWObUGUdK6",
  "+c4YiYgyw+sxVMBXytnwtnau9WXDOa70p8LE3gslqV2F5PB3WrBE\nYVWObUGUdK6Sk0QJTnl60laY3DinPRHuAxoAXZkLFV2R3atLkLG7N+Doeal5Ee/dH/l4+OqZ5aN+UeiCY6KMnM5Mub/4WgIT0WcX2DBk5dKNHlgqCcvlbC/o6ljOU5sY6nDgpCMSn0JVr+IlbtOrUFdzZNUF/BYPzClQmF\nJjmK2mJjMGK4wvPdkUAhGmQ4HWMo06LMOdn8",
  "Vr+IlbtOrUFdzZNUF/BYPzClQmF\nJjmK2mJjMGK4wvPdkUAhGmQ4HWMo06LMOdn8UD6DpZabTEX5mHV3lClEbT3DS7ntaAMD4dzfk31AEU0mMYzSEs1ZDkK5thM6fidX2hYq7VX0/5tOhUxfxso2kP+gWzU4YhPzvZwPMRExXVSOQLDlROX5KoHO2Br3m6Xu1Zt\nfHuJ5LasUPrVkrit+mlW+3QXtMDfrbp6O0m0REV1Ujkq+kh1RGVoz3w5Y7jp",
  "1Zt\nfHuJ5LasUPrVkrit+mlW+3QXtMDfrbp6O0m0REV1Ujkq+kh1RGVoz3w5Y7jpmsUDq1bKYnfWRydaod2rkTpH+2Zw6k5JqVyaI59qfSnJizUVKidwjThMRJOTViYlG0V3GPJroCHR1s1NWHhdiHaMmPAoiGXeAhTExZOl3Bb2d\niwdNMh3XRLmcxGSDk1YeEzluBRT01YGFNh7BSesixDwqmJxHGE4zicywKHOJ8IxkjhkhKeVKqHyUtkXGgE",
  "YeEzluBRT01YGFNh7BSesixDwqmJxHGE4zicywKHOJ8IxkjhkhKeVKqHyUtkXGgEVj1NrY0Rj0QKYKNdgYsbigmVc4M0+hLFY0iweuhgfXNKwZcmgMWLRF1pjnbzkXWYBDbF6lHUHOBFJlNIDbWLN\nNbPTXxBV5CQXRJeWXlJ6YekFpQeWHlCaW0reCILotaXk7SIzi09p3Tf0n1KS0tLSgeWDiNLI0ofWrpU0pDS0NK1yxdo1RbSk6k8ESwdI/S",
  "Xk7SIzi09p3Tf0n1KS0tLSgeWDiNLI0ofWrpU0pDS0NK1yxdo1RbSk6k8ESwdI/SkaUjSg8tPaT0jaVvKH1u6XNK31r6ltIPln6g9LGljyljJK1y1dp5RbSj4dB\nNGqpauUBpaSdz9Ya5ZuU5pZmlH6xNInlA4tJW/F8DyzlBxv4MFoqaT0haUvKBWkve3IHpl6StKE0sTSl9a+pLS95a+p/SZpc8ojS0l3wbgdGLpLqX2K1BVULpj6Q6lZ5aeub",
  "pl6StKE0sTSl9a+pLS95a+p/SZpc8ojS0l3wbgdGLpLqX2K1BVULpj6Q6lZ5aeub8L8Pk0Bq7E3LIOtihNLU0p3bCUvCnAUcLSU3\nKejFSzq82+NpF9LVJz7mBNxGe1ScwjNecO1uxOs9pkf4rUnI9I19f35x9SIKSw058srfTxV1ha2L/f7f/WfbjzcOXRavOF9mbn+84PnR87/c7vnUed53tzqATLqwujBbOFvLv/lnuLN9YvjWVfrTQ1Pm0/o",
  "vOF9mbn+84PnR87/c7vnUed53tzqATLqwujBbOFvLv/lnuLN9YvjWVfrTQ1Pm0/ot3/kPToYiAQ=\n=</latexit>2\n4\nh1\nh2\nh3\n3\n5 = a\n2\n4\n2\n4\n\u271310\n\u271320\n\u271330\n3\n5 +\n2\n4\n\u271311\n\u271321\n\u271331\n3\n5 x\n3\n5\n<latexit sha\n1_base64=\"eo6xbZtUDEx9R\nMeyKb8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6fGSmcSO",
  "b8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6fGSmcSO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/jAlwAJEUvkSLl5b+7d\n746Obtz65/encZ59/8eVXd\n+5+vZ/FRerzPT+WcXrosYxL\nofheLnLJD5OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0eOG\n3nx",
  "OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0eOG\n3nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6eO687p3H1b7v6V3H2\nUu49y96/PbDlHlzJPUC5By\nj3YJr75M7C0uJS9efQRq9pL\nHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEuQ\neRoRP",
  "WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEuQ\neRoRPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7igopJPHjp5+Zyh\nS7ufyEhrMTwWM1fFHLGV+Dot\nkzlX8wo+jiKlh6S6vbk1K1+\nOhUCU/K6oFM5m0ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE\nMcur6eI",
  "ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE\nMcur6eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1lBxTHuedowPMU\nZgGCh8czcFOwtRkelzOx3k\nalZmO4R5SpkJedQGn7DOpz6\nhtqEJKONRvWX9ia5up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\ntE/ghCNH",
  "up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\ntE/ghCNHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BVM+b+sr8cx2p\nsU5r3zd4GNnBJPVPoSlYX1a\n07grJrYhJpVrZBJqwWhNL5\nom3o0FpUnon2COoA3XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3+H4mGcA",
  "XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3+H4mGcAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7WOqCB5sHK\nGxQkDnhU8mFJrkIGjLOqBl+\nIRHAcsC8tFJ+vU5+jLOipST\nix9azxCpdH1ZTIW+WbUvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFbLu/mvK6",
  "UvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFbLu/mvK6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEf\nCzdcto14lHLOpIlKsZIfWIZ\nekPctnruG47C4trNyXJO62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGhb8B0rOw",
  "62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGhb8B0rOwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk1iEsPmcR\nPus6hMWQiqFVPGVJgsQ6ROo\n4wnUc0TomWEpsEp6RxDIjZE\nnZFlQ6ituSDmBpjHobWzqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth3WQeLjE",
  "zqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth3WQeLjE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5QmhpKfhF4\nwbah5NeJF5wbek7pvqH7lBaG\nFpTuGbpHaWBoQOkzQ59R6hv\nqU7pi6AqluaHkiRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoauUckPJq",
  "uUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoauUckPJqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/gxmiop\nPSloS8pFYaS329e8NrQ15RG\nhkaUvjL0FaXvDX1P6XNDn1M\naGkreDcDTiaE7lJq3QGVG6Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc1aZvm8h",
  "Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc1aZvm8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/qTOw\ns9/BaWNvb7i73fFh9uPVx4v\nNy8ob3d+a7zfenTq/ze+dx\n50Vns7PX8btR96/u391/5m/\nf/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1 +  12h2 +",
  "f/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]\n<latexit sh\na1_base64=\"qvab5eVFtFK\nvws6yR9bTatgWmaY=\">A\nAWuniclZhbU9w2FICdXtP0\nRtopL3xlMmk06Y7kKSXh\n7aTQMgNUpbAglLGNkrexV\nk2dgyLPHsv+mv",
  "ICdXtP0\nRtopL3xlMmk06Y7kKSXh\n7aTQMgNUpbAglLGNkrexV\nk2dgyLPHsv+mv6Wv70n/T\nI9u7is8RD90ZWO35PutyJP\nkWZFIUen53yvPve+x9\n8ePWjax9/8ulny9c/2KvS\nMs85IMwlWl+ELCS6H4QA\nst+UGWc5YEku8HJ2uG75/x\nvBCp2tUXGT9KWKxEJEKmI\nXS8PvFTf83f5iNxc3jan\n6fVtamY7r/7Pft+vft+e/\n79S/70yPF5",
  "KWKxEJEKmI\nXS8PvFTf83f5iNxc3jan\n6fVtamY7r/7Pft+vft+e/\n79S/70yPF5aWe8v1x6eFlb\naw5LWf/vH1r0bDURqWCVc\n6lKwoDleWM31UsVyLUPLpt\nWFZ8IyFJyzmh1BULOHFUV\nUPdOrfgMjIj9Ic/pT26+jb\nR1QsKYqLJAzYXpcYGaCL\nnZY6uiXo0qorNRchU1DUSl\n9nfoma/5I5DzU8gIKLMwF\n9NUPxyxnoYbcXhsqfh6mSc",
  "uiXo0qorNRchU1DUSl\n9nfoma/5I5DzU8gIKLMwF\n9NUPxyxnoYbcXhsqfh6mSc\nLUqBqurm9Pq2HAY6Eqflr\nWeZ5Ou8567XAoXmasPtmd1\nyI0T8QbTiqpFVPJQKPp1\nXFe3EPA8EBiB4nIFW8gDpN\nfoLIX0EU1pUEDxIJ9C5y\nH8+JVUrzWPISUd7STQoZJ\nPOtYasWAqk46yA4rv3/AN\n4DqHWYCuwhdHc7CTMTWdHa\nf5ROdJVZgYbiFnK",
  "oZJ\nPOtYasWAqk46yA4rv3/AN\n4DqHWYCuwhdHc7CTMTWdHa\nf5ROdJVZgYbiFnKuZ1EzD\nkEkzoq6hSinh0LBj/YGt5\n0ydtIlLs7qruYkgazfvOj\nqneVGjrlNHkAWLMO5adQRZ\nEs4CI5YwyHJbPoYBJ76Ju\nFWhsCrIwuznadBtOzMRvDY\nnGeyXrdekfSfMZQRE4Dd\nZ74FUyHv6mvp3PZnyTmrfV\nPgE38Mk9U9hOVxM6xZIzC\nqNjalZp",
  "fSfMZQRE4Dd\nZ74FUyHv6mvp3PZnyTmrfV\nPgE38Mk9U9hOVxM6xZIzC\nqNjalZp0rZNJsQShPz7um6\nY1D5ZnoDtAE8KYrc6Git7\nRbdQmWrAkPb8FQ81Lywx96\nP/LJUbVsto35R7IJFRVl5\nqrIhP9HRSO47uD1BRE8eal\nEkweBevJSCed3NHUsxwvb\nROq5g4JQTAp9gba/iFX3mD\nqCO5smqK8QMPXCNxMKTXI\nUdWUTMDJ8wxXUsYBCNM",
  "ROq5g4JQTAp9gba/iFX3mD\nqCO5smqK8QMPXCNxMKTXI\nUdWUTMDJ8wxXUsYBCNMiwG\nWMo06LMOTn5ofUMkVo3p8\nVcmItV94QqjdA9b3A5PwrK\ncHE45cHqCMBk0+g7RUI\n5ajZE7MlE5eDQsNW8y1+s\npb4pOK+anG2170C+YnTIM\n+enxBp6PmFjUkaguGVx1i\nWJ5WgP6pov17d7Vm28+o4\ns7djhuk1J6m176bYd7iU94\nKebjt5uEo9Y1",
  "uGVx1i\nWJ5WgP6pov17d7Vm28+o4\ns7djhuk1J6m176bYd7iU94\nKebjt5uEo9Y1JGoraH1C\nOWoz2oy53HTdcoHK7blKTe\nWR6dtsOdm2j5R7tjrpm5T\nUrlyNz2pXLYhLCoqaidYpr\nwGIlNCItJ2bXgN1Z2BFw8\nulYTwmK/EF3NBLA04hIPoQ\nlhsdnCXbONYXToW6VSa\nzMTKbEBYfsQSPuglhMaZi7\nBRPWJYhsQmRPI5xHsc0jx\nmWMpe",
  "XbONYXToW6VSa\nzMTKbEBYfsQSPuglhMaZi7\nBRPWJYhsQmRPI5xHsc0jx\nmWMpeEZyRzAhZUq4FlY/T\nrmQCWJqg1iaOxqAHMlWow\nTaI5YKuvMK58hRaxYqu4oG\nr4cElDWuGKjQBLG2RPeYP\nt5ybLMAphtsV5IzgayMJr\nCPnT51Znd/QVSRO7kgurD\n0gtJzS8p3bd0n9LcUvJE\nETPLSVPJ0F0ZukZpXuW7l\nFaWlpSOrB0QGlkaUTpQ0s",
  "0gtJzS8p3bd0n9LcUvJE\nETPLSVPJ0F0ZukZpXuW7l\nFaWlpSOrB0QGlkaUTpQ0sf\nUhpaGlK6ZukapdpSckcKV\nwRLdykdWzqm9MDSA0pfWPq\nC0seWPqb0paUvKX1j6RtK\n71t6n1JmKaN03dJ1Srml5N\nVBEK1aukpYCl59oO9Zm\nf0szSjNIHlj6gdGQpeSqG6\n5ml5PYGLoyWSkqfWPqEUm\nEpeX4LomeWPqM0sTSh9Kml\nTyl9belrSh9Z",
  "peSqG6\n5ml5PYGLoyWSkqfWPqEUm\nEpeX4LomeWPqM0sTSh9Kml\nTyl9belrSh9Z+ojS2FLyb\ngDuTizdodS+BaoKSrct3ab\n01NJT93sBPp/GwLUwt2wF\nW5SmlqaUblhKnhTgVsLSE3\nI/Gan2rDZ720TOa5Gacwd\nrMz47muQ8UnPuYO3ZaXY0O\nT9Fas7HpOvre/MXKZBSON\nMfLyt4LewtLB3u7fyU+/u\n9t2le6vtG9qr3tfeN963\nor3s",
  "7HpOvre/MXKZBSON\nMfLyt4LewtLB3u7fyU+/u\n9t2le6vtG9qr3tfeN963\nor3s3fPe+z1vYEXen96f3l\n/e/8s/roYLIrFk0Z950p7\nxit>zJde57Oo/wPjzuXS</late\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\nVector Notation",
  "Notation change #1\n<latexit sha\n1_base64=\"OexYJKlfph/Yp\nHs7sx68KlEAH9c=\">AXC\n3iclZhbU9w2FICX9JbSG2mn\nvOTFU5pOp0ZlqSXl84kEHK\nDFAgskLCEkb2yV0GWjS3DEs\n/+hE5/TN86fe2P6A/pe49s7\nwqfIx6yM8mK8326Hcm21n4\nqRa6Xlv6dufbOu+9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd",
  "9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd8/41kuErWrL1J\n+FLNIiVAETEPoeO6P4XHZH\nXvf/Or1Yz8ZlWx82NdDrhmE\nl8be970r+54dOT1+7NQYdl\ndYblVYflShTvuCndaFe6YCs\ndzC0uLS9XHo4VuU1joNJ+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI",
  "+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI39m5BZOCFS\nQb/lPaq6OUaJYvz/CL2wYyZ\nHuaYmaCLHRY6/OWoFCotNF\ndB3VFYSE8nlkGbyAyHmh5A\nQUWZALG6gVDlrFAw2LN9hU/\nD5I4ZmpQ9lfWtsdl3+eRUCU\n/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBi",
  "/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBieI5tGn\ny4deF1HYqBJwW+MPhjPx6\nRpXkEOWlpL4kGhVTyUctaJ\nRYsZdxSdkDxvFueAVxnsAow\nVPjiaA12UqbGk3qaj3QWl7\nmJ4R4ypiJedQFTDpg0M2obq\npASqgYt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab",
  "Yt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab8j\nFMOPZMxK0KhVBNuZWlvjt\nvlMTwXtzlML10vbWSpL+M4Y\nyYgJw9ZlvwVTA2/pqMrW9SX\nLOKt8U+MgbwmK1q7Asqc16\nQRm1cTG1KxyhUyaLQhlyXn\nbNKNxqDwV7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\ny",
  "7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\nyITfoqEBPMjw/oIXrxEosW\nDQLV4iYT7O1o6luGNbSLV2\nkFBKCaFvkCXv4hUu04VwYN\nYjRWCJh24ZsJhRY5DNuyCRg\nZvuGR7NhAZpkUM8xkEleZJ\nzc/NB+hkilm9tiJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzX",
  "iJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzXf3VktdFpxXx0/WmP\nxgXrE4RBPz0eB2vR0Qs6kjU\nFpyBnG1JYjn6g7am2/XyMr\n1V9+RrR05XLcpSbvNKN2w\n71iBPx0wzHaDeIRizoStdWM\nkHrEcvQHbnzuOGahcN1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B",
  "1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B31jZEfDwaFt\n1CItbuWhrJoClAZd4CnUIi/\nUl3DabGFY3HOqGW2UyHSKzD\nmHxEYvxrOsQFiMqRk7xhKUp\nEusQyeMQ53FI85hiKXVJeE\nVSx4qQLeXaUNkwaUsmgKUR6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzI",
  "R6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzIvNxiuGY5U\npyKpCV0gRuYWeLOpPTnx+W\n5CTnhxeWXlB6buk5pfuW7lO\naWUp+Efjhc0vJrxM/PLP0jN\nI9S/coLSwtKO1Z2qM0tDSk9\nKGlDykNLA0oXbV0lVJtKTmR\nwhPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jV",
  "whPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jVJuKXl1\n4Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nl\npLjDTwYLZWUPrH0CaXCUvL7\nzQ+fWfqM0tjSmNKnlj6l9LW\nlryl9ZOkjSiNLybsBOJ1Yuk\nOpfQtU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42",
  "tU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd\n7Dm7jSpTe5PoZryIRn62t70\nRQqkFO70x3MLXfwWlhb2lhe\n7Py3e3b67cG+leUN7vXOz8\n1Xn206383PnXudxZ6vT6wSd\n/2Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 +",
  "Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sha1_base64=\"/Ce9HI1M4ktH7enbrhPJjvhwRDE=\">AXWHiclZhbU9w2FMeX3pLQG0mn5aEvn\njKZ6bTpzm6SXl46k0DIDVIgsECSN7Za+CLBtbhiWe/aDt9MP0yOtd4XPEQ3cGLJ/fX0fS0ZEsO8ikKHSv9/",
  "DVIgsECSN7Za+CLBtbhiWe/aDt9MP0yOtd4XPEQ3cGLJ/fX0fS0ZEsO8ikKHSv9/fCRx9/8ulnN27eWvz8iy+/+nrp9p39Ii3zkA/CVKb5YcAKLoXiAy205IdZzlkSH4QnK4ZfnDO80Kkak9fZv\nw4YbESkQiZBtPJ0r+eH/BYqCpImM7FeLojU6q/sTzfVO4Pys8AOBzNbyi+9PzkyAdV34QeWziSx7pI+TM1yOuGfjrTcDN7O5+6+5Bb9L2/PN",
  "4Pys8AOBzNbyi+9PzkyAdV34QeWziSx7pI+TM1yOuGfjrTcDN7O5+6+5Bb9L2/PN1TvpQbe7j6s0DuLnqYeznIh7p45OlV63V/8Wug3hZVO89s+uf3t0B+mYZl\nwpUPJiuKo38v0cVyLULJ4t+WfCMhacs5kdQVCzhxXFVT8PEuwuWoRelOfwp7dXWqzUqlhTFZRKAEjo6KjAzRhc7KnX0x3ElVFZqrsJpQ1EpPZ16Zk69och5qOUlFiYC+irF45",
  "TFZRKAEjo6KjAzRhc7KnX0x3ElVFZqrsJpQ1EpPZ16Zk69och5qOUlFiYC+irF45YzkINM7/oK34RpknCIEL+6vrOpGoCzM/KO\ngsmk7ZmvdaYgF6nWH2xN/ciNE/EB06c1BLj5BoBjydVxbtxFwPBAYguJyBVvACfJj6Qcn1EIesl4Mom5esJca0jyEmLdlbIoNCJvm4pVojKpjKpCXZBYn3fUM4JCIoVfnY8jRHOxmTE1m9TQf6zypCmPDLeRM",
  "bIoNCJvm4pVojKpjKpCXZBYn3fUM4JCIoVfnY8jRHOxmTE1m9TQf6zypCmPDLeRMxbxuAoYcMm\nlG1FaoUkqoGrZUf2HVa6ZOm8ClWd3V3FiQai9va3RO46KGbU1tQSpIwritqi1IJWGPGrKEQZSb8gkMOPGMxS0VCksFScztPA3abWfGgnNznMF6aevWKxL+c4YiYgyw+sxVMBXytnwtnau9WXDOa70p8LE3gslqV2F5PB3WrBE\nYVWObUGUdK6",
  "+c4YiYgyw+sxVMBXytnwtnau9WXDOa70p8LE3gslqV2F5PB3WrBE\nYVWObUGUdK6Sk0QJTnl60laY3DinPRHuAxoAXZkLFV2R3atLkLG7N+Doeal5Ee/dH/l4+OqZ5aN+UeiCY6KMnM5Mub/4WgIT0WcX2DBk5dKNHlgqCcvlbC/o6ljOU5sY6nDgpCMSn0JVr+IlbtOrUFdzZNUF/BYPzClQmF\nJjmK2mJjMGK4wvPdkUAhGmQ4HWMo06LMOdn8",
  "Vr+IlbtOrUFdzZNUF/BYPzClQmF\nJjmK2mJjMGK4wvPdkUAhGmQ4HWMo06LMOdn8UD6DpZabTEX5mHV3lClEbT3DS7ntaAMD4dzfk31AEU0mMYzSEs1ZDkK5thM6fidX2hYq7VX0/5tOhUxfxso2kP+gWzU4YhPzvZwPMRExXVSOQLDlROX5KoHO2Br3m6Xu1Zt\nfHuJ5LasUPrVkrit+mlW+3QXtMDfrbp6O0m0REV1Ujkq+kh1RGVoz3w5Y7jp",
  "1Zt\nfHuJ5LasUPrVkrit+mlW+3QXtMDfrbp6O0m0REV1Ujkq+kh1RGVoz3w5Y7jpmsUDq1bKYnfWRydaod2rkTpH+2Zw6k5JqVyaI59qfSnJizUVKidwjThMRJOTViYlG0V3GPJroCHR1s1NWHhdiHaMmPAoiGXeAhTExZOl3Bb2d\niwdNMh3XRLmcxGSDk1YeEzluBRT01YGFNh7BSesixDwqmJxHGE4zicywKHOJ8IxkjhkhKeVKqHyUtkXGgE",
  "YeEzluBRT01YGFNh7BSesixDwqmJxHGE4zicywKHOJ8IxkjhkhKeVKqHyUtkXGgEVj1NrY0Rj0QKYKNdgYsbigmVc4M0+hLFY0iweuhgfXNKwZcmgMWLRF1pjnbzkXWYBDbF6lHUHOBFJlNIDbWLN\nNbPTXxBV5CQXRJeWXlJ6YekFpQeWHlCaW0reCILotaXk7SIzi09p3Tf0n1KS0tLSgeWDiNLI0ofWrpU0pDS0NK1yxdo1RbSk6k8ESwdI/S",
  "Xk7SIzi09p3Tf0n1KS0tLSgeWDiNLI0ofWrpU0pDS0NK1yxdo1RbSk6k8ESwdI/SkaUjSg8tPaT0jaVvKH1u6XNK31r6ltIPln6g9LGljyljJK1y1dp5RbSj4dB\nNGqpauUBpaSdz9Ya5ZuU5pZmlH6xNInlA4tJW/F8DyzlBxv4MFoqaT0haUvKBWkve3IHpl6StKE0sTSl9a+pLS95a+p/SZpc8ojS0l3wbgdGLpLqX2K1BVULpj6Q6lZ5aeub",
  "pl6StKE0sTSl9a+pLS95a+p/SZpc8ojS0l3wbgdGLpLqX2K1BVULpj6Q6lZ5aeub8L8Pk0Bq7E3LIOtihNLU0p3bCUvCnAUcLSU3\nKejFSzq82+NpF9LVJz7mBNxGe1ScwjNecO1uxOs9pkf4rUnI9I19f35x9SIKSw058srfTxV1ha2L/f7f/WfbjzcOXRavOF9mbn+84PnR87/c7vnUed53tzqATLqwujBbOFvLv/lnuLN9YvjWVfrTQ1Pm0/o",
  "vOF9mbn+84PnR87/c7vnUed53tzqATLqwujBbOFvLv/lnuLN9YvjWVfrTQ1Pm0/ot3/kPToYiAQ=\n=</latexit>2\n4\nh1\nh2\nh3\n3\n5 = a\n2\n4\n2\n4\n\u271310\n\u271320\n\u271330\n3\n5 +\n2\n4\n\u271311\n\u271321\n\u271331\n3\n5 x\n3\n5\n<latexit sha1_base64=\"OWbyUlTVn0iy+klb+PXSAXibkU=\">AX0XiclZhJb9w2FIDHXVN3S1oUPvQi1\nEhatKkxY6fLpUBix9ns1E6",
  "SAXibkU=\">AX0XiclZhJb9w2FIDHXVN3S1oUPvQi1\nEhatKkxY6fLpUBix9ns1E68JpZjUBpKw5iZC32OIKAotf+pP6SHnt/0QfJc1w9B59qIHE9Ps+PVKPpDYvkSL+/2/5t56+513v/2gfzH3708SefXr/x2X4WF6nP9/xYxumhxzIuheJ7ucglP0xSziJP8gPvdE3zg3OeZi\nJWu/lwo8jFioRCJ/lEDq5MbfruB4PhSq9iOWpGFfzujrk3JQO",
  "8gPvdE3zg3OeZi\nJWu/lwo8jFioRCJ/lEDq5MbfruB4PhSq9iOWpGFfzujrk3JQOa5bt5Z1SzdWKmfecbkazqi/uJEXj0vXCxWuZIH+RFK5yaZgHT9Ol/zx3K/mrZXoD2b0vkOj6dNAO61TZhRLfawcrOm+b1ijLM8ryjLJilJUZUrnVH\nQmkxL0lZEN1aqeXxcKsJRfnxyfbG/1K9/HNoYtI3FXvuzfXLji6E7jP0i4ir3Jcuyo0E/yY9LlubCl",
  "XxcKsJRfnxyfbG/1K9/HNoYtI3FXvuzfXLji6E7jP0i4ir3Jcuyo0E/yY9LlubCl7yad4uMJ8w/ZSE/gqZiEc+Oy3r+K+cmRIZOEKfwT+VOHZ09omRl1GHpgw0FGmQ7a2FGRBz8fl0IlRc6V3QUFNLJY\n0cvJmcoUu7n8hIazE8FjNXxRyxlfg5Lbt5V/MKPo4hBhdzV9WdV2ZaVnxX18quqrNeO7qgVxmrj3enWUTOI/GkyS1opNcIfCwKku+FC",
  "o4hBhdzV9WdV2ZaVnxX18quqrNeO7qgVxmrj3enWUTOI/GkyS1opNcIfCwKku+FC5hIDgAscQJiBXPIKeuDyz0AaKw3STg0myF5xVJrXIeQk062kuiQSORfNyx1ogFUx\nl1lB1QHOemowGHheg79Xr0OZqDnYSpanJczsd5GpWZjuEeUqZCXncBp+wzqc+oa6hCSjU71i/Yus5U6dt4eKkHmqI8jaTbtOntK6qGHXqSPIgkUYdq06giwJF8chixh",
  "6hCSjU71i/Yus5U6dt4eKkHmqI8jaTbtOntK6qGHXqSPIgkUYdq06giwJF8chixhUuW2fwAlHjo7YVaGwKsjC3E5jr9t3oiN4bY4T2C9\ndb70k5T9nqCI6ALtP/xZM+byr8VT25kU57z2dYOPnRFMVvcQlobNaU06gbNqYxU161ohk1YLQml80TX1aCwqT0T3BHUAb7oiFSqY0W7XLViyOuzehlNC8mPvl/6gY+Py7eNvo/Uk1IlBWJLZEO/49EQ",
  "3BHUAb7oiFSqY0W7XLViyOuzehlNC8mPvl/6gY+Py7eNvo/Uk1IlBWJLZEO/49EQ7gd4/UFETx5sUST\nB4F68mIJ13c0dSzFC1tH6rmDhlBMivwSbX8Rqu4xdQPNo7QWCGg8JvJhSa5CDoyjqgZfgNDxaWBeSjk/Sbc/RlnBUpJxc/tJ4hUuv6spgKfbPqXlClFrXDS6nR0Ebg7n/IrDPVRr6mnFxdqyFJUzLGe0vErN8thi9l2f\nz3lTdNqhfxs",
  "ClFrXDS6nR0Ebg7n/IrDPVRr6mnFxdqyFJUzLGe0vErN8thi9l2f\nz3lTdNqhfxso+0PxgWzU/g+PzvZwPMREos6EuWCJzlrLksS3+Qa7pcZ0dWbrz6lizt0OLaTUnytqO02xb3ihHws03LaDeJRyzqSJSrHSH1iGXpD3LZ67hpOwuLazclyTupo9W2uFMTLf9gd8Rzph+TYjnUj32xdJsQFnMq5l\nYxjniIxCaExajoWvA3VnYE3Dy6VhPC4nYm",
  "f9gd8Rzph+TYjnUj32xdJsQFnMq5l\nYxjniIxCaExajoWvA3VnYE3Dy6VhPC4nYmupoOYGnIJT6FJoTFZgt3zTaG1U2LumlXmUxGyGxCWHzInzWTQiLIRVDq3jKkgSJTYjUcYTrOKJ1TLCU2CQ8I4lRsiSsi2odBR3JR3A0hj1NrZ0BiOQsUIdtkEsZ3TlZdaVp9Aq\nVnQV79k63rui45yhDqApS2yx3y7rJPFxi/bpqKXIikJXQAm5jZ5s6k6c/",
  "aVp9Aq\nVnQV79k63rui45yhDqApS2yx3y7rJPFxi/bpqKXIikJXQAm5jZ5s6k6c/LyjJk5wXBp6SemFoReUHh6QGlqKHkj8ILnhpK3Ey84N/Sc0n1D9yktDC0o3TN0j9LA0IDSB4Y+oNQ31Kd0zdA1SnNDyRMp3BEM3aV0ZOiI0\nkNDyl9YegLSh8Z+ojSl4a+pPSNoW8ovWfoPUqZoYzSdUPXKeWGk8HXrBq6CqlnqHk3Q/2mqHblCaGJpTeN/Q",
  "a+pPSNoW8ovWfoPUqZoYzSdUPXKeWGk8HXrBq6CqlnqHk3Q/2mqHblCaGJpTeN/Q+pUNDyVsx3M8MJY83cGM0VFL62NDHlApDyfubFzw19CmlkaERpU8MfULpa0NfU/rQ0IeUhoaSbwPwdGLoDq\nXmK1CZUfrM0GeUnhl6Zv8uwKfT6NkW5pZJsEVpbGhM6Yah5E0BHiUMPSXPk4Fqr2qTr03kuhaoKbewtuKTo0nNAzXlFtZenSZHk+tToKZ8RI",
  "ah5E0BHiUMPSXPk4Fqr2qTr03kuhaoKbewtuKTo0nNAzXlFtZenSZHk+tToKZ8RIa+vj/9kAIlhSv9yfXFAf4KSxv7y0uDH5fuPLuzeHe1/UJ7rfdl76veN71B76f\ne3d6j3nZvr+fP/Tn39w/c/8u7CxcLvy28HujvjXHvN5r/Oz8Md/Mk9LA=</latexit>2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5 = a\n2\n4\n2\n4\n 10\n 20\n 30\n3\n5 +\n2\n4\n 11\n 12\n 13\n 21",
  "t>2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5 = a\n2\n4\n2\n4\n 10\n 20\n 30\n3\n5 +\n2\n4\n 11\n 12\n 13\n 21\n 22\n 23\n 31\n 32\n 33\n3\n5\n2\n4\nh1\nh2\nh3\n3\n5\n3\n5\n<latexit sha\n1_base64=\"eo6xbZtUDEx9R\nMeyKb8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6fGSmcSO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/",
  "1GNJ6fGSmcSO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/jAlwAJEUvkSLl5b+7d\n746Obtz65/encZ59/8eVXd\n+5+vZ/FRerzPT+WcXrosYxL\nofheLnLJD5OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0eOG\n3nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6e",
  "xPnh0eOG\n3nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6eO687p3H1b7v6V3H2\nUu49y96/PbDlHlzJPUC5By\nj3YJr75M7C0uJS9efQRq9pL\nHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEuQ\neRoRPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7ig",
  "zdjEuQ\neRoRPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7igopJPHjp5+Zyh\nS7ufyEhrMTwWM1fFHLGV+Dot\nkzlX8wo+jiKlh6S6vbk1K1+\nOhUCU/K6oFM5m0ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE\nMcur6eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1l",
  "AWPE\nMcur6eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1lBxTHuedowPMU\nZgGCh8czcFOwtRkelzOx3k\nalZmO4R5SpkJedQGn7DOpz6\nhtqEJKONRvWX9ia5up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\ntE/ghCNHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BV",
  "Krc\ntE/ghCNHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BVM+b+sr8cx2p\nsU5r3zd4GNnBJPVPoSlYX1a\n07grJrYhJpVrZBJqwWhNL5\nom3o0FpUnon2COoA3XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3+H4mGcAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7",
  "bI\nh3+H4mGcAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7WOqCB5sHK\nGxQkDnhU8mFJrkIGjLOqBl+\nIRHAcsC8tFJ+vU5+jLOipST\nix9azxCpdH1ZTIW+WbUvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFbLu/mvK6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65X",
  "bDFbLu/mvK6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEf\nCzdcto14lHLOpIlKsZIfWIZ\nekPctnruG47C4trNyXJO62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGhb8B0rOwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk",
  "TFqGhb8B0rOwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk1iEsPmcR\nPus6hMWQiqFVPGVJgsQ6ROo\n4wnUc0TomWEpsEp6RxDIjZE\nnZFlQ6ituSDmBpjHobWzqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth3WQeLjE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5Qmh",
  "jrth3WQeLjE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5QmhpKfhF4\nwbah5NeJF5wbek7pvqH7lBaG\nFpTuGbpHaWBoQOkzQ59R6hv\nqU7pi6AqluaHkiRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoauUckPJqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/g",
  "umqoauUckPJqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/gxmiop\nPSloS8pFYaS329e8NrQ15RG\nhkaUvjL0FaXvDX1P6XNDn1M\naGkreDcDTiaE7lJq3QGVG6Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc1aZvm8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/",
  "nA9Vc1aZvm8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/qTOw\ns9/BaWNvb7i73fFh9uPVx4v\nNy8ob3d+a7zfenTq/ze+dx\n50Vns7PX8btR96/u391/5m/\nf/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +",
  "+  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]\nVector Notation\nVector & Matrix Notation",
  "Notation change #1\n<latexit sha\n1_base64=\"OexYJKlfph/Yp\nHs7sx68KlEAH9c=\">AXC\n3iclZhbU9w2FICX9JbSG2mn\nvOTFU5pOp0ZlqSXl84kEHK\nDFAgskLCEkb2yV0GWjS3DEs\n/+hE5/TN86fe2P6A/pe49s7\nwqfIx6yM8mK8326Hcm21n4\nqRa6Xlv6dufbOu+9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd",
  "9/8H1D2\nc/+viTz+bu/H5Xp4UWcB7Q\nSKT7MBnOZdC8Z4WvKDNOMs\n9iXf909WDd8/41kuErWrL1J\n+FLNIiVAETEPoeO6P4XHZH\nXvf/Or1Yz8ZlWx82NdDrhmE\nl8be970r+54dOT1+7NQYdl\ndYblVYflShTvuCndaFe6YCs\ndzC0uLS9XHo4VuU1joNJ+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI",
  "+t4\nxtfDvqDJChirnQgWZ4fdpd\nSfVSyTItA8vFsv8h5yoITFv\nFDKCoW8/yorDI39m5BZOCFS\nQb/lPaq6OUaJYvz/CL2wYyZ\nHuaYmaCLHRY6/OWoFCotNF\ndB3VFYSE8nlkGbyAyHmh5A\nQUWZALG6gVDlrFAw2LN9hU/\nD5I4ZmpQ9lfWtsdl3+eRUCU\n/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBi",
  "/LaqFG4/bzlrlcCheZaw82Z\n2IjSPxRtOGqkU08gVAo/G\nZckXo0UMBAcgFjkBieI5tGn\ny4deF1HYqBJwW+MPhjPx6\nRpXkEOWlpL4kGhVTyUctaJ\nRYsZdxSdkDxvFueAVxnsAow\nVPjiaA12UqbGk3qaj3QWl7\nmJ4R4ypiJedQFTDpg0M2obq\npASqgYt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab",
  "Yt6zdsPWfqpElcklZD\nzUwEWbtZ29EZzYsatJ0qgiz\nYhFHbqiLIknBbGbCYQZab8j\nFMOPZMxK0KhVBNuZWlvjt\nvlMTwXtzlML10vbWSpL+M4Y\nyYgJw9ZlvwVTA2/pqMrW9SX\nLOKt8U+MgbwmK1q7Asqc16\nQRm1cTG1KxyhUyaLQhlyXn\nbNKNxqDwV7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\ny",
  "7QmaAL7oikyo8J\nJ2uyrBljXh/m2YalZIfvjD4\no98dFQumcvG/EeyCQ3lRepq\nyITfoqEBPMjw/oIXrxEosW\nDQLV4iYT7O1o6luGNbSLV2\nkFBKCaFvkCXv4hUu04VwYN\nYjRWCJh24ZsJhRY5DNuyCRg\nZvuGR7NhAZpkUM8xkEleZJ\nzc/NB+hkilm9tiJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzX",
  "iJszDqn1Dl\nUZo3ze4nNaCMjwczvgV1X2\nUb/Op58UasAylMyRWdLRq3\n6u4RJzXf3VktdFpxXx0/WmP\nxgXrE4RBPz0eB2vR0Qs6kjU\nFpyBnG1JYjn6g7am2/XyMr\n1V9+RrR05XLcpSbvNKN2w\n71iBPx0wzHaDeIRizoStdWM\nkHrEcvQHbnzuOGahcN1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B",
  "1m5K\n0O8mj03a4UxNt/3DXnETNMS\nmRA3PsS2S/DmFRU1E7xSTm\nERLrEBbjom3B31jZEfDwaFt\n1CItbuWhrJoClAZd4CnUIi/\nUl3DabGFY3HOqGW2UyHSKzD\nmHxEYvxrOsQFiMqRk7xhKUp\nEusQyeMQ53FI85hiKXVJeE\nVSx4qQLeXaUNkwaUsmgKUR6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzI",
  "R6\nm3k6AxGIBOFOmyCWM7pzsud\nO0+hXazoLu65Ou5d0bFmqE\nTwNImuca8/qbzIvNxiuGY5U\npyKpCV0gRuYWeLOpPTnx+W\n5CTnhxeWXlB6buk5pfuW7lO\naWUp+Efjhc0vJrxM/PLP0jN\nI9S/coLSwtKO1Z2qM0tDSk9\nKGlDykNLA0oXbV0lVJtKTmR\nwhPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jV",
  "whPB0l1Kh5YOKT2w9IDSF5\na+oPSxpY8pfWnpS0rfWPqG0\nvuW3qeUWcoXbN0jVJuKXl1\n4Icrlq5Q6ltKfvBtWbpFqW\npSmlDyx9QOnAUvKrGJ5nl\npLjDTwYLZWUPrH0CaXCUvL7\nzQ+fWfqM0tjSmNKnlj6l9LW\nlryl9ZOkjSiNLybsBOJ1Yuk\nOpfQtU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42",
  "tU5pRuW7pN6amlp+73A\nny6jL5rY27aBjYpTSxNKF2\n3lPxSgKOEpSfkPBmq5q42ed\ntE7muhmnIHazI+qU1yHqopd\n7Dm7jSpTe5PoZryIRn62t70\nRQqkFO70x3MLXfwWlhb2lhe\n7Py3e3b67cG+leUN7vXOz8\n1Xn206383PnXudxZ6vT6wSd\n/2Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 +",
  "Zuznw9c2v+9/k/5/+a/7t\nWr80db7otD7z/wPvt8CjQ\n=</latexit>h1 = a[\u271310 + \u271311x]\nh2 = a[\u271320 + \u271321x]\nh3 = a[\u271330 + \u271331x]\n<latexit sh\na1_base64=\"qvab5eVFtFK\nvws6yR9bTatgWmaY=\">A\nAWuniclZhbU9w2FICdXtP0\nRtopL3xlMmk06Y7kKSXh\n7aTQMgNUpbAglLGNkrexV\nk2dgyLPHsv+mv6Wv70n/T\nI9u7is8RD90ZWO35PutyJP",
  "QMgNUpbAglLGNkrexV\nk2dgyLPHsv+mv6Wv70n/T\nI9u7is8RD90ZWO35PutyJP\nkWZFIUen53yvPve+x9\n8ePWjax9/8ulny9c/2KvS\nMs85IMwlWl+ELCS6H4QA\nst+UGWc5YEku8HJ2uG75/x\nvBCp2tUXGT9KWKxEJEKmI\nXS8PvFTf83f5iNxc3jan\n6fVtamY7r/7Pft+vft+e/\n79S/70yPF5aWe8v1x6eFlb\naw5LWf/vH1r0bDURqWC",
  "6fVtamY7r/7Pft+vft+e/\n79S/70yPF5aWe8v1x6eFlb\naw5LWf/vH1r0bDURqWCVc\n6lKwoDleWM31UsVyLUPLpt\nWFZ8IyFJyzmh1BULOHFUV\nUPdOrfgMjIj9Ic/pT26+jb\nR1QsKYqLJAzYXpcYGaCL\nnZY6uiXo0qorNRchU1DUSl\n9nfoma/5I5DzU8gIKLMwF\n9NUPxyxnoYbcXhsqfh6mSc\nLUqBqurm9Pq2HAY6Eqflr\nWeZ5Ou8567",
  "8gIKLMwF\n9NUPxyxnoYbcXhsqfh6mSc\nLUqBqurm9Pq2HAY6Eqflr\nWeZ5Ou8567XAoXmasPtmd1\nyI0T8QbTiqpFVPJQKPp1\nXFe3EPA8EBiB4nIFW8gDpN\nfoLIX0EU1pUEDxIJ9C5y\nH8+JVUrzWPISUd7STQoZJ\nPOtYasWAqk46yA4rv3/AN\n4DqHWYCuwhdHc7CTMTWdHa\nf5ROdJVZgYbiFnKuZ1EzD\nkEkzoq6hSinh0LBj/YGt5\n0yd",
  "CuwhdHc7CTMTWdHa\nf5ROdJVZgYbiFnKuZ1EzD\nkEkzoq6hSinh0LBj/YGt5\n0ydtIlLs7qruYkgazfvOj\nqneVGjrlNHkAWLMO5adQRZ\nEs4CI5YwyHJbPoYBJ76Ju\nFWhsCrIwuznadBtOzMRvDY\nnGeyXrdekfSfMZQRE4Dd\nZ74FUyHv6mvp3PZnyTmrfV\nPgE38Mk9U9hOVxM6xZIzC\nqNjalZp0rZNJsQShPz7um6\nY1D5ZnoDtAE8KYrc",
  "fV\nPgE38Mk9U9hOVxM6xZIzC\nqNjalZp0rZNJsQShPz7um6\nY1D5ZnoDtAE8KYrc6Git7\nRbdQmWrAkPb8FQ81Lywx96\nP/LJUbVsto35R7IJFRVl5\nqrIhP9HRSO47uD1BRE8eal\nEkweBevJSCed3NHUsxwvb\nROq5g4JQTAp9gba/iFX3mD\nqCO5smqK8QMPXCNxMKTXI\nUdWUTMDJ8wxXUsYBCNMiwG\nWMo06LMOTn5ofUMkVo3p8\nVcmItV",
  "QMPXCNxMKTXI\nUdWUTMDJ8wxXUsYBCNMiwG\nWMo06LMOTn5ofUMkVo3p8\nVcmItV94QqjdA9b3A5PwrK\ncHE45cHqCMBk0+g7RUI\n5ajZE7MlE5eDQsNW8y1+s\npb4pOK+anG2170C+YnTIM\n+enxBp6PmFjUkaguGVx1i\nWJ5WgP6pov17d7Vm28+o4\ns7djhuk1J6m176bYd7iU94\nKebjt5uEo9Y1JGoraH1C\nOWoz2oy53HTdcoHK7blKTe",
  "jhuk1J6m176bYd7iU94\nKebjt5uEo9Y1JGoraH1C\nOWoz2oy53HTdcoHK7blKTe\nWR6dtsOdm2j5R7tjrpm5T\nUrlyNz2pXLYhLCoqaidYpr\nwGIlNCItJ2bXgN1Z2BFw8\nulYTwmK/EF3NBLA04hIPoQ\nlhsdnCXbONYXToW6VSa\nzMTKbEBYfsQSPuglhMaZi7\nBRPWJYhsQmRPI5xHsc0jx\nmWMpeEZyRzAhZUq4FlY/T\nrmQCWJqg1iaOxqA",
  "aZi7\nBRPWJYhsQmRPI5xHsc0jx\nmWMpeEZyRzAhZUq4FlY/T\nrmQCWJqg1iaOxqAHMlWow\nTaI5YKuvMK58hRaxYqu4oG\nr4cElDWuGKjQBLG2RPeYP\nt5ybLMAphtsV5IzgayMJr\nCPnT51Znd/QVSRO7kgurD\n0gtJzS8p3bd0n9LcUvJE\nETPLSVPJ0F0ZukZpXuW7l\nFaWlpSOrB0QGlkaUTpQ0sf\nUhpaGlK6ZukapdpSckcKV\nwRLdykdW",
  "ZukZpXuW7l\nFaWlpSOrB0QGlkaUTpQ0sf\nUhpaGlK6ZukapdpSckcKV\nwRLdykdWzqm9MDSA0pfWPq\nC0seWPqb0paUvKX1j6RtK\n71t6n1JmKaN03dJ1Srml5N\nVBEK1aukpYCl59oO9Zm\nf0szSjNIHlj6gdGQpeSqG6\n5ml5PYGLoyWSkqfWPqEUm\nEpeX4LomeWPqM0sTSh9Kml\nTyl9belrSh9Z+ojS2FLyb\ngDuTizdodS+BaoKSrct3ab",
  "X4LomeWPqM0sTSh9Kml\nTyl9belrSh9Z+ojS2FLyb\ngDuTizdodS+BaoKSrct3ab\n01NJT93sBPp/GwLUwt2wF\nW5SmlqaUblhKnhTgVsLSE3\nI/Gan2rDZ720TOa5Gacwd\nrMz47muQ8UnPuYO3ZaXY0O\nT9Fas7HpOvre/MXKZBSON\nMfLyt4LewtLB3u7fyU+/u\n9t2le6vtG9qr3tfeN963\nor3s3fPe+z1vYEXen96f3l\n/e/8s/roYLIrF",
  "fyU+/u\n9t2le6vtG9qr3tfeN963\nor3s3fPe+z1vYEXen96f3l\n/e/8s/roYLIrFk0Z950p7\nxit>zJde57Oo/wPjzuXS</late\ny0 = \u03c60\n0 + \u03c60\n1h0\n1 + \u03c60\n2h0\n2 + \u03c60\n3h0\n3\n<latexit sha1_base64=\"/Ce9HI1M4ktH7enbrhPJjvhwRDE=\">AXWHiclZhbU9w2FMeX3pLQG0mn5aEvn\njKZ6bTpzm6SXl46k0DIDVIgsECSN7Za+CLBtbhiWe/",
  "U9w2FMeX3pLQG0mn5aEvn\njKZ6bTpzm6SXl46k0DIDVIgsECSN7Za+CLBtbhiWe/aDt9MP0yOtd4XPEQ3cGLJ/fX0fS0ZEsO8ikKHSv9/fCRx9/8ulnN27eWvz8iy+/+nrp9p39Ii3zkA/CVKb5YcAKLoXiAy205IdZzlkSH4QnK4ZfnDO80Kkak9fZv\nw4YbESkQiZBtPJ0r+eH/BYqCpImM7FeLojU6q/sTzfVO4Pys8AOBzNbyi+9PzkyAdV3",
  "bESkQiZBtPJ0r+eH/BYqCpImM7FeLojU6q/sTzfVO4Pys8AOBzNbyi+9PzkyAdV34QeWziSx7pI+TM1yOuGfjrTcDN7O5+6+5Bb9L2/PN1TvpQbe7j6s0DuLnqYeznIh7p45OlV63V/8Wug3hZVO89s+uf3t0B+mYZl\nwpUPJiuKo38v0cVyLULJ4t+WfCMhacs5kdQVCzhxXFVT8PEuwuWoRelOfwp7dXWqzUqlhTFZRKAEjo6KjAzRhc7KnX0x",
  "acs5kdQVCzhxXFVT8PEuwuWoRelOfwp7dXWqzUqlhTFZRKAEjo6KjAzRhc7KnX0x3ElVFZqrsJpQ1EpPZ16Zk69och5qOUlFiYC+irF45YzkINM7/oK34RpknCIEL+6vrOpGoCzM/KO\ngsmk7ZmvdaYgF6nWH2xN/ciNE/EB06c1BLj5BoBjydVxbtxFwPBAYguJyBVvACfJj6Qcn1EIesl4Mom5esJca0jyEmLdlbIoNCJvm4pVojKpjKpCXZBY",
  "YguJyBVvACfJj6Qcn1EIesl4Mom5esJca0jyEmLdlbIoNCJvm4pVojKpjKpCXZBYn3fUM4JCIoVfnY8jRHOxmTE1m9TQf6zypCmPDLeRMxbxuAoYcMm\nlG1FaoUkqoGrZUf2HVa6ZOm8ClWd3V3FiQai9va3RO46KGbU1tQSpIwritqi1IJWGPGrKEQZSb8gkMOPGMxS0VCksFScztPA3abWfGgnNznMF6aevWKxL+c4YiYgyw+sxVMBXytnwtna",
  "kMOPGMxS0VCksFScztPA3abWfGgnNznMF6aevWKxL+c4YiYgyw+sxVMBXytnwtnau9WXDOa70p8LE3gslqV2F5PB3WrBE\nYVWObUGUdK6Sk0QJTnl60laY3DinPRHuAxoAXZkLFV2R3atLkLG7N+Doeal5Ee/dH/l4+OqZ5aN+UeiCY6KMnM5Mub/4WgIT0WcX2DBk5dKNHlgqCcvlbC/o6ljOU5sY6nDgpCMSn0JVr+IlbtOrUFdzZNUF/BYPzC",
  "cX2DBk5dKNHlgqCcvlbC/o6ljOU5sY6nDgpCMSn0JVr+IlbtOrUFdzZNUF/BYPzClQmF\nJjmK2mJjMGK4wvPdkUAhGmQ4HWMo06LMOdn8UD6DpZabTEX5mHV3lClEbT3DS7ntaAMD4dzfk31AEU0mMYzSEs1ZDkK5thM6fidX2hYq7VX0/5tOhUxfxso2kP+gWzU4YhPzvZwPMRExXVSOQLDlROX5KoHO2Br3m6Xu1Zt\nfHuJ5LasUPrVkrit+ml",
  "gWzU4YhPzvZwPMRExXVSOQLDlROX5KoHO2Br3m6Xu1Zt\nfHuJ5LasUPrVkrit+mlW+3QXtMDfrbp6O0m0REV1Ujkq+kh1RGVoz3w5Y7jpmsUDq1bKYnfWRydaod2rkTpH+2Zw6k5JqVyaI59qfSnJizUVKidwjThMRJOTViYlG0V3GPJroCHR1s1NWHhdiHaMmPAoiGXeAhTExZOl3Bb2d\niwdNMh3XRLmcxGSDk1YeEzluBRT01YGFNh7BSesix",
  "MmPAoiGXeAhTExZOl3Bb2d\niwdNMh3XRLmcxGSDk1YeEzluBRT01YGFNh7BSesixDwqmJxHGE4zicywKHOJ8IxkjhkhKeVKqHyUtkXGgEVj1NrY0Rj0QKYKNdgYsbigmVc4M0+hLFY0iweuhgfXNKwZcmgMWLRF1pjnbzkXWYBDbF6lHUHOBFJlNIDbWLN\nNbPTXxBV5CQXRJeWXlJ6YekFpQeWHlCaW0reCILotaXk7SIzi09p3Tf0n1KS0tLSg",
  "bPTXxBV5CQXRJeWXlJ6YekFpQeWHlCaW0reCILotaXk7SIzi09p3Tf0n1KS0tLSgeWDiNLI0ofWrpU0pDS0NK1yxdo1RbSk6k8ESwdI/SkaUjSg8tPaT0jaVvKH1u6XNK31r6ltIPln6g9LGljyljJK1y1dp5RbSj4dB\nNGqpauUBpaSdz9Ya5ZuU5pZmlH6xNInlA4tJW/F8DyzlBxv4MFoqaT0haUvKBWkve3IHpl6StKE0sTSl9a+pLS95a+p",
  "xNInlA4tJW/F8DyzlBxv4MFoqaT0haUvKBWkve3IHpl6StKE0sTSl9a+pLS95a+p/SZpc8ojS0l3wbgdGLpLqX2K1BVULpj6Q6lZ5aeub8L8Pk0Bq7E3LIOtihNLU0p3bCUvCnAUcLSU3\nKejFSzq82+NpF9LVJz7mBNxGe1ScwjNecO1uxOs9pkf4rUnI9I19f35x9SIKSw058srfTxV1ha2L/f7f/WfbjzcOXRavOF9mbn+84PnR87/c7vnUed",
  "19f35x9SIKSw058srfTxV1ha2L/f7f/WfbjzcOXRavOF9mbn+84PnR87/c7vnUed53tzqATLqwujBbOFvLv/lnuLN9YvjWVfrTQ1Pm0/ot3/kPToYiAQ=\n=</latexit>2\n4\nh1\nh2\nh3\n3\n5 = a\n2\n4\n2\n4\n\u271310\n\u271320\n\u271330\n3\n5 +\n2\n4\n\u271311\n\u271321\n\u271331\n3\n5 x\n3\n5\n<latexit sha1_base64=\"G0sK8Q21pndWhXzb45WVjx5lI6g=\">AX0XiclZhJb9",
  "sha1_base64=\"G0sK8Q21pndWhXzb45WVjx5lI6g=\">AX0XiclZhJb9w2FIDHXVN3S1oUPvQi1\nEhatKkxY6fLpUBix9ns1E68JpZjUBpKw5iZC32OIKAotf+pP6SHnt/0QfJc1w9B59qIHE9Ps+PVKPpDYvkSL+/2/5t56+513v/2gfzH3708SefXr/x2X4WF6nP9/xYxumhxzIuheJ7ucglP0xSziJP8gPvdE3zg3OeZi\nJWu/lwo8jFioRCJ",
  "4WF6nP9/xYxumhxzIuheJ7ucglP0xSziJP8gPvdE3zg3OeZi\nJWu/lwo8jFioRCJ/lEDq5MbfruB4PhSq9iOWpGFfzujrk3JQOa5bt5Z1SzdWKmfecbkazqi/uJEXj0vXCxWuZIH+RFK5yaZgHT9Ol/zx3K/mrZXoD2b0vkOj6dNAO61TZhRLfawcrOm+b1ijLM8ryjLKyPFVWZpQVrXR\nGQWsyLUlbEd1YqebxcakIR/nxyfXF/lK/nFoY9A2Fn",
  "jLM8ryjLKyPFVWZpQVrXR\nGQWsyLUlbEd1YqebxcakIR/nxyfXF/lK/nFoY9A2Fnvtz/bJjS+G7jD2i4ir3Jcsy4G/SQ/LlmaC1/yat4tMp4w/5SF/AiaikU8Oy7r+a+cmxAZOkGcwj+VO3V09oiSRVl2GXlgwkBHGWY6aGNHR78fFwKlRQ5V37TUVBIJ\n48dvZicoUi5n8tLaDA/FTBWx+xlPk5Ll5V/ELP4iBhVyV9efVWVbVn5W1MuvqrOeu3",
  "dvZicoUi5n8tLaDA/FTBWx+xlPk5Ll5V/ELP4iBhVyV9efVWVbVn5W1MuvqrOeu3ogl5lrD7enWYROY/EG06S1IpOcoXAw6os+VK4hIHgAMQSJyBWPIOcuj6w0AeIwnaTgEuzFZ5XJLXKeQg16WgviQaNRPJx1ojFk\nxl1F2QHGcm4GHBai79Tr0edoDnYSpqrJcTkf52lUZjqGe0iZCndBZyz6Q+o6hCinhUL9j/Yqt50ydtoWLk3qoqY4gazft",
  "qrJcTkf52lUZjqGe0iZCndBZyz6Q+o6hCinhUL9j/Yqt50ydtoWLk3qoqY4gazftOnlK6KGXaeOIAsWYdi16giyJFwchyxiUOW2fQInHDk6YleFwqogC3M7jb1u34mO4LU5TmC\n/dL31kpT/nKGK6ADsPv1bMOXzr4WT21nUpz2tcNPnZGMFndQ1gaNqc16QTOqo1V1KxrhUxaLQil8UX1KOxqDwR3RPUAbzpilSoYEa7XbdgyeqwextONS0kP/p",
  "TOqo1V1KxrhUxaLQil8UX1KOxqDwR3RPUAbzpilSoYEa7XbdgyeqwextONS0kP/p+6Qc+Pi7etvo/0g1IVFWJLZEOvw/Eg3hdozXF0Tw5MUS\nTR4E6smLJVzf0dSxFC9sHanDhpCMSnyS7T9Rai6x9QRPNg4QmOFgM4Lv5lQaJKDoCvrgJbhNzxYWBaQj07Sb87Rl3FWpJxc/NB6hkit68tiKvTNqntBlVroXje4nB4Fbg5nPMrDvdQRb2mnl5cq",
  "b87Rl3FWpJxc/NB6hkit68tiKvTNqntBlVroXje4nB4Fbg5nPMrDvdQRb2mnl5cqCFLUTHekrHr9wshy1m2\n/31lDdNqxXys42PxgXzE7h+/zsZAPR0gs6kiUC57krLksSz9Qa7pcp0dWbnx6luytEOLazclyduO0m5b3CtGwM82LaPdJB6xqCNRrnaE1COWpT/IZa/jpu0sLK7dlCTvpI5W2+JOTbT8g90Rz5l+TIrlUD/2xdJtQljMqZ\nhbxTj",
  "pT/IZa/jpu0sLK7dlCTvpI5W2+JOTbT8g90Rz5l+TIrlUD/2xdJtQljMqZ\nhbxTjiIRKbEBajomvB31jZEXDz6FpNCIvbmehqOoClIZf4FJoQFpst3DXbGFY3LeqmXWUyGSGzCWHxIYvwWTchLIZUDK3iKUsSJDYhUscRruOI1jHBUmKT8IwklhkhS8q2oNJR3JV0AEtj1NvY0hmMQMYKdgGsZzRlZdZV5C\nq1jRVbxn63jvio5zhLqAJa2yB5z3",
  "R3JV0AEtj1NvY0hmMQMYKdgGsZzRlZdZV5C\nq1jRVbxn63jvio5zhLqAJa2yB5z3C3rJvNwifXrqXIiUBWQgu4jZ1t6kye/rygJE9yXnBp6CWlF4ZeUHpg6AGlqaHkjcALnhtK3k684NzQc0r3Dd2ntDC0oHTP0D1KA0MDSh8Y+oBS31Cf0jVD1yjNDSVPpHBHMHSX0pGhI\n0oPDT2k9IWhLyh9ZOgjSl8a+pLSN4a+ofSeofcoZYyStcNXaeUG0",
  "BHMHSX0pGhI\n0oPDT2k9IWhLyh9ZOgjSl8a+pLSN4a+ofSeofcoZYyStcNXaeUG0o+HXjBqGrlHqGknc/2GuGblOaGJpQet/Q+5QODSVvxXA/M5Q83sCN0VBJ6WNDH1MqDCXvb17w1NCnlEaGRpQ+MfQJpa8NfU3pQ0MfUhoaSr4NwNOJoT\nuUmq9AZUbpM0OfUXpm6Jn9uwCfTqNnW5hbJsEWpbGhMaUbhpI3BXiUMPSUPE8Gqr2qTb42ketaoK",
  "OfUXpm6Jn9uwCfTqNnW5hbJsEWpbGhMaUbhpI3BXiUMPSUPE8Gqr2qTb42ketaoKbcwtqKT4mNQ/UlFtYe3WaHE2uT4Ga8hEZ+vr+9EMKlBSu9CfXFwf4Kyxt7C8vDX5cuvPszuLd1fYL7bXel72vet/0Br2\nfend7j3rbvb2eP/fn3N9z/8z9u7CzcLnw28LvjfrWXHvM573Oz8If/wFIzEsB</latexit>2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5 =",
  "Lnw28LvjfrWXHvM573Oz8If/wFIzEsB</latexit>2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5 = a\n2\n4\n2\n4\n 10\n 20\n 30\n3\n5 +\n2\n4\n 11\n 12\n 13\n 21\n 22\n 23\n 32\n 32\n 33\n3\n5\n2\n4\nh1\nh2\nh3\n3\n5\n3\n5\n<latexit sha1_base64=\"eCyEQI+ojXNqWPyJfH2h0yK0JNg=\">AXAniclZhJU9xGFICHrI6zYafCxTm\noQjlOJc7U4DjLJVU2G/gAIYBbISplqYltWm1",
  "AniclZhJU9xGFICHrI6zYafCxTm\noQjlOJc7U4DjLJVU2G/gAIYBbISplqYltWm1hBYrJpbKj8mt1Su+SP5D/kReS1ptF7zSFUYdrv+3p73a3NS6XIi8Hgn7m3n7n3fev/LB1Q8/+viT+evXd/NkzLz+dBPZJLteyznUig+LEQh+X6acRZ7ku95xyua\n753yLBeJ2inOU34Ys1CJQPisgNDR/O/O+S3nF8dNI3HrqBpMnG8d1+OhUJUXsyIT48mU",
  "LBeJ2inOU34Ys1CJQPisgNDR/O/O+S3nF8dNI3HrqBpMnG8d1+OhUJUXsyIT48mULU2cr6blOxfK34PB1Wjmo+pRU9V1naip6LpRXatT6Wh+cdAf1D8OLSy1hcVe+7N5dO3zkTtK/DLmqvAly/ODpUFaHFYsK4Qv+\neSqW+Y8Zf4xC/kBFBWLeX5Y1QmbODchMnKCJINfVTh19GKNisV5fh57YMIAoxwzHbSxg7Ifj6shErLgiu/6SgopVMkjs6+",
  "hMnKCJINfVTh19GKNisV5fh57YMIAoxwzHbSxg7Ifj6shErLgiu/6SgopVMkjs6+MxIZ9wt5DgXmZwLG6vgRy5hfwBpdRU/85M4ZpAZd3l1a1K1yeQnZb1ek0nXWa0dncjLj\nOUnO7NWRMFj8YaTRmpFN3KJwMNJVfF+2MdAcACizwlIFM+hTZ0fL3CWEIX9KQED95IxDC5wnk9I06rgIeSko70kGhRSycda4VYsJRxR9kGxXFuOhpw2IC+U+9Dn",
  "KQED95IxDC5wnk9I06rgIeSko70kGhRSycda4VYsJRxR9kGxXFuOhpw2IC+U+9Dn6M12E6ZmkzrFXxcZHGV6xjuIWMq5HUXMGWfST2jrq\nFKaGq37F+xdZzpo7bxCVpPdRMR5C1k3WdIqN5UaOuU0eQBZsw7Fp1BFkSriYjFjPIcls+gnHjo7YVaGwKsjG3MwSr9t3qiN4b45TOC9db7Ui6T9lKCM6AKdP/xVM+byryQz25km57T2dYGPnQgWq1u",
  "wSr9t3qiN4b45TOC9db7Ui6T9lKCM6AKdP/xVM+byryQz25km57T2dYGPnQgWq1uFZWEzrWknMKs\n2NqFmnStk0mxBKEvOuqYejUXlqehOUAfwoSszoYIL2u26BFtWh93bMNWslPzgu/4PfHxYDfSx0f+QbEJDeZnaGtLh/9HQCO5feH9BC9eItHiQaBevETC9R0tHcvwxtaReu2gIBSTojhHx1+EqlunjuDBJjEaKwR0u/CX\nCYUWOQi6sg5oGf",
  "C9R0tHcvwxtaReu2gIBSTojhHx1+EqlunjuDBJjEaKwR0u/CX\nCYUWOQi6sg5oGf7CndiygXw0Sb+Zoy+TvMw4ufih/QyRWteXxUzom1X3giq10L1ucDmrBW4OZzyS6p7KNek08vKdWIZSiZY72k41duXsARs53+esmbotUK+cla2x+MC1an9H1+crSG1yMkFnUkagsefaxtSWJZ+oO2Z\ntv14siqtVfkK0dWly7KUm7SjtsW9ZAT8ZN0y2nXiEY",
  "FnUkagsefaxtSWJZ+oO2Z\ntv14siqtVfkK0dWly7KUm7SjtsW9ZAT8ZN0y2nXiEYs6ErXVjpB6xL0B23Z87hum4XFtZuStDvNo9W2uDMTbf9gJ+IF049JiRzpx75Euk0IiwUVC6uYxDxEYhPCYlx2Lfg/VrYF3Dy6VhPC4mYupoOYGnEJZ5CE\n8Jic4S7ZhvD6rpFXberTKYRMpsQFh+xGM+6CWExpGJoFY9ZmiKxCZE8RjiPEc1jiqXUJuEVS0r",
  "D6rpFXberTKYRMpsQFh+xGM+6CWExpGJoFY9ZmiKxCZE8RjiPEc1jiqXUJuEVS0rQraUbUNlUdKVdABLY9Tb2NIZjEAmCnXYBrGc052XW3eQrtY0V08tHU8vKTjgqEGdQBLG+SMOe6G9ZB5OMX6HdeS5F\nQgK6UJ3MTOJnWmT39eUJEnOS84N/Sc0jNDzyjdM3SP0sxQ8kbgBc8NJW8nXnBq6Cmlu4buUloaWlI6NHRIaWBoQOlDQx9S6hvqU7pi",
  "M3SP0sxQ8kbgBc8NJW8nXnBq6Cmlu4buUloaWlI6NHRIaWBoQOlDQx9S6hvqU7pi6AqlhaHkiRTuCIbuUBoZGlG6b+g+pS8MfUHpY0MfU/rS0JeUvjH0DaX3Db1PKTO\nUbpq6Cql3FDy6cALlg1dptQzlLz7wVkzdJPS1NCU0geGPqB0ZCh5K4b7maHk8QZujIZKSp8Y+oRSYSh5f/OCZ4Y+ozQ2NKb0qaFPKX1t6GtKHxn6iNLQUPJtAJ5ODN2m",
  "jIZKSp8Y+oRSYSh5f/OCZ4Y+ozQ2NKb0qaFPKX1t6GtKHxn6iNLQUPJtAJ5ODN2m1HwFqnJKtwzdovTE0BP7dwE+W0bPtjE3TAMb\nlCaGJpSuGUreFOBRwtBj8jwZqPaqNv3aRK5rgZpxC2szPq1Nch6oGbew9uo0rU2uT4Ga8YgMfXV39iEFUgpX+qP5xSX8FZYWdu/0l37s3926u3hvuf1Ce6V3o/dl7+veUu+n3r3e495mb9jze/OXZ+7MfFw",
  "8FZYWdu/0l37s3926u3hvuf1Ce6V3o/dl7+veUu+n3r3e495mb9jze/OXZ+7MfFwm8Lfyz8u\nfBXo7419b5rNf5Wfj7P59mAD4=</latexit>\ny0 = \u03c60\n0 +\n\u21e5\u03c60\n1\n\u03c60\n2\n\u03c60\n3\n\u21e4\n2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5\n<latexit sha\n1_base64=\"eo6xbZtUDEx9R\nMeyKb8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6",
  "9R\nMeyKb8msDGOEPA=\">AXg3\niclZhZc9s2EICltElT90ra6\nfihL5x60na1GNJ6fGSmcSO\nc9mp7yMxHQ9IgRiEKR52HI4\n+hV9bX9Y/0XJCWYu/BDPZM\nI2u/jAlwAJEUvkSLl5b+7d\n746Obtz65/encZ59/8eVXd\n+5+vZ/FRerzPT+WcXrosYxL\nofheLnLJD5OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0",
  "eLnLJD5OUs8iT/MA7XdH\n84JynmYjVbn6Z8OIhUoEwm\nc5hE7udm+NfjwpexPnh0eOG\n3nxuGSTIzfJBASXJs59p2n3\nJiOtzb739fe+T7Q3weTY8dV\nsSoij6eO687p3H1b7v6V3H2\nUu49y96/PbDlHlzJPUC5By\nj3YJr75M7C0uJS9efQRq9pL\nHSav82Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEu",
  "2Tu98O3WHsFxFXuS9Z\nlh31lpL8uGRpLnzJ3NukfG\nE+acs5EfQVCzi2XFZzdjEuQ\neRoRPEKfxTuVNFrx5RsijL\niMPzIjlowzHbSxoyIP/jgu\nhUqKnCu/7igopJPHjp5+Zyh\nS7ufyEhrMTwWM1fFHLGV+Dot\nkzlX8wo+jiKlh6S6vbk1K1+\nOhUCU/K6oFM5m0ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE",
  "6oFM5m0ndXK4dC8z\nlh+uTvLInIeiQ+cJKkUneQa\ngYeTsuSL4SIGgMQi5yAWPE\nMcur6eIHTQxQ2iARc1gvHBW\nN7QlKrnIdQk5b2lmjQSCQft\n6wVYsFURi1lBxTHuedowPMU\nZgGCh8czcFOwtRkelzOx3k\nalZmO4R5SpkJedQGn7DOpz6\nhtqEJKONRvWX9ia5up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\nt",
  "vWX9ia5up06ZwcV\nINdURZO2mbSdPaV3UsO1UE\nWTBIgzbVhVBloTL2ZBFDKrc\ntE/ghCNHR+yqUFgVZGFuprH\nX7jvREbw2xwnsl7a3WpLynz\nNUER2A3ac/BVM+b+sr8cx2p\nsU5r3zd4GNnBJPVPoSlYX1a\n07grJrYhJpVrZBJqwWhNL5\nom3o0FpUnon2COoA3XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3",
  "n2COoA3XZEKFV\nzRHlQtWLI67D6AU0LyY9+Wf\nyVj4/LJb1t9H+kmpAoKxJbI\nh3+H4mGcAPF6wsiePJiSYP\nAtXkxRKu72jqWIoXto5Ucwc\nNoZgU+SXa/iJU7WOqCB5sHK\nGxQkDnhU8mFJrkIGjLOqBl+\nIRHAcsC8tFJ+vU5+jLOipST\nix9azxCpdH1ZTIW+WbUvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFb",
  "ZTIW+WbUvqFI\nL7esGl7OjoA03h3N+zeEeq\nhX19OLCzVkKSrmWE/p+J2b5\nbDFbLu/mvK6abVCfrbW9Afjg\ntkpfJ+fnazh+QiJR2JcsGz\nlzWXJalP8g1W65XR1auvfu\nZLO3Q4tpNSfI2o7TbFveaEf\nCzdcto14lHLOpIlKsZIfWIZ\nekPctnruG47C4trNyXJO62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGh",
  "trNyXJO62j\n1ba4MxMt/2B3xHOmH5NiOdS\nPfbF06xAWcyrmVjGOeIjEOo\nTFqGhb8B0rOwJuHm2rDmFxM\nxNtTQewNOQSn0IdwmK9hdtm\nE8PqukVdt6tMJiNk1iEsPmcR\nPus6hMWQiqFVPGVJgsQ6ROo\n4wnUc0TomWEpsEp6RxDIjZE\nnZFlQ6ituSDmBpjHobWzqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth",
  "BpjHobWzqDE\nchYoQ6bIJYzuvIy68pTaBUr\nuor3bB3vXdNxzlBCHcDSBtl\njrth3WQeLjE8ZtmKnAhkJb\nSAm9jZpM706c8LSvIk5wWXh\nl5SemHoBaUHh5QmhpKfhF4\nwbah5NeJF5wbek7pvqH7lBaG\nFpTuGbpHaWBoQOkzQ59R6hv\nqU7pi6AqluaHkiRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoa",
  "iRTuCIbuUj\noydETpoaGHlL4x9A2lLwx9Q\nelbQ9S+sHQD5Q+MfQJpcxQ\nRumqoauUckPJqwMvWDZ0mVL\nPUPLbD/aoZuUJoYmlD419C\nmlQ0PJr2K4nxlKHm/gxmiop\nPSloS8pFYaS329e8NrQ15RG\nhkaUvjL0FaXvDX1P6XNDn1M\naGkreDcDTiaE7lJq3QGVG6Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc",
  "q3QGVG6Za\nhW5SeGXpmfy/AZ9Po2Rbmhk\nmwQWlsaEzpmqHklwI8Sh6S\np4nA9Vc1aZvm8h1LVAzbmFN\nxadHk5oHasYtrLk6TY8m16d\nAzfiIDH1f/YiBUoKV/qTOw\ns9/BaWNvb7i73fFh9uPVx4v\nNy8ob3d+a7zfenTq/ze+dx\n50Vns7PX8btR96/u391/5m/\nf/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1",
  "91/5m/\nf/AXCKek=</latexit>O35/vz+s1Rvd5phvOq2/+U\nh0\n1 = a[ 10 +  11h1 +  12h2 +  13h3]\nh0\n2 = a[ 20 +  21h1 +  22h2 +  23h3]\nh0\n3 = a[ 30 +  31h1 +  32h2 +  33h3]",
  "Notation change #2\n<latexit sh\na1_base64=\"/Ce9HI1M4k\ntH7enbrhPJjvhwRDE=\">A\nAXWHiclZhbU9w2FMeX3pL\nQG0mn5aEvnjKZ6bTpzm6S\nXl46k0DIDVIgsECSN7Z\na+CLBtbhiWe/aDt9MP0yO\ntd4XPEQ3cGLJ/fX0fS0ZE\nsO8ikKHSv9/fCRx9/8uln\nN27eWvz8iy+/+nrp9p39I\ni3zkA/CVKb5YcAKLoXiAy2\n05IdZzlkSH4QnK4ZfnDO\n80K",
  "vz8iy+/+nrp9p39I\ni3zkA/CVKb5YcAKLoXiAy2\n05IdZzlkSH4QnK4ZfnDO\n80Kkak9fZvw4YbESkQiZB\ntPJ0r+eH/BYqCpImM7FeL\nLojU6q/sTzfVO4Pys8AOB\nzNbyi+9PzkyAdV34QeWzi\nSx7pI+TM1yOuGfjrTcDN7\nO5+6+5Bb9L2/PN1TvpQbe7\nj6s0DuLnqYeznIh7p45Ol\nlV63V/8Wug3hZVO89s+u\nf3t0B+mYZlwpUPJiuKo",
  "7\nj6s0DuLnqYeznIh7p45Ol\nlV63V/8Wug3hZVO89s+u\nf3t0B+mYZlwpUPJiuKo38\nv0cVyLULJ4t+WfCMhac\ns5kdQVCzhxXFVT8PEuwuW\noRelOfwp7dXWqzUqlhTFZ\nRKAEjo6KjAzRhc7KnX0x3E\nlVFZqrsJpQ1EpPZ16Zk69\noch5qOUlFiYC+irF45Yz\nkINM7/oK34RpknCIEL+6v\nrOpGoCzM/KOgsmk7Zmvda\nYgF6nWH2xN/ciNE",
  "45Yz\nkINM7/oK34RpknCIEL+6v\nrOpGoCzM/KOgsmk7Zmvda\nYgF6nWH2xN/ciNE/EB06c\n1BLj5BoBjydVxbtxFwPBA\nYguJyBVvACfJj6Qcn1EIes\nl4Mom5esJca0jyEmLdlb\nIoNCJvm4pVojKpjKpCXZB\nYn3fUM4JCIoVfnY8jRHO\nxmTE1m9TQf6zypCmPDLeR\nMxbxuAoYcMmlG1FaoUkqo\nGrZUf2HVa6ZOm8ClWd3V3F\niQai9va3R",
  "ypCmPDLeR\nMxbxuAoYcMmlG1FaoUkqo\nGrZUf2HVa6ZOm8ClWd3V3F\niQai9va3RO46KGbU1tQSp\nIwritqi1IJWGPGrKEQZSb\n8gkMOPGMxS0VCksFScztP\nA3abWfGgnNznMF6aevWKx\nL+c4YiYgyw+sxVMBXytnw\ntnau9WXDOa70p8LE3gslq\nV2F5PB3WrBEYVWObUGUdK6\nSk0QJTnl60laY3DinPRHu\nAxoAXZkLFV2R3atLkLG\n7N+D",
  "B3WrBEYVWObUGUdK6\nSk0QJTnl60laY3DinPRHu\nAxoAXZkLFV2R3atLkLG\n7N+Doeal5Ee/dH/l4+OqZ\n5aN+UeiCY6KMnM5Mub/4W\ngIT0WcX2DBk5dKNHlgqCc\nvlbC/o6ljOU5sY6nDgpC\nMSn0JVr+IlbtOrUFdzZNUF\n/BYPzClQmFJjmK2mJjMGK\n4wvPdkUAhGmQ4HWMo06LM\nOdn8UD6DpZabTEX5mHV3\nlClEbT3DS7ntaAMD4dzfk",
  "4wvPdkUAhGmQ4HWMo06LM\nOdn8UD6DpZabTEX5mHV3\nlClEbT3DS7ntaAMD4dzfk\n31AEU0mMYzSEs1ZDkK5th\nM6fidX2hYq7VX0/5tOhU\nxfxso2kP+gWzU4YhPzvZwP\nMRExXVSOQLDlROX5KoHO2\nBr3m6Xu1ZtfHuJ5LasUPr\nVkrit+mlW+3QXtMDfrbp6\nO0m0REV1Ujkq+kh1RGVoz\n3w5Y7jpmsUDq1bKYnfWRy\ndaod2rkTpH+2Zw6",
  "rbp6\nO0m0REV1Ujkq+kh1RGVoz\n3w5Y7jpmsUDq1bKYnfWRy\ndaod2rkTpH+2Zw6k5JqVy\naI59qfSnJizUVKidwjThMR\nJOTViYlG0V3GPJroCHR1s\n1NWHhdiHaMmPAoiGXeAhT\nExZOl3Bb2diwdNMh3XRLm\ncxGSDk1YeEzluBRT01YGF\nNh7BSesixDwqmJxHGE4zi\nicywKHOJ8IxkjhkhKeVK\nqHyUtkXGgEVj1NrY0Rj0QK\nYKNdgYsb",
  "wqmJxHGE4zi\nicywKHOJ8IxkjhkhKeVK\nqHyUtkXGgEVj1NrY0Rj0QK\nYKNdgYsbigmVc4M0+hLFY\n0iweuhgfXNKwZcmgMWLRF\n1pjnbzkXWYBDbF6lHUHOB\nFJlNIDbWLNbPTXxBV5C\nQXRJeWXlJ6YekFpQeWHlC\naW0reCILotaXk7SIzi09\np3Tf0n1KS0tLSgeWDiNLI\n0ofWrpU0pDS0NK1yxdo1R\nbSk6k8ESwdI/SkaUjSg8t\nPaT0j",
  "KS0tLSgeWDiNLI\n0ofWrpU0pDS0NK1yxdo1R\nbSk6k8ESwdI/SkaUjSg8t\nPaT0jaVvKH1u6XNK31r6l\ntIPln6g9LGljyljJK1y\n1dp5RbSj4dBNGqpauUBpa\nSdz9Ya5ZuU5pZmlH6xNInl\nA4tJW/F8DyzlBxv4MFoqa\nT0haUvKBWkve3IHpl6St\nKE0sTSl9a+pLS95a+p/SZ\npc8ojS0l3wbgdGLpLqX2K\n1BVULpj6Q6lZ5aeub8L8P\nk",
  "sTSl9a+pLS95a+p/SZ\npc8ojS0l3wbgdGLpLqX2K\n1BVULpj6Q6lZ5aeub8L8P\nk0Bq7E3LIOtihNLU0p3bC\nUvCnAUcLSU3KejFSzq82+N\npF9LVJz7mBNxGe1ScwjNe\ncO1uxOs9pkf4rUnI9I19f\n35x9SIKSw058srfTxV1ha\n2L/f7f/WfbjzcOXRavOF9\nmbn+84PnR87/c7vnUed5\n3tzqATLqwujBbOFvLv/ln\nuLN9YvjWVfrTQ1Pm0",
  "OF9\nmbn+84PnR87/c7vnUed5\n3tzqATLqwujBbOFvLv/ln\nuLN9YvjWVfrTQ1Pm0/ot3\n/kPToYiAQ=</latexit>2\n4\nh1\nh2\nh3\n3\n5 = a\n2\n4\n2\n4\n\u271310\n\u271320\n\u271330\n3\n5 +\n2\n4\n\u271311\n\u271321\n\u271331\n3\n5 x\n3\n5\n<latexit sh\na1_base64=\"G0sK8Q21pn\ndWhXzb45WVjx5lI6g=\">A\nAX0XiclZhJb9w2FIDHXVN\n3S1oUPvQi1EhatKkxY6fL\npUBi",
  "45WVjx5lI6g=\">A\nAX0XiclZhJb9w2FIDHXVN\n3S1oUPvQi1EhatKkxY6fL\npUBix9ns1E68JpZjUBpKw\n5iZC32OIKAotf+pP6SHn\nt/0QfJc1w9B59qIHE9Ps\n+PVKPpDYvkSL+/2/5t56\n+513v/2gfzH3708SefX\nr/x2X4WF6nP9/xYxumhxzI\nuheJ7ucglP0xSziJP8gPv\ndE3zg3OeZiJWu/lwo8jF\nioRCJ/lEDq5MbfruB4PhS\nq9i",
  "7ucglP0xSziJP8gPv\ndE3zg3OeZiJWu/lwo8jF\nioRCJ/lEDq5MbfruB4PhS\nq9iOWpGFfzujrk3JQOa5\nbt5Z1SzdWKmfecbkazqi/\nuJEXj0vXCxWuZIH+RFK5\nyaZgHT9Ol/zx3K/mrZXoD2\nb0vkOj6dNAO61TZhRLfa\nwcrOm+b1ijLM8ryjLKyP\nFVWZpQVrXRGQWsyLUlbEd\n1YqebxcakIR/nxyfXF/lK\n/nFoY9A2Fnvtz/bJjS+G\n7",
  "WZpQVrXRGQWsyLUlbEd\n1YqebxcakIR/nxyfXF/lK\n/nFoY9A2Fnvtz/bJjS+G\n7jD2i4ir3Jcsy4G/SQ/L\nlmaC1/yat4tMp4w/5SF/Ai\naikU8Oy7r+a+cmxAZOkGc\nwj+VO3V09oiSRVl2GXlgw\nkBHGWY6aGNHR78fFwKlR\nQ5V37TUVBIJ48dvZicoUi\n5n8tLaDA/FTBWx+xlPk5\nLl5V/ELP4iBhVyV9efV\nWVbVn5W1MuvqrOeu3ogl5",
  "i\n5n8tLaDA/FTBWx+xlPk5\nLl5V/ELP4iBhVyV9efV\nWVbVn5W1MuvqrOeu3ogl5\nlrD7enWYROY/EG06S1IpO\ncoXAw6os+VK4hIHgAMQSJ\nyBWPIOcuj6w0AeIwnaTgE\nuzFZ5XJLXKeQg16WgviQa\nNRPJx1ojFkxl1F2QHGc\nm4GHBai79Tr0edoDnYSpq\nrJcTkf52lUZjqGe0iZCn\ndBZyz6Q+o6hCinhUL9j\n/Yqt50ydtoWLk3qoqY4g",
  "pq\nrJcTkf52lUZjqGe0iZCn\ndBZyz6Q+o6hCinhUL9j\n/Yqt50ydtoWLk3qoqY4ga\nzftOnlK6KGXaeOIAsWYd\ni16giyJFwchyxiUOW2fQI\nnHDk6YleFwqogC3M7jb1u\n34mO4LU5TmC/dL31kpT/nK\nGK6ADsPv1bMOXzr4WT21\nnUpz2tcNPnZGMFndQ1ga\nNqc16QTOqo1V1KxrhUxaL\nQil8UX1KOxqDwR3RPUAb\nzpilSoYEa7Xbdgyeq",
  "1ga\nNqc16QTOqo1V1KxrhUxaL\nQil8UX1KOxqDwR3RPUAb\nzpilSoYEa7Xbdgyeqwext\nONS0kP/p+6Qc+Pi7etvo\n/0g1IVFWJLZEOvw/Eg3hdo\nzXF0Tw5MUSTR4E6smLJVz\nf0dSxFC9sHanDhpCMSny\nS7T9Rai6x9QRPNg4QmOFg\nM4Lv5lQaJKDoCvrgJbhNz\nxYWBaQj07Sb87Rl3FWpJx\nc/NB6hkit68tiKvTNqntB\nlVroXje4nB4F",
  "rgJbhNz\nxYWBaQj07Sb87Rl3FWpJx\nc/NB6hkit68tiKvTNqntB\nlVroXje4nB4Fbg5nPMrDv\ndQRb2mnl5cqCFLUTHekr\nHr9wshy1m2/31lDdNqxXy\ns42PxgXzE7h+/zsZAPR\n0gs6kiUC57krLksSz9Qa\n7pcp0dWbnx6luytEOLazc\nlyduO0m5b3CtGwM82LaPd\nJB6xqCNRrnaE1COWpT/IZa\n/jpu0sLK7dlCTvpI5W2+J\nOTbT8g90R",
  "GwM82LaPd\nJB6xqCNRrnaE1COWpT/IZa\n/jpu0sLK7dlCTvpI5W2+J\nOTbT8g90Rz5l+TIrlUD/2\nxdJtQljMqZhbxTjiIRKbE\nBajomvB31jZEXDz6FpNCI\nvbmehqOoClIZf4FJoQFps\nt3DXbGFY3LeqmXWUyGSGz\nCWHxIYvwWTchLIZUDK3iKU\nsSJDYhUscRruOI1jHBUmK\nT8IwklhkhS8q2oNJR3JV0\nAEtj1NvY0hmMQMYKdgGs\nZzR",
  "YhUscRruOI1jHBUmK\nT8IwklhkhS8q2oNJR3JV0\nAEtj1NvY0hmMQMYKdgGs\nZzRlZdZV5Cq1jRVbxn63\njvio5zhLqAJa2yB5z3C3\nrJvNwifXrqXIiUBWQgu4\njZ1t6kye/rygJE9yXnBp6C\nWlF4ZeUHpg6AGlqaHkjcA\nLnhtK3k684NzQc0r3Dd2n\ntDC0oHTP0D1KA0MDSh8Y+\noBS31Cf0jVD1yjNDSVPpH\nBHMHSX0pGhI0oPDT2k9IW",
  "DC0oHTP0D1KA0MDSh8Y+\noBS31Cf0jVD1yjNDSVPpH\nBHMHSX0pGhI0oPDT2k9IW\nhLyh9ZOgjSl8a+pLSN4a+o\nfSeofcoZYyStcNXaeUG0\no+HXjBqGrlHqGknc/2Gu\nGblOaGJpQet/Q+5QODSVv\nxXA/M5Q83sCN0VBJ6WNDH\n1MqDCXvb17w1NCnlEaGRp\nQ+MfQJpa8NfU3pQ0MfUho\naSr4NwNOJoTuUmq9AZUbpM\n0OfUXpm6Jn9uwCf",
  "GRp\nQ+MfQJpa8NfU3pQ0MfUho\naSr4NwNOJoTuUmq9AZUbpM\n0OfUXpm6Jn9uwCfTqNnW5\nhbJsEWpbGhMaUbhpI3BXi\nUMPSUPE8Gqr2qTb42keta\noKbcwtqKT4mNQ/UlFtYe\n3WaHE2uT4Ga8hEZ+vr+9E\nMKlBSu9CfXFwf4Kyxt7C8\nvDX5cuvPszuLd1fYL7bXel\n72vet/0Br2fend7j3rbvb\n2eP/fn3N9z/8z9u7CzcLn\nw28LvjfrW",
  "d1fYL7bXel\n72vet/0Br2fend7j3rbvb\n2eP/fn3N9z/8z9u7CzcLn\nw28LvjfrWXHvM573Oz8If\n/wFIzEsB</latexit>2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5 = a\n2\n4\n2\n4\n 10\n 20\n 30\n3\n5 +\n2\n4\n 11\n 12\n 13\n 21\n 22\n 23\n 32\n 32\n 33\n3\n5\n2\n4\nh1\nh2\nh3\n3\n5\n3\n5\n<latexit sh\na1_base64=\"eCyEQI+oj\nXNqWPyJfH2h0yK0JNg=\">\nAXAniclZh",
  "sh\na1_base64=\"eCyEQI+oj\nXNqWPyJfH2h0yK0JNg=\">\nAXAniclZhJU9xGFICHr\nI6zYafCxTmoQjlOJc7U4\nDjLJVU2G/gAIYBbISplq\nYltWm1hBYrJpbKj8mt1\nSu+SP5D/kReS1ptF7zSF\nUYdrv+3p73a3NS6XIi8Hg\nn7m3n7n3fev/LB1Q8/\n+viT+evXd/NkzLz+dBPZ\nJLteyznUig+LEQh+X6ac\nRZ7ku95xyua753yLBeJ2i\nn",
  "viT+evXd/NkzLz+dBPZ\nJLteyznUig+LEQh+X6ac\nRZ7ku95xyua753yLBeJ2i\nnOU34Ys1CJQPisgNDR/O/\nO+S3nF8dNI3HrqBpMnG8\nd1+OhUJUXsyIT48mULU2c\nr6blOxfK34PB1Wjmo+pR\nU9V1naip6LpRXatT6Wh+c\ndAf1D8OLSy1hcVe+7N5dO\n3zkTtK/DLmqvAly/ODpU\nFaHFYsK4Qv+eSqW+Y8Zf4\nxC/kBFBWLeX5Y1QmbODc",
  "3zkTtK/DLmqvAly/ODpU\nFaHFYsK4Qv+eSqW+Y8Zf4\nxC/kBFBWLeX5Y1QmbODc\nhMnKCJINfVTh19GKNisV5\nfh57YMIAoxwzHbSxg7If\nj6shErLgiu/6SgopVMkj\ns6+MxIZ9wt5DgXmZwLG6v\ngRy5hfwBpdRU/85M4Zp\nAZd3l1a1K1yeQnZb1ek0n\nXWa0dncjLjOUnO7NWRMFj\n8YaTRmpFN3KJwMNJVfF+\n2MdAcACizwlIFM+hTZ0f",
  "XWa0dncjLjOUnO7NWRMFj\n8YaTRmpFN3KJwMNJVfF+\n2MdAcACizwlIFM+hTZ0fL\n3CWEIX9KQED95IxDC5wn\nk9I06rgIeSko70kGhRSyc\ncda4VYsJRxR9kGxXFuOhp\nw2IC+U+9Dn6M12E6Zmkz\nrFXxcZHGV6xjuIWMq5HUX\nMGWfST2jrqFKaGq37F+x\ndZzpo7bxCVpPdRMR5C1k\n3WdIqN5UaOuU0eQBZsw7F\np1BFkSriYjFjPIcls+",
  "+x\ndZzpo7bxCVpPdRMR5C1k\n3WdIqN5UaOuU0eQBZsw7F\np1BFkSriYjFjPIcls+g\nnHjo7YVaGwKsjG3MwSr9t\n3qiN4b45TOC9db7Ui6T9l\nKCM6AKdP/xVM+byryQz\n25km57T2dYGPnQgWq1uFZ\nWEzrWknMKs2NqFmnStk0\nmxBKEvOuqYejUXlqehOUA\nfwoSszoYIL2u26BFtWh93\nbMNWslPzgu/4PfHxYDfS\nx0f+QbEJDeZnaGtLh/",
  "UA\nfwoSszoYIL2u26BFtWh93\nbMNWslPzgu/4PfHxYDfS\nx0f+QbEJDeZnaGtLh/9HQ\nCO5feH9BC9eItHiQaBe\nvETC9R0tHcvwxtaReu2gI\nBSTojhHx1+EqlunjuDBJj\nEaKwR0u/CXCYUWOQi6sg\n5oGf7CndiygXw0Sb+Zoy+\nTvMw4ufih/QyRWteXxUz\nom1X3giq10L1ucDmrBW4\nOZzyS6p7KNek08vKdWIZ\nSiZY72k41duXsARs53",
  "xUz\nom1X3giq10L1ucDmrBW4\nOZzyS6p7KNek08vKdWIZ\nSiZY72k41duXsARs53+e\nsmbotUK+cla2x+MC1an9H\n1+crSG1yMkFnUkagsefa\nxtSWJZ+oO2Ztv14siqtVf\nfkK0dWly7KUm7SjtsW9\nZAT8ZN0y2nXiEYs6ErXV\njpB6xL0B23Z87hum4XFt\nZuStDvNo9W2uDMTbf9gJ\n+IF049JiRzpx75Euk0Iiw\nUVC6uYxDxEYhPCYlx2L",
  "t\nZuStDvNo9W2uDMTbf9gJ\n+IF049JiRzpx75Euk0Iiw\nUVC6uYxDxEYhPCYlx2Lfg\n/VrYF3Dy6VhPC4mYupo\nOYGnEJZ5CE8Jic4S7ZhvD\n6rpFXberTKYRMpsQFh+x\nGM+6CWExpGJoFY9ZmiKxC\nZE8RjiPEc1jiqXUJuEVS\n0rQraUbUNlUdKVdABLY9\nTb2NIZjEAmCnXYBrGc052\nXW3eQrtY0V08tHU8vKT\njgqEGdQBLG+SMOe6G9ZB",
  "9\nTb2NIZjEAmCnXYBrGc052\nXW3eQrtY0V08tHU8vKT\njgqEGdQBLG+SMOe6G9ZB5\nOMX6HdeS5FQgK6UJ3MTOJ\nnWmT39eUJEnOS84N/Sc0\njNDzyjdM3SP0sxQ8kbgBc\n8NJW8nXnBq6Cmlu4buUl\noaWlI6NHRIaWBoQOlDQx9\nS6hvqU7pi6AqlhaHkiRTu\nCIbuUBoZGlG6b+g+pS8M\nfUHpY0MfU/rS0JeUvjH0D\naX3Db1PKTOUbpq6Cq",
  "RTu\nCIbuUBoZGlG6b+g+pS8M\nfUHpY0MfU/rS0JeUvjH0D\naX3Db1PKTOUbpq6Cql3\nFDy6cALlg1dptQzlLz7wV\nkzdJPS1NCU0geGPqB0ZCh\n5K4b7maHk8QZujIZKSp8\nY+oRSYSh5f/OCZ4Y+ozQ2\nNKb0qaFPKX1t6GtKHxn6\niNLQUPJtAJ5ODN2m1HwFq\nnJKtwzdovTE0BP7dwE+W0\nbPtjE3TAMblCaGJpSuGU\nreFOBRwtBj8jwZqP",
  "HwFq\nnJKtwzdovTE0BP7dwE+W0\nbPtjE3TAMblCaGJpSuGU\nreFOBRwtBj8jwZqPaqNv3\naRK5rgZpxC2szPq1Nch6\noGbew9uo0rU2uT4Ga8YgM\nfXV39iEFUgpX+qP5xSX8F\nZYWdu/0l37s3926u3hvu\nf1Ce6V3o/dl7+veUu+n3r\n3e495mb9jze/OXZ+7Mf\nfFwm8Lfyz8ufBXo7419b\n5rNf5Wfj7P59mAD4=</la\ntexit>\ny0 = \u03c60\n0",
  "XZ+7Mf\nfFwm8Lfyz8ufBXo7419b\n5rNf5Wfj7P59mAD4=</la\ntexit>\ny0 = \u03c60\n0 +\n\u21e5\u03c60\n1\n\u03c60\n2\n\u03c60\n3\n\u21e4\n2\n4\nh0\n1\nh0\n2\nh0\n3\n3\n5\nNotation Reminder\n\ud835\udc65, \ud835\udf13 : normal lower case -- scalar\n\ud835\udc99, \ud835\udf4d : bold face lower case -- vector\n\ud835\udc7f, \ud835\udebf : bold face upper case -- matrix\n\ud835\udc21 = \ud835\udc1a[\ud835\udf3d\ud835\udfce + \ud835\udf3d\ud835\udc65]\n\ud835\udc21\u2032 = \ud835\udc1a[\ud835\udf4d\ud835\udfce + \ud835\udebf\ud835\udc21]\n\ud835\udc66\" = \ud835\udf19\u2032# + \ud835\udf53\"\ud835\udc7b\ud835\udc21\"",
  "Notation change #3\n<latexi\nt sha1_base64\n=\"vsvFeJd7CA\nhbv2jJIiHEd2W\nOncw=\">AWtX\niclZhb9s2FI\nDV7tZ1t3TD8rI\nXYUGBYesMZ+jW\n7WFAmzS9JV2cJ\nk7Sxq5ByZTMh\nqIUiUqcCv4t+z\nV73Z73b3Yoy2Z\n1DvMwA4np83\ni5ZCSKAWZFIXu\ndv+9dv29z/48\nKMbH9/85NPv\n9i5daXh0Va5i\nHvh6lM8+OAFVw\nKxftaMmPs5yz\nJD8KDjd",
  "z/48\nKMbH9/85NPv\n9i5daXh0Va5i\nHvh6lM8+OAFVw\nKxftaMmPs5yz\nJD8KDjdNPzo\nnOeFSNWBvsz4M\nGxEpEImYbQaO\nU3fxBE/93f5A\nE6bSCHz6bDS\nP9AmU9YRrNq6\nsx8WP/zpIBfxR\nA/90cpat9OtP\nz4trDeFNa/59E\na3vh4PxmlYJlz\npULKiOFnvZnpY\nsVyLUPLZzUFZ\n8IyFpyzmJ1BUL\nOHFsKrHOPNvQ2\nTsR2kOf0r7df",
  "KiOFnvZnpY\nsVyLUPLZzUFZ\n8IyFpyzmJ1BUL\nOHFsKrHOPNvQ2\nTsR2kOf0r7df\nTdIyqWFMVlEoC\nZMD0pMDNBFzsp\ndfTrsBIqKzVX4\nbyhqJS+Tn2TM\nH8sch5qeQkFu\nYC+uqHE5azUEN\nabw4UvwjTJGF\nqXA02tvZmkEIe\nC1Xxs7JO8WzWd\nrZqh0PxKmPj6c\nGyFqF5It5yUk\nmtmEquEHg8qyr\neiTsYCA5AdDgB\nqeIF1GnyA2tg",
  "0PxKmPj6c\nGyFqF5It5yUk\nmtmEquEHg8qyr\neiTsYCA5AdDgB\nqeIF1GnyA2tg\nHVFYUhJwZVfJi\nxmpWmkeQ05a2i\nuiQSGTfNqyNok\nFU5m0lH1QfP+\n2bwDXOcwCdBW+\nOJqD/Yyp2eI4z\nac6T6rCxHALO\nVMxr5uAIYdMmh\nG1DVKCYeGLes\nPbL1g6rRJXJrV\nXc1NBFkHedvR\nOc2LGredOoIsW\nIRx26ojyJwAR\nizhEGWm/IBp\nz4J",
  "6rRJXJrV\nXc1NBFkHedvR\nOc2LGredOoIsW\nIRx26ojyJwAR\nizhEGWm/IBp\nz4JuJWhcKqIAu\nzl6dBu+3MRPDa\nnGZwvrS9rYqk/\n5yhjJgAnH3mW\nzAV8ra+mS5tf5\nGc89o3BT71JzB\nZ7UNYHs+HtWg\nERtXEZtSsc4VM\nmi0I5elF2zS9c\nag8E+0BmgA+6c\npcqOgd7U5dgi\nVrwoM7MNS8lPz\nkx87PfDqsua0\nMf9INqGiosxc\nFZnw",
  "mgA+6c\npcqOgd7U5dgi\nVrwoM7MNS8lPz\nkx87PfDqsua0\nMf9INqGiosxc\nFZnw/6hoDLcv\nL4gicvlWjyIF\nBPXirh+o6mjuV\n4YZtIPXdQEIp\nJoS/R6S9i1T6m\njuDOpgnqKwRMv\nfDNhEKTHEVt2\nQSMDN9w83QsoB\nANMpyPMZRpUea\ncXPzQeoZIrZvL\nYi7Mzap9QZVG\naF83uFweBW4O\nZzKw4PUEaDeT\n6DtFRjlqNkTs\n2UTl8PCg",
  "ZvL\nYi7Mzap9QZVG\naF83uFweBW4O\nZzKw4PUEaDeT\n6DtFRjlqNkTs\n2UTl8PCg2nmOv\nsr6d8XnRaMT/b\nbtqDfsHslGHI\nz0beD5iYlFHo\nrpgt+KsSxL0R\n7UtVyu7/as2n7\n9PVnascN1m5L\nU2/TSbTvcK3rA\nz3Ycvd0hHrGoI\n1FdTQ+pRyxHe\n1CXO487rlE4XL\ncpSb2LPDpth7s\n0fKPDsxm1GyT\nUjk275UDuYh\nLGoqaqeYJjxG",
  "CXO487rlE4XL\ncpSb2LPDpth7s\n0fKPDsxm1GyT\nUjk275UDuYh\nLGoqaqeYJjxG4\njyExaRsW/AbK/\nsCbh5tax7CYq\n8Qbc0EsDTmEg9\nhHsLi/BRum0M\nqzsOdcetMplNk\nDkPYfExS/Co5\nyEsxlSMneIpyz\nIkzkMkjxOcxwn\nNY4alzCXhGck\ncM0KWlGtB5ZO0\nLZkAlqaotamjM\neiBTBVqsAliua\nAr3CuPIVWsa\nKruO9quH9Fw5q",
  "WlGtB5ZO0\nLZkAlqaotamjM\neiBTBVqsAliua\nAr3CuPIVWsa\nKruO9quH9Fw5q\nhCk0AS7vkHPMH\nu86TLMAphm2W\nK8mZQFZGE9jDT\no86i91fEFVkJx\ndEl5ZeUnph6QW\nlR5YeUZpbSp4\nIguiFpeTpJIjO\nLT2n9NDSQ0pLS\n0tK+5b2KY0sj\nSh9ZOkjSkNLQ0\no3Ld2kVFtKdqR\nwR7D0gNKJpRNK\njy09pvSlpS8p\nfWLpE0pfWfqK0",
  "kjSkNLQ0\no3Ld2kVFtKdqR\nwR7D0gNKJpRNK\njy09pvSlpS8p\nfWLpE0pfWfqK0\nreWvqX0gaUPKG\nWMkq3LN2ilF\ntKXh0E0YalG5Q\nGlpJnPzjXLO1R\nmlmaUfrQ0oeUj\ni0lT8VwP7OUb\nG/gxmipPSpU\n8pFZaS57cgem7\npc0oTSxNKn1n\n6jNI3lr6h9LGl\njymNLSXvBmB3Y\nuk+pfYtUFVQum\nfpHqVnlp653w\nvw5TQGroW5ayv\nYpT",
  "h9LGl\njymNLSXvBmB3Y\nuk+pfYtUFVQum\nfpHqVnlp653w\nvw5TQGroW5ayv\nYpTS1NKV021Ly\npABbCUtPyX4y\nUs1VbfG2iVzXI\nrXkDtZkfHE0yX\nmkltzBmqvT4mh\nyfYrUk9I17c\nOly9SIKVwpR+t\nrK3jt7C0cPhTZ\n/2Xzt29u2v3N\n5o3tDe8b7xve\n+8de+ed974vW\n8vhd6f3p/eX97\n/6zeWx2ujlej\nuXr9WnPMV17rs\n5r+BwJ",
  "xve\n+8de+ed974vW\n8vhd6f3p/eX97\n/6zeWx2ujlej\nuXr9WnPMV17rs\n5r+BwJv5Xc=</\nlatexit>h = a [\u27130 + \u2713x]\n<latexi\nt sha1_base64\n=\"V3JmSB8gD8\nhg/vLGAqXrvlg\noUI0=\">AWtH\niclZhb9s2FI\nDVXbvulm5YXvY\niLCg2bJ3hDF3X\nlwFt0vSWdEmaO\nEkbpwYlUzIbi\nlIkKnEq+K/s1+\nx1e9+/2aEkm9U\n5zMtGbO94m\nXQ",
  "0vSWdEmaO\nEkbpwYlUzIbi\nlIkKnEq+K/s1+\nx1e9+/2aEkm9U\n5zMtGbO94m\nXQ1KiFWRSFLrf\n/fae+9/8OFH\n1/5Man3+xZ\ndLN786KNIyD/\nkgTGWaHwWs4FI\noPtBCS36U5Zwl\ngeSHwem64Yfn\nPC9Eqvb1ZcZPE\nhYrEYmQaQiNlu\n75wyCafO/7g+\nTIJ1W8JfPZkP\nJI30M5awQo6o/\n+wmKO4Wo3WEu4\nok+GS2t9Hv9+\nuPTwmpbW",
  "+\nTIJ1W8JfPZkP\nJI30M5awQo6o/\n+wmKO4Wo3WEu4\nok+GS2t9Hv9+\nuPTwmpbWPHaz8\n7o5jfj4TgNy4Q\nrHUpWFMer/Uyf\nVCzXIpR8dmNY\nFjxj4SmL+TEUF\nUt4cVLVQ5z5ty\nAy9qM0h39K+3\nX03SsqlhTFZRK\nAmTA9KTAzQRc7\nLnV076QSKis1V\n2HTUFRKX6e+y\nZc/FjkPtbyEAg\ntzAX31wnLWag\nhqzeGil+EaZI\nwNa6Gaxu7",
  "V\n2HTUFRKX6e+y\nZc/FjkPtbyEAg\ntzAX31wnLWag\nhqzeGil+EaZI\nwNa6Gaxu7M0g\nj4Wq+FlZ3g26\nzobtcOheJWx9n\nR/UYvQPBFvOa\nmkVkwlVwg8nlU\nV78U9DAQHIHqc\ngFTxAuo0+YEV\nsIorCgJuLJr5\nMWMVK0jyEnHe\n0V0aCQST7tWOv\nEgqlMOsoeKL5\n/yzeA6xmAboK\nXxzNwV7G1Gx+n\neZTnSdVYWK4h\nZypmNdNwJBDJs",
  "lMOsoeKL5\n/yzeA6xmAboK\nXxzNwV7G1Gx+n\neZTnSdVYWK4h\nZypmNdNwJBDJs\n2IuoYqpYRLw47\n1B7ZeMHXaJi7N\n6q7mJoKs/bzr\n6JzmRY27Th1BF\nizCuGvVEWRJ2P\n9jljDIclsewY\nAT30TcqlBYFWR\nh7uRp0G07MxG8\nNqcZ7Jeut1GR9\nJ8zlBETgN1nv\ngVTIe/q6+nC9u\nfJOa9U+BTfwK\nT1b2E5XEzrHk\njMKo2NqNmnStk\n0",
  "BETgN1nv\ngVTIe/q6+nC9u\nfJOa9U+BTfwK\nT1b2E5XEzrHk\njMKo2NqNmnStk\n0mxBKE8vuqbpj\nUPlmegO0ATwpi\ntzoaJ3tNt1CZ\nasCQ9vw1DzUvL\njn3u/8ulJ1Tfb\nxvxHsgkVFWXm\nqsiE/0dFY3ji4\nPUFETx5qUSTB4\nF68lIJ93c0dSz\nHC9tE6rmDglB\nMCn2Jtr+IVfea\nOoI7myaorxAw9\ncI3EwpNchR1Z\nRMwMnzDs9OxgE\nI",
  "6rmDglB\nMCn2Jtr+IVfea\nOoI7myaorxAw9\ncI3EwpNchR1Z\nRMwMnzDs9OxgE\nI0yLAZYyjTosw\n5ufmh9QyRWje3\nxVyYh1X3hiqN\n0L1vcLm4Csrwc\nDjnV1weoIwGT\n6DtFRjlqNkTs\n2UTl8PCw1bzLX\n76ylvik4r5meb\nbXvQL5idMgz5\n2WgTz0dMLOpIV\nBcVpx1SWI52o\nO6Fsv13Z5Vm69\n/JEs7drhuU5J\n6216bYd7RQ/4\n2Zaj",
  "MLOpIV\nBcVpx1SWI52o\nO6Fsv13Z5Vm69\n/JEs7drhuU5J\n6216bYd7RQ/4\n2Zajt1vEIxZ1J\nKqr7SH1iOVoD\n+py53HLNQqH6z\nYlqXeR6ftcBc\nmWv7R/oRrZo5J\nqRybY18qh0I\ni5qK2imCY+R2\nISwmJRdC/7Gyp\n6Ah0fXakJYhD\nN1VzMBLI25xEN\noQlhstnDXbGNY\n3XKoW26VyWyCz\nCaExcswaNuQ\nliMqRg7xVOWZU\nhsQiSPE5",
  "N\noQlhstnDXbGNY\n3XKoW26VyWyCz\nCaExcswaNuQ\nliMqRg7xVOWZU\nhsQiSPE5zHCc1\njhqXMJeEZyRw\nzQpaUa0Hlk7Qr\nmQCWpqi1qaMx6\nIFMFWqwDWK5oC\nuvcK48hVaxoq\nt4Gp4cEXDmqE\nKTQBL2SP+cNt\n5yYLcIrhmOVK\nciaQldE7mBnh\nzrz018QVeQkF0\nSXl5SemHpBaW\nHlh5SmltKfhE\nE0QtLya+TIDq3\n9JzSA0sPKC0t",
  "z018QVeQkF0\nSXl5SemHpBaW\nHlh5SmltKfhE\nE0QtLya+TIDq3\n9JzSA0sPKC0tL\nSkdWDqgNLI0o\nvSRpY8oDS0NKV\n23dJ1SbSk5kcI\nTwdJ9SieWTig9\nsvSI0peWvqT0\niaVPKH1l6StK3\n1r6ltIHlj6glF\nnKN2wdINSbi\nl5dRBEa5auURp\nYSn7wV6zdIfS\nzNKM0oeWPqR0b\nCn5VQzPM0vJ8\nQYejJZKSp9a+p\nRSYSn5/RZEzy1",
  "wV6zdIfS\nzNKM0oeWPqR0b\nCn5VQzPM0vJ8\nQYejJZKSp9a+p\nRSYSn5/RZEzy1\n9TmliaULpM0u\nfUfrG0jeUPrb0\nMaWxpeTdAJxOL\nN2j1L4FqgpKdy\n3dpfTM0jP3ew\nG+mMbAtTC3bQX\nblKaWpRuWkp+\nKcBRwtJTcp6M\nVHtXm79tIve1S\nC24g7UZn19Nch\n6pBXew9u40v5r\ncnyK14BPS9Y2\nDxYsUSCnc6UdL\nK6v4LSwtHPzSW\n7",
  "Zn19Nch\n6pBXew9u40v5r\ncnyK14BPS9Y2\nDxYsUSCnc6UdL\nK6v4LSwtHPzSW\n73bu7N7Z+X+W\nvuG9r3rfed94\nO36v3m3feDv\newAu9P72/vL+9\nf5bvLg+Xw2Xe\nqO9da6/52ut8l\nlatexit>tV/cAvk1A=</\nh0 = a [ 0 +  h]\n<latexi\nt sha1_base64\n=\"itKSwzHaSz\nm5UIhVM1UlAW\nBEiQ=\">AWnH\niclZhb9s2FI\nDVXbvulm5YXgY",
  "=\"itKSwzHaSz\nm5UIhVM1UlAW\nBEiQ=\">AWnH\niclZhb9s2FI\nDVXbvulm5YXgY\nMwoKiw9YZztBd\nXga0SdNb0sVp4\niRtnBqUTMlsK\nEqRqMSu4Nf9mr\n1u/2X/ZoeybFb\nnMA8z0Io53yd\neDkndgkyKQne7\n/157593v/gw\n+sf3fj4k08/+3\nzl5heHRVrmIe\n+HqUz4AVXAr\nF+1poyY+znLMk\nkPwoONs0/OiC\n54VI1YGeZvw0Y\nbESk",
  "RVrmIe\n+HqUz4AVXAr\nF+1poyY+znLMk\nkPwoONs0/OiC\n54VI1YGeZvw0Y\nbESkQiZhtBwxf\nen/u/+Iiysbg\n9rLoz/4fFX+Y\n4vj1cWet2uvXP\np4X1prDmNb/e8\nOZXo8EoDcuEK\nx1KVhQn691Mn1\nYs1yKUfHZjUBY\n8Y+EZi/kJFBVL\neHFa1UOZ+bcg\nMvKjNId/Svt19\nO0zKpYUxTQJwE\nyYHheYmaCLnZ\nQ6+u20EiorNVf\nhvKGo",
  "bcg\nMvKjNId/Svt19\nO0zKpYUxTQJwE\nyYHheYmaCLnZ\nQ6+u20EiorNVf\nhvKGolL5OfZMX\nfyRyHmo5hQILc\nwF9cMxy1moI\nXs3BopfhmSMD\nWqBhtbe7NqEPB\nYqIqfl3UmZ7O\n2s1U7HIpXGRtP\nDpa1CM0T8YaTS\nmrFVHKFwONZVf\nFO3MFAcACiw\nlIFS+gTpOfIPL\nXEYWVIwED9IJ\ndC7yn89I1Urz\nGHLS0l4SDQqZ5\nJOWtUksmM",
  "lIFS+gTpOfIPL\nXEYWVIwED9IJ\ndC7yn89I1Urz\nGHLS0l4SDQqZ5\nJOWtUksmMqkpe\nyD4vu3fAO4zmE\nWoKtw4GgO9jO\nmZovzNJ/oPKkK\nE8Mt5EzFvG4Ch\nhwyaUbUNlQpJ\nZwatqw/sPWcqb\nMmcWlWdzU3EWQ\nd5G1H5zQvatR2\n6giyYBHGbauO\nIEvCPh+xhEGWm\n/IQBpz4JuJWhc\nKqIAuzl6dBu+\n3MRPDanGSwX9r\neVkXSf8FQ",
  "IEvCPh+xhEGWm\n/IQBpz4JuJWhc\nKqIAuzl6dBu+\n3MRPDanGSwX9r\neVkXSf8FQRkwA\ndp85CqZC3tY30\n6XtL5JzUfumw\nCf+GCarfQrL4/\nmwFo3AqJrYjJp\n1rpBJswWhPL1\nsm6Y3DpVnoj1A\nE8CbrsyFit7S7\ntQlWLImPLgDQ8\n1LyU9+7PzMJ6\ndV12wb8x/JlR\nUlJmrIhP+HxWN\n4M6C1xdE8OSl\nEk0eBOrJSyVc3\n9HUsRwvbBO",
  "dV12wb8x/JlR\nUlJmrIhP+HxWN\n4M6C1xdE8OSl\nEk0eBOrJSyVc3\n9HUsRwvbBOp5w\n4KQjEp9BRtfxG\nr9jl1BHc2TVB\nfIWDqhSMTCk1y\nFLVlEzAyHOEe6\nVhAIRpkOB9jK\nNOizDm5+KH1DJ\nFaN5fFXJibVfu\nCKo3Qvm5wuTwL\nynBzuOBXnB6g\njAbzfAZpqUYsR\n8mcmCmdvBoUGr\naYa/fXUz4vOq\n2Yn2837UG/YHb\nKMOTnw208H",
  "AbzfAZpqUYsR\n8mcmCmdvBoUGr\naYa/fXUz4vOq\n2Yn2837UG/YHb\nKMOTnw208HzGx\nqCNRXfBQ4qxL\nEsvRHtS1XK5v9\n6zafvU9Wdqxw3\nWbktTb9NJtO9w\nresDPdxy93SE\nesagjUV1ND6lH\nLEd7UJc7jzuU\nThctylJvYs8O\nm2HuzTR8o8Oxl\nwz85iUypF57Ev\nlYB7Coqaidop\nwmMkzkNYTMq2\nBX9jZV/AzaNtz\nUNY7BWirZkAl",
  "85iUypF57Ev\nlYB7Coqaidop\nwmMkzkNYTMq2\nBX9jZV/AzaNtz\nUNY7BWirZkAlk\nZc4iHMQ1icb+\nG2cSwuNQd9w\nqk9kYmfMQFh+x\nBI96HsJiTMXYK\nZ6xLEPiPETyO\nMZ5HNM8ZljKXB\nKekcwxI2RJuRZ\nUPk7bkglgaYJ\namzgagx7IVKEG\nmyCWC7ryCufKU\n2gVK7qK+6G+1\nc0rBmq0ASwtE\nv2mD/YdW6yAKc\nYHrNcSc4EsjKa\nw",
  "7ryCufKU\n2gVK7qK+6G+1\nc0rBmq0ASwtE\nv2mD/YdW6yAKc\nYHrNcSc4EsjKa\nwB52etRZP0F\nUWe5IJoaumU0\nktLyk9svSI0t\nxS8kYQRM8tJW8\nnQXRh6QWlh5Y\neUlpaWlLat7RP\naWRpROlDSx9SG\nloaUrp6Sal2\nlLyRAp3BEsPKB\n1bOqb02NJjSl9\nY+oLSx5Y+pvSl\npS8pfWPpG0rv\nW3qfUmYpo3TL0\ni1KuaXk0EQbV\ni6QWlg",
  "Sl9\nY+oLSx5Y+pvSl\npS8pfWPpG0rv\nW3qfUmYpo3TL0\ni1KuaXk0EQbV\ni6QWlgKXn3g7\n1maY/SzNKM0ge\nWPqB0ZCl5K4b7\nmaXk8QZujJZKS\np9Y+oRSYSl5f\nwuiZ5Y+ozSxNK\nH0qaVPKX1t6Wt\nKH1n6iNLYUvJ\ntAJ5OLN2n1H4F\nqgpK9yzdo/Tc0\nnP3dwG+nMbAtT\nB3bQW7lKaWp\nRuW0reFOBRwtI\nz8jwZqeaqtvja\nRK5rkVp",
  "c0\nnP3dwG+nMbAtT\nB3bQW7lKaWp\nRuW0reFOBRwtI\nz8jwZqeaqtvja\nRK5rkVpyB2sy\nvjib5DxS+5gz\ndVpcTa5PkVqyc\nek61uHyw8pkFK\n40g9X1tbxV1h\naOPyps/5L5+7e\n3bV7G80X2uve1\n963nfeuverd\n897PW8vhd6f3\np/eX97/6x+s/p\ngdXv12Vx951pz\nzpde67d6+B8B\n>pdmB</latexit\ny = \u03c60\n0 + \u03c60h0\n<latexit sha1_ba",
  "dXv12Vx951pz\nzpde67d6+B8B\n>pdmB</latexit\ny = \u03c60\n0 + \u03c60h0\n<latexit sha1_base64=\"UGmj8AB2OH/n/0YCMA80y0J0vI=\"\n>AWq3iclZhbU9tGFICV9JbSG2mnvPRFUyYznTZx7U56elMAiE3SIGAgYCJZyWv5A2rlZBWYKLxz+iv6Wv7I/pvelaWvdE5y0M9A\n16f79Nezq6klYJMikJ3u/euPne+x98+NGtj5c+fSz79Yv3lQZGWecj7YS",
  "5y0M9A\n16f79Nezq6klYJMikJ3u/euPne+x98+NGtj5c+fSz79Yv3lQZGWecj7YSrT/ChgBZdC8b4WvKjLOcsCSQ/DM7WDT+84HkhUrW\nvrzJ+mrBYiUiETENouPyjPwi8bDqTf3fTZGdwL+AazasulP/B/ixnfC4/gXlyelwebXb6dYfnxZ6TWHVaz47w9tfjwajNCwTrnQo\nWVGc9LqZPq1YrkUo+XRpUBY8Y+EZi/kJFBVLeHFa1SOb+ncgMv",
  "fjwajNCwTrnQo\nWVGc9LqZPq1YrkUo+XRpUBY8Y+EZi/kJFBVLeHFa1SOb+ncgMvKjNIc/pf06+u4RFUuK4ioJwEyYHheYmaCLnZQ6+u20EiorNVfhrK\nGolL5OfZMmfyRyHmp5BQUW5gL6odjlrNQzKXBopfhmSMDWqBmsbu9NqEPBYqIqfl3Vip9O2s1E7HIrXGWvP9he1CM0T8ZaTSmrF\nVHKNwONpVfFO3MFAcACiwlIFS+gTpOfIPJ7iMJC",
  "HIrXGWvP9he1CM0T8ZaTSmrF\nVHKNwONpVfFO3MFAcACiwlIFS+gTpOfIPJ7iMJCkoCB+kEOhf5L6ekaqV5DlpacdEg0Im+aRlrRMLpjJpKXug+P4d3wCuc5gF6C\np8cTQHexlT0/lxmk90nlSFieEWcqZiXjcBQw6ZNCNqG6qUEg4NW9Yf2HrJ1FmTuDSru5qbCL287ajc5oXNWo7dQRZsAjtlVHkCX\nhtB+xhEGWm/IQBpz4JuJWhcKqIAtzJ0",
  "5qbCL287ajc5oXNWo7dQRZsAjtlVHkCX\nhtB+xhEGWm/IQBpz4JuJWhcKqIAtzJ0+DdtuZieC1OcngfGl7GxVJ/wVDGTEBOPvMt2Aq5G19PV3Y/jw5F7VvCnzij2Gy2oewPJ4Na\n94IjKqJTalZ5wqZNFsQytPLtml641B5JtoDNAF80pW5UNE72t26BEvWhAd3Yah5KfnJvc7PfHJadc1pY/6RbEJFRZm5KjLh/1HRCG4\n0eH1BE9eKtHkQaCev",
  "Ad3Yah5KfnJvc7PfHJadc1pY/6RbEJFRZm5KjLh/1HRCG4\n0eH1BE9eKtHkQaCevFTC9R1NHcvxwjaReu6gIBSTQl+h01/Eqn1MHcGdTRPUVwiYeuGbCYUmOYrasgkYGb7hlulYQCEaZDgbYyjT\nosw5ufih9QyRWjeXxVyYm1X7giqN0L5ucLk4Cspwc7jg1xweoIwGs3wGalGLEfJnJgpnbweFBpOMdfZX0/5rOi0Yn6+2bQH/YLZKc\nOQnw83",
  "1xweoIwGs3wGalGLEfJnJgpnbweFBpOMdfZX0/5rOi0Yn6+2bQH/YLZKc\nOQnw838XzExKORHXBHsVZlySWoz2oa7Fc3+1Ztfn6e7K0Y4frNiWpt+ml23a41/SAn285ertFPGJR6K6mh5Sj1iO9qAudx63XKNw\nuG5TknrneXTaDndhouUf7Y9hZ2q2SakcmW1fKgezEBY1FbVTM3uti3OQlhMyrYFv7GyJ+Dm0bZmISzuFKtmQCWRlziIcxCWJy",
  "W1fKgezEBY1FbVTM3uti3OQlhMyrYFv7GyJ+Dm0bZmISzuFKtmQCWRlziIcxCWJydwm\n2ziWF1y6FuVUmszEyZyEsPmEJHvUshMWYirFTPGNZhsRZiORxjPM4pnMsJS5JDwjmWNGyJyLah8nLYlE8DSBLU2cTQGPZCpQg0\n2QSwXdOUVzpWn0CpWdBX3XQ3r2lYM1ShCWBpm5xj/mDbeZIFOMWwzXIlORPIymgCd7CzQ5357i+IKrKTC6IrS68ov",
  "2lYM1ShCWBpm5xj/mDbeZIFOMWwzXIlORPIymgCd7CzQ5357i+IKrKTC6IrS68ovbT0ktJDSw8pz\nS0lTwRB9NJS8nQSRBeWXlB6YOkBpaWlJaV9S/uURpZGlD629DGloaUhpeuWrlOqLSU7UrgjWLpP6djSMaVHlh5R+srSV5Q+tfQpce\nWHlP61tK3lD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8uwH5qlO5RmlmaUPrL0EaUjS8lTMdzPL",
  "9CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8uwH5qlO5RmlmaUPrL0EaUjS8lTMdzPLCXbG7gxWiopfWbpM0qFpeT5\nLYheWPqC0sTShNLnlj6n9I2lbyh9YukTSmNLybsB2J1YukepfQtUFZTuWrpL6bml5+73AnwxjYFrYW7bCrYpTS1NKd20lDwpwFbC0j\nOyn4xUc1Wbv20i17VILbiDNRmfH01yHqkFd7Dm6jQ/mlyfIrXgY9L1jYPFixRIKVz",
  "yn4xUc1Wbv20i17VILbiDNRmfH01yHqkFd7Dm6jQ/mlyfIrXgY9L1jYPFixRIKVzph8urPfwWlhYOfur0func372/+mCteUN7y/vG\n+9b7zut5v3oPvKfejtf3Qu9P7y/vb+flXsreyvHK4OZevNGc8xXuzwv8DIu/g1A=</latexit>h1 = a[\u03b20 + \u23260x]\n<latexit sha1_base64=\"UwfMPn/m/6auzwL2FXGTyWOmVOc=\"\n>AWr3iclZhbU9w2FICd",
  "sha1_base64=\"UwfMPn/m/6auzwL2FXGTyWOmVOc=\"\n>AWr3iclZhbU9w2FICdXtP0RtopL3xlMlMp0132DS9vHQmgZAbpEBgYQlW9krexVk2dgyLPHsD+mv6Wv7E/pvemR7V/E54qE7A\n6s932djmRbdpBJUejV1X+vfPue+9/8OH1j258/Mmn32+dPOLgyIt85APwlSm+VHACi6F4gMtORHWc5ZEkh+GJyuG354zvNCpGp\nfX2b8JGxEpEImY",
  "yIt85APwlSm+VHACi6F4gMtORHWc5ZEkh+GJyuG354zvNCpGp\nfX2b8JGxEpEImYbQaOlHfxhEk1F1Z+b/ZorsGP4FXLNR1Z/538OP7YTH9a9G7M9ORksrq73V+uPTQr8trHjtZ2d086vxcJyGZcKV\nDiUriuP+aqZPKpZrEUo+uzEsC56x8JTF/BiKiW8OKnq0c38WxAZ+1Gaw5/Sfh19+4iKJUVxmQRgJkxPCsxM0MWOSx39elIJlZWaq7\nBpK",
  "nq0c38WxAZ+1Gaw5/Sfh19+4iKJUVxmQRgJkxPCsxM0MWOSx39elIJlZWaq7\nBpKCqlr1PfpMofi5yHWl5CgYW5gL764YTlLNSQ0BtDxS/CNEmYGlfDtY3dWTUMeCxUxc/KOrmzWdfZqB0OxauMtSf7i1qE5ol4w0kl\ntWIquULg8ayqeC/uYSA4ANHjBKSKF1CnyU8Q+X1EYTFJwMCDdAqdi/znM1K10jyGnHS0l0SDQib5tGOtEwumMukoe6D4",
  "F1CnyU8Q+X1EYTFJwMCDdAqdi/znM1K10jyGnHS0l0SDQib5tGOtEwumMukoe6D4/i3fAK5zmA\nXoKnxNAd7GVOz+XGaT3WeVIWJ4RZypmJeNwFDpk0I+oaqpQSDg071u/Yes7UaZu4NKu7mpsIsvbzrqNzmhc17jp1BFmwCOuVUe\nQJeHUH7OEQZb8gGnPgm4laFwqogC3MnT4Nu25mJ4LU5zeB86XobFUn/OUMZMQE4+8y3YCrkX09Xdj+PDnt",
  "m4laFwqogC3MnT4Nu25mJ4LU5zeB86XobFUn/OUMZMQE4+8y3YCrkX09Xdj+PDntW8KfOpPYLK6h7A8b\noY1bwRG1cZm1KxzhUyaLQjl6UXNL1xqDwT3QGaAD7pylyo6C3tdl2CJWvCw9sw1LyU/PiH3k98elKtmtPG/CPZhIqKMnNVZML/o6I\nx3Gzw+oInrxUosmDQD15qYTrO5o6luOFbSL13EFBKCaFvkSnv4hV95g6gjubJqivED1wjcTCk",
  "rxUosmDQD15qYTrO5o6luOFbSL13EFBKCaFvkSnv4hV95g6gjubJqivED1wjcTCk1yFHVlEzAyfMNt07GAQjTIsBlj\nKNOizDm5+KH1DJFaN5fFXJibVfeCKo3QvW5wuTgKynBzOdXHB6gjAZNPoO0VGOWo2ROzZROXw0LDaeY6+yvp7wpOq2Yn27UG/YH\nbKMORno08HzGxqCNRXbBPcdYlieVoD+paLNe3e1ZtvqOLO3Y4bpNSepte+m2He4VP",
  "KMORno08HzGxqCNRXbBPcdYlieVoD+paLNe3e1ZtvqOLO3Y4bpNSepte+m2He4VPeBnW47ebhGPWNSRqK62h9QjlqM9qMudxy3X\nKByu25Sk3nkenbDXZho+Uf7E9idm1SKsdm25fKYRPCoqaidoqp2eF2xSaExaTsWvAbK3sCbh5dqwlhcacQXc0EsDTmEg+hCWGxOY\nW7ZhvD6pZD3XKrTGYTZDYhLD5iCR51E8JiTMXYKZ6yLENiEyJ5nOA8T",
  "+hCWGxOY\nW7ZhvD6pZD3XKrTGYTZDYhLD5iCR51E8JiTMXYKZ6yLENiEyJ5nOA8TmgeMyxlLgnPSOaYEbKkXAsqn6RdyQSwNEWtTR2NQ9kqlC\nDbRDLBV15hXPlKbSKFV3FA1fDgysa1gxVaAJY2ibnmD/cdp5kAU4xbLNcSc4EsjKawB3s7FBnvsLors5ILo0tJLSi8svaD0NJDS\nnNLyRNBED23lDydBNG5peUHlh6QGlpaUnpwNIBpZGlEaUP",
  "o0tJLSi8svaD0NJDS\nnNLyRNBED23lDydBNG5peUHlh6QGlpaUnpwNIBpZGlEaUPLX1IaWhpSOm6peuUakvJjhTuCJbuUzqxdELpkaVHlL6w9AWljy19TOl\nLS19S+sbSN5Tet/Q+pcxSRumGpRuUckvJq4MgWrN0jdLAUvLsB+eapTuUZpZmlD6w9AGlY0vJUzHczywl2xu4MVoqKX1i6RNKhaXk\n+S2Inln6jNLE0oTSp5Y+pfS1pa8pfWTpI0",
  "JUzHczywl2xu4MVoqKX1i6RNKhaXk\n+S2Inln6jNLE0oTSp5Y+pfS1pa8pfWTpI0pjS8m7AdidWLpHqX0LVBWU7lq6S+mZpWfu9wJ8MY2Ba2Fu2wq2KU0tTSndtJQ8KcBWwt\nJTsp+MVHtVm79tIte1SC24g7UZnx9Nch6pBXew9uo0P5pcnyK14BPS9Y2DxYsUSClc6UdLK38FpYWDu70+j/37u7eXbm31r6hve59\n7X3jfev1vV+8e95jb8cbe",
  "YsUSClc6UdLK38FpYWDu70+j/37u7eXbm31r6hve59\n7X3jfev1vV+8e95jb8cbeKH3p/eX97f3z3J/+XD51fIfjfrOtfaYL73OZ1n8BwXj4nc=</latexit>h2 = a[\u03b21 + \u23261h1]\n<latexit sha1_base64=\"bnyc6wZgKO+WNFXnqbelP1nMXEQ=\"\n>AWpXiclZhbU9w2FICdXtP0RtopL3xlMn0lu5AJr28dCaBkBukLIEFEpbsyF7ZqyDLxpZhiWd",
  "hbU9w2FICdXtP0RtopL3xlMn0lu5AJr28dCaBkBukLIEFEpbsyF7ZqyDLxpZhiWd/Q39NX9vf0X/TI9u7is8RD2Ums\nfZ8n3U5km3ZQSZFoVdX/732zrvf/Bh9c/uvHxJ59+9vnSzS8OirTMQz4IU5nmRwEruBSKD7TQkh9lOWdJIPlhcLph+OE5zwuRqn1\n9mfGThMVKRCJkGkKjpe/9YRBd+r+bQ8A1G1V3Zv6P8GMn4XH9C8oTcxwtraz2Vus",
  "9mfGThMVKRCJkGkKjpe/9YRBd+r+bQ8A1G1V3Zv6P8GMn4XH9C8oTcxwtraz2Vus/nxbW2sK1/71Rze/Gg/HaVgmXOlQsqI4XlvN\n9EnFci1CyWc3hmXBMxaespgfQ1GxhBcnVT2mX8LImM/SnP4p7RfR98+o2JUVwmAZgJ05MCMxN0seNSR7+dVEJlpeYqbBqKSunr1D\ncJ8sci56GWl1BgYS6gr34YTkLNaTxlDxizBNEqbG1XB9c3dWDQMeC1",
  "BqKSunr1D\ncJ8sci56GWl1BgYS6gr34YTkLNaTxlDxizBNEqbG1XB9c3dWDQMeC1Xxs7JO6WzWdTZrh0PxKmP9yf6iFqF5It5wUkmtmEquEHg8\nqyrei3sYCA5A9DgBqeIF1GnyE0T+GqKwhCRg4E6hc5F/vMZqVpHkNOtpLokEhk3zasTaIBVOZdJQ9UHz/lm8A1znMAnQVDhzNwV\n7G1Gx+nuZTnSdVYWK4hZypmNdNwJBDJs2IuoYqpYRTw47",
  "lm8A1znMAnQVDhzNwV\n7G1Gx+nuZTnSdVYWK4hZypmNdNwJBDJs2IuoYqpYRTw471B7aeM3XaJi7N6q7mJoKs/bzr6JzmRY27Th1BFizCuGvVEWRJuODHLG\nQ5bY8gEnvom4VaGwKsjC7Odp0G07MxG8NqcZXC9db7Mi6T9nKCMmAFefOQqmQt7VN9KF7c+Tc17psCn/gQmq3sKy+NmWPNGYFRtb\nEbNOlfIpNmCUJ5edE3TG4fKM9EdoAngi67M",
  "c17psCn/gQmq3sKy+NmWPNGYFRtb\nEbNOlfIpNmCUJ5edE3TG4fKM9EdoAngi67MhYre0m7XJViyJjy8DUPNS8mPf+r9zKcn1aq5bMx/JtQUVFmropM+H9UNIZHDF5fEMG\nTl0o0eRCoJy+VcH9HU8dyvLBNpJ47KAjFpNCX6PIXseqeU0dwZ9ME9RUCpl4MqHQJEdRVzYBI8MRHpaOBRSiQYbNGEOZFmXOyc0P\nrWeI1Lq5LebCPKy6N1RphO59",
  "qHQJEdRVzYBI8MRHpaOBRSiQYbNGEOZFmXOyc0P\nrWeI1Lq5LebCPKy6N1RphO59g8vFWVCGh8M5v+L0AGU0aPIZpKUasxwlc2qmdPpqWGi4xFxXfz3lTdFpxfxsq20P+gWzU4YhPxt4f\nmIiUdieqC3YmzLksR3tQ12K5vt2zauvVD2Rpxw7XbUpSb9tLt+1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN2mJPXO\n8+i0He7CRMs/2p",
  "1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN2mJPXO\n8+i0He7CRMs/2p/A5tRsk1I5Ntu+VA6bEBY1FbVTM0Gtys2ISwmZdeC31jZE/Dw6FpNCIv9QnQ1E8DSmEs8hCaExeYS7ptDKvbDn\nXbrTKZTZDZhLD4iCV41E0IizEVY6d4yrIMiU2I5HGC8zihecywlLkPCOZY0bIknItqHySdiUTwNIUtTZ1NAY9kKlCDbZBLBd05RX\nOlaf",
  "C8zihecywlLkPCOZY0bIknItqHySdiUTwNIUtTZ1NAY9kKlCDbZBLBd05RX\nOlafQKlZ0FQ9cDQ+uaFgzVKEJYGmHXGP+cMd5kQU4xbDNciU5E8jKaAL72OlTZ7C6K7OTghdjS0ovL2g9NDSQ0pzS8kbQRA9t\n5S8nQTRuaXnlB5YekBpaWlJ6cDSAaWRpRGlDy19SGloaUjphqUblGpLyY4UngiW7lM6sXRC6ZGlR5S+sPQFpY8tfUzpS0tfUvr",
  "lDy19SGloaUjphqUblGpLyY4UngiW7lM6sXRC6ZGlR5S+sPQFpY8tfUzpS0tfUvrG0je\nU3rf0PqXMUkbpqWblHJLyaeDIFq3dJ3SwFLy7gfXmqV9SjNLM0ofWPqA0rGl5K0YnmeWku0NPBgtlZQ+sfQJpcJS8v4WRM8sfUZp\nYmlC6VNLn1L62tLXlD6y9BGlsaXk2wDsTizdo9R+BaoKSnct3aX0zNIz93cBvpjGwLUwd2wFO5SmlqaUblK3hR",
  "GlsaXk2wDsTizdo9R+BaoKSnct3aX0zNIz93cBvpjGwLUwd2wFO5SmlqaUblK3hRgK2HpKdlPRq9q8\n2/NpH7WqQW3MHajM/PJjmP1I7WHt3mp9N7k+RWvAJ6frmweJDCqQU7vSjpZU1/BWFg7u9NZ+6d3dvbtyb739Qnvd+9r7xvOW/N+\n9e5j72+N/BC70/vL+9v75/lb5efLe8vHzTqO9fac70On/Lo/8AePXeYQ=</latexit>y = \u03b22 +",
  "/BC70/vL+9v75/lb5efLe8vHzTqO9fac70On/Lo/8AePXeYQ=</latexit>y = \u03b22 + \u23262h2\n\ud835\udf14 : omega\n\u03a9 : Omega",
  "Notation change #3\n<latexi\nt sha1_base64\n=\"vsvFeJd7CA\nhbv2jJIiHEd2W\nOncw=\">AWtX\niclZhb9s2FI\nDV7tZ1t3TD8rI\nXYUGBYesMZ+jW\n7WFAmzS9JV2cJ\nk7Sxq5ByZTMh\nqIUiUqcCv4t+z\nV73Z73b3Yoy2Z\n1DvMwA4np83\ni5ZCSKAWZFIXu\ndv+9dv29z/48\nKMbH9/85NPv\n9i5daXh0Va5i\nHvh6lM8+OAFVw\nKxftaMmPs5yz\nJD8KDjd",
  "z/48\nKMbH9/85NPv\n9i5daXh0Va5i\nHvh6lM8+OAFVw\nKxftaMmPs5yz\nJD8KDjdNPzo\nnOeFSNWBvsz4M\nGxEpEImYbQaO\nU3fxBE/93f5A\nE6bSCHz6bDS\nP9AmU9YRrNq6\nsx8WP/zpIBfxR\nA/90cpat9OtP\nz4trDeFNa/59E\na3vh4PxmlYJlz\npULKiOFnvZnpY\nsVyLUPLZzUFZ\n8IyFpyzmJ1BUL\nOHFsKrHOPNvQ2\nTsR2kOf0r7df",
  "KiOFnvZnpY\nsVyLUPLZzUFZ\n8IyFpyzmJ1BUL\nOHFsKrHOPNvQ2\nTsR2kOf0r7df\nTdIyqWFMVlEoC\nZMD0pMDNBFzsp\ndfTrsBIqKzVX4\nbyhqJS+Tn2TM\nH8sch5qeQkFu\nYC+uqHE5azUEN\nabw4UvwjTJGF\nqXA02tvZmkEIe\nC1Xxs7JO8WzWd\nrZqh0PxKmPj6c\nGyFqF5It5yUk\nmtmEquEHg8qyr\neiTsYCA5AdDgB\nqeIF1GnyA2tg",
  "0PxKmPj6c\nGyFqF5It5yUk\nmtmEquEHg8qyr\neiTsYCA5AdDgB\nqeIF1GnyA2tg\nHVFYUhJwZVfJi\nxmpWmkeQ05a2i\nuiQSGTfNqyNok\nFU5m0lH1QfP+\n2bwDXOcwCdBW+\nOJqD/Yyp2eI4z\nac6T6rCxHALO\nVMxr5uAIYdMmh\nG1DVKCYeGLes\nPbL1g6rRJXJrV\nXc1NBFkHedvR\nOc2LGredOoIsW\nIRx26ojyJwAR\nizhEGWm/IBp\nz4J",
  "6rRJXJrV\nXc1NBFkHedvR\nOc2LGredOoIsW\nIRx26ojyJwAR\nizhEGWm/IBp\nz4JuJWhcKqIAu\nzl6dBu+3MRPDa\nnGZwvrS9rYqk/\n5yhjJgAnH3mW\nzAV8ra+mS5tf5\nGc89o3BT71JzB\nZ7UNYHs+HtWg\nERtXEZtSsc4VM\nmi0I5elF2zS9c\nag8E+0BmgA+6c\npcqOgd7U5dgi\nVrwoM7MNS8lPz\nkx87PfDqsua0\nMf9INqGiosxc\nFZnw",
  "mgA+6c\npcqOgd7U5dgi\nVrwoM7MNS8lPz\nkx87PfDqsua0\nMf9INqGiosxc\nFZnw/6hoDLcv\nL4gicvlWjyIF\nBPXirh+o6mjuV\n4YZtIPXdQEIp\nJoS/R6S9i1T6m\njuDOpgnqKwRMv\nfDNhEKTHEVt2\nQSMDN9w83QsoB\nANMpyPMZRpUea\ncXPzQeoZIrZvL\nYi7Mzap9QZVG\naF83uFweBW4O\nZzKw4PUEaDeT\n6DtFRjlqNkTs\n2UTl8PCg",
  "ZvL\nYi7Mzap9QZVG\naF83uFweBW4O\nZzKw4PUEaDeT\n6DtFRjlqNkTs\n2UTl8PCg2nmOv\nsr6d8XnRaMT/b\nbtqDfsHslGHI\nz0beD5iYlFHo\nrpgt+KsSxL0R\n7UtVyu7/as2n7\n9PVnascN1m5L\nU2/TSbTvcK3rA\nz3Ycvd0hHrGoI\n1FdTQ+pRyxHe\n1CXO487rlE4XL\ncpSb2LPDpth7s\n0fKPDsxm1GyT\nUjk275UDuYh\nLGoqaqeYJjxG",
  "CXO487rlE4XL\ncpSb2LPDpth7s\n0fKPDsxm1GyT\nUjk275UDuYh\nLGoqaqeYJjxG4\njyExaRsW/AbK/\nsCbh5tax7CYq\n8Qbc0EsDTmEg9\nhHsLi/BRum0M\nqzsOdcetMplNk\nDkPYfExS/Co5\nyEsxlSMneIpyz\nIkzkMkjxOcxwn\nNY4alzCXhGck\ncM0KWlGtB5ZO0\nLZkAlqaotamjM\neiBTBVqsAliua\nAr3CuPIVWsa\nKruO9quH9Fw5q",
  "WlGtB5ZO0\nLZkAlqaotamjM\neiBTBVqsAliua\nAr3CuPIVWsa\nKruO9quH9Fw5q\nhCk0AS7vkHPMH\nu86TLMAphm2W\nK8mZQFZGE9jDT\no86i91fEFVkJx\ndEl5ZeUnph6QW\nlR5YeUZpbSp4\nIguiFpeTpJIjO\nLT2n9NDSQ0pLS\n0tK+5b2KY0sj\nSh9ZOkjSkNLQ0\no3Ld2kVFtKdqR\nwR7D0gNKJpRNK\njy09pvSlpS8p\nfWLpE0pfWfqK0",
  "kjSkNLQ0\no3Ld2kVFtKdqR\nwR7D0gNKJpRNK\njy09pvSlpS8p\nfWLpE0pfWfqK0\nreWvqX0gaUPKG\nWMkq3LN2ilF\ntKXh0E0YalG5Q\nGlpJnPzjXLO1R\nmlmaUfrQ0oeUj\ni0lT8VwP7OUb\nG/gxmipPSpU\n8pFZaS57cgem7\npc0oTSxNKn1n\n6jNI3lr6h9LGl\njymNLSXvBmB3Y\nuk+pfYtUFVQum\nfpHqVnlp653w\nvw5TQGroW5ayv\nYpT",
  "h9LGl\njymNLSXvBmB3Y\nuk+pfYtUFVQum\nfpHqVnlp653w\nvw5TQGroW5ayv\nYpTS1NKV021Ly\npABbCUtPyX4y\nUs1VbfG2iVzXI\nrXkDtZkfHE0yX\nmkltzBmqvT4mh\nyfYrUk9I17c\nOly9SIKVwpR+t\nrK3jt7C0cPhTZ\n/2Xzt29u2v3N\n5o3tDe8b7xve\n+8de+ed974vW\n8vhd6f3p/eX97\n/6zeWx2ujlej\nuXr9WnPMV17rs\n5r+BwJ",
  "xve\n+8de+ed974vW\n8vhd6f3p/eX97\n/6zeWx2ujlej\nuXr9WnPMV17rs\n5r+BwJv5Xc=</\nlatexit>h = a [\u27130 + \u2713x]\n<latexi\nt sha1_base64\n=\"V3JmSB8gD8\nhg/vLGAqXrvlg\noUI0=\">AWtH\niclZhb9s2FI\nDVXbvulm5YXvY\niLCg2bJ3hDF3X\nlwFt0vSWdEmaO\nEkbpwYlUzIbi\nlIkKnEq+K/s1+\nx1e9+/2aEkm9U\n5zMtGbO94m\nXQ",
  "0vSWdEmaO\nEkbpwYlUzIbi\nlIkKnEq+K/s1+\nx1e9+/2aEkm9U\n5zMtGbO94m\nXQ1KiFWRSFLrf\n/fae+9/8OFH\n1/5Man3+xZ\ndLN786KNIyD/\nkgTGWaHwWs4FI\noPtBCS36U5Zwl\ngeSHwem64Yfn\nPC9Eqvb1ZcZPE\nhYrEYmQaQiNlu\n75wyCafO/7g+\nTIJ1W8JfPZkP\nJI30M5awQo6o/\n+wmKO4Wo3WEu4\nok+GS2t9Hv9+\nuPTwmpbW",
  "+\nTIJ1W8JfPZkP\nJI30M5awQo6o/\n+wmKO4Wo3WEu4\nok+GS2t9Hv9+\nuPTwmpbWPHaz8\n7o5jfj4TgNy4Q\nrHUpWFMer/Uyf\nVCzXIpR8dmNY\nFjxj4SmL+TEUF\nUt4cVLVQ5z5ty\nAy9qM0h39K+3\nX03SsqlhTFZRK\nAmTA9KTAzQRc7\nLnV076QSKis1V\n2HTUFRKX6e+y\nZc/FjkPtbyEAg\ntzAX31wnLWag\nhqzeGil+EaZI\nwNa6Gaxu7",
  "V\n2HTUFRKX6e+y\nZc/FjkPtbyEAg\ntzAX31wnLWag\nhqzeGil+EaZI\nwNa6Gaxu7M0g\nj4Wq+FlZ3g26\nzobtcOheJWx9n\nR/UYvQPBFvOa\nmkVkwlVwg8nlU\nV78U9DAQHIHqc\ngFTxAuo0+YEV\nsIorCgJuLJr5\nMWMVK0jyEnHe\n0V0aCQST7tWOv\nEgqlMOsoeKL5\n/yzeA6xmAboK\nXxzNwV7G1Gx+n\neZTnSdVYWK4h\nZypmNdNwJBDJs",
  "lMOsoeKL5\n/yzeA6xmAboK\nXxzNwV7G1Gx+n\neZTnSdVYWK4h\nZypmNdNwJBDJs\n2IuoYqpYRLw47\n1B7ZeMHXaJi7N\n6q7mJoKs/bzr\n6JzmRY27Th1BF\nizCuGvVEWRJ2P\n9jljDIclsewY\nAT30TcqlBYFWR\nh7uRp0G07MxG8\nNqcZ7Jeut1GR9\nJ8zlBETgN1nv\ngVTIe/q6+nC9u\nfJOa9U+BTfwK\nT1b2E5XEzrHk\njMKo2NqNmnStk\n0",
  "BETgN1nv\ngVTIe/q6+nC9u\nfJOa9U+BTfwK\nT1b2E5XEzrHk\njMKo2NqNmnStk\n0mxBKE8vuqbpj\nUPlmegO0ATwpi\ntzoaJ3tNt1CZ\nasCQ9vw1DzUvL\njn3u/8ulJ1Tfb\nxvxHsgkVFWXm\nqsiE/0dFY3ji4\nPUFETx5qUSTB4\nF68lIJ93c0dSz\nHC9tE6rmDglB\nMCn2Jtr+IVfea\nOoI7myaorxAw9\ncI3EwpNchR1Z\nRMwMnzDs9OxgE\nI",
  "6rmDglB\nMCn2Jtr+IVfea\nOoI7myaorxAw9\ncI3EwpNchR1Z\nRMwMnzDs9OxgE\nI0yLAZYyjTosw\n5ufmh9QyRWje3\nxVyYh1X3hiqN\n0L1vcLm4Csrwc\nDjnV1weoIwGT\n6DtFRjlqNkTs\n2UTl8PCw1bzLX\n76ylvik4r5meb\nbXvQL5idMgz5\n2WgTz0dMLOpIV\nBcVpx1SWI52o\nO6Fsv13Z5Vm69\n/JEs7drhuU5J\n6216bYd7RQ/4\n2Zaj",
  "MLOpIV\nBcVpx1SWI52o\nO6Fsv13Z5Vm69\n/JEs7drhuU5J\n6216bYd7RQ/4\n2Zajt1vEIxZ1J\nKqr7SH1iOVoD\n+py53HLNQqH6z\nYlqXeR6ftcBc\nmWv7R/oRrZo5J\nqRybY18qh0I\ni5qK2imCY+R2\nISwmJRdC/7Gyp\n6Ah0fXakJYhD\nN1VzMBLI25xEN\noQlhstnDXbGNY\n3XKoW26VyWyCz\nCaExcswaNuQ\nliMqRg7xVOWZU\nhsQiSPE5",
  "N\noQlhstnDXbGNY\n3XKoW26VyWyCz\nCaExcswaNuQ\nliMqRg7xVOWZU\nhsQiSPE5zHCc1\njhqXMJeEZyRw\nzQpaUa0Hlk7Qr\nmQCWpqi1qaMx6\nIFMFWqwDWK5oC\nuvcK48hVaxoq\nt4Gp4cEXDmqE\nKTQBL2SP+cNt\n5yYLcIrhmOVK\nciaQldE7mBnh\nzrz018QVeQkF0\nSXl5SemHpBaW\nHlh5SmltKfhE\nE0QtLya+TIDq3\n9JzSA0sPKC0t",
  "z018QVeQkF0\nSXl5SemHpBaW\nHlh5SmltKfhE\nE0QtLya+TIDq3\n9JzSA0sPKC0tL\nSkdWDqgNLI0o\nvSRpY8oDS0NKV\n23dJ1SbSk5kcI\nTwdJ9SieWTig9\nsvSI0peWvqT0\niaVPKH1l6StK3\n1r6ltIHlj6glF\nnKN2wdINSbi\nl5dRBEa5auURp\nYSn7wV6zdIfS\nzNKM0oeWPqR0b\nCn5VQzPM0vJ8\nQYejJZKSp9a+p\nRSYSn5/RZEzy1",
  "wV6zdIfS\nzNKM0oeWPqR0b\nCn5VQzPM0vJ8\nQYejJZKSp9a+p\nRSYSn5/RZEzy1\n9TmliaULpM0u\nfUfrG0jeUPrb0\nMaWxpeTdAJxOL\nN2j1L4FqgpKdy\n3dpfTM0jP3ew\nG+mMbAtTC3bQX\nblKaWpRuWkp+\nKcBRwtJTcp6M\nVHtXm79tIve1S\nC24g7UZn19Nch\n6pBXew9u40v5r\ncnyK14BPS9Y2\nDxYsUSCnc6UdL\nK6v4LSwtHPzSW\n7",
  "Zn19Nch\n6pBXew9u40v5r\ncnyK14BPS9Y2\nDxYsUSCnc6UdL\nK6v4LSwtHPzSW\n73bu7N7Z+X+W\nvuG9r3rfed94\nO36v3m3feDv\newAu9P72/vL+9\nf5bvLg+Xw2Xe\nqO9da6/52ut8l\nlatexit>tV/cAvk1A=</\nh0 = a [ 0 +  h]\n<latexi\nt sha1_base64\n=\"itKSwzHaSz\nm5UIhVM1UlAW\nBEiQ=\">AWnH\niclZhb9s2FI\nDVXbvulm5YXgY",
  "=\"itKSwzHaSz\nm5UIhVM1UlAW\nBEiQ=\">AWnH\niclZhb9s2FI\nDVXbvulm5YXgY\nMwoKiw9YZztBd\nXga0SdNb0sVp4\niRtnBqUTMlsK\nEqRqMSu4Nf9mr\n1u/2X/ZoeybFb\nnMA8z0Io53yd\neDkndgkyKQne7\n/157593v/gw\n+sf3fj4k08/+3\nzl5heHRVrmIe\n+HqUz4AVXAr\nF+1poyY+znLMk\nkPwoONs0/OiC\n54VI1YGeZvw0Y\nbESk",
  "RVrmIe\n+HqUz4AVXAr\nF+1poyY+znLMk\nkPwoONs0/OiC\n54VI1YGeZvw0Y\nbESkQiZhtBwxf\nen/u/+Iiysbg\n9rLoz/4fFX+Y\n4vj1cWet2uvXP\np4X1prDmNb/e8\nOZXo8EoDcuEK\nx1KVhQn691Mn1\nYs1yKUfHZjUBY\n8Y+EZi/kJFBVL\neHFa1UOZ+bcg\nMvKjNId/Svt19\nO0zKpYUxTQJwE\nyYHheYmaCLnZ\nQ6+u20EiorNVf\nhvKGo",
  "bcg\nMvKjNId/Svt19\nO0zKpYUxTQJwE\nyYHheYmaCLnZ\nQ6+u20EiorNVf\nhvKGolL5OfZMX\nfyRyHmo5hQILc\nwF9cMxy1moI\nXs3BopfhmSMD\nWqBhtbe7NqEPB\nYqIqfl3UmZ7O\n2s1U7HIpXGRtP\nDpa1CM0T8YaTS\nmrFVHKFwONZVf\nFO3MFAcACiw\nlIFS+gTpOfIPL\nXEYWVIwED9IJ\ndC7yn89I1Urz\nGHLS0l4SDQqZ5\nJOWtUksmM",
  "lIFS+gTpOfIPL\nXEYWVIwED9IJ\ndC7yn89I1Urz\nGHLS0l4SDQqZ5\nJOWtUksmMqkpe\nyD4vu3fAO4zmE\nWoKtw4GgO9jO\nmZovzNJ/oPKkK\nE8Mt5EzFvG4Ch\nhwyaUbUNlQpJ\nZwatqw/sPWcqb\nMmcWlWdzU3EWQ\nd5G1H5zQvatR2\n6giyYBHGbauO\nIEvCPh+xhEGWm\n/IQBpz4JuJWhc\nKqIAuzl6dBu+\n3MRPDanGSwX9r\neVkXSf8FQ",
  "IEvCPh+xhEGWm\n/IQBpz4JuJWhc\nKqIAuzl6dBu+\n3MRPDanGSwX9r\neVkXSf8FQRkwA\ndp85CqZC3tY30\n6XtL5JzUfumw\nCf+GCarfQrL4/\nmwFo3AqJrYjJp\n1rpBJswWhPL1\nsm6Y3DpVnoj1A\nE8CbrsyFit7S7\ntQlWLImPLgDQ8\n1LyU9+7PzMJ6\ndV12wb8x/JlR\nUlJmrIhP+HxWN\n4M6C1xdE8OSl\nEk0eBOrJSyVc3\n9HUsRwvbBO",
  "dV12wb8x/JlR\nUlJmrIhP+HxWN\n4M6C1xdE8OSl\nEk0eBOrJSyVc3\n9HUsRwvbBOp5w\n4KQjEp9BRtfxG\nr9jl1BHc2TVB\nfIWDqhSMTCk1y\nFLVlEzAyHOEe6\nVhAIRpkOB9jK\nNOizDm5+KH1DJ\nFaN5fFXJibVfu\nCKo3Qvm5wuTwL\nynBzuOBXnB6g\njAbzfAZpqUYsR\n8mcmCmdvBoUGr\naYa/fXUz4vOq\n2Yn2837UG/YHb\nKMOTnw208H",
  "AbzfAZpqUYsR\n8mcmCmdvBoUGr\naYa/fXUz4vOq\n2Yn2837UG/YHb\nKMOTnw208HzGx\nqCNRXfBQ4qxL\nEsvRHtS1XK5v9\n6zafvU9Wdqxw3\nWbktTb9NJtO9w\nresDPdxy93SE\nesagjUV1ND6lH\nLEd7UJc7jzuU\nThctylJvYs8O\nm2HuzTR8o8Oxl\nwz85iUypF57Ev\nlYB7Coqaidop\nwmMkzkNYTMq2\nBX9jZV/AzaNtz\nUNY7BWirZkAl",
  "85iUypF57Ev\nlYB7Coqaidop\nwmMkzkNYTMq2\nBX9jZV/AzaNtz\nUNY7BWirZkAlk\nZc4iHMQ1icb+\nG2cSwuNQd9w\nqk9kYmfMQFh+x\nBI96HsJiTMXYK\nZ6xLEPiPETyO\nMZ5HNM8ZljKXB\nKekcwxI2RJuRZ\nUPk7bkglgaYJ\namzgagx7IVKEG\nmyCWC7ryCufKU\n2gVK7qK+6G+1\nc0rBmq0ASwtE\nv2mD/YdW6yAKc\nYHrNcSc4EsjKa\nw",
  "7ryCufKU\n2gVK7qK+6G+1\nc0rBmq0ASwtE\nv2mD/YdW6yAKc\nYHrNcSc4EsjKa\nwB52etRZP0F\nUWe5IJoaumU0\nktLyk9svSI0t\nxS8kYQRM8tJW8\nnQXRh6QWlh5Y\neUlpaWlLat7RP\naWRpROlDSx9SG\nloaUrp6Sal2\nlLyRAp3BEsPKB\n1bOqb02NJjSl9\nY+oLSx5Y+pvSl\npS8pfWPpG0rv\nW3qfUmYpo3TL0\ni1KuaXk0EQbV\ni6QWlg",
  "Sl9\nY+oLSx5Y+pvSl\npS8pfWPpG0rv\nW3qfUmYpo3TL0\ni1KuaXk0EQbV\ni6QWlgKXn3g7\n1maY/SzNKM0ge\nWPqB0ZCl5K4b7\nmaXk8QZujJZKS\np9Y+oRSYSl5f\nwuiZ5Y+ozSxNK\nH0qaVPKX1t6Wt\nKH1n6iNLYUvJ\ntAJ5OLN2n1H4F\nqgpK9yzdo/Tc0\nnP3dwG+nMbAtT\nB3bQW7lKaWp\nRuW0reFOBRwtI\nz8jwZqeaqtvja\nRK5rkVp",
  "c0\nnP3dwG+nMbAtT\nB3bQW7lKaWp\nRuW0reFOBRwtI\nz8jwZqeaqtvja\nRK5rkVpyB2sy\nvjib5DxS+5gz\ndVpcTa5PkVqyc\nek61uHyw8pkFK\n40g9X1tbxV1h\naOPyps/5L5+7e\n3bV7G80X2uve1\n963nfeuverd\n897PW8vhd6f3\np/eX97/6x+s/p\ngdXv12Vx951pz\nzpde67d6+B8B\n>pdmB</latexit\ny = \u03c60\n0 + \u03c60h0\n<latexit sha1_ba",
  "dXv12Vx951pz\nzpde67d6+B8B\n>pdmB</latexit\ny = \u03c60\n0 + \u03c60h0\n<latexit sha1_base64=\"UGmj8AB2OH/n/0YCMA80y0J0vI=\"\n>AWq3iclZhbU9tGFICV9JbSG2mnvPRFUyYznTZx7U56elMAiE3SIGAgYCJZyWv5A2rlZBWYKLxz+iv6Wv7I/pvelaWvdE5y0M9A\n16f79Nezq6klYJMikJ3u/euPne+x98+NGtj5c+fSz79Yv3lQZGWecj7YS",
  "5y0M9A\n16f79Nezq6klYJMikJ3u/euPne+x98+NGtj5c+fSz79Yv3lQZGWecj7YSrT/ChgBZdC8b4WvKjLOcsCSQ/DM7WDT+84HkhUrW\nvrzJ+mrBYiUiETENouPyjPwi8bDqTf3fTZGdwL+AazasulP/B/ixnfC4/gXlyelwebXb6dYfnxZ6TWHVaz47w9tfjwajNCwTrnQo\nWVGc9LqZPq1YrkUo+XRpUBY8Y+EZi/kJFBVLeHFa1SOb+ncgMv",
  "fjwajNCwTrnQo\nWVGc9LqZPq1YrkUo+XRpUBY8Y+EZi/kJFBVLeHFa1SOb+ncgMvKjNIc/pf06+u4RFUuK4ioJwEyYHheYmaCLnZQ6+u20EiorNVfhrK\nGolL5OfZMmfyRyHmp5BQUW5gL6odjlrNQzKXBopfhmSMDWqBmsbu9NqEPBYqIqfl3Vip9O2s1E7HIrXGWvP9he1CM0T8ZaTSmrF\nVHKNwONpVfFO3MFAcACiwlIFS+gTpOfIPJ7iMJC",
  "HIrXGWvP9he1CM0T8ZaTSmrF\nVHKNwONpVfFO3MFAcACiwlIFS+gTpOfIPJ7iMJCkoCB+kEOhf5L6ekaqV5DlpacdEg0Im+aRlrRMLpjJpKXug+P4d3wCuc5gF6C\np8cTQHexlT0/lxmk90nlSFieEWcqZiXjcBQw6ZNCNqG6qUEg4NW9Yf2HrJ1FmTuDSru5qbCL287ajc5oXNWo7dQRZsAjtlVHkCX\nhtB+xhEGWm/IQBpz4JuJWhcKqIAtzJ0",
  "5qbCL287ajc5oXNWo7dQRZsAjtlVHkCX\nhtB+xhEGWm/IQBpz4JuJWhcKqIAtzJ0+DdtuZieC1OcngfGl7GxVJ/wVDGTEBOPvMt2Aq5G19PV3Y/jw5F7VvCnzij2Gy2oewPJ4Na\n94IjKqJTalZ5wqZNFsQytPLtml641B5JtoDNAF80pW5UNE72t26BEvWhAd3Yah5KfnJvc7PfHJadc1pY/6RbEJFRZm5KjLh/1HRCG4\n0eH1BE9eKtHkQaCev",
  "Ad3Yah5KfnJvc7PfHJadc1pY/6RbEJFRZm5KjLh/1HRCG4\n0eH1BE9eKtHkQaCevFTC9R1NHcvxwjaReu6gIBSTQl+h01/Eqn1MHcGdTRPUVwiYeuGbCYUmOYrasgkYGb7hlulYQCEaZDgbYyjT\nosw5ufih9QyRWjeXxVyYm1X7giqN0L5ucLk4Cspwc7jg1xweoIwGs3wGalGLEfJnJgpnbweFBpOMdfZX0/5rOi0Yn6+2bQH/YLZKc\nOQnw83",
  "1xweoIwGs3wGalGLEfJnJgpnbweFBpOMdfZX0/5rOi0Yn6+2bQH/YLZKc\nOQnw838XzExKORHXBHsVZlySWoz2oa7Fc3+1Ztfn6e7K0Y4frNiWpt+ml23a41/SAn285ertFPGJR6K6mh5Sj1iO9qAudx63XKNw\nuG5TknrneXTaDndhouUf7Y9hZ2q2SakcmW1fKgezEBY1FbVTM3uti3OQlhMyrYFv7GyJ+Dm0bZmISzuFKtmQCWRlziIcxCWJy",
  "W1fKgezEBY1FbVTM3uti3OQlhMyrYFv7GyJ+Dm0bZmISzuFKtmQCWRlziIcxCWJydwm\n2ziWF1y6FuVUmszEyZyEsPmEJHvUshMWYirFTPGNZhsRZiORxjPM4pnMsJS5JDwjmWNGyJyLah8nLYlE8DSBLU2cTQGPZCpQg0\n2QSwXdOUVzpWn0CpWdBX3XQ3r2lYM1ShCWBpm5xj/mDbeZIFOMWwzXIlORPIymgCd7CzQ5357i+IKrKTC6IrS68ov",
  "2lYM1ShCWBpm5xj/mDbeZIFOMWwzXIlORPIymgCd7CzQ5357i+IKrKTC6IrS68ovbT0ktJDSw8pz\nS0lTwRB9NJS8nQSRBeWXlB6YOkBpaWlJaV9S/uURpZGlD629DGloaUhpeuWrlOqLSU7UrgjWLpP6djSMaVHlh5R+srSV5Q+tfQpce\nWHlP61tK3lD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8uwH5qlO5RmlmaUPrL0EaUjS8lTMdzPL",
  "9CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8uwH5qlO5RmlmaUPrL0EaUjS8lTMdzPLCXbG7gxWiopfWbpM0qFpeT5\nLYheWPqC0sTShNLnlj6n9I2lbyh9YukTSmNLybsB2J1YukepfQtUFZTuWrpL6bml5+73AnwxjYFrYW7bCrYpTS1NKd20lDwpwFbC0j\nOyn4xUc1Wbv20i17VILbiDNRmfH01yHqkFd7Dm6jQ/mlyfIrXgY9L1jYPFixRIKVz",
  "yn4xUc1Wbv20i17VILbiDNRmfH01yHqkFd7Dm6jQ/mlyfIrXgY9L1jYPFixRIKVzph8urPfwWlhYOfur0func372/+mCteUN7y/vG\n+9b7zut5v3oPvKfejtf3Qu9P7y/vb+flXsreyvHK4OZevNGc8xXuzwv8DIu/g1A=</latexit>h1 = a[\u03b20 + \u23260x]\n<latexit sha1_base64=\"UwfMPn/m/6auzwL2FXGTyWOmVOc=\"\n>AWr3iclZhbU9w2FICd",
  "sha1_base64=\"UwfMPn/m/6auzwL2FXGTyWOmVOc=\"\n>AWr3iclZhbU9w2FICdXtP0RtopL3xlMlMp0132DS9vHQmgZAbpEBgYQlW9krexVk2dgyLPHsD+mv6Wv7E/pvemR7V/E54qE7A\n6s932djmRbdpBJUejV1X+vfPue+9/8OH1j258/Mmn32+dPOLgyIt85APwlSm+VHACi6F4gMtORHWc5ZEkh+GJyuG354zvNCpGp\nfX2b8JGxEpEImY",
  "yIt85APwlSm+VHACi6F4gMtORHWc5ZEkh+GJyuG354zvNCpGp\nfX2b8JGxEpEImYbQaOlHfxhEk1F1Z+b/ZorsGP4FXLNR1Z/538OP7YTH9a9G7M9ORksrq73V+uPTQr8trHjtZ2d086vxcJyGZcKV\nDiUriuP+aqZPKpZrEUo+uzEsC56x8JTF/BiKiW8OKnq0c38WxAZ+1Gaw5/Sfh19+4iKJUVxmQRgJkxPCsxM0MWOSx39elIJlZWaq7\nBpK",
  "nq0c38WxAZ+1Gaw5/Sfh19+4iKJUVxmQRgJkxPCsxM0MWOSx39elIJlZWaq7\nBpKCqlr1PfpMofi5yHWl5CgYW5gL764YTlLNSQ0BtDxS/CNEmYGlfDtY3dWTUMeCxUxc/KOrmzWdfZqB0OxauMtSf7i1qE5ol4w0kl\ntWIquULg8ayqeC/uYSA4ANHjBKSKF1CnyU8Q+X1EYTFJwMCDdAqdi/znM1K10jyGnHS0l0SDQib5tGOtEwumMukoe6D4",
  "F1CnyU8Q+X1EYTFJwMCDdAqdi/znM1K10jyGnHS0l0SDQib5tGOtEwumMukoe6D4/i3fAK5zmA\nXoKnxNAd7GVOz+XGaT3WeVIWJ4RZypmJeNwFDpk0I+oaqpQSDg071u/Yes7UaZu4NKu7mpsIsvbzrqNzmhc17jp1BFmwCOuVUe\nQJeHUH7OEQZb8gGnPgm4laFwqogC3MnT4Nu25mJ4LU5zeB86XobFUn/OUMZMQE4+8y3YCrkX09Xdj+PDnt",
  "m4laFwqogC3MnT4Nu25mJ4LU5zeB86XobFUn/OUMZMQE4+8y3YCrkX09Xdj+PDntW8KfOpPYLK6h7A8b\noY1bwRG1cZm1KxzhUyaLQjl6UXNL1xqDwT3QGaAD7pylyo6C3tdl2CJWvCw9sw1LyU/PiH3k98elKtmtPG/CPZhIqKMnNVZML/o6I\nx3Gzw+oInrxUosmDQD15qYTrO5o6luOFbSL13EFBKCaFvkSnv4hV95g6gjubJqivED1wjcTCk",
  "rxUosmDQD15qYTrO5o6luOFbSL13EFBKCaFvkSnv4hV95g6gjubJqivED1wjcTCk1yFHVlEzAyfMNt07GAQjTIsBlj\nKNOizDm5+KH1DJFaN5fFXJibVfeCKo3QvW5wuTgKynBzOdXHB6gjAZNPoO0VGOWo2ROzZROXw0LDaeY6+yvp7wpOq2Yn27UG/YH\nbKMORno08HzGxqCNRXbBPcdYlieVoD+paLNe3e1ZtvqOLO3Y4bpNSepte+m2He4VP",
  "KMORno08HzGxqCNRXbBPcdYlieVoD+paLNe3e1ZtvqOLO3Y4bpNSepte+m2He4VPeBnW47ebhGPWNSRqK62h9QjlqM9qMudxy3X\nKByu25Sk3nkenbDXZho+Uf7E9idm1SKsdm25fKYRPCoqaidoqp2eF2xSaExaTsWvAbK3sCbh5dqwlhcacQXc0EsDTmEg+hCWGxOY\nW7ZhvD6pZD3XKrTGYTZDYhLD5iCR51E8JiTMXYKZ6yLENiEyJ5nOA8T",
  "+hCWGxOY\nW7ZhvD6pZD3XKrTGYTZDYhLD5iCR51E8JiTMXYKZ6yLENiEyJ5nOA8TmgeMyxlLgnPSOaYEbKkXAsqn6RdyQSwNEWtTR2NQ9kqlC\nDbRDLBV15hXPlKbSKFV3FA1fDgysa1gxVaAJY2ibnmD/cdp5kAU4xbLNcSc4EsjKawB3s7FBnvsLors5ILo0tJLSi8svaD0NJDS\nnNLyRNBED23lDydBNG5peUHlh6QGlpaUnpwNIBpZGlEaUP",
  "o0tJLSi8svaD0NJDS\nnNLyRNBED23lDydBNG5peUHlh6QGlpaUnpwNIBpZGlEaUPLX1IaWhpSOm6peuUakvJjhTuCJbuUzqxdELpkaVHlL6w9AWljy19TOl\nLS19S+sbSN5Tet/Q+pcxSRumGpRuUckvJq4MgWrN0jdLAUvLsB+eapTuUZpZmlD6w9AGlY0vJUzHczywl2xu4MVoqKX1i6RNKhaXk\n+S2Inln6jNLE0oTSp5Y+pfS1pa8pfWTpI0",
  "JUzHczywl2xu4MVoqKX1i6RNKhaXk\n+S2Inln6jNLE0oTSp5Y+pfS1pa8pfWTpI0pjS8m7AdidWLpHqX0LVBWU7lq6S+mZpWfu9wJ8MY2Ba2Fu2wq2KU0tTSndtJQ8KcBWwt\nJTsp+MVHtVm79tIte1SC24g7UZnx9Nch6pBXew9uo0P5pcnyK14BPS9Y2DxYsUSClc6UdLK38FpYWDu70+j/37u7eXbm31r6hve59\n7X3jfev1vV+8e95jb8cbe",
  "YsUSClc6UdLK38FpYWDu70+j/37u7eXbm31r6hve59\n7X3jfev1vV+8e95jb8cbeKH3p/eX97f3z3J/+XD51fIfjfrOtfaYL73OZ1n8BwXj4nc=</latexit>h2 = a[\u03b21 + \u23261h1]\n<latexit sha1_base64=\"bnyc6wZgKO+WNFXnqbelP1nMXEQ=\"\n>AWpXiclZhbU9w2FICdXtP0RtopL3xlMn0lu5AJr28dCaBkBukLIEFEpbsyF7ZqyDLxpZhiWd",
  "hbU9w2FICdXtP0RtopL3xlMn0lu5AJr28dCaBkBukLIEFEpbsyF7ZqyDLxpZhiWd/Q39NX9vf0X/TI9u7is8RD2Ums\nfZ8n3U5km3ZQSZFoVdX/732zrvf/Bh9c/uvHxJ59+9vnSzS8OirTMQz4IU5nmRwEruBSKD7TQkh9lOWdJIPlhcLph+OE5zwuRqn1\n9mfGThMVKRCJkGkKjpe/9YRBd+r+bQ8A1G1V3Zv6P8GMn4XH9C8oTcxwtraz2Vus",
  "9mfGThMVKRCJkGkKjpe/9YRBd+r+bQ8A1G1V3Zv6P8GMn4XH9C8oTcxwtraz2Vus/nxbW2sK1/71Rze/Gg/HaVgmXOlQsqI4XlvN\n9EnFci1CyWc3hmXBMxaespgfQ1GxhBcnVT2mX8LImM/SnP4p7RfR98+o2JUVwmAZgJ05MCMxN0seNSR7+dVEJlpeYqbBqKSunr1D\ncJ8sci56GWl1BgYS6gr34YTkLNaTxlDxizBNEqbG1XB9c3dWDQMeC1",
  "BqKSunr1D\ncJ8sci56GWl1BgYS6gr34YTkLNaTxlDxizBNEqbG1XB9c3dWDQMeC1Xxs7JO6WzWdTZrh0PxKmP9yf6iFqF5It5wUkmtmEquEHg8\nqyrei3sYCA5A9DgBqeIF1GnyE0T+GqKwhCRg4E6hc5F/vMZqVpHkNOtpLokEhk3zasTaIBVOZdJQ9UHz/lm8A1znMAnQVDhzNwV\n7G1Gx+nuZTnSdVYWK4hZypmNdNwJBDJs2IuoYqpYRTw47",
  "lm8A1znMAnQVDhzNwV\n7G1Gx+nuZTnSdVYWK4hZypmNdNwJBDJs2IuoYqpYRTw471B7aeM3XaJi7N6q7mJoKs/bzr6JzmRY27Th1BFizCuGvVEWRJuODHLG\nQ5bY8gEnvom4VaGwKsjC7Odp0G07MxG8NqcZXC9db7Mi6T9nKCMmAFefOQqmQt7VN9KF7c+Tc17psCn/gQmq3sKy+NmWPNGYFRtb\nEbNOlfIpNmCUJ5edE3TG4fKM9EdoAngi67M",
  "c17psCn/gQmq3sKy+NmWPNGYFRtb\nEbNOlfIpNmCUJ5edE3TG4fKM9EdoAngi67MhYre0m7XJViyJjy8DUPNS8mPf+r9zKcn1aq5bMx/JtQUVFmropM+H9UNIZHDF5fEMG\nTl0o0eRCoJy+VcH9HU8dyvLBNpJ47KAjFpNCX6PIXseqeU0dwZ9ME9RUCpl4MqHQJEdRVzYBI8MRHpaOBRSiQYbNGEOZFmXOyc0P\nrWeI1Lq5LebCPKy6N1RphO59",
  "qHQJEdRVzYBI8MRHpaOBRSiQYbNGEOZFmXOyc0P\nrWeI1Lq5LebCPKy6N1RphO59g8vFWVCGh8M5v+L0AGU0aPIZpKUasxwlc2qmdPpqWGi4xFxXfz3lTdFpxfxsq20P+gWzU4YhPxt4f\nmIiUdieqC3YmzLksR3tQ12K5vt2zauvVD2Rpxw7XbUpSb9tLt+1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN2mJPXO\n8+i0He7CRMs/2p",
  "1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN2mJPXO\n8+i0He7CRMs/2p/A5tRsk1I5Ntu+VA6bEBY1FbVTM0Gtys2ISwmZdeC31jZE/Dw6FpNCIv9QnQ1E8DSmEs8hCaExeYS7ptDKvbDn\nXbrTKZTZDZhLD4iCV41E0IizEVY6d4yrIMiU2I5HGC8zihecywlLkPCOZY0bIknItqHySdiUTwNIUtTZ1NAY9kKlCDbZBLBd05RX\nOlaf",
  "C8zihecywlLkPCOZY0bIknItqHySdiUTwNIUtTZ1NAY9kKlCDbZBLBd05RX\nOlafQKlZ0FQ9cDQ+uaFgzVKEJYGmHXGP+cMd5kQU4xbDNciU5E8jKaAL72OlTZ7C6K7OTghdjS0ovL2g9NDSQ0pzS8kbQRA9t\n5S8nQTRuaXnlB5YekBpaWlJ6cDSAaWRpRGlDy19SGloaUjphqUblGpLyY4UngiW7lM6sXRC6ZGlR5S+sPQFpY8tfUzpS0tfUvr",
  "lDy19SGloaUjphqUblGpLyY4UngiW7lM6sXRC6ZGlR5S+sPQFpY8tfUzpS0tfUvrG0je\nU3rf0PqXMUkbpqWblHJLyaeDIFq3dJ3SwFLy7gfXmqV9SjNLM0ofWPqA0rGl5K0YnmeWku0NPBgtlZQ+sfQJpcJS8v4WRM8sfUZp\nYmlC6VNLn1L62tLXlD6y9BGlsaXk2wDsTizdo9R+BaoKSnct3aX0zNIz93cBvpjGwLUwd2wFO5SmlqaUblK3hR",
  "GlsaXk2wDsTizdo9R+BaoKSnct3aX0zNIz93cBvpjGwLUwd2wFO5SmlqaUblK3hRgK2HpKdlPRq9q8\n2/NpH7WqQW3MHajM/PJjmP1I7WHt3mp9N7k+RWvAJ6frmweJDCqQU7vSjpZU1/BWFg7u9NZ+6d3dvbtyb739Qnvd+9r7xvOW/N+\n9e5j72+N/BC70/vL+9v75/lb5efLe8vHzTqO9fac70On/Lo/8AePXeYQ=</latexit>y = \u03b22 +",
  "/BC70/vL+9v75/lb5efLe8vHzTqO9fac70On/Lo/8AePXeYQ=</latexit>y = \u03b22 + \u23262h2\nBias \nvector\nWeight \nmatrix\n\ud835\udf14 : omega\n\u03a9 : Omega",
  "General equations for deep network\n<latexit sha1_base64=\"zA1nsyQYzyTQ\nTxAr4O/Qwo0w87c=\">AXxXiclZhb9s2FMft7tZlt3bDkIe9CAtaDFsbxFl3eRnQJk\n2TNuniNc2Tg1KpmQ1FKVIlONUMPaR9mn2sJftq+xQls3yHOZhBhJR5/fnIXl4SFHyMx\nEXamXl7/aN97/4MOPbn68Mmn3+xa3bXx4VaZkH/DBIRZqf+KzgIpb8UMVK8JMs5y\nzxBT/2z9c1Px7xvIhTeaC",
  "mn3+xa3bXx4VaZkH/DBIRZqf+KzgIpb8UMVK8JMs5y\nzxBT/2z9c1Px7xvIhTeaCuMn6WsEjGYRwBab+7fbWgtfzw2G/6ky8u7/pMjuFfz5Xr\nF+tTLwf4GY34VF9B+XxWU+mskx8nu93qzyqNyx6rcmcyacTr40eFg1XKwOpk1hRzc7\nY0GqSruzq3G67bD6/Z9u2P6vlFD6czDbq4aF01tu+6s5uTeQv/W0srySv3zaKHTFJZaz\na/bv/31oDd",
  "6/Z9u2P6vlFD6czDbq4aF01tu+6s5uTeQv/W0srySv3zaKHTFJZaz\na/bv/31oDdIgzLhUgWCFcVpZyVTZxXLVRwIPlnolQXPWHDOIn4KRckSXpxV9ZxPvDtg\nGXhmsOfVF5tfbdGxZKiuEp8UCZMDQvMtNHFTksV/npWxTIrFZfBtKGwFJ5KPZ1A3iDO\neaDEFRYkMfQVy8YspwFCtJsoSf5ZAmCZODqre2sTepej6PYlnxi7JOucnE1mzUGg7F",
  "DEFRYkMfQVy8YspwFCtJsoSf5ZAmCZODqre2sTepej6PYlnxi7JOucnE1mzUGg7F\n6xRrTw/mXmLFk/gtJ05qiXZyjYBHk6riy9EyBjEHEC9zAlLJC/Cp4+OHXgdRWGICMHA/\nHUPnQu/FhLiWikcQE0v2isigkAk+tlTrRAVTmViSfZB43h1PA65ymAXoKlw4moP9jMn\nJrJ7iY5UnVaFtuIWcyYjXTcCQAyb0iGyFLIWAqoGl+h2rXjB53gQuz",
  "w4moP9jMn\nJrJ7iY5UnVaFtuIWcyYjXTcCQAyb0iGyFLIWAqoGl+h2rXjB53gQuzequ5tqCVAe5rVE\n5jYsc2JraglSQhJGtqi1IJWBDHLCEQZSbch8GnHja4pbGEktjkpjdPXtjNtwbk5zmC\n92LqNioR/xFBEtAFWn7GTAbclq+nc7U3C86o1usCH3tDmCy7Csuj6bBmjcCoGtuEKut\nYISWNFpjy9NJW6t4pDyL7QFqA150ZR7L8B3ZvboEKa",
  "y7Csuj6bBmjcCoGtuEKut\nYISWNFpjy9NJW6t4pDyL7QFqA150ZR7L8B3ZvboEKavNvXsw1LwU/PT+8k98fFat6G\nWj/5FogqOizFyOtPl/OBrAIxjnF1jw5KUCTR4Y6slLBezvaOpYjhNbW+q5g0IsmYjVFV\nr+cSTtOrUFdzZNUF/BoP3ClcUSTXIY2mJt0GK4wmHCkUABGmQwHWMg0qLMOdn8UD6DpZ\nbrbTGP9cPK3lCFtj7BhfzWlCGh8OI",
  "0GK4wmHCkUABGmQwHWMg0qLMOdn8UD6DpZ\nbrbTGP9cPK3lCFtj7BhfzWlCGh8OIX1PdRxH1p/H01IOWI6COdZTOn7dKxQsMdfqr6\nd8WnSqIn6x3bQH/YLZKYOAX/S38XxEREU1AvmC05vTlyAqR3vga56u7/as2n79PUnty\nKF1KwXx2/TSrXZor+kBv9hx9HaH6IiKagTy1fSQ6ojK0R74csdxzUKh9atFMTvLI5Ot\nUM7V6L0Dw+GcDzVx6R",
  "x9HaH6IiKagTy1fSQ6ojK0R74csdxzUKh9atFMTvLI5Ot\nUM7V6L0Dw+GcDzVx6RUDPSxLxW9qQkLFRUqpzDVR1xbODVhYVLaKrjHkv0YHh62amrCw\nm4R2zJtwKIBF3gIUxMWTpewrWxsWLrjkO64pUxkQ6ScmrBwkyV41FMTFkZUGDmF5yzL\nkHBqInEc4jgOaRwzLMpcIjwjmWNGSEq5EiofprZIG7BojFobOxqDHohUogYbIxYXNPMK\nZ+ZJ",
  "jgOaRwzLMpcIjwjmWNGSEq5EiofprZIG7BojFobOxqDHohUogYbIxYXNPMK\nZ+ZJlMWSZvGhq+HDaxpWDnUBizaJWvM6+06F5mPQwzHLFeQsxipMhrALtZ0qWZ2+vPD\nipzk4J3Y0CtKLw29pPTY0GNKc0PJG4EfvjCUvJ34cjQEaVHh5RWhpaUnpo6CGloaEh\npU8MfUJpYGhA6bqh65QqQ8mJFJ4Ih5QOjR0SOmJoSeUvjT0JaVbhm5R+srQV5",
  "Eh\npU8MfUJpYGhA6bqh65QqQ8mJFJ4Ih5QOjR0SOmJoSeUvjT0JaVbhm5R+srQV5S+NfQ\ntpY8MfUQpM5RumHoBqXcUPLpwA/XDF2j1DeUvPvBWjO0S2lmaEbpY0MfUzowlLwVw/P\nMUHK8gQejoYLSp4Y+pTQ2lLy/+eFzQ59TmhiaUPrM0GeUvjH0DaWbhm5SGhlKvg3A6cT\nQfUrNV6CqoHTP0D1KLwy9cH8X4PNp9F2JuWsc7FKaGpSum0oe",
  "m5SGhlKvg3A6cT\nQfUrNV6CqoHTP0D1KLwy9cH8X4PNp9F2JuWsc7FKaGpSum0oeVOAo4Sh5+Q8GcpmV5t\n9bSL7Wijn3MGaiM9qk5iHcs4drNmdZrXJ/hTKOR+Srm8czT+kQEhp+/fWurgr7C0cL\nS63Pl5+cHeg6WHa80X2putb1rftr5rdVq/tB62tlrd1mEraP/Z/qv9T/vfxc3FZFEtjq\nbSG+2mzlct67f4x3/GvUPF</latexit> h1 =",
  "rd1mEraP/Z/qv9T/vfxc3FZFEtjq\nbSG+2mzlct67f4x3/GvUPF</latexit> h1 = a[\u03b20 + \u23260x]\nh2 = a[\u03b21 + \u23261h1]\nh3 = a[\u03b22 + \u23262h2]\n.\nhK = a[\u03b2K\u22121 + \u2326K\u22121hK\u22121]\ny = \u03b2K + \u2326KhK,\n<late\nxit sha1_\nbase64=\"\nLp7vHjH5\niMJ6eWM5g\nQJrOrnQS\ns=\">AXW\nXiclZhb9\ns2FICd7t\nZlt3TFloe\n9CAsKDFsb\n2EV3eRnQ\nJk1vSZeku\nTZx",
  "AXW\nXiclZhb9\ns2FICd7t\nZlt3TFloe\n9CAsKDFsb\n2EV3eRnQ\nJk1vSZeku\nTZxGlAyJb\nOhKEWiEq\neCf+gw7L/\nsUJbN8Bzm\nYQYS0+f7\nxMshJVEKc\nylK3e3+PX\nfro48/+f\nSz25/Pf/H\nlV19/s3Dn\n2/0yq4qI\n70WZzIrDk\nJVcCsX3tN\nCSH+YFZ2\nko+UF4tmr\n4wQUvSpGp\nX2V85OU\nJUrEImIaQ\nqcL/wb9ML\n76E/6FXL\nPTe",
  "FZ2\nko+UF4tmr\n4wQUvSpGp\nX2V85OU\nJUrEImIaQ\nqcL/wb9ML\n76E/6FXL\nPTen0c/AI\n/NlOeNL+g\nzPqSx/rY\nKg96rgS/r\n2lykOlyPp\njpDx35oa9\nGt76eT+k\n6Stco34h\nkqE+mX41D\nbux04Wl7\nnK3+QS0G\nsLS532s3V\n657tBf5B\nFVcqVjiQr\ny+NeN9cnN\nSu0iCQfz\n/erkucsOm\nMJP4aiYik\nvT+pmHsb\nBPYgMgjgr\n4E/",
  "Qr\ny+NeN9cnN\nSu0iCQfz\n/erkucsOm\nMJP4aiYik\nvT+pmHsb\nBPYgMgjgr\n4E/poIleP\n6JmaVlep\nSGYKdPDEj\nMT9LHjSsd\n/nNRC5ZX\nmKpo0Fcy\n0FlgJjUYi\nIJHWl5Bg\nUWFgL4G0Z\nAVLNIw9fN\n9xS+jLE2\nZGtT9lbXt\ncd0PeSJUz\nc+rZhmMx\n6z1jgcij\ncZKy93Z7U\nIzVPxgZN\nKGsVUcoPA\nk3Fd8+VkG\nQPBAYhlTk\nCme",
  "Mx\n6z1jgcij\ncZKy93Z7U\nIzVPxgZN\nKGsVUcoPA\nk3Fd8+VkG\nQPBAYhlTk\nCmeAl1mv\nyEcdBDFJa\n9BAw8zEbQ\nuTh4MyZV\nK80TyImjH\nRENCrnkI8\ndaJRZMZe\noO6AEwb3\nAK4LmAXo\nKnxNAc7\nOVPj6XGaj\n3SR1qWJ4R\nYKphLeNA\nFDjpg0I3I\nNVUkJh0aO\n9Re23jB1\n1iYuy5uF\niaCrN3CdX\nRB86IGrt\nNEkAWLMHG\ntJoIsC",
  "NVUkJh0aO\n9Re23jB1\n1iYuy5uF\niaCrN3CdX\nRB86IGrt\nNEkAWLMHG\ntJoIsCRep\nAUsZLkt\nn8KA08BE/\nKpQWBVkYW\n4VWei2nZ\nsIXpujHM4\nX1urSfov\nGMqICcDZ\nZ74FUxF39\ndVsZgfT5F\nw0vinwUT\nCEyXIPYU\nyGda0ERhV\nGxtTs8kV\nMm2IFRkl\n65peuNReS\n7cAZoAPum\nqQqj4mna\n/KcGSNeH+\nfRhqUl+/\nGD5Vz46q",
  "2IFRkl\n65peuNReS\n7cAZoAPum\nqQqj4mna\n/KcGSNeH+\nfRhqUl+/\nGD5Vz46q\nbvmtDH/SD\nahorLKfRW\nZ8P+oaAC\n3Rby+In\nL5No8iDQT\nF4m4fqOp\no4VeGbSD\nN3UBCKSaG\nv0OkvEuU\ne0RwZ7MU\n9RUCpl74Z\nkKhSY5jV\nzYBI8M3O\nA9CyhCg4w\nmY4xkVlY\nFJxc/tJ4h\n0ujmslgIc\n7NyL6jSC\nO51g8vZUV\nCGm8MFv+H\nwEG",
  "4w\nmY4xkVlY\nFJxc/tJ4h\n0ujmslgIc\n7NyL6jSC\nO51g8vZUV\nCGm8MFv+H\nwEGU0nOQ\nzCo1YAVK\n5shM6ehdv\n9RwivnO/\nmbKJ0Wvlf\nDz9bY96Bf\nMThVF/Px\n0Hc9HQizq\nSFQX7Ki8d\nUliedqDu\nmbL9XrP6v\nV3P5OlnXh\ncvylJvW0\nv/bHvaEH\n/HzD09sN4\nhGLOhLV1f\naQesTytA\nd1+fO4Ru\nFx/WbktQ7\nzaPX9rgz\nEy3/e",
  "H\n/HzD09sN4\nhGLOhLV1f\naQesTytA\nd1+fO4Ru\nFx/WbktQ7\nzaPX9rgz\nEy3/eHcIW\n1ezTcrkwG\nz7MtmfhL\nCoqai9Yma\n2v64CWEx\nrVwLfmNl\nR8DNw7UmI\nSxulcLVTA\nBLAy7xEC\nYhLE5OYd\nsY1jd8Kgb\nfpXJfIjM\nSQiLz1mKR\nz0JYTGhYu\nIVz1ieI3\nESInkc4jw\nOaR5zLOU+\nCc9I7pkR\nsqR8C6oYZ\nq5kAlgaod\nZGnsa",
  "IVz1ieI3\nESInkc4jw\nOaR5zLOU+\nCc9I7pkR\nsqR8C6oYZ\nq5kAlgaod\nZGnsagBz\nJTqME2iOW\nSrzSu/IU\nWsWKruI9\nX8N7NzSsG\narQBLC0Sc\n6xoL/pPc\nlCnGLYZvm\nSnAtk5TSB\nW9jZos50\n9xfGNdnJw\nUO8pVeUXl\np6SemBpQe\nUFpaSJ4I\nwfmMpeToJ\n4wtLyjdt\n3Sf0srSi\ntI9S/cojS\n2NKX1m6TN\nKI0sjSlc\ntXaVUW",
  "wfmMpeToJ\n4wtLyjdt\n3Sf0srSi\ntI9S/cojS\n2NKX1m6TN\nKI0sjSlc\ntXaVUW0p2\npHBHsHSX0\nqGlQ0oPL\nT2k9K2lby\nl9YekLSo8\nsPaL0g6U\nfKH1i6RNK\nmaWM0jVL1\nyjlpJXB\n2G8YukKpa\nGl5NkPzjV\nLtyjNLc0\npfWrpU0oH\nlpKnYrifW\nUq2N3Bjt\nFRS+tLSl5\nQKS8nzWxi\n/tvQ1pam\nlKaWvLH1F\n6XtL31P63\nNLnlC",
  "Uq2N3Bjt\nFRS+tLSl5\nQKS8nzWxi\n/tvQ1pam\nlKaWvLH1F\n6XtL31P63\nNLnlCaWk\nncDsDuxdI\ndS+xaoLin\ndtnSb0nN\nLz/3vBfhs\nGkPfwty0F\nWxSmlmaU\nbpuKXlSgK\n2EpWdkPxm\nr9qo2fdtE\nrmuxmnEP\nazM+PZrkP\nFYz7mHt1W\nl6NLk+xW\nrGh6Tra/u\nzFymQUrjS\nny4s9fBb\nWFrYf7jc+\n2350fajpc\ncr7Rva25\n0fOj",
  "W\nrGh6Tra/u\nzFymQUrjS\nny4s9fBb\nWFrYf7jc+\n2350fajpc\ncr7Rva25\n0fOj92fur\n0Or93Hnde\ndLY6e51o\nbnVOzBVz5\nf/LM4t3l\n6cn6i35t\npj7nacz+L\nd/wDjLCGv\nt></latexi\ny = \u03b2K + \u2326Ka\n\u21e5\n\u03b2K\u22121 + \u2326K\u22121a [. . . \u03b22 + \u23262a [\u03b21 + \u23261a [\u03b20 + \u23260x]] . . .]\n\u21e4",
  "Example",
  "Deep neural networks\n\u2022 Composing two networks\n\u2022 Combining the two networks into one\n\u2022 Hyperparameters\n\u2022 Notation change and general case\n\u2022 Shallow vs. deep networks",
  "Shallow vs. deep networks\nThe best results are created by deep networks with many layers. \n\u2022 50-1000 layers for most applications\n\u2022 Best results in \n\u2022 Computer vision\n\u2022 Natural language processing\n\u2022 Graph neural networks\n\u2022 Generative models\n\u2022 Reinforcement learning\nAll use deep networks.  \nBut why?",
  "Shallow vs. deep networks\n1. Ability to approximate different functions?\nBoth obey the universal approximation theorem.\nArgument:  One layer is enough, and for deep networks could \narrange for the other layers to compute the identity function.",
  "Shallow vs. deep networks\n2.  Number of linear regions per parameter",
  "Number of linear regions per parameter\n5 layers\n10 hidden units per layer\n471 parameters\n161,501 linear regions",
  "Number of linear regions per parameter\n5 layers\n10 hidden units per layer\n471 parameters\n161,501 linear regions\n5 layers\n50 hidden units per layer\n10,801 parameters\n>10!\" linear regions",
  "Question from last lecture on Shallow Nets",
  "Shallow vs. deep networks\n2.  Number of linear regions per parameter\n\u2022 Deep networks create many more regions per parameters\n\u2022 But there are dependencies between them\n\u2022 Think of folding example\n\u2022 Perhaps similar symmetries in real-world functions? Unknown",
  "Shallow vs. Deep Networks\n3. Depth efficiency\n\u2022 There are some functions that require a shallow network with \nexponentially more hidden units than a deep network to achieve an \nequivalent approximation\n\u2022 This is known as the depth efficiency of deep networks\n\u2022 But do the real-world functions we want to approximate have this \nproperty?  Unknown.",
  "Shallow vs. Deep Networks\n4. Large structured networks\n\u2022 Think about images as input \u2013 might be 1M pixels\n\u2022 Fully connected works not practical\n\u2022 Answer is to have weights that only operate locally, and share across image\n\u2022 This leads to convolutional networks\n\u2022 Gradually integrate information from across the image \u2013 needs multiple layers",
  "Shallow vs. Deep Networks\n5. Fitting and generalization\n\u2022 Fitting of deep models seems to be easier up to about 20 layers\n\u2022 Then needs various tricks to train deeper networks, so (in vanilla \nform), fitting becomes harder\n\u2022 Generalization is good in deep networks. Why?",
  "Shallow vs. Deep Networks\n5. Fitting and generalization\n\u2022 Fitting of deep models is also faster",
  "Tensorflow Playground Example?\n\u2022 Try 2 inputs, 3 hidden units, 1 output\n\u2022 You can inspect and/or edit weights and biases\n69\nplayground.tensorflow.org\nDo you ever get stuck in local minima?\nAre you getting the expected number of \nregions?",
  "Where are we going?\n\u2022 We have defined families of very flexible networks that map multiple \ninputs to multiple outputs\n\u2022 Now we need to train them\n\u2022 How to choose loss functions for different types of targets (Read Ch. 5)\n\u2022 How to find minima of the loss function\n\u2022 How to do this efficiently with deep networks\n\u2022 Then how do we evaluate them?",
  "Feedback?\nLink",
  "Lecture 05\nLoss Functions\n(and probability models)\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "Recap\n\u2022 So far, we talked about linear regression, shallow neural networks and \ndeep neural networks\n\u2022 Each have parameters, \ud835\udf19, that we want to choose for a best possible \nmapping between input and output training data\n\u2022 A loss function or cost function, \ud835\udc3f[\ud835\udf19], returns a single number that \ndescribes a mismatch between \ud835\udc53[\ud835\udc65!, \ud835\udf19]and the ground truth outputs, \n\ud835\udc66!.\n2",
  "We need to find a loss function \nthat works with\u2026\n3",
  "Univariate and Multivariate Regression\nDepth Map\n4",
  "Binary Classification\n5",
  "Multiclass Classification\n6",
  "So far, we thought about \nfitting a model to the data\u2026\n7",
  "Alternatively, we can think about \nfitting a probability model to the \ndata.\nPr(\ud835\udc66|\ud835\udc65)\nWhy?\n8",
  "Alternatively, we can think about \nfitting a probability model to the \ndata.\nPr(\ud835\udc66|\ud835\udc65)\nWhy?\nBecause this provides a framework \nto build loss functions for other \nprediction types\u2026\n9",
  "Alternatively, we can think about \nfitting a probability model to the \ndata.\nPr(\ud835\udc66|\ud835\udc65)\nWhy?\nBecause this provides a framework \nto build loss functions for other \nprediction types\u2026\n\u2026 and justifies least squares for \nreal-valued regression models.\n10",
  "Brief Probability Review\n\u2022 Random variables, e.g. \ud835\udc65 and \ud835\udc66\n\u2022 Pr \ud835\udc65 is a probability distribution over \ud835\udc65\n\u2022 0 \u2264 Pr \ud835\udc65 \u2264 1\n\u2022 \u222b\" Pr \ud835\udc65 \ud835\udc51\ud835\udc65 = 1 or  \u2211! Pr \ud835\udc65! = 1\n\u2022 Pr \ud835\udc65, \ud835\udc66 = Pr \ud835\udc65 \u22c5 Pr(\ud835\udc66) when \ud835\udc65 and \ud835\udc66 are independent\n\u2022 Pr \ud835\udc65 \ud835\udc66) Pr \ud835\udc66 = Pr \ud835\udc65, \ud835\udc66 = Pr \ud835\udc66 \ud835\udc65) Pr(\ud835\udc65)\n\u2022 And\u2026\n11",
  "Joint and Marginal Probability Distributions\nJoint Distribution\nMarginal distribution\nPr \ud835\udc66 = %\n!\nPr \ud835\udc65, \ud835\udc66 \ud835\udc51\ud835\udc65\nMarginal distribution\nPr \ud835\udc65 = %\n\"\nPr \ud835\udc65, \ud835\udc66 \ud835\udc51\ud835\udc66\n12",
  "Conditional Probabilities\n!\n)\nPr \ud835\udc65 \ud835\udc66 = 3.0) \ud835\udc51\ud835\udc65 = 1\n!\n)\nPr \ud835\udc65 \ud835\udc66 = \u22121.0) \ud835\udc51\ud835\udc65 = 1\n13",
  "Continuous\nPr(\ud835\udc66|\ud835\udc65)\n14",
  "Continuous\nPr(\ud835\udc66|\ud835\udc65)\n15",
  "Continuous\nPr(\ud835\udc66|\ud835\udc65)\n16",
  "Continuous\nPr(\ud835\udc66|\ud835\udc65)\n17",
  "Continuous\nPr(\ud835\udc66|\ud835\udc65)\n18",
  "19",
  "0\n1\nDiscrete\nPr(\ud835\udc66|\ud835\udc65)\n20",
  "0\n1\nDiscrete\nPr(\ud835\udc66|\ud835\udc65)\n21",
  "0\n1\nDiscrete\nPr(\ud835\udc66|\ud835\udc65)\n22",
  "0\n1\nDiscrete\nPr(\ud835\udc66|\ud835\udc65)\n23",
  "0\n1\nDiscrete\nPr(\ud835\udc66|\ud835\udc65)\n24",
  "25",
  "Discrete\nPr(\ud835\udc66|\ud835\udc65)\n26",
  "Discrete\nPr(\ud835\udc66|\ud835\udc65)\n27",
  "Discrete\nPr(\ud835\udc66|\ud835\udc65)\n28",
  "Discrete\nPr(\ud835\udc66|\ud835\udc65)\n29",
  "30",
  "31",
  "32",
  "33",
  "Loss function\n\u2022 Training dataset of I pairs of input/output examples:\n\u2022 Loss function or cost function measures how bad model is:\nor for short:\n<latexit sha1_base64=\"aYKJ\nls3gxo4H7A6/dQhUn1+H0GU=\">ACFHicbVDLSsNAFJ3UV62vq\nEs3g0UQlJIUTdC0Y3uKtgHNDFMpN26OTBzEQsIR/hxl9x40IR\nty7c+TdO0gjaemCGc8+9l3vcSNGhTSML60N7+wuFRerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhc",
  "Rerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhcCPHEbF9NAioRzGSnL0AyuxfCSHrpfcp05C08OfcJyHVvaf\nmeltcpU6etWoGTngLDELUgUFmo7+afVDHPskJghIXqmEUk7QVx\nSzEhasWJBIoRHaEB6igbIJ8JO8qNSuKeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzC",
  "KeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzCPYpJ1iys\nSIc6p2hXiIOMJS+VhRJpjTJ8+S9lHNPK7Vr+vVxnlhRxnsgF2w\nCgCA=</latexit>D0xwAhrgEjRBC2DwAJ7AC3jVHrVn7U17n5SWtKJnG/yB9vEN+g\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"hEz8\n6CO3UCw8TPROwDlyfIH4Vs8=\">AX/Hic",
  "i, yi}I\ni=1\n<latexit sha1_base64=\"hEz8\n6CO3UCw8TPROwDlyfIH4Vs8=\">AX/HiclZjNbtw2EMfX/UzdL\n6dF4UMvQo0AReAudoOk7aVAYsdJHDu1nfgrsWyDkigtY4qSJcre\njaCem2forei175L36EP0GOHknZpcWigXcNen5/DsnhkKLopZ\nzlcjD4a+6t95973b3w/+FH3/y6cLNz/bzpMh8ucnPMkOP\nZJTzgTdk0xyephmlMQepwfe2ari",
  "t95973b3w/+FH3/y6cLNz/bzpMh8ucnPMkOP\nZJTzgTdk0xyephmlMQepwfe2ariBxc0y1kiduUkpcxiQLmU8k\nmE4X/nE9GjERZUmRurW7MiYZWOq1XzZGj1O/LNq01hET9yvTA\ndsWncgsR0MzLiE+7Qje5DIrZHmnP6Bj+NdLxmVYqZrj5ab6cX\nVaNvY4CSivqmXnP7srlZ/TklXK10QVXPDGfhWJ+V6NXMsM8KE\nxBJqkr1PDued6kI6sGe",
  "4CSivqmXnP7srlZ/TklXK10QVXPDGfhWJ+V6NXMsM8KE\nxBJqkr1PDued6kI6sGeLiwN+oP64+DCsC0s9drP9unNL/52g8Qv\nYiqkz0meHw0HqTwuSaZz2k17xY5TaG3JKJHUBQkpvlxWQ+jcm\n6BJXDCJINfIZ3aerVGSeI8n8QeKGMiR7nJlNHGjgoZfn9cMpEWk\ngq/aSgsuCMTR823E7CM+pJPoED8jEFfHX9EIMQSsmLeFfTST+KY\niKB0V",
  "cMpEWk\ngq/aSgsuCMTR823E7CM+pJPoED8jEFfHX9EIMQSsmLeFfTST+KY\niKB0V9Z2qrLJh5KeF3WGVFVXs1ZrIjXKlbWd2demKQxe0ORk1q\ninFwjoFVlrQf9U3AKADWpwgkgubgU8XHC52hQWFcMBtWkDOM\n8r5FpIGkFMOrJXSAaFlEMOXlWtIhVMZdyRvACJ49xyFKAyg1mAr\nsIXNebgRUpENa0n6VhmcZkrm9lCRkRE6yZgyD7hakRdhS",
  "RvACJ49xyFKAyg1mAr\nsIXNebgRUpENa0n6VhmcZkrm9lCRkRE6yZgyD7hakRdhSg4h6p+\nR/WjqXpOxFkbuCStu5opi6HazboameG4iKCrqS2GCpIw6qpqi6\nHisH8FJCYQ5bYMKzqLHWxS5kwpQwl5naWeN2U2Uxc3Ocwnrp6\ntZKFP4LYkREGWD1qW9GhE+78tVkpnamwbmo9apAx84IJqtbBXbh\nZljTRmBUra3CyjpWhJHC0xZctlVqt5Yp",
  "E+78tVkpnamwbmo9apAx84IJqtbBXbh\nZljTRmBUra3CyjpWhJHC0xZctlVqt5YpDRl3QEqg7noioyJ8Ip\nsuS5ByiqzuwxDzQpOj7p36Pj41Jt3fUfFE1wlBepzZEy/w9HAT\nwxzfwCizl5CTcmDwz15CUc9ndj6khmJray1HMHBSYIZ3JiLH8Wi\nW6d2mJ2NomNvoJB+YVveHAZkxyGXbEyKDF8w7PfkC+MUi/GaP\nk7zIKNr8jHwGSy1X2LG1",
  "2NomNvoJB+YVveHAZkxyGXbEyKDF8w7PfkC+MUi/GaP\nk7zIKNr8jHwGSy1X2LG1MOqu6FyJejuG5TPakEZHg4X9Jrqnh\nFRr4mnl8ABgGRGMdqSscnbi5hidlWfz3lTdGqiuj5Rtse9Atmp\n/B9en6Yc5HhFRYw1fcNiy+uJIZWkPfM3S9WrPyo2T2yi1I4vW\nruTIb9tLu9qivaYH9HzT0tNpEMqrOGr7aHWIdUlvbAlz2Om7Z\nRWLR2JUd+p3",
  "vW\nruTIb9tLu9qivaYH9HzT0tNpEMqrOGr7aHWIdUlvbAlz2Om7Z\nRWLR2JUd+p3G0qi3amdJI/3B3ROFQCqWEB+rYl3C3MZlCiYXSKk\nxiGhnCxmQK46Krgv9NyQsGD4+uqjGZwu2cdWXKYIrgcG8OoTGZw\nmYJd5WtzZRuWqSbdinh6chQNiZT+JjE5qgbkymMsDCyCs9ImhrC\nxoTiODLjOMJxTE1RahOZM5JaZgSlC2hslHSFSmDKRobrY0t",
  "kymMsDCyCs9ImhrC\nxoTiODLjOMJxTE1RahOZM5JaZgSlC2hslHSFSmDKRobrY0tjU\nEPeCKMBlujKc5x5uXWzBNGFgucxXu2hveuaVgSw6EymKItMYcd\n8u6yDwzxHDMsgU5ZYqxQHcNjXbWDM9/XlhiU5y8Ias6QTS0v\nMT3Q9ADTFP0RuCFzVFbydeKHpBab7mu5jWmhaYLqn6R6moaY\nhpo80fYSpr6mP6aqmq5hKTdGJFJ4Imu5iOtJ0hOm",
  "Bab7mu5jWmhaYLqn6R6moaY\nhpo80fYSpr6mP6aqmq5hKTdGJFJ4Imu5iOtJ0hOmhpoeYvtT0Ja\nZPNH2C6StNX2H6RtM3mD7Q9AGmRFOC6Zqma5hSTdHVgReuaLqCq\nacpeveDtabpNqapimDzV9iGmgKXorhueZpuh4Aw9GTm65q\nuY8o0Re9vXvhM02eYxprGmD7V9CmrzV9jeljTR9jGmK7gbgdK\nLpC0z1LVCZY7qj6Q6m5qe2+8F6GwaPV",
  "xprGmD7V9CmrzV9jeljTR9jGmK7gbgdK\nLpC0z1LVCZY7qj6Q6m5qe2+8F6GwaPVtibmkHW5gmiaYbmiK3\nhTgKHpGTpPhqLd1a3TWhfC8WMW1gb8WltFPNQzLiFtbvTtDba\nn0Ix4yPU9bX92UKhLTe6QOqXmjxXTpxMIw6B/dzBcHqgfVL\nvjasTIuRl4oDXNBFUyM6l2u3p7fHR8Bje/dVFgnPJAjkapNIZUR\naNpCoFNJWjpaF6tSXjav50Y",
  "DXNBFUyM6l2u3p7fHR8Bje/dVFgnPJAjkapNIZUR\naNpCoFNJWjpaF6tSXjav50YWlo3gbjwv6d/vDb/r2du0v3V9qb4\n/aKRvzbV1Pu91Pot/guMHWbl</latexit>hu9L3tf9b7uDXvf9e73nvS2e3s9f+5k7ue5X+Z+Xfxp8bfF3xf\nL\nh\n\u03c6, f[x, \u03c6]\n| {z }\nmodel\n, {xi, yi}I\ni=1\n|\n{z\n}\ntrain data\ni\n34",
  "Loss function\n\u2022 Training dataset of I pairs of input/output examples:\n\u2022 Loss function or cost function measures how bad model is:\nor for short:\n<latexit sha1_base64=\"aYKJ\nls3gxo4H7A6/dQhUn1+H0GU=\">ACFHicbVDLSsNAFJ3UV62vq\nEs3g0UQlJIUTdC0Y3uKtgHNDFMpN26OTBzEQsIR/hxl9x40IR\nty7c+TdO0gjaemCGc8+9l3vcSNGhTSML60N7+wuFRerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhc",
  "Rerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhcCPHEbF9NAioRzGSnL0AyuxfCSHrpfcp05C08OfcJyHVvaf\nmeltcpU6etWoGTngLDELUgUFmo7+afVDHPskJghIXqmEUk7QVx\nSzEhasWJBIoRHaEB6igbIJ8JO8qNSuKeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzC",
  "KeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzCPYpJ1iys\nSIc6p2hXiIOMJS+VhRJpjTJ8+S9lHNPK7Vr+vVxnlhRxnsgF2w\nCgCA=</latexit>D0xwAhrgEjRBC2DwAJ7AC3jVHrVn7U17n5SWtKJnG/yB9vEN+g\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnic",
  "i, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnicbVDLSsNAFJ34rPUVd\nSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc\n+g3u/BsnbRbaemCYwzn3cu89QcaZAtv+NpaWV1bX1isb1c2t7Z\n1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvn",
  "0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvnl063EagesFgodqlOjPy2LmSTaIwe+b\nNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5Yp\nmAzxgLqapjihyh9Pz5hYJ1oJrUhI/VKwpurvjFOVLGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR",
  "wxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR2CM3/yIumc1Z3zeuOuUWtelXFU0CE6RqfIQReoiW5Q\nC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</late\nxit>L [\u03c6]\nReturns a scalar that is smaller \nwhen model maps inputs to \noutputs better\n35",
  "Training\n\u2022 Loss function:\n\u2022 Find the parameters that minimize the loss:\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnicbVDLSsNAFJ34rPUVd\nSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc\n+g3u/BsnbRbaemCYwzn3cu89QcaZAtv+NpaWV1bX1isb1c2t7Z\n1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYw",
  "c2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvnl063EagesFgodqlOjPy2LmSTaIwe+b\nNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5Yp\nmAzxgLqapjihyh9Pz5hYJ1oJrUhI/VKwpurvjFOVLGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEB",
  "LGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR2CM3/yIumc1Z3zeuOuUWtelXFU0CE6RqfIQReoiW5Q\nC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</late\nxit>L [\u03c6]\nReturns a scalar that is smaller \nwhen model maps inputs to \noutputs better\n<latexit sha1_base64=\"xKCk\nVHPMbZlPS6XRxAf2oJtIcZs=\">AXTHiclZhNb9xEGIDdAqW",
  "ha1_base64=\"xKCk\nVHPMbZlPS6XRxAf2oJtIcZs=\">AXTHiclZhNb9xEGIDdAqWEr\nxREOHCxiCqhqx2UQtckNqkadMmJUnz2cbaOwd29OMx49TnZr\n7V/ip3DiwgF+AUduCIl3bO9OPO/kQKo20/d5PDN+58P2+Blnhe\nz3f792/Z137vx/s0PFj786ONPl289dlBkZ5QPeDlKf5kU8Ky\npmg+5JTo+ynJLE5/TQP1V/PCc5gVLxZ6cZHSYkEi",
  "l289dlBkZ5QPeDlKf5kU8Ky\npmg+5JTo+ynJLE5/TQP1V/PCc5gVLxZ6cZHSYkEiwkAVEQuhk\ncd31YiIrzw+zmE3dn1yP5FHCxMks5K2wiB97iZ+Oq82px2kojxv\nk5SyK5dBVRj48WVzu9/r1j4sLg7aw7LQ/2ye3vjLG6VBmVAhA0\n6K4njQz+SwIrlkAafTBa8saEaCUxLRYygKktBiWNW3PHVvQ2Tkh\nmkOf4V06+jlKyqSFMUk8cFMiIwL",
  "afTBa8saEaCUxLRYygKktBiWNW3PHVvQ2Tkh\nmkOf4V06+jlKyqSFMUk8cFMiIwLk6mgjR2XMvxWDGRlZKoGko\nLkrU1flzx2xnAaST6BAgpxBX90gJjkJGR5wRP0IkiThIhR5a\n2s7UwhjTRioqJnZ3x6bTrNUOheJVxsrTvXktTNKEvaWoklpRl\nVwh0GhaVbQX9UzAKADWowikghZQp8qPH7oDg8IM4CrZlrAdHBf\nTFHVQtIctLRXiENChmn",
  "aVbQX9UzAKADWowikghZQp8qPH7oDg8IM4CrZlrAdHBf\nTFHVQtIctLRXiENChmn461iwYyqSj7ILiurdBajMYRSgq/C\nLGmOwmxExnV0n6VjmSVWomNlCTkRE6ybglgPC1R1DVFyDpcGHe\ntn03pBxGmbuDSru5qriGHt5V1H5jgvYtR16ohwSMulYdMSwO+\n8GIJASy3JZheJqyJ2lQlTZWhibuep3207UxFzbo4zWC9db61C\n6T8nRkZUAFaf",
  "+\n8GIJASy3JZheJqyJ2lQlTZWhibuep3207UxFzbo4zWC9db61C\n6T8nRkZUAFaf+s2ICGhX03ntjtLzntqwIduzEMVveSZl+71A\njcVRubYrPOlWHibEoTy+6puqNRaUZ696gCpiLrsyZC9pd+sST\nFkV9u7CreYlp8f9u7T8bDq2Wj/kHZhIqKMrNVpML/o6IRPIHM\n+QURc/BSbgweBOrBSzns78bQkdyc2CpSjx0UmCcyYmx/Fkutf\nUEbOz",
  "o6IRPIHM\n+QURc/BSbgweBOrBSzns78bQkdyc2CpSjx0UmCcyYmx/Fkutf\nUEbOzaWL0FQKqXvhNmDAGOQy7sgoGX7Ds9QygQLjJoPmHgOeFm\nVO0eZnzGeI1LraFnOmHlbdDZUrobtvUD6/CsrwcDinV1zuGxn1m\n3z6aSlGJDeSOVZDOn7tFRKWmG310PeFK1WRM82vagXzA6ZRDQ\ns5MNczwiZGHG3XBy4u1Lo4sS3tQ13y6Xu5ZtfH6Dprak",
  "FK1WRM82vagXzA6ZRDQ\ns5MNczwiZGHG3XBy4u1Lo4sS3tQ13y6Xu5ZtfH6DprakcW1mx\nzV2/bSblvcK3pAzYtvd1EHrKw4262h5iD1mW9qAuex43bXdhc\ne0mR/XO8mi1Le7cNKZ/uBdTSdRrUspH6rUv5V4TMkWJRWkV04RG\nhtiETDEpuxb831R2GTw8ulYTMsXtgnU1FTClEeXmLTQhU2yWcNd\nsY6a6aVE37SrhWyYTcgUn5DEvOsmZIoR",
  "ulYTMsXtgnU1FTClEeXmLTQhU2yWcNd\nsY6a6aVE37SrhWyYTcgUn5DEvOsmZIoRFiOreEqyzBCbEMpjbO\nYxnMTCmzSeaIZJYRQVPKNqHyO1KmBKY6O1saUx6AFPhdFgG\nzTlAs+8wjrzhDGLBZ7F+7aG969oWBKjQhUwpS20xlxvy7rIfDPF\n8JplS3LGDCvDCdw2nW3szN7+/LBCb3J+ONF0gumFpheYHmp6iG\nmuKfoi8MXmqKvEz81/Qc0w",
  "CvDCdw2nW3szN7+/LBCb3J+ONF0gumFpheYHmp6iG\nmuKfoi8MXmqKvEz81/Qc0wNDzAtNS0x3d0H9NQ0xDTx5o+x\njTQNMB0VdNVTKWm6I0Ungia7mEaxpjeqTpEaYvNX2J6bqm65i+\n0vQVpm81fYvpQ0fYko0JZiuabqGKdUHR34YqmK5j6mqJvP1h\nrm5jmaYfpI0eYjRFX8XwPNMUvd7Ag1FTjulTZ9iyjRF32\n9+FzT5gmiaYPtP0GaZvN",
  "5jmaYfpI0eYjRFX8XwPNMUvd7Ag1FTjulTZ9iyjRF32\n9+FzT5gmiaYPtP0GaZvNH2D6RNn2AaYrOBuDtRNdTPUpU\nFVguqPpDqZnmp7ZzwXofBh928Tc0hVsYZpqmK6oSn6UoBXCU1\nP0ftkKNpdbXbahPa1UMy5hbUZn12Nch6KObewdneaXY32p1DMeY\ny6vnYwP0iBlNY7/YiqD9r6hLRKSB4xUR8LT6sgmUAa+r17/cHdv\nvqjSuq7cXVCh",
  "y6vnYwP0iBlNY7/YiqD9r6hLRKSB4xUR8LT6sgmUAa+r17/cHdv\nvqjSuq7cXVChLxIXag1SwUVsnOodsdLJoXMS3k8GMK3vzpIcC/Y\nSMb9TLoxVYfCqjSimYyXB+rTloynCyeLywPzNBgXDr7rDb7v3d+\n8v/bP0b6Nev9Ze87nT+fnyxn/BiBF</latexit>5t/xgpT0pvul85XztfOMnB+cB86s+3sO4Hzi/Ob84fz59KvS3\n\u02c6\u03c6 =",
  "</latexit>5t/xgpT0pvul85XztfOMnB+cB86s+3sO4Hzi/Ob84fz59KvS3\n\u02c6\u03c6 = argmin\n\u03c6\nh\nL [\u03c6]\ni\n36",
  "Example:  1D Linear regression loss function\n<latexit sha1_base64=\"/hMh896NPSehdG/Bg07J5FjBuSo=\">AW9HiclZhbc9Q2FI\nA3lLY0LTS07z0xdMHdpCJsvQywszkBAgJDQJuUKc7Mhe2Ssiy4vyQaP/0nfOn3t/+lLf0uPbO8Kn6M8dGfCivN9uh1JtdeIkWLy39M3Pto+sf/Lpjc9mP/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVb",
  "/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hAZz40j1wuSkTh2vn/kuFkRDUrxqF+dlGvVXTfy4nEZVEdjCFb3XC+Ww+wSgtLVNe5f6vAPJw9cd\n9ZSGZRBuVT91BT6Vd3ItM5gbmFpcan+OLTQbwsLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+Oyz",
  "sLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+OyzpFlXMHIkMniFP4U7lTRz+sUbIo01MDM2L5KMNM\nB23sqMiD345LoZIi58pvOgoK6eSxo/PtDEXK/VxeQoH5qYCxOv6IpczPYVmXcUv/DiKmBqW7vLqdlW6Hg+FKvlZUa9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0q",
  "9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0qfPjBU4fUdiREnDZbCjYcM7rijStch5CTjraW6JBIZF83LFWiAVLGXWUHVAc546jAc9TWAUYKnxtAY7CVPVpF7Ox3kalZmO4R5SpkJedwFT9pnUM+oaqpASqvod63d\nsvWbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK",
  "WbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK4KhVBNuZWGnvdvhMdwXtznMB56XqrJUn/OUMZ0QE4fpbMOXzr4ST21nkpz2tcFPnZGsF\njdKiwNm2lNOoFZtbGKmnWukEmzBaE0vuiaejQWlSeiO0EdwIeuSIUKPtDu1SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4",
  "SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4E6sWLJVzf0dKxF\nG9sHanXDgpCMSnyS3T8Rai6deoIHmwcobFCQLcL30wotMhB0JV1QMvwDfdeywby0ST9Zo6+jLMi5eTih/YzRGpdXxZToW9W3Quq1EL3usHltBaU4eZwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/G",
  "Zwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/GywjtcjJBZ1JGoLHnasbUliWfqDtqb9cORlesnP5KtHVpcuylJu+0o7bFvWIE/GzDMtoN4hGLOhK1Y6QesSy9Adt2fO4YZuFxbWb\nkrQ7yaPVtrhTE23/YHfEc6Yfk8wTbRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu",
  "bRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu2cawumFRN+wqk8kImU0Ii89ZhGfdhLAYUjG0iqcsSZDYhEg\neRziPI5rHBEuJTcIrklhWhGwp24ZKR3FX0gEsjVFvY0tnMAIZK9RhG8RyRndeZt15Cu1iRXfxnq3jvSs6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4",
  "6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4YekHpgaEHlKaGkl8EXvDaUPLrxAvODT2ndN/QfUoLQwtK9wzdozQwNKD0maHPKPUN9SldMXSF0txQ8kQKdwRDdykdGTqi9NDQ0rfGPqG0heGvqD0raFvKX1v6HtKnxj6hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8M",
  "hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8MJY83cGM0VFK6ZugapcJQ8vNC14Z+orSyNCI0peGvqT0naHvKH1u6HNKQ0PJuwF4OjF0h1LzFqjMKN02d\nJvSM0P7O8F+HQZPdvG3DQNbFIaGxpTum4o+aUAjxKGnpLnyUC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfz",
  "UC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfzC308VtYWth/sNj/ZfHh9sOFx8vtG\n9obvW973/Xu9vq9X3uPey96W729nt/7d+b6zM2ZW/Pn83/M/zn/V6Nem2nrfN3rfOb/g9NB/xE</latexit>\nL[\u03c6] =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\nLoss function:\n\u201cLeast squares loss function\u201d\n37",
  "Example: 1D Linear regression training\nThis technique is known as gradient descent\n38",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n39",
  "Maximum Likelihood Estimation\n\u2022 In statistics, maximum likelihood estimation (MLE) is a method of \nestimating the parameters of an assumed probability distribution, \ngiven some observed data. \n\u2022 This is achieved by maximizing a likelihood function so that, under the \nassumed statistical model, the observed data is most probable.\n40",
  "How do we do this?\n\u2022 Model predicts output y given input x\n41",
  "How do we do this?\n\u2022 Model predicts output y given input x\n42",
  "How do we do this?\n\u2022 Model predicts output y given input x\n\u2022 Model predicts a conditional probability distribution:\nover outputs y given inputs x.\n\u2022 Define and minimize a loss function that makes the outputs have high \nprobability\n<latexit sha1_base64=\"0mryZJN\nvwRC/ELK5bncNAZe/8Y=\">AWh3iclZhbU9w2FICd9Jakt6Sd8tIXT5nM\npJ10C50zWMCITdIWQILJCxhZK/sVZBlry3DEnd/R1/bn9V/0yPbu4rPEQ/\ndGXbF+T7rciT5FmRSFHpl5d8rVz/6+JNP7t2/cbnX3z51dc3b32zX6RlHv\nJBmMo0PwxY",
  "+T7rciT5FmRSFHpl5d8rVz/6+JNP7t2/cbnX3z51dc3b32zX6RlHv\nJBmMo0PwxYwaVQfKCFlvwyzlLAskPgtN1w/OeF6IVO3pi4wfJyxWIhIh\n0xA67ud3hkF08Sd8TX8ubm80lupPz4trLaFZa/9E9ufTcajtKwTLjSoWR\nFcbS6kunjiuVahJLPbgzLgmcsPGUxP4KiYgkvjqu61zP/NkRGfpTm8Ke0X\n0c/PKJiSVFcJAGYCdPjAjMTdLGjU",
  "csPGUxP4KiYgkvjqu61zP/NkRGfpTm8Ke0X\n0c/PKJiSVFcJAGYCdPjAjMTdLGjUkcPjiuhslJzFTYNRaX0deqbFPgjkfNQ\nywsosDAX0Fc/HLOchRoSdWOo+HmYJglTo2q4trEzq4YBj4Wq+KSskzabdZ2\nN2uFQvMxYe763qEVonoj3nFRSK6aSwQez6qK9+IeBoIDED1OQKp4AXWa/\nASRv4oLBIJGHiQTqFzkf9qRqpWmseQk472hmhQyCSf",
  "9+IeBoIDED1OQKp4AXWa/\nASRv4oLBIJGHiQTqFzkf9qRqpWmseQk472hmhQyCSfdqx1YsFUJh1lFxTf\nv+0bwHUOswBdhR+O5mA3Y2o2P07zqc6TqjAx3ELOVMzrJmDIZNmRF1DlVL\nCoWH+gNbr5g6bROXZnVXcxNB1l7edXRO86JGXaeOIAsWYdy16giyJGzpE\nUsYZLktn8CAE9E3KpQWBVkYfbzNOi2nZkIXpvTDPZL19uoSPrPGMqICcDu",
  "JGzpE\nUsYZLktn8CAE9E3KpQWBVkYfbzNOi2nZkIXpvTDPZL19uoSPrPGMqICcDu\nM7+CqZB39fV0Yfvz5JzVvinwqT+GyeoewvK4Gda8ERhVG5tRs84VMm2IJ\nSn513T9Mah8kx0B2gCeNOVuVDRB9rdugRL1oSHd2GoeSn50c+93/j0uFox2\n8Z8kWxCRUWZuSoy4f9R0QguInh9QRPXirR5EGgnrxUwvkdTR3L8cI2kXru\noCAUk0JfoO0vYt",
  "WZuSoy4f9R0QguInh9QRPXirR5EGgnrxUwvkdTR3L8cI2kXru\noCAUk0JfoO0vYtU9po7gzqYJ6isETL3wy4RCkxFXdkEjAy/cDl0LKAQDT\nJsxhjKtChzTk5+aD1DpNbNaTEX5mLVPaFKI3TPG1wujoIyXBzO+CWHByijQ\nZPIC3ViOUomVMzpdO3w0LDFnPt/nrKm6LTivlks20P+gWzU4Yhn5xs4vm\nIiUdieqC+w9nXZJYjvagrsVy/bBn1e",
  "Pt/nrKm6LTivlks20P+gWzU4Yhn5xs4vm\nIiUdieqC+w9nXZJYjvagrsVy/bBn1ebn8jSjh2u25Sk3raXbtvhXtIDPt\nly9HaLeMSijkR1tT2kHrEc7UFd7jxuUbhcN2mJPXO8+i0He7CRMs/2htz\ncxtUipH5rYvlcMmhEVNRe0U04THSGxCWEzKrgX/Y2VXwMWjazUhLPYL0dV\nMAEsjLvEQmhAWmy3cNdsYVrc6pZbZTIbI7MJYfEpS/ComxA",
  "XwMWjazUhLPYL0dV\nMAEsjLvEQmhAWmy3cNdsYVrc6pZbZTIbI7MJYfEpS/ComxAWYyrGTvGUZR\nkSmxDJ4xjncUzmGEpc0l4RjLHjJAl5VpQ+TjtSiaApSlqbepoDHogU4Uab\nINYLujK5wrT6FVrOgqHrgaHlzSsGaoQhPA0jbZY/5w27nJApxiuM1yJTk\nTyMpoAvY6VNnfvcXRBW5k4NHXksvKD239JzSA0sPKM0tJU8EQfTKUvJ0Ek\nRnlp",
  "TyMpoAvY6VNnfvcXRBW5k4NHXksvKD239JzSA0sPKM0tJU8EQfTKUvJ0Ek\nRnlp5Rum/pPqWlpSWlA0sHlEaWRpQ+sfQJpaGlIaXrlq5Tqi0ld6RwRbB0\nj9KxpWNKDy09pPS1pa8pfWbpM0rfWPqG0veWvqf0kaWPKGWMko3LN2glFt\nKXh0E0Zqla5QGlpJnP9hrlvYpzSzNKH1s6WNKR5aSp2K4nlKbm/gwmipP\nS5pc8pFZaS57cgemnpS0",
  "pJnP9hrlvYpzSzNKH1s6WNKR5aSp2K4nlKbm/gwmipP\nS5pc8pFZaS57cgemnpS0oTSxNKX1j6gtJ3lr6j9KmlTymNLSXvBuDuxNJd\nSu1boKqgdMfSHUonlk7c7wX4YhoD18LcthVsU5pamlK6aSl5UoBbCUtPyf1\nkpNqz2vxtEzmvRWrBHazN+PxokvNILbiDtWen+dHk/BSpBR+Trm/sL16kQ\nErhTH9yc3kVv4Wlhf1fe6v3e/d27i0/XGv",
  "LbiDtWen+dHk/BSpBR+Trm/sL16kQ\nErhTH9yc3kVv4Wlhf1fe6v3e/d27i0/XGvf0F7zvd+8O54q97v3kPvmdf3\nBl7oTby/vL+9f5auL/2ydH/pQaNevdIe863X+Sw9+g8YN9Nn</latexit>Pr(y|x)\n43",
  "How can a model predict a probability \ndistribution? \u00e8 Parametric Models\n1.\nPick a known distribution (e.g., normal distribution) to model output y \nwith parameters \ne.g., the normal distribution\n2.  Use model to predict parameters    of probability distribution\n<latexit sha1_base64=\"VhcSfedKJAm\n7Hk8ez5AJ9hTnr0=\">AWgniclZhb9s2FIDVrtu67tJ2w/KyF2FBgaHojHjrLg97aJ\nOmt6SL08RJmjgNKJmS2VCUIlGJU8F/Yq/bH9u/2aEkm9U5zMpGbP94mXQ1KiFWRSFH\npl5d9r1z+68fEn9787NbnX3z51e07d7/eK9",
  "2aEkm9U5zMpGbP94mXQ1KiFWRSFH\npl5d9r1z+68fEn9787NbnX3z51e07d7/eK9IyD/kwTGWaHwSs4FIoPtRCS36Q5Zwlg\neT7wema4fvnPC9Eqnb1ZcaPExYrEYmQaQgdjIJIT7hmJ3eWV3or9cenhX5bWPbaz+Dk7\nrfj0TgNy4QrHUpWFEf9lUwfVyzXIpR8dmtUFjxj4SmL+REUFUt4cVzVHZ759yAy9qM0\nhz+l/Tr64RUVS4riMgnATJi",
  "IpR8dmtUFjxj4SmL+REUFUt4cVzVHZ759yAy9qM0\nhz+l/Tr64RUVS4riMgnATJieFJiZoIsdlTr6/bgSKis1V2HTUFRKX6e+Gb0/FjkPtbyE\nAgtzAX31wnLWaghR7dGil+EaZIwNa5Gq+vbs2oU8Fioip+Vdb5ms6zXjscilcZqy92\nF7UIzRPxnpNKasVUcoXA41lV8V7cw0BwAKLHCUgVL6BOk58g8vuIwvqQgIEH6RQ6F/m\nvZ6RqpXkMOe",
  "UcoXA41lV8V7cw0BwAKLHCUgVL6BOk58g8vuIwvqQgIEH6RQ6F/m\nvZ6RqpXkMOeloh0SDQib5tGOtEQumMukoO6D4/j3fAK5zmAXoKnxNAc7GVOz+XWaT3W\neVIWJ4RZypmJeNwFDpk0I+oaqpQSLg071p/Yes3UaZu4NKu7mpsIsnbzrqNzmhc17jp\n1BFmwCOuVUeQJWE3j1nCIMt+QGnPgm4laFwqogC3OQp0G37cxE8NqcZrBfut56Rd\nJ/",
  "mwCOuVUeQJWE3j1nCIMt+QGnPgm4laFwqogC3OQp0G37cxE8NqcZrBfut56Rd\nJ/zlBGTAB2n/kWTIW8q6+lC9ufJ+e89k2BT/0JTFb3EpbHzbDmjcCo2tiMmnWukEmzBa\nE8veiapjcOlWeiO0ATwJuzIWKPtAe1CVYsiY8egBDzUvJj37s/cKnx9WK2TbmH5JNq\nKgoM1dFJvw/KhrD8wOvL4jgyUslmjwI1JOXSri/o6ljOV7YJlLPHRSEYlL",
  "H5JNq\nKgoM1dFJvw/KhrD8wOvL4jgyUslmjwI1JOXSri/o6ljOV7YJlLPHRSEYlLoS7T9Ray61\n9QR3Nk0QX2FgKkXvplQaJKjqCubgJHhG56EjgUokGzRhDmRZlzsnND61niNS6uS3mw\njysujdUaYTufYPLxVQhofDOb/i8gBlNGjyGaSlGrMcJXNqpnT6dlRo2GKu3V9PeVN0\nWjE/2jbg37B7JRhyM9ONvB8xMSijkR1wdHDWZcklqM9qGux",
  "dlRo2GKu3V9PeVN0\nWjE/2jbg37B7JRhyM9ONvB8xMSijkR1wdHDWZcklqM9qGuxXD/sWbXx9j5Z2rHDdZuS\n1Nv20m073Ct6wM82Hb3dJB6xqCNRXW0PqUcsR3tQlzuPm65ROFy3KUm98zw6bYe7MNHy\nj3bNEdQck1I5Nse+VI6aEBY1FbVTBMeI7EJYTEpuxb8Hys7Ah4eXasJYXFQiK5mAlg\nac4mH0ISw2GzhrtnGsLrpUDfdKpPZBJlNCI",
  "puxb8Hys7Ah4eXasJYXFQiK5mAlg\nac4mH0ISw2GzhrtnGsLrpUDfdKpPZBJlNCIvPWIJH3YSwGFMxdoqnLMuQ2IRIHic4jxO\naxwxLmUvCM5I5ZoQsKdeCyidpVzIBLE1Ra1NHY9ADmSrUYBvEckFXuFceQqtYkVX8d\nDV8PCKhjVDFZoAlrbIHvNHW85NFuAUwzHLleRMICujCRxgZ0Cd+ekviCpykguiS0svKb\n2w9ILSfUv3Kc0tJb8Igui1",
  "uAUwzHLleRMICujCRxgZ0Cd+ekviCpykguiS0svKb\n2w9ILSfUv3Kc0tJb8Igui1peTXSRCdW3pO6Z6le5SWlpaUDi0dUhpZGlH61NKnlIaWhp\nSuWbpGqbaUnEjhiWDpLqUTSyeUHlh6QOkbS9Q+tzS5QeWnpI6XtL31P62NLHlDJLG\naXrlq5Tyi0lrw6CaNXSVUoDS8lvP9hrlg4ozSzNKH1i6RNKx5aSX8XwPLOUHG/gwWip\nPSFpS8oFZaS",
  "aNXSVUoDS8lvP9hrlg4ozSzNKH1i6RNKx5aSX8XwPLOUHG/gwWip\nPSFpS8oFZaS329B9MrSV5QmliaUvrT0JaXvLH1H6TNLn1EaW0reDcDpxNIdSu1boKqg\ndNvSbUrPLD1zvxfgi2kMXAtzy1awRWlqaUrphqXklwIcJSw9JefJSLV3tfnbJnJfi9SC\nO1ib8fnVJOeRWnAHa+9O86vJ/SlSCz4hXV/fW7xIgZTCnf7kznIfv4Wlhb2fev1few",
  "ib8fnVJOeRWnAHa+9O86vJ/SlSCz4hXV/fW7xIgZTCnf7kznIfv4Wlhb2fev1few+3\nHy4/Wm3f0N70vO+937w+t5v3iPvuTfwhl7oSe8v72/vn6UbS/eX+ks/N+r1a+013i\ndz9If/wGNWNGn</latexit>\u2713\n<latexit sha1_base64=\"VhcSfedKJAm7\nHk8ez5AJ9hTnr0=\">AWgniclZhb9s2FIDVrtu67tJ2w/KyF2FBgaHojHjrLg97",
  "7\nHk8ez5AJ9hTnr0=\">AWgniclZhb9s2FIDVrtu67tJ2w/KyF2FBgaHojHjrLg97aJOmt\n6SL08RJmjgNKJmS2VCUIlGJU8F/Yq/bH9u/2aEkm9U5zMpGbP94mXQ1KiFWRSFHpl5d9\nr1z+68fEn9787NbnX3z51e07d7/eK9IyD/kwTGWaHwSs4FIoPtRCS36Q5ZwlgeT7wema\n4fvnPC9Eqnb1ZcaPExYrEYmQaQgdjIJIT7hmJ3eWV3or9cen",
  "36Q5ZwlgeT7wema\n4fvnPC9Eqnb1ZcaPExYrEYmQaQgdjIJIT7hmJ3eWV3or9cenhX5bWPbaz+Dk7rfj0TgNy4\nQrHUpWFEf9lUwfVyzXIpR8dmtUFjxj4SmL+REUFUt4cVzVHZ759yAy9qM0hz+l/Tr64RUV\nS4riMgnATJieFJiZoIsdlTr6/bgSKis1V2HTUFRKX6e+Gb0/FjkPtbyEAgtzAX31wnLWa\nghR7dGil+EaZIwNa5Gq+vbs2oU8F",
  "HTUFRKX6e+Gb0/FjkPtbyEAgtzAX31wnLWa\nghR7dGil+EaZIwNa5Gq+vbs2oU8Fioip+Vdb5ms6zXjscilcZqy92F7UIzRPxnpNKasV\nUcoXA41lV8V7cw0BwAKLHCUgVL6BOk58g8vuIwvqQgIEH6RQ6F/mvZ6RqpXkMOeloh0SDQ\nib5tGOtEQumMukoO6D4/j3fAK5zmAXoKnxNAc7GVOz+XWaT3WeVIWJ4RZypmJeNwFDpk\n0I+oaqpQSLg",
  "6D4/j3fAK5zmAXoKnxNAc7GVOz+XWaT3WeVIWJ4RZypmJeNwFDpk\n0I+oaqpQSLg071p/Yes3UaZu4NKu7mpsIsnbzrqNzmhc17jp1BFmwCOuVUeQJWE3j1nC\nIMt+QGnPgm4laFwqogC3OQp0G37cxE8NqcZrBfut56RdJ/zlBGTAB2n/kWTIW8q6+lC9\nufJ+e89k2BT/0JTFb3EpbHzbDmjcCo2tiMmnWukEmzBaE8veiapjcOlWeiO0ATwJu",
  "fJ+e89k2BT/0JTFb3EpbHzbDmjcCo2tiMmnWukEmzBaE8veiapjcOlWeiO0ATwJuzIWK\nPtAe1CVYsiY8egBDzUvJj37s/cKnx9WK2TbmH5JNqKgoM1dFJvw/KhrD8wOvL4jgyUslm\njwI1JOXSri/o6ljOV7YJlLPHRSEYlLoS7T9Ray619QR3Nk0QX2FgKkXvplQaJKjqCubgJH\nhG56EjgUokGzRhDmRZlzsnND61niNS6uS3mwjysujdUaYT",
  "kXvplQaJKjqCubgJH\nhG56EjgUokGzRhDmRZlzsnND61niNS6uS3mwjysujdUaYTufYPLxVQhofDOb/i8gBlN\nGjyGaSlGrMcJXNqpnT6dlRo2GKu3V9PeVN0WjE/2jbg37B7JRhyM9ONvB8xMSijkR1wdH\nDWZcklqM9qGuxXD/sWbXx9j5Z2rHDdZuS1Nv20m073Ct6wM82Hb3dJB6xqCNRXW0PqUcs\nR3tQlzuPm65ROFy3KUm98zw6bYe7MN",
  "20m073Ct6wM82Hb3dJB6xqCNRXW0PqUcs\nR3tQlzuPm65ROFy3KUm98zw6bYe7MNHyj3bNEdQck1I5Nse+VI6aEBY1FbVTBMeI7EJYT\nEpuxb8Hys7Ah4eXasJYXFQiK5mAlgac4mH0ISw2GzhrtnGsLrpUDfdKpPZBJlNCIvPWIJH\n3YSwGFMxdoqnLMuQ2IRIHic4jxOaxwxLmUvCM5I5ZoQsKdeCyidpVzIBLE1Ra1NHY9ADm\nSrUYBvEckFX",
  "IRIHic4jxOaxwxLmUvCM5I5ZoQsKdeCyidpVzIBLE1Ra1NHY9ADm\nSrUYBvEckFXuFceQqtYkVX8dDV8PCKhjVDFZoAlrbIHvNHW85NFuAUwzHLleRMICujCRx\ngZ0Cd+ekviCpykguiS0svKb2w9ILSfUv3Kc0tJb8Igui1peTXSRCdW3pO6Z6le5SWlpaUD\ni0dUhpZGlH61NKnlIaWhpSuWbpGqbaUnEjhiWDpLqUTSyeUHlh6QOkbS9Q+tzS",
  "D\ni0dUhpZGlH61NKnlIaWhpSuWbpGqbaUnEjhiWDpLqUTSyeUHlh6QOkbS9Q+tzS5QeWn\npI6XtL31P62NLHlDJLGaXrlq5Tyi0lrw6CaNXSVUoDS8lvP9hrlg4ozSzNKH1i6RNKx5aS\nX8XwPLOUHG/gwWipPSFpS8oFZaS329B9MrSV5QmliaUvrT0JaXvLH1H6TNLn1EaW0reDc\nDpxNIdSu1boKqgdNvSbUrPLD1zvxfgi2kMXAtzy1awRWl",
  "LH1H6TNLn1EaW0reDc\nDpxNIdSu1boKqgdNvSbUrPLD1zvxfgi2kMXAtzy1awRWlqaUrphqXklwIcJSw9JefJSLV\n3tfnbJnJfi9SCO1ib8fnVJOeRWnAHa+9O86vJ/SlSCz4hXV/fW7xIgZTCnf7kznIfv4Wlh\nb2fev1few+3Hy4/Wm3f0N70vO+937w+t5v3iPvuTfwhl7oSe8v72/vn6UbS/eX+ks/N+r\n1a+013idz9If/wGNWNGn</late",
  "5v3iPvuTfwhl7oSe8v72/vn6UbS/eX+ks/N+r\n1a+013idz9If/wGNWNGn</latexit>\u2713\n<latexit sha1_base64=\"fuZW1quo5Po\nqM6jniErbZ+1qlA=\">AWlXiclZhbT9xGFICdXtP0RlqVquqLVRSpqtIVROnlJVICI\nTdIgcACSZo7B17J4zHxh7DEmvVX9PX9vf03/SM7d2JzxkeuhLZyfk+z+XMjD3rMJei\n1MvL/157/0Pvzo4+uf",
  "DEmvVX9PX9vf03/SM7d2JzxkeuhLZyfk+z+XMjD3rMJei\n1MvL/157/0Pvzo4+uf3Pj0s8+/+HLh5lf7ZVYVER9GmcyKw5CVXArFh1poyQ/zgrM\n0lPwgPF0z/OCcF6XI1J6+zPlxyhIlYhExDaGThe+CMNZjrtm9oA7S6nZQiRlr+8E05\nOFpeXBcvPxaWGlKyx53Wf75OY3o2CURVXKlY4kK8ujleVcH9es0CKSfHojqEqes+iUJ\nfwIioqlvD",
  "lKyx53Wf75OY3o2CURVXKlY4kK8ujleVcH9es0CKSfHojqEqes+iUJ\nfwIioqlvDyumzFM/VsQGflxVsCf0n4TfeKmqVleZmGYKZMj0vMTNDFjiod/35cC5VX\nmquobSiupK8z3yTEH4mCR1peQoFhYC+tGYFSzSkLYbgeIXUZamTI3qYHV9Z1oHIU+\nEqvlZ1aRwOu07643DoXiVsfp0b16L0DwVbzmpFMJVcIPJnWNR8kAwEByAGnIBM8R\nLqN",
  "Z1aRwOu07643DoXiVsfp0b16L0DwVbzmpFMJVcIPJnWNR8kAwEByAGnIBM8R\nLqNPkJY38FUVgyEjDwMJtA52L/xZRUrTRPICc97RXRoJBLPulZa8SCqUx7yi4ovn/LN\n4DrAmYBugpfHM3Bbs7UdHad5hNdpHVpYriFgqmEN03AkCMmzYj6hqkhEujnvUHtl4wd\ndolLsubrhYmgqy9ou/oguZFjfpOE0EWLMKkbzURZEnY4COWMshyVz6BAae+",
  "l4wd\ndolLsubrhYmgqy9ou/oguZFjfpOE0EWLMKkbzURZEnY4COWMshyVz6BAae+ibhVobAq\nyMLcLrKw3ZuInhtTnLYL31vSbpP2coIyYAu898C6Yi3tfXsrntz5Jz3vimwCf+GCa\nrfwkrknZYs0ZgVF1sSs0mV8ik2YJQkV30TdMbh8pz0R+gCeBNVxVCxe9ot5sSLFkTDm\n7DUItK8qOfB7/wyXG9bLaN+YdkEyoq9xVkQn/j4pG8EjB6wsi",
  "xe9ot5sSLFkTDm\n7DUItK8qOfB7/wyXG9bLaN+YdkEyoq9xVkQn/j4pG8EjB6wsiePIyiSYPAs3kZRLu7\n2jqWIEXtok0cwcFoZgU+hJtf5Go/jVNBHc2S1FfIWDqhW8mFJrkO7LJmBk+IaHo2MB\nRWiQUTvGSGZlVXBy80PrGSKNbm6LhTAPq/4NVRqhf9/gcn4VlOHhcM6vuDxEGQ3bfIZ\nZpUasQMmcmCmdvA5KDVvMtfubKW+LTivhZxtde9",
  "gcn4VlOHhcM6vuDxEGQ3bfIZ\nZpUasQMmcmCmdvA5KDVvMtfubKW+LTivhZxtde9AvmJ0qivjZyQaej4RY1JGoLjiNO\nuSxHK0B3XNl+u7Pas3Xv9ElnbicN2mJPV2vXTbDveKHvCzTUdvN4lHLOpIVFfXQ+oRy\n9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20/OM9cyY1x6RMjsyxL5NBG8KipqJ2ilnKEyS2\nISymVd+C/2Nl1xyS+1YbwuJ2KfqaC",
  "M9cyY1x6RMjsyxL5NBG8KipqJ2ilnKEyS2\nISymVd+C/2Nl1xyS+1YbwuJ2KfqaCWBpxCUeQhvCYruF+2YXw+qmQ910q0zmY2S2ISw\n+ZikedRvCYkLFxCmesjxHYhsieRzjPI5pHnMs5S4Jz0jumBGypFwLqhnfckEsDRBrU\n0cjUEPZKZQg10QyVdeaVz5Sm0ihVdxUNXw8MrGtYMVWgCWNoie8wPtpybLMQphmOWK\n8m5QFZOE7iNnW3qzE5/",
  "Sm0ihVdxUNXw8MrGtYMVWgCWNoie8wPtpybLMQphmOWK\n8m5QFZOE7iNnW3qzE5/YVyTk1wYX1p6SemFpReUHlh6QGlhKflFEMYvLCW/TsL43NJz\nSvct3ae0srSidGjpkNLY0pjSR5Y+ojSyNKJ0zdI1SrWl5EQKTwRL9ygdWzqm9NDSQ0p\nfWvqS0ieWPqH0laWvKH1r6VtKH1j6gFJmKaN03dJ1Srml5NVBGK9aukpaCn57Qd7zd\nJtSnNLc0",
  "H0laWvKH1r6VtKH1j6gFJmKaN03dJ1Srml5NVBGK9aukpaCn57Qd7zd\nJtSnNLc0ofWvqQ0pGl5FcxPM8sJcbeDBaKil9aulTSoWl5PdbGD+39DmlqaUpc8sf\nUbpG0vfUPrY0seUJpaSdwNwOrF0l1L7FqguKd2xdIfSM0vP3O8F+HwaQ9fC3LIVbFGa\nWZpRumEp+aUARwlLT8l5MlbdXW32tonc12I15w7WZXx2Ncl5rObcwbq70+xqcn+K1Z",
  "pRumEp+aUARwlLT8l5MlbdXW32tonc12I15w7WZXx2Ncl5rObcwbq70+xqcn+K1Zy\nTb9oZe5P3p/eX97f2z+O3ivcWHi49a9b1r3TVfe73P4tZ/kgnY1g=</latexit>PSdfX9+cvUiClcKc/WVhawW9haWH/zmDl18HdnbtL91e7N7TXve+9H7wfvRXvN+98\n\u2713 = {\u00b5, \u03c32}\n44",
  "Maximize the joint, conditional probability\n\u2022 We know we picked a good model and the right parameters \nwhen the joint conditional probability is high for the \nobserved (e.g. training) data.\nPr \ud835\udc66#, \ud835\udc66$, \u2026 , \ud835\udc66% \ud835\udc65#, \ud835\udc65$, \u2026 , \ud835\udc65%)\n45",
  "Two simplifying assumptions\nPr \ud835\udc66#, \ud835\udc66$, \u2026 , \ud835\udc66% \ud835\udc65#, \ud835\udc65$, \u2026 , \ud835\udc65%) = A\n!&#\n%\nPr \ud835\udc66! \ud835\udc65!)\nIdentically distributed (the form of \nthe probably distribution is the same \nfor each input/output pair)\nIndependent \nIndependent and identically distributed (i.i.d)\n46",
  "Maximum likelihood criterion\n<latexit sha1_base6\n4=\"7mr1QUEuofibXjI6egoym0oeHA=\">A\nXhnicnZhbU9w2FICX9JbQW9JOh4e+eMqk3b\nSHegkTV8yk0BIQiBlCSyQYMLIXtmrIMvGlmG\nJ67/R1/Zv9d/0yHsRPkc8NMykq57vsyQfXSw7\nyKQo9NLSv3PXPvr4k08/u35j/vMvzq65u3\nvtkr0jIPeT9MZofBKzgUije10JLfpDlnCWB\n5PvByarh+2c8L0SqdvVFx",
  "zq65u3\nvtkr0jIPeT9MZofBKzgUije10JLfpDlnCWB\n5PvByarh+2c8L0SqdvVFxo8SFisRiZBpCB3fm\nrvh+UOmKz+IsqGoPe/Hh57P8jho+Np0Jc80\nod+lqeD40o8XK7fVu18vAL+ASP0nFEam8\nJOfi3ioj3x/oMr0kOu2eXKPF+lqkwCnu+/\n4HVJkE6MroX1YfTzt4dX340bef45uJSd6n582\nheVJY7Ez+ese3vhv4gzQsE650KFlRHC4",
  "6MroX1YfTzt4dX340bef45uJSd6n582\nheVJY7Ez+ese3vhv4gzQsE650KFlRHC4vZf\nqoYrkWoeT1vF8WPGPhCYv5IRQVS3hxVDWDVn\nu3ITLwojSHf0p7TfTyFRVLiuIiCcBMmB4WmJm\ngix2WOvr9qBIqKzVX4bihqJSeTj0zA7yByHm\no5QUWJgL6KsXDlnOQg3zZN5X/DxMk4SpQeW\nvrG3XkDYeC1Xx07KZM3XdtYah0PxKmNlfXd\nWi9A",
  "DlnOQg3zZN5X/DxMk4SpQeW\nvrG3XkDYeC1Xx07KZM3XdtYah0PxKmNlfXd\nWi9A8Ee85qaRTCVXCDyuq4p34y4GgMQXU5A\nqngBdZr8wLgvIwprRAKu7Mx4VZOqleYx5KSl\nvSEaFDLJRy1rlVgwlElL2QHF8257BnCdwyhA\nV+GHozHYyZiqp9dpPtJ5UhUmhlvImYp50wTc\ncsikuaO2oUop4dKwZf2BrVdMnUwSl2ZNV3MTQ\ndZu3nZ0TvOi",
  "UmhlvImYp50wTc\ncsikuaO2oUop4dKwZf2BrVdMnUwSl2ZNV3MTQ\ndZu3nZ0TvOiBm2niSALJmHctpoIsiTsaAMGi\n76elmH154lnIm5VKwKMjF7eRq0285MBM/NU\nQbrpe2tVST9ZwxlxARg9ZlfwVTI2/pqOrO9aX\nLOGt8U+MgbwmC1LzHbXnNb0bgriaxmpNrp\nBJswWhPD1vm6Y3DpVnon2DJoAXZkLFV3S7j\nYlmLIm7N+FW81LyQ9/6d7",
  "xmpNrp\nBJswWhPD1vm6Y3DpVnon2DJoAXZkLFV3S7j\nYlmLIm7N+FW81LyQ9/6d7no6NqySwb8x+STa\nioKDNXRSb8PyoawDMUzy+I4MFLJRo8CDSDl0r\nY39HQsRxPbBNpxg4KQjEp9AVa/iJW7WuaCO5\nsmqC+QsDUC79MKDTIUdSWTcDI8AunAcECtF\nNhuN7DGValDknmx+azxBpdLMt5sI8rNobqjR\nCe9/gcnYVlOHhcMavuDxAGQ3G+Qz",
  "huN7DGValDknmx+azxBpdLMt5sI8rNobqjR\nCe9/gcnYVlOHhcMavuDxAGQ3G+QzSUg1YjpI5\nMkM6eusXGpaYa/U3Qz4uOq2Yn25M2oN+weiU\nYchPjzfweMTEo5EdcHxy1mXJajPahrNl0v\n96zaePszmdqxw3WbktQ76aXbdrhX9ICfbjp6u\n0k8YlFHoromPaQesRztQV3uPG67sLhuk1J6\np3m0Wk73JmJpn+0a4695piUyoE59qXSH4ewq",
  "romPaQesRztQV3uPG67sLhuk1J6\np3m0Wk73JmJpn+0a4695piUyoE59qXSH4ewq\nKmonWKa8BiJ4xAWk7Jtwf9jZUfAw6NtjUNY7\nBWirZkAlgZc4lsYh7A4XsJtcxLD6qZD3XSrTG\nZDZI5DWHzGEnzX4xAWYyrGTvGEZRkSxyGSxy\nHO45DmMcNS5pLwiGSOESFTyjWh8mHalkwASy\nPU2sjRGPRApgo1OAliuaAzr3DOPIVmsaKzuO\n9quH9",
  "GSOESFTyjWh8mHalkwASy\nPU2sjRGPRApgo1OAliuaAzr3DOPIVmsaKzuO\n9quH9Fw5qhCk0AS1tkjXn+lnORBTjFcMxyJTk\nTyMpoAnvY6VFnevoLoqc5OC12NILSs8tPad\n039J9SnNLyRtBEL2ylLydBNGZpWeU7lm6R2l\npaUlp39I+pZGlEaVPLX1KaWhpSOmqpauUakvJ\niRSeCJbuUjq0dEjpgaUHlL629DWlzy19Tukb\nS9Q+t7S95Q+",
  "WhpSOmqpauUakvJ\niRSeCJbuUjq0dEjpgaUHlL629DWlzy19Tukb\nS9Q+t7S95Q+tvQxpcxSRumapWuUckvJp4Mg\nWrF0hdLAUvLuB2vN0h6lmaUZpU8sfULpwFLy\nVgzPM0vJ8QYejJZKStctXadUWEre34LopaUvK\nU0sTSh9YekLSt9Z+o7SZ5Y+ozS2lHwbgNOJp\nTuU2q9AVUHptqXblJ5aeur+LsBnwxi4JuaWr\nWCL0tTSlNINS8mbAhw",
  "HwbgNOJp\nTuU2q9AVUHptqXblJ5aeur+LsBnwxi4JuaWr\nWCL0tTSlNINS8mbAhwlLD0h58lITXY1+92zxs\naMO9gk49OrSc4jNeMONtmdpleT/SlSMz4kXV\n/bm31IgZTCTn98c3EZf4Wlhb1fu8u/de9t31\nt8tDL5Qnu983nh86dznLnQedR53mn1+l3wr\nls7q+5v+f+Wbi+0F24v/BgrF6bm1zbaf1t/D\noP4A+M0Y=</latexit>\n\u02c6\u03c6 =",
  "wr\nls7q+5v+f+Wbi+0F24v/BgrF6bm1zbaf1t/D\noP4A+M0Y=</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi)\n#\n= argmax\n\u03c6\n\" IY\ni=1\nPr(yi|\u2713i)\n#\n= argmax\n\u03c6\n\" IY\ni=1\nPr(yi|f[xi, \u03c6])\n#\nWhen we consider this probability as a function of the \nparameters \ud835\udf19, we call it a likelihood.\n\ud835\udf03*are the parameters of the \nprobability distribution\n\ud835\udf19 are the parameters of the \nneural network, e.g.\n\ud835\udf03! = f[x!, \ud835\udf19]\n47",
  "Problem:\n\u2022 The terms in this product might all be small\n\u2022 The product might get so small that we can\u2019t easily represent it in \nfixed precision arithmetic\n<latexit sha1_\nbase64=\"cV60uOWAdFLGj2E7xzId\ndRa2vpA=\">AX13icrZjZbtw2F\nEDH6Za6W9Ki8ENfhBop0iId2EW6v\nARI7DiJY6ceJ95iyzEoDaVhTJEyR\ndnjqELfir72k/oZ/YK+tn/QS81C\ni5cuGqADOGLuOSKpy0VLlHNW6IWF\nP2auvPHmW2+/c/Xd2fe/+Dj65\nd/3inkKWK6XYsuVR7",
  "OGLuOSKpy0VLlHNW6IWF\nP2auvPHmW2+/c/Xd2fe/+Dj65\nd/3inkKWK6XYsuVR7ESkoZ4Jua6Y\n53csVJVnE6W50vGz47ilVBZNiS5/\nn9DAjqWAJi4mG0NH1mf0gHBdhV\nGSD1gdfHEnCIlKMzI8msRCThN9EO\nZK9o8qdmexflGt1kFP3QR+DpH6p\nzCL5NDoQVIfwGFordGpx9+GSqWD\nvRhEAopyiyiKgjD2X9piMv0f2pyc\nviPDRdldqG",
  "5NDoQVIfwGFordGpx9+GSqWD\nvRhEAopyiyiKgjD2X9piMv0f2pyc\nviPDRdldqG1ph9LOUHr9Eu+GrS\nbPfo2vxCd6H5BbiwOC7Md8a/3tH1\nT/thX8ZlRoWOSmKg8WFXB9WRGk\nWc1rPhmVBcxIfk5QeQFGQjBaHVTM\nL6uAGRPpBIhX8CR0YtnVCQrivM\nsAjMjelC4zAR97KDUyQ+HFRN5qa\nmIRw0lJQ+0DMyUCvpM0VjzcyiQWD\nHoaxAP",
  "sAjMjelC4zAR97KDUyQ+HFRN5qa\nmIRw0lJQ+0DMyUCvpM0VjzcyiQWD\nHoaxAPiCKxhok3Gwp6FsI6Jfh\nUsrmzVkj6ZMVPSkbCZhXbedlcahU\nLzMWFrdmtbCNM3YK4oqaRTySUC\nTeuqot206wJGAbAuRUAKWkCdJj8w\n/IsOhUXHAVd2gjytUdVC0xRy0tL2\nkQaFnNhy1pGFgxl1lKegRIENwI\nDqFYwCtBVOFBnDJ7lRNST8zQdapV\nVh",
  "y0tL2\nkQaFnNhy1pGFgxl1lKegRIENwI\nDqFYwCtBVOFBnDJ7lRNST8zQdapV\nVhYm5LSgiUto0AZcE26uqG2Ikn\nM4NW5ZP7rWUyKOx4mTedNVZSKOta\nXajlY4L6LfdpqIY8EkTNtWE3EsDl\ntkn8AmUE/KsBuoLDARv8qEqzI0M\nXtKRu2cxNx5+Ywh/XS9lYqlP5T4\nmTEBGD1mSMjIqZtfVlO7WCSnNPG\nNwU6DAYwWO1TzDbYXNakEbiqc",
  "S9lYqlP5T4\nmTEBGD1mSMjIqZtfVlO7WCSnNPG\nNwU6DAYwWO1TzDbYXNakEbiqcazG\nZpMrx8TZgpCSZ23T9Maj0py1L9AE\n3EVXKiaSC9qtpgRT1oTDW3CpquT\n04Ovut3R4WC2YZWP+QdmEioy91V\nkwq9RUR9uyu78gog7eJI7gweBZv\nAkh/3dGTqi3IltIs3YQYEJwpk+d5\nY/S0X7nCbidlZmTl8hYOqFI2HCGe\nQkacsmYGQ4wuOFZwL",
  "IltIs3YQYEJwpk+d5\nY/S0X7nCbidlZmTl8hYOqFI2HCGe\nQkacsmYGQ4wuOFZwLFzkXGo2uMu\nSxKRdHm58xniDS62RYVMzer9obKj\ndDeNyifngVluDmc0ktOj5yMRqN8\nRrIUfaKcZA7NkA5fhIWGJeZb/c2Q\nj4peK6Una+P2oF8wOmUc05OjNXc\n8UmRhzt1wfOcty6OLE97UNd0ul7\nsWbX24is0tVOP6zc5qnfcS7/tcS/\npAT1Z9/R2H",
  "Rhzt1wfOcty6OLE97UNd0ul7\nsWbX24is0tVOP6zc5qnfcS7/tcS/\npAT1Z9/R2HXnIwg536hr3EHvI8r\nQHdfnzuO67Co/rNzmqd5JHr+1xp6\nYz/ZOtAdXEPCZJ3jePfZKHo5Ara\nixqrygzmjriKOSKWdm24P+u8ozBz\naNtjUKu2CtYWzMBV+pT7l7CKOSKo\nyXcNscxV13qOt+lfB84JijkCs+\nJl71aOQK6ZYTL3iMclzRxyFUB4H\nbh4H",
  "OSKo\nyXcNscxV13qOt+lfB84JijkCs+\nJl71aOQK6ZYTL3iMclzRxyFUB4H\nbh4HOI+5K+U+yR2R3DMiaEr5JpQ\nayLZkAq40dFobehqDHnApnAbHQVc\nu8MwrvDNPOLNY4Fm87Wt4+5KGNXE\nqNAFX2kBrLAg3vIsclMj1m+JO\nfMsXKcwJ7r9LAzefqLkgo9ycHbsa\nXnmJ5ZeobprqW7mCpL0RtBlDy1F\nL2dRMmpaeY7li6g2lpaYnptqXbm",
  "o9ycHbsa\nXnmJ5ZeobprqW7mCpL0RtBlDy1F\nL2dRMmpaeY7li6g2lpaYnptqXbm\nCaWJpg+sPQBprGlMabLli5jqi1FT\n6RwR7B0C9OBpQNM9yzdw/S5pc8x\nfWTpI0z3Ld3H9JWlrzC9Z+k9TIml\nBNMVS1cwpZaiTwdRsmTpEqaRpej\ndD9apT1Mc0tzTO9beh/TvqXorRj\nuZ5aixu4MVrKMV21dBVTZil6f4\nuSJ5Y+wTSzNMP0saWPMX1p",
  "O9beh/TvqXorRj\nuZ5aixu4MVrKMV21dBVTZil6f4\nuSJ5Y+wTSzNMP0saWPMX1p6UtMH1\nr6ENPUvRtAJ5OLH2Gqf0KVBWYbl\nq6iemJpSf+7wJ0OoyRb2Ju2Ao2M\nJWSkzXLEVvCvAoYekxep5MxHhXs\n58/a9eYcg8bZ3xyNsp5Iqbcw8a7\n0+RstD8lYsoHqOsrO9MPKZBS2OmP\nrs0vul9hcWHnm+7id93bm7fn7y6N\nv9Be7XzW+bxzs7P",
  "soHqOsrO9MPKZBS2OmP\nrs0vul9hcWHnm+7id93bm7fn7y6N\nv9Be7XzW+bxzs7PY+b5zt/Oo0+t\nsd+KZ32f+nPlr5u+53M/z/0y9+t\nIvTIzPueTus39s/CFJVTQ=</\nlatexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|f[xi, \u03c6])\n#\n= argmax\n\u03c6\n\"\nlog\n\" IY\ni=1\nPr(yi|f[xi, \u03c6])\n##\n= argmax\n\u03c6\n\" I\nX\ni=1\nlog\nh\nPr(yi|f[xi, \u03c6])\ni#\n.\n48",
  "The log function is monotonic\nMaximum of the logarithm of a function is in the same place as maximum of function \n50",
  "Maximum log likelihood\n<latexit sha1_b\nase64=\"cV60uOWAdFLGj2E7xzIdRa\n2vpA=\">AX13icrZjZbtw2FEDH6Za\n6W9Ki8ENfhBop0iId2EW6vARI7DiJ\nY6ceJ95iyzEoDaVhTJEyRdnjqELfir\n72k/oZ/YK+tn/QS81Ci5cuGqADOGLu\nOSKpy0VLlHNW6IWFP2auvPHmW2+/c\n/Xd2fe/+Dj65d/3inkKWK6XYsuVR\n7ESkoZ4Jua6Y53csVJVnE6W50vGz47",
  "+/c\n/Xd2fe/+Dj65d/3inkKWK6XYsuVR\n7ESkoZ4Jua6Y53csVJVnE6W50vGz47\nilVBZNiS5/n9DAjqWAJi4mG0NH1mf\n0gHBdhVGSD1gdfHEnCIlKMzI8msRC\nThN9EOZK9o8qdmexflGt1kFP3QR+Dp\nH6pzCL5NDoQVIfwGFordGpx9+GSq\nWDvRhEAopyiyiKgjD2X9piMv0f2pyc\nviPDRdldqG1ph9LOUHr9Eu+GrSbPf\no2vxCd6H5Bb",
  "pyiyiKgjD2X9piMv0f2pyc\nviPDRdldqG1ph9LOUHr9Eu+GrSbPf\no2vxCd6H5BbiwOC7Md8a/3tH1T/th\nX8ZlRoWOSmKg8WFXB9WRGkWc1rPhm\nVBcxIfk5QeQFGQjBaHVTML6uAGRPpB\nIhX8CR0YtnVCQrivMsAjMjelC4z\nAR97KDUyQ+HFRN5qamIRw0lJQ+0DMy\nUCvpM0VjzcyiQWDHoaxAPiCKxhok3G\nwp6FsI6JfhUsrmzVkj6ZMV",
  "w0lJQ+0DMy\nUCvpM0VjzcyiQWDHoaxAPiCKxhok3G\nwp6FsI6JfhUsrmzVkj6ZMVPSkbC\nZhXbedlcahULzMWFrdmtbCNM3YK4oq\naRTySUCTeuqot206wJGAbAuRUAKWk\nCdJj8w/IsOhUXHAVd2gjytUdVC0xR\ny0tL2kQaFnNhy1pGFgxl1lKegRIEN\nwIDqFYwCtBVOFBnDJ7lRNST8zQdapV\nVhYm5LSgiUto0AZcE26uqG2IknM4\nNW5ZP",
  "IDqFYwCtBVOFBnDJ7lRNST8zQdapV\nVhYm5LSgiUto0AZcE26uqG2IknM4\nNW5ZP7rWUyKOx4mTedNVZSKOtaXajl\nY4L6LfdpqIY8EkTNtWE3EsDltkn8Am\nUE/KsBuoLDARv8qEqzI0MXtKRu2c\nxNx5+Ywh/XS9lYqlP5T4mTEBGD1mSM\njIqZtfVlO7WCSnNPGNwU6DAYwWO1Tz\nDbYXNakEbiqcazGZpMrx8TZgpCSZ2\n3T9Maj0py1L9AE",
  "CSnNPGNwU6DAYwWO1Tz\nDbYXNakEbiqcazGZpMrx8TZgpCSZ2\n3T9Maj0py1L9AE3EVXKiaSC9qtpgRT\n1oTDW3CpquT04Ovut3R4WC2YZWP+Qd\nmEioy91Vkwq9RUR9uyu78gog7eJI\n7gweBZvAkh/3dGTqi3IltIs3YQYEJw\npk+d5Y/S0X7nCbidlZmTl8hYOqFI2H\nCGeQkacsmYGQ4wuOFZwLFzkXGo2uM\nuSxKRdHm58xniDS62RYVMze",
  "l8hYOqFI2H\nCGeQkacsmYGQ4wuOFZwLFzkXGo2uM\nuSxKRdHm58xniDS62RYVMzer9obKjd\nDeNyifngVluDmc0ktOj5yMRqN8RrIU\nfaKcZA7NkA5fhIWGJeZb/c2Qj4peK\n6Una+P2oF8wOmUc05OjNXc8UmRhzt\n1wfOcty6OLE97UNd0ul7sWbX24is0t\nVOP6zc5qnfcS7/tcS/pAT1Z9/R2HX\nnIwg536hr3EHvI8rQHdfnzuO67Co/r\nN",
  "t\nVOP6zc5qnfcS7/tcS/pAT1Z9/R2HX\nnIwg536hr3EHvI8rQHdfnzuO67Co/r\nNzmqd5JHr+1xp6Yz/ZOtAdXEPCZJ3j\nePfZKHo5AraixqrygzmjriKOSKWdm\n24P+u8ozBzaNtjUKu2CtYWzMBV+pT7\nl7CKOSKoyXcNscxV13qOt+lfB84Ji\njkCs+Jl71aOQK6ZYTL3iMclzRxyF\nUB4Hbh4HOI+5K+U+yR2R3DMiaEr5Jp\nQayLZkAq40d",
  "71aOQK6ZYTL3iMclzRxyF\nUB4Hbh4HOI+5K+U+yR2R3DMiaEr5Jp\nQayLZkAq40dFobehqDHnApnAbHQVcu\n8MwrvDNPOLNY4Fm87Wt4+5KGNXEqN\nAFX2kBrLAg3vIsclMj1m+JOfMsXK\ncwJ7r9LAzefqLkgo9ycHbsaXnmJ5Ze\nobprqW7mCpL0RtBlDy1FL2dRMmpa\neY7li6g2lpaYnptqXbmCaWJpg+sPQB\nprGlMabLli5jqi1FT6RwR7",
  "1FL2dRMmpa\neY7li6g2lpaYnptqXbmCaWJpg+sPQB\nprGlMabLli5jqi1FT6RwR7B0C9OBpQ\nNM9yzdw/S5pc8xfWTpI0z3Ld3H9JW\nlrzC9Z+k9TImlBNMVS1cwpZaiTwdRs\nmTpEqaRpejdD9apT1Mc0tzTO9beh/\nTvqXorRjuZ5aixu4MVrKMV21dBVT\nZil6f4uSJ5Y+wTSzNMP0saWPMX1p6U\ntMH1r6ENPUvRtAJ5OLH2Gqf0KVBWY\nbl",
  "Zil6f4uSJ5Y+wTSzNMP0saWPMX1p6U\ntMH1r6ENPUvRtAJ5OLH2Gqf0KVBWY\nblq6iemJpSf+7wJ0OoyRb2Ju2Ao2M\nJWSkzXLEVvCvAoYekxep5MxHhXs58\n/a9eYcg8bZ3xyNsp5Iqbcw8a70+Rst\nD8lYsoHqOsrO9MPKZBS2OmPrs0vul\n9hcWHnm+7id93bm7fn7y6Nv9Be7XzW\n+bxzs7PY+b5zt/Oo0+tsd+KZ32f+n\nPlr5u+53M/z/",
  "id93bm7fn7y6Nv9Be7XzW\n+bxzs7PY+b5zt/Oo0+tsd+KZ32f+n\nPlr5u+53M/z/0y9+tIvTIzPueTus\n39s/CFJVTQ=</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|f[xi, \u03c6])\n#\n= argmax\n\u03c6\n\"\nlog\n\" IY\ni=1\nPr(yi|f[xi, \u03c6])\n##\n= argmax\n\u03c6\n\" I\nX\ni=1\nlog\nh\nPr(yi|f[xi, \u03c6])\ni#\n.\nNow it\u2019s a sum of terms, so doesn\u2019t matter so much if the terms are small \n51",
  "Minimizing negative log likelihood\n\u2022 By convention, we minimize things (i.e. a loss)\n<latexit sha1_b\nase64=\"YPdSblfErEamlr68Ud09En\ngZAU=\">AXpnicvZjbUtw2GICXHlN\n6StrpcNEbT5l0k7CsJ30cJOZBEISA\nglLYIEb7ayV/YqyLKRZVji+h36NL1\ntX6Nv01/27grFxe9KTPpqv/3WZJ/H\nSw7yDjL1erqPwvf/Bhx9fO2TxU8\n/+/yL6/f+OogTwsZ0n6Y8lQeBSn\nAnaV0xepRJSpKA08Pg",
  "qPwvf/Bhx9fO2TxU8\n/+/yL6/f+OogTwsZ0n6Y8lQeBSn\nAnaV0xepRJSpKA08PgZF3zwzMqc5a\nKfXWR0UFCYsEiFhIFoeGNhdv+mKjSD\n6JszCrv+/ueT2SckMlwFvM5jdSxnx\nfJsGT3u9WbcrPyfJ7G/hqL+XFP3gLx\nAlj1u58E6URf50XVMfxMdPROU8/gtv\nblwJcsHquBL1JRJAGVnu8vzlplwmr1\n7v/ebF15U+F2XZmuo6lieH15dW",
  "U8/gtv\nblwJcsHquBL1JRJAGVnu8vzlplwmr1\n7v/ebF15U+F2XZmuo6lieH15dW1/v\nNwoTstLHemf73hjW9G/igNi4QKFXKS\n58fd1UwNSiIVCzmtFv0ipxkJT0hMj6\nEoSELzQVkPauXdhMjIi1IJ/4Ty6ujl\nK0qS5PlFEoCZEDXObaDLnZcqOjXQc\nlEVigqwqahqOCeSj09Q7wRkzRU/AIK\nJQM+uqFYyJqGAeLfqCnodpkhAxK\nv21jd0",
  "EVigqwqahqOCeSj09Q7wRkzRU/AIK\nJQM+uqFYyJqGAeLfqCnodpkhAxK\nv21jd0KUkdjJkp6WtRzqrazkbtUCh\neZaxt7s9rYom7B1FldSKruQKgcZVW\ndKVeMUGjAJgKxSBVNAc6tT5gZnTtSi\nsIQ64NHPrZYWqForGkJOW9hpUMg4n\nbSsdWTBUCYtZQ8Uz7vpaUCVhFGArsI\nPtcZgLyOiml2n6ETJpMx1zG5BEhHTu\ngm45ZBwfUdtQxSc",
  "8Uz7vpaUCVhFGArsI\nPtcZgLyOiml2n6ETJpMx1zG5BEhHTu\ngm45ZBwfUdtQxScw6Vhy3phWy+JOJk\nmLs3qrkodsax92XaUxHkRo7ZTRywLJ\nmHctuqIZXHY8UYEtq1qVoYlLRNPR9w\nqE7bK0MTsyTRot53piD03Jxmsl7a3\nUaL0nxErIzoAq0/MiJC2tbX07ntzZ\nJzVvu6QCfeGAarfUmzl1qBO5qGquw\nWefKMnG2ICT87ape+NQacbaN",
  "tbX07ntzZ\nJzVvu6QCfeGAarfUmzl1qBO5qGquw\nWefKMnG2ICT87ape+NQacbaN6gD9q\nIrJBPRJe1OXYIpq8P+HbhVWXB6fHfl\nJzoZlKt62ej/oGxCRXmRuSrS4f9Q0Q\niesfb8gog9eCm3Bg8C9eClHPZ3a+iI\ntCe2jtRjBwUmCGfqwlr+LBbta+qI3d\nk0sfoKAV0v/BImrEGOorasA1qGXzgt\nOCZQaN1k2NxjyNO8kBRtftZ8hkit62",
  "d\nk0sfoKAV0v/BImrEGOorasA1qGXzgt\nOCZQaN1k2NxjyNO8kBRtftZ8hkit62\n1RMv2wam+oXAvtfYPy+VQhofDGb3\ni8sDKaNDkM0gLMSLSuZED+nkjZ8rW\nGKu1V8PeVN0WjE93Zq2B/2C0SnCkJ4\nOt+zxiJGFHW7VBczZ10cWY72oK75d\nL3cs3LrzQ9oascO121yVO+0l27b4V7\nRA3q67ejtNvKQhR1u1TXtIfaQ5WgP6\nnLncdt1Fw7",
  "Q9oascO121yVO+0l27b4V7\nRA3q67ejtNvKQhR1u1TXtIfaQ5WgP6\nnLncdt1Fw7XbXJU7yPTtvhzk1r+kf\n7Y6qIPialfKSPfSn3m5AtKiwqp5gmN\nLbEJmSLSdG24P9tZY/Bw6NtNSFb7OW\nsremALY0ot2+hCdlis4Tb5jRmq9sOd\ndutEp6NLbMJ2eITkth3YRsMcZi7B\nRPSJZYhNCeRzbeRzjPGa2lLke0Qy\nx4igKeWaUHKctiUdsKWJ1",
  "kth3YRsMcZi7B\nRPSJZYhNCeRzbeRzjPGa2lLke0Qy\nx4igKeWaUHKctiUdsKWJ1drE0Rj0gK\nfCanAatOUcz7zcOfOENYsFnsV9V8P9\nKxpWxKpQB2xpB60xz9xLrLATjEcs1\nxJzphlZTiBPdvpYWd2+guiEp3k4MXa\n0AtMzw09x/TQ0ENMpaHojSCIXhqK3k\n6C6MzQM0wPD3AtDC0wLRvaB/TyNAI\n08eGPsY0NDTEdN3QdUyVoehECk8EQ",
  "qK3k\n6C6MzQM0wPD3AtDC0wLRvaB/TyNAI\n08eGPsY0NDTEdN3QdUyVoehECk8EQ/\ncxHRs6xvTI0CNMXxn6CtOnhj7F9LWh\nrzF9Z+g7TB8a+hBTYijBdMPQDUypo\nejTQRCtGbqGaWAoeveDtWZoD9PM0Az\nTR4Y+wnRkKHorhueZoeh4Aw9GQzm\n4ZuYsoMRe9vQfTc0OeYJoYmD4z9Bm\nmbw19i+kTQ59gGhuKvg3A6cTQPUzNV\n6Ayx3TX0",
  "oMRe9vQfTc0OeYJoYmD4z9Bm\nmbw19i+kTQ59gGhuKvg3A6cTQPUzNV\n6Ayx3TX0F1MTw09dX8XoPNhDFwTc8d\nUsINpamiK6Zah6E0BjhKGnqDzZCSmu\n5r5clrZxpw72DTjs6tRziMx5w423Z1\nmV6P9KRJzPkZd3ziYf0iBlMJOP7y+3\nLW/wuLCwY8r3Z9X7u3eW36wNv1Ce63\nzbe7zq1Ot/NL50HnafX6XfChT8W\n2/pd/+BU5sQSA=</l",
  "Z9X7u3eW36wNv1Ce63\nzbe7zq1Ot/NL50HnafX6XfChT8W\n2/pd/+BU5sQSA=</latexit>/lz4a+HvpVtL5b6S4eN+t7C9JqvO6\n\u02c6\u03c6 = argmax\n\u03c6\n\" I\nX\ni=1\nlog\nh\nPr(yi|f[xi, \u03c6])\ni#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\nh\nPr(yi|f[xi, \u03c6])\ni#\n= argmin\n\u03c6\nh\nL[\u03c6]\ni\n52",
  "Inference\n\u2022 But now we predict a probability distribution\n\u2022 We need an actual prediction (point estimate)\n\u2022 Find the peak of the probability distribution (i.e., mean for normal)\n)\ud835\udc66 = \u0302\ud835\udf07 = argmax\n\"\n[Pr \ud835\udc66 \ud835\udc1f \ud835\udc31, \ud835\udf19 )]]\n53",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n54",
  "Recipe for loss functions\n55",
  "Recipe for loss functions\n56",
  "Recipe for loss functions\n57",
  "Recipe for loss functions\n58",
  "Let\u2019s apply this recipe to\n\u2022 Example 1: Real valued univariate regression\n\u2022 Example 2: Binary Classification\n\u2022 Example 3: Multiclass Classification\n59",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n60",
  "Example 1: univariate regression\n61",
  "Example 1:  univariate regression\n\u2022 Predict scalar output:\n\u2022 Sensible probability distribution: \n\u2022 Normal distribution\n<latexit sha1_base64=\"qnWtp5AC\n6bjVzWxDT8S09MRwyzM=\">AWiHiclZhb9s2FIDV7tZl7Yblpe9CAsK\nDENnOEO3bm9t0vSWdHuaeM0oGRKZkNRikQldgX/j71u/2r/ZoeSbFbnMA\n8zkJo93ydeDkmJVpBJUeh+/98bNz/6+JNP7v1+dIX3719e07d785LNIy\nD/lBmMo0Pw5YwaVQ/EALflxlnOWBJIfBefr",
  "/6+JNP7v1+dIX3719e07d785LNIy\nD/lBmMo0Pw5YwaVQ/EALflxlnOWBJIfBefrh9d8rwQqdrX04yfJixWIh\nIh0xB6Ox0KNUyYHgdBtTs7u7PS7/Xrj08Lq21hxWs/g7O7342GozQsE650\nKFlRnKz2M31asVyLUPLZ0rAseMbCcxbzEygqlvDitKq7PfPvQWTkR2kOf0r\n7dfTDKyqWFMU0CcA0XSwM0EXOyl19PtpJVRWaq7CpqGolL5OfZM",
  "vQWTkR2kOf0r\n7dfTDKyqWFMU0CcA0XSwM0EXOyl19PtpJVRWaq7CpqGolL5OfZMDfyRyH\nmo5hQILcwF9cMxy1moIVNLQ8WvwjRJmBpVw7WNnVk1DHgsVMUvyjprs1n\nX2agdDsXrjLUX+4tahOaJeM9JbViKrlG4PGsqngv7mEgOADR4wSkihdQZ\nz29kb+KwSCRh4kE6gc5G/OyNVK81jyElHe0M0KGSTzrWOrFgKpOsge\nK79/zDeA6h1mAr",
  "+KwSCRh4kE6gc5G/OyNVK81jyElHe0M0KGSTzrWOrFgKpOsge\nK79/zDeA6h1mArsIXR3OwlzE1m1+n+UTnSVWYG4hZyrmdRMw5JBJM6Kuo\nUop4dKwY/2JrV2mztvEpVnd1dxEkLWfdx2d07yoUdepI8iCRh3rTqCLAl7\nesQSBluy2cw4MQ3EbcqFYFWZiDPA26bWcmgtfmJIP90vU2KpL+S4YyYg\nKw+8y3YCrkX09Xdj+PDmXtW8KfOKPYbK",
  "DPA26bWcmgtfmJIP90vU2KpL+S4YyYg\nKw+8y3YCrkX09Xdj+PDmXtW8KfOKPYbK6l7A8boY1bwRG1cZm1KxzhUya\nLQjl6VXNL1xqDwT3QGaAN50ZS5U9IF2vy7BkjXh4X0Yal5KfvJz71c+Oa\n36ZtuYf0g2oaKizFwVmfD/qGgETxG8viCJy+VaPIgUE9eKuH+jqaO5Xh\nm0g9d1AQikmhp2j7i1h1r6kjuLNpgvoKAVMvfDOh0CRHUVc2ASP",
  "eKuH+jqaO5Xh\nm0g9d1AQikmhp2j7i1h1r6kjuLNpgvoKAVMvfDOh0CRHUVc2ASPDNzwPHQ\nsoRIMmzGMi3KnJObH1rPEKl1c1vMhXlYdW+o0gjd+waXi6ugDA+HS37N\n5QHKaNDkM0hLNWI5SubETOnk7bDQsMVcu7+e8qbotGJ+sdm2B/2C2SnDkF+\ncbeL5iIlFHYnqgOIsy5JLEd7UNdiuX7Ys2rz7U9kacO121KUm/bS7ftc\nK/pAb/YcvR",
  "L5iIlFHYnqgOIsy5JLEd7UNdiuX7Ys2rz7U9kacO121KUm/bS7ftc\nK/pAb/YcvR2i3jEo5EdbU9pB6xHO1BXe48brlG4XDdpiT1zvPotB3uwkT\nLP9ofc83MSmVI3PsS+WwCWFRU1E7xThMRKbEBaTsmvB/7GyJ+Dh0bWaE\nBYHhehqJoClEZd4CE0Ii80W7ptDKtbDnXLrTKZjZHZhLD4jCV41E0IizE\nVY6d4zrIMiU2I5HGM8zimecywlLkP",
  "7ptDKtbDnXLrTKZjZHZhLD4jCV41E0IizE\nVY6d4zrIMiU2I5HGM8zimecywlLkPCOZY0bIknItqHycdiUTwNIEtTZxN\nAY9kKlCDbZBLBd05RXOlafQKlZ0FR+4Gj64pmHNUIUmgKVtsf84bZzkwU4\nxXDMciU5E8jKaAIH2BlQZ376C6KnOSCaGrplNIrS68oPbL0iNLcUvKLI\nh2LSW/ToLo0tJLSg8tPaS0tLSk9MDSA0ojSyNKn1r6lNLQ0",
  "68oPbL0iNLcUvKLI\nh2LSW/ToLo0tJLSg8tPaS0tLSk9MDSA0ojSyNKn1r6lNLQ0pDSdUvXKdW\nkhMpPBEs3ad0bOmY0mNLjyl9belrSp9b+pzSN5a+ofS9pe8pfWzpY0qZpY\nzSDUs3KOWklcHQbRm6RqlgaXktx/sNUsHlGaWZpQ+sfQJpSNLya9ieJ5Z\nSo438GC0VFL6wtIXlApLye+3IHpl6StKE0sTSl9a+pLSd5a+o/SZpc8ojS\n0l7wb",
  "So438GC0VFL6wtIXlApLye+3IHpl6StKE0sTSl9a+pLSd5a+o/SZpc8ojS\n0l7wbgdGLpHqX2LVBVULpj6Q6lF5ZeuN8L8MU0Bq6FuW0r2KY0tTSldNS8\nksBjhKWnpPzZKTau9r8bRO5r0VqwR2szfj8apLzSC24g7V3p/nV5P4UqQU\nfk65vHC5epEBK4U5/dmdlFb+FpYXDX3qrv/Ue7DxYebTWvqG95X3v/eD96\nK16D71H3nNv4B14oZd7f",
  "5/dmdlFb+FpYXDX3qrv/Ue7DxYebTWvqG95X3v/eD96\nK16D71H3nNv4B14oZd7f3l/e/8sLy3lx8u/9GoN2+013zrdT7La/8Bwib\nUaQ=</latexit>y 2 R\n<latexit s\nha1_base64=\"sL5nmFJ\nHMlFiWTdFxaDUKnbUrJ\ng=\">AW4XiclZjLbtw\n2FECVPtP05bSoN90INQ\nIkRWLYRvrYFEjsOC879\nTh+JpZjUBpKw5iZImy\nZ6LOB3RXdNtP",
  "P05bSoN90INQ\nIkRWLYRvrYFEjsOC879\nTh+JpZjUBpKw5iZImy\nZ6LOB3RXdNtP6hf0M7p\ntN73UaIbRvfSiAzjD3H\nPExyUpcRTmUpR6aemvK\n+8+97H3x49aNrH3/y\n6Wefz13/Yr/MqiLie1E\nms+IwZCWXQvE9LbTkh3\nnBWRpKfhCerhl+cM6LUm\nRqV49yfpyRIlYRExD6\nGSO94qbo1+CtLodlCJ\n2auVW/5PfhAXLKqXx3V\nQnhW6Xg",
  "qV49yfpyRIlYRExD6\nGSO94qbo1+CtLodlCJ\n2auVW/5PfhAXLKqXx3V\nQnhW6Xgly0eJ6ZTweB3\nyYB5LH+ujOxLw5ugNV3\nGo2FYNCpEM9PHJ3MLS\n4lLz8WlhuS0seO2nd3L\n9q37Qz6Iq5UpHkpXl0f\nJSro9rVmgRST6+FlQlz\n1l0yhJ+BEXFUl4e10+\nxv4NiPT9OCvgT2m/ib5\n9Rc3SshylIZgp04MSMx\nN0saNKxz8e10LleYqm\njQ",
  "0+\nxv4NiPT9OCvgT2m/ib5\n9Rc3SshylIZgp04MSMx\nN0saNKxz8e10LleYqm\njQUV9LXmW+S6/dFwSMt\nR1BgUSGgr340YJAhDVN\nwLVD8IsrSlKl+Hayub0\nN6Q54IVfOzqpmO8bjr\nDcOh+JlxuqT3VktQvNU\nvOGkYxlVwi8GRc13w\nxWcRAcABikROQKV5CnS\nY/YewvIwrLTwIGHmZD6\nFzsPx+TqpXmCeSko70k\nGhRyYcda41YMJVpR9",
  "KV5CnS\nY/YewvIwrLTwIGHmZD6\nFzsPx+TqpXmCeSko70k\nGhRyYcda41YMJVpR9k\nBxfdv+AZwXcAsQFfhi6\nM52MmZGk+v03yoi7QuTQ\ny3UDCV8KYJGHLEpBlR1\n1CVlHBp1LF+xtZzpk7b\nxGV509XCRJC1W3QdXdC\n8qH7XaSLIgkWYdK0mgi\nwJN4s+SxlkuS2fwIBT3\n0TcqlBYFWRh9os7Lad\nmwhem8Mc9kvXW69J+s8\nZyogJwO4z3",
  "+SxlkuS2fwIBT3\n0TcqlBYFWRh9os7Lad\nmwhem8Mc9kvXW69J+s8\nZyogJwO4z34KpiHf1tW\nxm+9PknDe+KfChP4DJ6\nl7CimQyrGkjMKo2NqZm\nkytk0mxBqMguqbpjUP\nluegO0ATwpqsKoeK3tN\ntNCZasCQe3YahFJfnRn\ncXv+PC4XjLbxvxDsgkV\nlVXuqsiE/0dFfXg84fU\nFETx5mUSTB4Fm8jIJ93\nc0dazAC9tEmrmDglBMC\nj1C",
  "lVXuqsiE/0dFfXg84fU\nFETx5mUSTB4Fm8jIJ93\nc0dazAC9tEmrmDglBMC\nj1C218kqntNE8GdzVLU\nVwiYeuGbCYUmOY67sgk\nYGb7hQetYQBEaZDQZYy\nSzsio4ufmh9QyRje3x\nUKYh1X3hiqN0L1vcDm7\nCsrwcDjnl1weoyGk3y\nGWaX6rEDJHJopHb4KSg\n1bzLX7mymfFJ1Wws82\nvagXzA7VRTxs5MNPB8Js\nagjUV1wsnHWJYnlaA",
  "Hb4KSg\n1bzLX7mymfFJ1Wws82\nvagXzA7VRTxs5MNPB8Js\nagjUV1wsnHWJYnlaA/q\nmi3Xt3tWb7z6liztxOG\n6TUnqbXvpth3uJT3gZ5\nuO3m4Sj1jUkaiutofUI\n5ajPajLncdN1ygcrtuU\npN5pHp2w52ZaPnHuwO\numTkmZbJvjn2ZDCYhLG\noqaqeYpTxB4iSExbTqW\nvB/rOyYw3LXmoSw2CtF\nVzMBLPW5xEOYhLA42cJ\nds41hdOhbr",
  "pTxB4iSExbTqW\nvB/rOyYw3LXmoSw2CtF\nVzMBLPW5xEOYhLA42cJ\nds41hdOhbrpVJvMBMi\nchLD5iKR71JITFhIqJU\nzxleY7ESYjkcYDzOKB5\nzLGUuyQ8I7ljRsiSci2\noYpB1JRPA0hC1NnQ0Bj\n2QmUINtkEsl3Tlc6Vp\n9AqVnQV7ka3rukYc1Q\nhSaApS2yx/xgy7nJQpx\niOGa5kpwLZOU0gT3s9K\ngzPf2FcU1OcmE8snRE6\nYWlF5",
  "SaApS2yx/xgy7nJQpx\niOGa5kpwLZOU0gT3s9K\ngzPf2FcU1OcmE8snRE6\nYWlF5QeWHpAaWEp+UQ\nxs8tJb9Owvjc0nNK9y3\ndp7SytKJ0z9I9SmNLY0\nofWvqQ0sjSiNI1S9co1\nZaSEyk8ESzdpXRg6YDS\nQ0sPKX1h6QtKH1v6mNKX\nlr6k9I2lbyi9b+l9Spm\nljNJ1S9cp5ZaSVwdhvG\nrpKqWhpeS3H+w1S3uU5\npbmlD6w9AGlfUvJr2",
  "+l9Spm\nljNJ1S9cp5ZaSVwdhvG\nrpKqWhpeS3H+w1S3uU5\npbmlD6w9AGlfUvJr2J4\nnlKjfwYLRUvrE0ie\nUCkvJ7cwfmbpM0pTS1\nNKn1r6lNLXlr6m9JGlj\nyhNLCXvBuB0YukOpfYt\nUF1Sum3pNqVnlp653wv\nw2TSGroW5ZSvYojSzNK\nN0w1LySwGOEpaekvNkr\nNq72vRtE7mvxWrGHazN\n+PRqkvNYzbiDtXen6dX\nk/hSrGR+Qrq/v",
  "OEpaekvNkr\nNq72vRtE7mvxWrGHazN\n+PRqkvNYzbiDtXen6dX\nk/hSrGR+Qrq/vz16kQE\nrhTn8yt7CM38LSwv7K4\nvL3i3e37y7cW23f0F71\nva+8W56y94P3j3vsdf\nz9rzI+9P72/vH+3c+mv\n91/rf53yfqO1fa70O\np/5P/4D+G/4mg=</la\ntexit>\nPr(y|\u00b5, \u03c32) =\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(y \u2212 \u00b5)2\n2\u03c32\n\ufffd\n62",
  "Example 1:  univariate regression\n<latexit sha1_ba\nse64=\"VrHufIDtMmytgQ3rO/QqNtGUe\nl4=\">AXBXiclZjZbtw2FEDHXdN0c1\nrELwVaoUaApHAM20iXlwKJHWezU0/iN\nbEcg9JQGsYUJUuUPRN1not+TN+KvY7\n+hX9hV5K8jC6l37oAPEw9xul5TEUZB\nJUeilpX9m3n3vfc/+PDKR1c/uTz\n6fvfbFXpGWech3w1Sm+UHACi6F4rta\nMkPspyzJB8PzhZM3z/jO",
  "/+PDKR1c/uTz\n6fvfbFXpGWech3w1Sm+UHACi6F4rta\nMkPspyzJB8PzhZM3z/jOeFSNWOHmf8\nKGxEpEImYbQ8ezv/fzm+Fc/CdJRFU0\nO/SAaLcCfbCiOFvxCxAl7tXL+9nzo5\nyF1fKk8ovTXFcrfiZaXK1MJhOfjzJf8\nkgf3m7Mm+PbzkZv1T7Ut5X9XMRDfXQ8\nO7+0uFR/PFpYbgvzvfbTP752feAP0rB\nMuNKhZEVxuLyU6aOK5VqEk+",
  "XMRDfXQ8\nO7+0uFR/PFpYbgvzvfbTP752feAP0rB\nMuNKhZEVxuLyU6aOK5VqEk+u+mXBMx\naesJgfQlGxhBdHVZ2ziXcDIgMvSnP4p\n7RXR9+uUbGkKMZJAGbC9LDAzARd7LDU\n0U9HlVBZqbkKm46iUno69cwCeAOR81D\nLMRYmAsYqxcOGeRMwzJd9RU/D9MkYW\npQ+avrzyDhAY+FqvhpWS/ZNJ1muHQ\n/EyY/XxzrQVoXki3nDSK2YRi4Re",
  "D9MkYW\npQ+avrzyDhAY+FqvhpWS/ZNJ1muHQ\n/EyY/XxzrQVoXki3nDSK2YRi4ReDyp\nKr4YL2IgOACxyAlIFS+gTZOfIPKWEYU\ntKgFXzR6BveE9n5CmleYx5KSjvSQaFD\nLJRx1rjViwlElH2QbF8254BnCdwyrAU\nOGLozXYzpiaXNTfKTzpCpMDPeQMxXz\nuguYcsikmVHXUKWUDXsWL9g6zlTJ23\ni0qweam4iyNrJu47OaV7UoOvUEWTBJ",
  "Xz\nuguYcsikmVHXUKWUDXsWL9g6zlTJ23\ni0qweam4iyNrJu47OaV7UoOvUEWTBJo\ny7Vh1BloQbyoAlDLclo9hwolnIm5VK\nKwKsjH7eRp0+85MBO/NUQbXS9dbr0j6\nzxjKiAnA1We+BVMh7+pr6dT2LpJzVvu\nmwEfeEBarW4XlcTOti05gVm1sQs06V8\nik2YJQnp53TMah8oz0Z2gCeCLrsyFi\nt7SFuoSbFkT9hdgqnkp+eHtxe/56Kha",
  "ik2YJQnp53TMah8oz0Z2gCeCLrsyFi\nt7SFuoSbFkT9hdgqnkp+eHtxe/56Kha\nMpeN+UOyCQ0VZeZqyIT/R0MDeITh/QU\nRvHipRIsHgXrxUgn3d7R0LMcb20TqtY\nOCUEwKPUaXv4hVt04dwYNEzRWCJh24\nZsJhRY5irqyCRgZvuFh7NhAIZpk2Mwx\nlGlR5pzc/NB+hkitm9tiLszDqntDlUb\no3je4nNaCMjwczvgl1QOU0aDJZ5CWas\nB",
  "GlR5pzc/NB+hkitm9tiLszDqntDlUb\no3je4nNaCMjwczvgl1QOU0aDJZ5CWas\nBylMyRWdLRK7/QcIm5rv56yZui04r56\nUbH4wLVqcMQ356vIHXIyYWdSRqC04/\nzrYksRz9QVvT7fr2yKqNV9+RrR07XLc\npSbvtKN2w71kBPx0zHaTeIRizoStd\nWOkHrEcvQHbnzuOmahcN1m5K0e5FHp\n+1wpyba/tHOkGtmjkmpHJhjXyr9JoRF\nTUXtF",
  "rEcvQHbnzuOmahcN1m5K0e5FHp\n+1wpyba/tHOkGtmjkmpHJhjXyr9JoRF\nTUXtFNOEx0hsQlhMyq4F/8fKtjksd60\nmhMV+IbqaCWBpwCWeQhPCYnMJd802ht\nVNh7rpVpnMhshsQlh8yBI86yaExZiKs\nVM8YVmGxCZE8jEeRzSPGZYylwSXpHM\nsSJkS7k2VD5Mu5IJYGmEehs5OoMRyFS\nhDtsglgu68wrnzlNoFyu6i3dHe9e0r\nFmqETwN",
  "k2VD5Mu5IJYGmEehs5OoMRyFS\nhDtsglgu68wrnzlNoFyu6i3dHe9e0r\nFmqETwNIWucY8f8t5kQU4xXDMciU5E\n8jKaAL72OlT5+L0F0QVOckF0djSMaXn\nlp5Tum/pPqW5peQXQRA9t5T8OgmiM0v\nPKN2zdI/S0tKS0l1LdymNLI0ofWDpA0\npDS0NK1yxdo1RbSk6k8ESwdIfSoaVDS\ng8sPaD0haUvKH1k6SNKX1r6ktI3lr6h\n9J6l9yhl",
  "xdo1RbSk6k8ESwdIfSoaVDS\ng8sPaD0haUvKH1k6SNKX1r6ktI3lr6h\n9J6l9yhljJK1y1dp5RbSl4dBNGqpau\nUBpaS35wrVnapzSzNKP0vqX3KR1YSn\n4Vw/PMUnK8gQejpZLSx5Y+plRYSn6/B\ndFTS59SmliaUPrE0ieUvrb0NaUPLX1I\naWwpeTcApxNLtym1b4GqgtJnlj6j9NT\nSU/d7AT5dxsC1MbdsA1uUpamlG5YSn\n4pwFHC0hNyn",
  "NLtym1b4GqgtJnlj6j9NT\nSU/d7AT5dxsC1MbdsA1uUpamlG5YSn\n4pwFHC0hNynoxUe1e7eNtE7muRmnIHa\nzN+UZvkPFJT7mDt3emiNrk/RWrKh2To\n63vTFymQUrjTH8/OL+O3sLSwt7K4/MP\ninWd35u+utm9or/S+6n3bu9lb7v3Yu9\nt71Ov3dnth79+Z6zNfz3wz9vcH3N/z\nv3VqO/MtHW+7HU+c3/B3l+BUI=</la\ntexit>\nPr(y|f",
  "9+Z6zNfz3wz9vcH3N/z\nv3VqO/MtHW+7HU+c3/B3l+BUI=</la\ntexit>\nPr(y|f[x, \u03c6], \u03c32) =\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(y \u2212 f[x, \u03c6])2\n2\u03c32\n\ufffd\n<latexit sha1_ba\nse64=\"sL5nmFJHMlFiWTdFxaDUKnbUr\nJg=\">AW4XiclZjLbtw2FECVPtP05b\nSoN90INQIkRWLYRvrYFEjsOC879Th+J\npZjUBpKw5iZImyZ6LOB3RXdNtP6hf0\nM7ptN73UaIbRv",
  "LYRvrYFEjsOC879Th+J\npZjUBpKw5iZImyZ6LOB3RXdNtP6hf0\nM7ptN73UaIbRvfSiAzjD3HPExyUpcRT\nmUpR6aemvK+8+97H3x49aNrH3/y6W\nefz13/Yr/MqiLie1Ems+IwZCWXQvE9L\nbTkh3nBWRpKfhCerhl+cM6LUmRqV49y\nfpyRIlYRExD6GSO94qbo1+CtLodlCJ\nJ2auVW/5PfhAXLKqXx3VQnhW6Xgly0e\nJ6ZTweB3yYB5LH+u",
  "94qbo1+CtLodlCJ\nJ2auVW/5PfhAXLKqXx3VQnhW6Xgly0e\nJ6ZTweB3yYB5LH+ujOxLw5ugNV3Go2\nFYNCpEM9PHJ3MLS4lLz8WlhuS0seO2n\nd3L9q37Qz6Iq5UpHkpXl0fJSro9rVmg\nRST6+FlQlz1l0yhJ+BEXFUl4e10+xv\n4NiPT9OCvgT2m/ib59Rc3SshylIZgp0\n4MSMxN0saNKxz8e10LleYqmjQUV9LX\nmW+S6/dFwSMtR1BgUSG",
  "Rc3SshylIZgp0\n4MSMxN0saNKxz8e10LleYqmjQUV9LX\nmW+S6/dFwSMtR1BgUSGgr340YJAhDVN\nwLVD8IsrSlKl+Hayub0N6Q54IVfOzqp\nmO8bjrDcOh+JlxuqT3VktQvNUvOGk\nkYxlVwi8GRc13wxWcRAcABikROQKV5C\nnSY/YewvIwrLTwIGHmZD6FzsPx+TqpX\nmCeSko70kGhRyYcda41YMJVpR9kBxf\ndv+AZwXcAsQFfhi6M52MmZ",
  "FzsPx+TqpX\nmCeSko70kGhRyYcda41YMJVpR9kBxf\ndv+AZwXcAsQFfhi6M52MmZGk+v03yoi\n7QuTQy3UDCV8KYJGHLEpBlR1CVlHBp\n1LF+xtZzpk7bxGV509XCRJC1W3QdXdC\n8qH7XaSLIgkWYdK0mgiwJN4s+SxlkuS\n2fwIBT30TcqlBYFWRh9os7Ladmwhem\n8Mc9kvXW69J+s8ZyogJwO4z34KpiHf1\ntWxm+9PknDe+KfChP4DJ6l7C",
  "admwhem\n8Mc9kvXW69J+s8ZyogJwO4z34KpiHf1\ntWxm+9PknDe+KfChP4DJ6l7CimQyrGk\njMKo2NqZmkytk0mxBqMguqbpjUPlue\ngO0ATwpqsKoeK3tNtNCZasCQe3YahFJ\nfnRncXv+PC4XjLbxvxDsgkVlVXuqsiE\n/0dFfXg84fUFETx5mUSTB4Fm8jIJ93c\n0dazAC9tEmrmDglBMCj1C218kqntNE8\nGdzVLUVwiYeuGbCYUmOY67sgk",
  "jIJ93c\n0dazAC9tEmrmDglBMCj1C218kqntNE8\nGdzVLUVwiYeuGbCYUmOY67sgkYGb7hQ\netYQBEaZDQZYySzsio4ufmh9QyRje3\nxUKYh1X3hiqN0L1vcDm7CsrwcDjnl1w\neoyGk3yGWaX6rEDJHJopHb4KSg1bzL\nX7mymfFJ1Wws82vagXzA7VRTxs5MNP\nB8JsagjUV1wsnHWJYnlaA/qmi3Xt3tW\nb7z6liztxOG6TUnqbXvpth3uJT3g",
  "MNP\nB8JsagjUV1wsnHWJYnlaA/qmi3Xt3tW\nb7z6liztxOG6TUnqbXvpth3uJT3gZ5u\nO3m4Sj1jUkaiutofUI5ajPajLncdN1y\ngcrtuUpN5pHp2w52ZaPnHuwOumTkmZ\nbJvjn2ZDCYhLGoqaqeYpTxB4iSExbTq\nWvB/rOyYw3LXmoSw2CtFVzMBLPW5xEO\nYhLA42cJds41hdOhbrpVJvMBMichLD\n5iKR71JITFhIqJUzxleY7ESYjkcYDz",
  "EO\nYhLA42cJds41hdOhbrpVJvMBMichLD\n5iKR71JITFhIqJUzxleY7ESYjkcYDzO\nKB5zLGUuyQ8I7ljRsiSci2oYpB1JRPA\n0hC1NnQ0Bj2QmUINtkEsl3Tlc6Vp9A\nqVnQV7ka3rukYc1QhSaApS2yx/xgy7\nnJQpxiOGa5kpwLZOU0gT3s9KgzPf2Fc\nU1OcmE8snRE6YWlF5QeWHpAaWEp+UQ\nxs8tJb9Owvjc0nNK9y3dp7SytKJ0z9I\n9",
  "U1OcmE8snRE6YWlF5QeWHpAaWEp+UQ\nxs8tJb9Owvjc0nNK9y3dp7SytKJ0z9I\n9SmNLY0ofWvqQ0sjSiNI1S9co1ZaSEy\nk8ESzdpXRg6YDSQ0sPKX1h6QtKH1v6m\nNKXlr6k9I2lbyi9b+l9SpmljNJ1S9cp\n5ZaSVwdhvGrpKqWhpeS3H+w1S3uU5pb\nmlD6w9AGlfUvJr2J4nlKjfwYLRUv\nrE0ieUCkvJ7cwfmbpM0pTS1NKn1r6l\nNLXlr",
  "lD6w9AGlfUvJr2J4nlKjfwYLRUv\nrE0ieUCkvJ7cwfmbpM0pTS1NKn1r6l\nNLXlr6m9JGljyhNLCXvBuB0YukOpfYt\nUF1Sum3pNqVnlp653wvw2TSGroW5ZSv\nYojSzNKN0w1LySwGOEpaekvNkrNq72v\nRtE7mvxWrGHazN+PRqkvNYzbiDtXen6\ndXk/hSrGR+Qrq/vz16kQErhTn8yt7CM\n38LSwv7K4vL3i3e37y7cW23f0F71va\n+8W56y",
  "SrGR+Qrq/vz16kQErhTn8yt7CM\n38LSwv7K4vL3i3e37y7cW23f0F71va\n+8W56y94P3j3vsdfz9rzI+9P72/vH+3\nc+mv91/rf53yfqO1fa70Op/5P/4D+\nG/4mg=</latexit>\nPr(y|\u00b5, \u03c32) =\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(y \u2212 \u00b5)2\n2\u03c32\n\ufffd\nIn this case, \njust the mean\nJust learn the mean, \ud835\udf07, and assume the variance is fixed,\n63",
  "Example 1:  univariate regression\n<latexit sha1_bas\ne64=\"r2Iwry2CQ7n2YOuZTj49M3VUwAI\n=\">AXbniclZhbU9w2FICX9JbSG2mn4\naHTqadMOkHGDaTXl4yk0BIQiANhGuCg\nZG9sldBlo0swxLXfW3/Yv9Fp7+gR7Z3F\neuIh+4MWeV8n25Hku1kHGWq6Wlv6euv\nf+Bx9+dP3j6U8+/ezL2ZufLmXp4UM6\nW6Y8lQeBCSnAm6q5ji9CTlCQBp/vB6\nYrm+dU5iwVO",
  "j6U8+/ezL2ZufLmXp4UM6\nW6Y8lQeBCSnAm6q5ji9CTlCQBp/vB6\nYrm+dU5iwVO+oyo0cJiQWLWEgUhE5m/\nt049IMoG7Ij74f73oKfF8lJye73q+Nyr\nfJ8nsY+p5E63JS3LwFUv/tJkI7KqNLVR\njoy39af93MWJ+S4vFvd8SWLh+rIF6kok\noBKz/enf7h/det+JElY9qvSz8+kKu/6G\nTOtVZVPR1kjLjQmjIUtdEfCxuO4U9eBN\nkwD4+E",
  "7h/det+JElY9qvSz8+kKu/6G\nTOtVZVPR1kjLjQmjIUtdEfCxuO4U9eBN\nkwD4+E0Xyczc0uLS/XHw4V+W5jrtZ/Nk\nxtfD/xBGhYJFSrkJM8P+0uZOiqJVCzkt\nJr2i5xmJDwlMT2EoiAJzY/KenEq7xZEB\nl6USvgTyquj79YoSZLnl0kAZkLUMLeZD\nrYaGiX49KJrJCURE2HUF91Tq6ZX2B\nkzSUPFLKJBQMhirFw4JpE/Bfpj2Bb0I0\nyQh",
  "aGiX49KJrJCURE2HUF91Tq6ZX2B\nkzSUPFLKJBQMhirFw4JpE/Bfpj2Bb0I0\nyQhYlD6y6tbkPuAxkyU9Kyo90ZVdZ3V2\nqFQvMpYXtuZtMIUTdhbihqpFd3IFQKNq\n7Kki/GiDRgFwBYpAqmgObSp8xNEXt+ic\nBY4LZLBFvJcValoGkNOtprpEh4\n3TUsVaQBUuZdJRtUDzvlqcBVRJWAYKX\n9Rag+2MiGpcT9GRkmZ65jdgyQipnUXM\nOWQcD",
  "aQBUuZdJRtUDzvlqcBVRJWAYKX\n9Rag+2MiGpcT9GRkmZ65jdgyQipnUXM\nOWQcD2jriEKzqFq2LF+s62XRJy2iUuze\nqhSRyxrR3YdJXFexKDr1BHLgk0Yd606Y\nlkcrlwDkhDIcls+gQkno64VSZslaGNu\nSnToNt3piP23hxlcF63mqJ0n9OrIzoA\nJw+/c2ICGlX0kntjdOznt6wIdeUNYrG\n4VIuNmWuNOYFZtrMJmnSvLxNmCkEwvuq\nYe",
  "/c2ICGlX0kntjdOznt6wIdeUNYrG\n4VIuNmWuNOYFZtrMJmnSvLxNmCkEwvuq\nYejUOlGetOUAfsQ1dIJqJ3tPm6BFtWh/\n15mKosOD1cWPyJjo7KJX1s9D8om9BQXm\nSuhnT4fzQ0gHulvb8gYi9eyq3Fg0C9eC\nmH67u1dETaG1tH6rWDAhOEM3VpHX8Wi2\n6dOmIPNk2sUJAtwvfhAlrkaOoK+uAlu\nEb7vqODRakwybOY8zQtJ0cXP2s8QqX",
  "2\n6dOmIPNk2sUJAtwvfhAlrkaOoK+uAlu\nEb7vqODRakwybOY8zQtJ0cXP2s8QqX\nV9WZRM36y6F1Suhe51g/JLSjDzeGcXl\nE9sDIaNPkM0kIMiLSOdJLOjr2cwVHzH\nX6yVvik4rpmfrbX8wLlidIgzp2cm6vR\n4xsrDrbgMcvZFkeWoz9oa7Jd3x1ZuX\n78I9rascN1mxy1247SbTvcK0ZAzYco9\n1AHrKw62hFiD1mO/qAtdx43XLNwuG\n6T",
  "78I9rascN1mxy1247SbTvcK0ZAzYco9\n1AHrKw62hFiD1mO/qAtdx43XLNwuG\n6To3bHeXTaDndiWts/2hlSRfRjUsoH+r\nEv5X4TskWFReU04TGltiEbDEpuhb831\na29TNz12pCtriZs6mA7Y0oNyeQhOyxe\nYId802ZqsbDnXDrRKeDS2zCdniE5LYs2\n5CthjMXaKpyTLEJoTwO7TwOcR4zW8\npckr0imWNF0JZybSg5TLuSDtjSyOpt5O\ng",
  "5CthjMXaKpyTLEJoTwO7TwOcR4zW8\npckr0imWNF0JZybSg5TLuSDtjSyOpt5O\ngMRsBTYXYBm05xzsvd+48Ye1igXfxrq\nvj3Ss6VsRqUAds6QU6Y57/wnIAjvF8J\njlSnLGLCvDCdy0nU3sjJ/+gqhET3JBdG\nnoJaYXhl5gum/oPqbSUPSLIheGop+nQ\nTRuaHnmO4ZuodpYWiB6a6hu5hGhkaYPj\nb0MahoSGmK4auYKoMRU+kcEcwdAfToa",
  "TRuaHnmO4ZuodpYWiB6a6hu5hGhkaYPj\nb0MahoSGmK4auYKoMRU+kcEcwdAfToa\nFDTA8MPcD0laGvMH1q6FNMXxv6GtO3hr\n7F9KGhDzElhJMVw1dxZQail4dBNGyoc\nuYBoai35w1gzdxDQzNMP0kaGPMB0Yin\n4Vw/3MUPR4AzdGQzma4auYcoMRb/fgu\ni5oc8xTQxNMH1m6DNM3xj6BtMnhj7BND\nYUvRuApxNDtzE1b4HKHNMtQ7cwPTP",
  "u\ni5oc8xTQxNMH1m6DNM3xj6BtMnhj7BND\nYUvRuApxNDtzE1b4HKHNMtQ7cwPTP0zP1\negE6WMXBtzBemgReYpoamK4bin4pwKO\nEoafoeTIS7Vt/LYJXdciMeEO1mZ8XBv\nlPBIT7mDt1WlcG12fIjHhQzT01b3JixR\nIKVzpT2bm+vZbWFzYu7vY/3nx3ta9uQf\nL7Rva671vet/3bvf6vV96D3pPe5u93V4\n4dTD1x9SfU3/d/Gf25uy3s98",
  "ta9uQf\nL7Rva671vet/3bvf6vV96D3pPe5u93V4\n4dTD1x9SfU3/d/Gf25uy3s9816rWpts5\nXvc5n9vZ/WBQsQw=</latexit>\nL[\u03c6] = \u2212\nI\nX\ni=1\nlog\n\u21e5\nPr(yi|f[xi, \u03c6], \u03c32)\n\u21e4\n= \u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n\ufffd\ufffd\n64",
  "65",
  "<latexit sha1_base64=\"okFq8nm4VOAX07HoPQ2po9cH+sY=\">A\nAWnHiclZhb9s2FIDV7tZ1t3TD8jJgEBYUGLbWSIbu8jKgTZreki5OEydpYzegZEpmQ1GKRCVOBb/u1+x1+y/7NzuUZLM6h3mYgVTs+T\n7xckjqFmRSFHp19d9r197/4MP7rx8c1Pv3s8y+Wbn15UKRlHvJBmMo0PwpYwaVQfKCFlvwoyzlLAskPg9MNw/PeV6IVO3ry4yPEh\nYr",
  "bn15UKRlHvJBmMo0PwpYwaVQfKCFlvwoyzlLAskPg9MNw/PeV6IVO3ry4yPEh\nYrEYmQaQidLPlDmcbHbBiOU+0HI/3NjD6sT4Go5OldXeav3zaWGtLax47a9/cuvr8XCchmXClQ4lK4rjtdVMjyqWaxFKPrs5LAuesfC\nUxfwYiolvBhV9VBm/m2IjP0ozeFPab+OvntGxZKiuEwCMBOmJwVmJuhix6WOfhtVQmWl5ipsGopK6evUN3nxyLnoZa",
  "FPab+OvntGxZKiuEwCMBOmJwVmJuhix6WOfhtVQmWl5ipsGopK6evUN3nxyLnoZaXUGBhLqCvfjh\nOQs1ZO/mUPGLME0SpsbVcH1zd1YNAx4LVfGzs7kbNZ1NmuHQ/EqY/3p/qIWoXki3nJSa2YSq4QeDyrKt6LexgIDkD0OAGp4gXUafITR\nP4aorByJGDgQTqFzkX+ixmpWmkeQ0462iuiQSGTfNqxNogFU5l0lD1QfP+2bwDXOcwCdBUOHM3B",
  "QTqFzkX+ixmpWmkeQ0462iuiQSGTfNqxNogFU5l0lD1QfP+2bwDXOcwCdBUOHM3BXsbUbH6e5lOdJ1VhYriFnKmY103AkE\nMmzYi6hiqlhFPDjvUHtl4wdomLs3qruYmgqz9vOvonOZFjbtOHUEWLMK4a9URZEnY52OWMhyWz6BASe+ibhVobAqyMLs52nQbTszEb\nw2pxnsl63WZH0nzOUEROA3WeOgqmQd/WNdGH78+Sc174p8Kk/gcnqnsLyu",
  "TszEb\nw2pxnsl63WZH0nzOUEROA3WeOgqmQd/WNdGH78+Sc174p8Kk/gcnqnsLyuBnWvBEYVRubUbPOFTJptiCUpxd0/TGofJMdAdoAnjTlbl\nQ0TvanboES9aEh3dgqHkp+fHd3s98OqpWzbYx/5BsQkVFmbkqMuH/UdEY7ix4fUET14q0eRBoJ68VML1HU0dy/HCNpF67qAgFJNCX6Lt\nL2LVPaeO4M6mCeorBEy9cGRCoUmOoq5sAkaGI9wjH",
  "0dy/HCNpF67qAgFJNCX6Lt\nL2LVPaeO4M6mCeorBEy9cGRCoUmOoq5sAkaGI9wjHQsoRIMmzGMi3KnJOLH1rPEKl1c1nMhblZdS+o0gjd6waXi7OgDeHc37F6QHKa\nNDkM0hLNWY5SubUTOn09bDQsMVcu7+e8qbotGJ+tW2B/2C2SnDkJ+dbOH5iIlFHYnqgocSZ12SWI72oK7Fcn23Z9XW6x/I0o4drtuUpN\n62l27b4V7RA3627ejtNvGIR2J",
  "gocSZ12SWI72oK7Fcn23Z9XW6x/I0o4drtuUpN\n62l27b4V7RA3627ejtNvGIR2J6mp7SD1iOdqDutx53HaNwuG6TUnqnefRaTvchYmWf7Q/4ZqZx6RUjs1jXyqHTQiLmoraKaYJj5HYhL\nCYlF0L/o+VPQE3j67VhLDYL0RXMwEsjbnEQ2hCWGy2cNdsY1jdqjbpXJbILMJoTFxyzBo25CWIypGDvFU5ZlSGxCJI8TnMcJzWOGpcw\nl4RnJHDNC",
  "dqjbpXJbILMJoTFxyzBo25CWIypGDvFU5ZlSGxCJI8TnMcJzWOGpcw\nl4RnJHDNClpRrQeWTtCuZAJamqLWpozHogUwVarANYrmgK69wrjyFVrGiq3jganhwRcOaoQpNAEs7ZI/5wx3nJgtwiuExy5XkTCArowns\nY6dPnfnTXxBV5EkuiC4tvaT0wtILSg8tPaQ0t5S8EQTRC0vJ20kQnVt6TumBpQeUlpaWlA4sHVAaWRpR+sjSR5SGloaUbli",
  "tPaQ0t5S8EQTRC0vJ20kQnVt6TumBpQeUlpaWlA4sHVAaWRpR+sjSR5SGloaUbli6Qam2lDyRw\nh3B0n1KJ5ZOKD2y9IjSl5a+pPSJpU8ofWXpK0rfWvqW0geWPqCUWco3bR0k1JuKfl0ETrlq5TGlhK3v1gr1napzSzNKP0oaUPKR1bS\nt6K4X5mKXm8gRujpZLSp5Y+pVRYSt7fgui5pc8pTSxNKH1m6TNK31j6htLHlj6mNLaUfBuApxNL9yi",
  "jpZLSp5Y+pVRYSt7fgui5pc8pTSxNKH1m6TNK31j6htLHlj6mNLaUfBuApxNL9yi1X4GqgtJdS3cpPbP0zP1dgC+mMX\nAtzB1bwQ6lqaUpVuWkjcFeJSw9JQ8T0aqvarNvzaR61qkFtzB2ozPzyY5j9SCO1h7dZqfTa5PkVrwCen65sHiQwqkFK70J0sra/grLC0\nc/NRb+6V3b/feyv319gvtDe8b7zve2/N+9W7z3x+t7AC70/vb+8v71/lr9d",
  "rLC0\nc/NRb+6V3b/feyv319gvtDe8b7zve2/N+9W7z3x+t7AC70/vb+8v71/lr9dfri8tfy8Ua9fa8/5yuv8lg/+A7/2uQ=</latexit>log[a \u00b7 b] = log[a] + log[b]\n66",
  "67",
  "Just a constant \noffset\n68",
  "69",
  "Just dividing by a \npositive constant\n70",
  "Least squares!\n71",
  "Least squares\nNegative log likelihood\n72",
  "Least squares\nMaximum likelihood\n73",
  "Example 1:  univariate regression\n<late\nxit sha1_\nbase64=\"V\nrHufIDtM\nmytgQ3rO/\nQqNtGUel4\n=\">AXBX\niclZjZbtw\n2FEDHXdN\n0c1rELwVa\noUaApHAM2\n0iXlwKJHW\nezU0/iNbE\ncg9JQGsY\nUJUuUPRN1\nnot+TN+Kv\nvY7+hX9hV\n5K8jC6l37\noAPEw9x\nul5TEUZBJ\nUeilpX9m3\nn3vfc/+P\nDKR1c/uT\nTz6fvfb\nFXpGWech3\nw1Sm+UHAC\ni6F4rta",
  "UeilpX9m3\nn3vfc/+P\nDKR1c/uT\nTz6fvfb\nFXpGWech3\nw1Sm+UHAC\ni6F4rtaM\nkPspyzJB\n8PzhZM3z\n/jOeFSNWO\nHmf8KGxE\npEImYbQ8e\nzv/fzm+Fc\n/CdJRFU0\nO/SAaLcCf\nbCiOFvxCx\nAl7tXL+9\nnzo5yF1fK\nk8ovTXFc\nrfiZaXK1M\nJhOfjzJf8\nkgf3m7Mm+\nPbzkZv1T7\nUt5X9XMR\nDfXQ8O7+0\nuFR/PFpYb\ngvzvfbT",
  "OfjzJf8\nkgf3m7Mm+\nPbzkZv1T7\nUt5X9XMR\nDfXQ8O7+0\nuFR/PFpYb\ngvzvfbTP7\n52feAP0rB\nMuNKhZEV\nxuLyU6aOK\n5VqEk+u+\nmXBMxaesJ\ngfQlGxhBd\nHVZ2ziXc\nDIgMvSnP4\np7RXR9+uU\nbGkKMZJAG\nbC9LDAzAR\nd7LDU0U9\nHlVBZqbkK\nm46iUno69\ncwCeAOR81\nDLMRYmAs\nYqxcOGeR\nMwzJd9RU/\nD9MkYWpQ+\navrzy",
  "m46iUno69\ncwCeAOR81\nDLMRYmAs\nYqxcOGeR\nMwzJd9RU/\nD9MkYWpQ+\navrzyDhAY\n+FqvhpWS/\nZNJ1mu\nHQ/EyY/Xx\nzrQVoXki3\nnDSK2YRi\n4ReDypKr4\nYL2IgOAC\nxyAlIFS+g\nTZOfIPKWE\nYUtKgFXzR\n6BveE9n5C\nmleYx5KS\njvSQaFDLJ\nRx1rjViwl\nElH2QbF82\n54BnCdwyr\nAUOGLozX\nYzpiaXNT\nfKTzpCpMD\nPeQMx",
  "Rx1rjViwl\nElH2QbF82\n54BnCdwyr\nAUOGLozX\nYzpiaXNT\nfKTzpCpMD\nPeQMxXzug\nuYcsikmVH\nXUKWUDX\nsWL9g6zlT\nJ23i0qwea\nm4iyNrJu4\n7OaV7UoOv\nUEWTBJoy\n7Vh1BloQb\nyoAlDLcl\no9hwolnIm\n5VKwKsjH\n7eRp0+85\nMBO/NUQbX\nS9dbr0j6z\nxjKiAnA1W\ne+BVMh7+p\nr6dT2LpJ\nzVvumwEfe\nEBarW4Xlc\nTOti",
  "S9dbr0j6z\nxjKiAnA1W\ne+BVMh7+p\nr6dT2LpJ\nzVvumwEfe\nEBarW4Xlc\nTOti05gVm\n1sQs06V8i\nk2YJQnp5\n3TMah8oz\n0Z2gCeCLr\nsyFit7SFu\noSbFkT9hd\ngqnkp+eH\ntxe/56Kha\nMpeN+UOyC\nQ0VZeZqyI\nT/R0MDeIT\nh/QURvHi\npRIsHgXrx\nUgn3d7R0L\nMcb20TqtY\nOCUEwKPUa\nXv4hVt04\ndwYNEzRW\nCJh24ZsJh\nRY",
  "Xrx\nUgn3d7R0L\nMcb20TqtY\nOCUEwKPUa\nXv4hVt04\ndwYNEzRW\nCJh24ZsJh\nRY5irqyCR\ngZvuFh7Nh\nAIZpk2Mw\nxlGlR5pzc\n/NB+hkitm\n9tiLszDqn\ntDlUbo3je\n4nNaCMjw\nczvgl1QOU\n0aDJZ5CWa\nsBylMyRWd\nLRK7/QcIm\n5rv56yZu\ni04r56Ub\nH4wLVqcMQ\n356vIHXIy\nYWdSRqC04\n/zrYksRz\n9QVvT7fr2\nyKqNV9+Rr",
  "r56Ub\nH4wLVqcMQ\n356vIHXIy\nYWdSRqC04\n/zrYksRz\n9QVvT7fr2\nyKqNV9+Rr\nR07XLcpSb\nvtKN2w71\nkBPx0zH\naTeIRizoS\ntdWOkHrEc\nvQHbnzuO\nmahcN1m5K\n0e5FHp+1\nwpyba/tHO\nkGtmjkmpH\nJhjXyr9Jo\nRFTUXtFNO\nEx0hsQlh\nMyq4F/8fK\ntjksd60mh\nMV+IbqaCW\nBpwCWeQhP\nCYnMJd80\n2htVNh7rp\nVpnMhshs",
  "4F/8fK\ntjksd60mh\nMV+IbqaCW\nBpwCWeQhP\nCYnMJd80\n2htVNh7rp\nVpnMhshsQ\nlh8yBI86y\naExZiKsVM\n8YVmGxCZ\nE8jEeRzS\nPGZYylwSX\npHMsSJkS7\nk2VD5Mu5I\nJYGmEehs\n5OoMRyFSh\nDtsglgu68\nwrnzlNoFy\nu6i3dHe9\ne0rFmqE\nTwNIWucY8\nf8t5kQU4x\nXDMciU5E8\njKaAL72Ol\nT5+L0F0Q\nVOckF0djS\nMaXnlp5",
  "NIWucY8\nf8t5kQU4x\nXDMciU5E8\njKaAL72Ol\nT5+L0F0Q\nVOckF0djS\nMaXnlp5Tu\nm/pPqW5pe\nQXQRA9t5T\n8OgmiM0v\nPKN2zdI/S\n0tKS0l1Ld\nymNLI0ofW\nDpA0pDS0N\nK1yxdo1R\nbSk6k8ESw\ndIfSoaVDS\ng8sPaD0ha\nUvKH1k6SN\nKX1r6ktI\n3lr6h9J6l\n9yhljJK1\ny1dp5RbSl\n4dBNGqpau\nUBpaS35\nwrVnapzSz\nNKP0v",
  "I\n3lr6h9J6l\n9yhljJK1\ny1dp5RbSl\n4dBNGqpau\nUBpaS35\nwrVnapzSz\nNKP0vqX3K\nR1YSn4Vw/\nPMUnK8gQe\njpZLSx5Y\n+plRYSn6/\nBdFTS59Sm\nliaUPrE0i\neUvrb0NaU\nPLX1IaWw\npeTcApxNL\ntym1b4Gqg\ntJnlj6j9N\nTSU/d7AT5\ndxsC1Mbd\nsA1uUpam\nlG5YSn4pw\nFHC0hNyno\nxUe1e7eNt\nE7muRmnI\nHazN+UZvk\nPF",
  "Mbd\nsA1uUpam\nlG5YSn4pw\nFHC0hNyno\nxUe1e7eNt\nE7muRmnI\nHazN+UZvk\nPFJT7mDt3\nemiNrk/RW\nrKh2To63v\nTFymQUrj\nTH8/OL+O3\nsLSwt7K4/\nMPinWd35u\n+utm9or/S\n+6n3bu9l\nb7v3Yu9t7\n1Ov3dnth7\n9+Z6zNfz3\nwz9vcH3N\n/zv3VqO/\nMtHW+7HU+\nc3/B3l+B\nUI=</late\nxit>\nPr(y|f[x, \u03c6],",
  "wz9vcH3N\n/zv3VqO/\nMtHW+7HU+\nc3/B3l+B\nUI=</late\nxit>\nPr(y|f[x, \u03c6], \u03c32) =\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(y \u2212 f[x, \u03c6])2\n2\u03c32\n\ufffd\n)\ud835\udc66 = \u0302\ud835\udf07 = f x \ud835\udf19] \nFull distribution:\nMax probability:\n74",
  "Estimating variance\n\u2022 Perhaps surprisingly, the variance term disappeared:\n<latexit sha1_base64=\"aHuoPscj2Z815iPb8W4FNF9T\nVlE=\">AYpXicvZjrbts2FIDd7NZlt3bDgAD7Iyxo1w1tEBfd5U+BNmnapkmXpM2tjRKDkimZDUpFJU4FfQMe5r93\nZ5jb7NDSTYjHgbsGEGUrPn+0RShxdTCjLOcrW4+MeVmXfefe/9D65+OPvRx598+tm165/v5mkhQ7oTpjyV+wHJKWeC\n7imON3PJCVJwOlecLys+d",
  "/9D65+OPvRx598+tm165/v5mkhQ7oTpjyV+wHJKWeC\n7imON3PJCVJwOlecLys+d4plTlLxbY6z+hQmLBIhYSBaHB9Zlb/oio0g+ibMQq7+Z9zycyTpgYTGI+p5E6uOPnRTI\no2f1+dVSuVp7P07ghfiRJWPar0s9PpCrv+hnzcxYn5Ki8W1WVT8fZpIravHU+YHf8JEjHZVQdQCPjAbvdtHX4bX0N1G\nEq8CWLR+qw+zXri1QUSUCl5/uzN+/13",
  "U+YHf8JEjHZVQdQCPjAbvdtHX4bX0N1G\nEq8CWLR+qw+zXri1QUSUCl5/uzN+/131uWvH+XY+7Xfybef0fmrRa/Mum2soH1+YXFxbrj4cL/bYw32s/m4PrXw79Y\nRoWCRUq5CTPD/qLmTosiVQs5LSa9YucZiQ8JjE9gKIgCc0Py3pOV94NiAy9KJXwJ5RXRy9eUZIkz8+TAMyEqFuMx10\nsYNCRT8dlkxkhaIibBqKCu6p1NMLxBsySUPFz6FAQ",
  "y9eUZIkz8+TAMyEqFuMx10\nsYNCRT8dlkxkhaIibBqKCu6p1NMLxBsySUPFz6FAQsmgr14IjAoCpYRTDx6FqZJQsSw9JdWtmAKBTRmoqQnRb2kqr\nrNQOheJlxtLq9rQWpmjC3lJUSa3oSi4RaFyVJV2IF2zAKAC2QBFIBc2hTp2fIPL6FoUthAMum3kBs8F7UaGqhaIx5K\nSjvUYaFDJOx1rGVkwlElHeQmK593wNKBKwihAV+GLWmPwMiOimly",
  "7UaGqhaIx5K\nSjvUYaFDJOx1rGVkwlElHeQmK593wNKBKwihAV+GLWmPwMiOimlyn6FjJpMx1zG5BEhHTugm45ZBwfUdQxScw6Vhx\n/rZtl4QcdwmLs3qrkodsaxt2XWUxHkRw65TRywLJmHcteqIZXHY8IckIZDltgxrXiaejrhVJmyVoYm5KdOg23amI/bc\nHGewXreSonSf0qsjOgArD79zYgIaVdfTqe2N0nOae3rAh17Ixis7iXNZnehEbi",
  "c\nHGewXreSonSf0qsjOgArD79zYgIaVdfTqe2N0nOae3rAh17Ixis7iXNZnehEbirNlZhs86VZeJsQUimZ1T98ah0ox\n1b1AH7EVXSCaiC9rtugRTVof923CrsuD04M7C93R8WC7qZaP/QdmEivIic1Wkw/+goiEcMez5BRF78FJuDR4E6sFLOe\nzv1tARaU9sHanHDgpMEM7UubX8WSy619QRu7NpYvUVArpe+CZMWIMcRV1ZB7QM3BYckyg0Lr",
  "9sHanHDgpMEM7UubX8WSy619QRu7NpYvUVArpe+CZMWIMcRV1ZB7QM3BYckyg0LrJsLnHkKd5ISna/Kz5D\nJFa19uiZPrHqruhci109w3Kp1dBGX4cTuklwdWRoMmn0FaiCGRVjLHekjHR36uYIm5Vn895E3RacX0ZK1tD/oFo1OE\nIT0ZrNnjESMLO9yqC06nzro4shztQV3T6XqxZ+Xa0XdoascO121yVG/bS7ftcC/pAT1Zd/R2HXnIwg636m",
  "6nzro4shztQV3T6XqxZ+Xa0XdoascO121yVG/bS7ftcC/pAT1Zd/R2HXnIwg636mp7iD1kOdq\nDutx5XHfdhcN1mxzVO8mj03a4U9Oa/tH2iCqij0kpH+pjX8r9JmSLCovKaYJjS2xCdliUnQt+L+tvNQn8a7VhGxM2\ndTQdsaUi5fQtNyBabJdw125itrjvUdbdKeDayzCZki09IYt91E7LFGIuxUzwmWaJTQjlcWTncYTzmNlS5pLsEckcI\n4K",
  "UdbdKeDayzCZki09IYt91E7LFGIuxUzwmWaJTQjlcWTncYTzmNlS5pLsEckcI\n4KmlGtCyVHalXTAlsZWa2NHY9ADngqrwTZoyzmeblz5glrFgs8i3dcDe9c0rAiVoU6YEsbaI15/oZzkQV2iuGY5Upy\nxiwrwnctJ1N7ExOf0FUopNcEJ0beo7pmaFnmO4ZuoepNBQ9EQTRC0PR0kQnRp6iumuobuYFoYWmO4YuoNpZGiE6WN\nDH2MaGhpiumzo",
  "uoepNBQ9EQTRC0PR0kQnRp6iumuobuYFoYWmO4YuoNpZGiE6WN\nDH2MaGhpiumzoMqbKUHQihV8EQ7cxHRk6wnTf0H1MXxn6CtOnhj7F9LWhrzF9a+hbTB8a+hBTYijBdMXQFUypoejVQR\nAtGbqEaWAoevaDtWboJqaZoRmjwx9hOnQUPRUDL9nhqLjDfwGsoxXTV0FVNmKHp+C6Lnhj7HNDE0wfSZoc8wfWPoG\n0yfGPoE09hQ9G4ATieGvsTU",
  "soxXTV0FVNmKHp+C6Lnhj7HNDE0wfSZoc8wfWPoG\n0yfGPoE09hQ9G4ATieGvsTUvAUqc0y3DN3C9MTQE/d7ATodxsA1MTdMBRuYpoamK4Zip4U4Ch6DE6T0ai3dUmb5vQ\nvhaJKXewNuOTq1HOIzHlDtbuTpOr0f4UiSkfoa6v7E5fpEBKYacfXJv29hcWH37kL/h4V7W/fmHy1b2iv9r7qfd2\n71ev3fuw96D3tbfZ2euHMLzO/zvw28/vcN3P",
  "37kL/h4V7W/fmHy1b2iv9r7qfd2\n71ev3fuw96D3tbfZ2euHMLzO/zvw28/vcN3P57bndht15kp7zRe9zmdu8CfJ/qNY</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n\ufffd\ufffd#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32\n\ufffd\n\u2212 (yi \u2212 f[xi, \u03c6])2\n2\u03c32\n#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\n\u2212(yi \u2212 f[xi,",
  "\u03c6])2\n2\u03c32\n#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n#\n= argmin\n\u03c6\n\" I\nX\ni=1\n(yi \u2212 f[xi, \u03c6])2\n#\n<latexit sha1_base64=\"aHuoPscj2Z815iPb8W4FNF9T\nVlE=\">AYpXicvZjrbts2FIDd7NZlt3bDgAD7Iyxo1w1tEBfd5U+BNmnapkmXpM2tjRKDkimZDUpFJU4FfQMe5r93\nZ5jb7NDSTYjHgbsGEGUrPn+0RShxdTCjLOcrW4+MeVmXfefe/9D65+O",
  "fQMe5r93\nZ5jb7NDSTYjHgbsGEGUrPn+0RShxdTCjLOcrW4+MeVmXfefe/9D65+OPvRx598+tm165/v5mkhQ7oTpjyV+wHJKWeC\n7imON3PJCVJwOlecLys+d4plTlLxbY6z+hQmLBIhYSBaHB9Zlb/oio0g+ibMQq7+Z9zycyTpgYTGI+p5E6uOPnRTI\no2f1+dVSuVp7P07ghfiRJWPar0s9PpCrv+hnzcxYn5Ki8W1WVT8fZpIravHU+YHf8J",
  "f1+dVSuVp7P07ghfiRJWPar0s9PpCrv+hnzcxYn5Ki8W1WVT8fZpIravHU+YHf8JEjHZVQdQCPjAbvdtHX4bX0N1G\nEq8CWLR+qw+zXri1QUSUCl5/uzN+/131uWvH+XY+7Xfybef0fmrRa/Mum2soH1+YXFxbrj4cL/bYw32s/m4PrXw79Y\nRoWCRUq5CTPD/qLmTosiVQs5LSa9YucZiQ8JjE9gKIgCc0Py3pOV94NiAy9KJXwJ5RXRy9eUZIk",
  "D/qLmTosiVQs5LSa9YucZiQ8JjE9gKIgCc0Py3pOV94NiAy9KJXwJ5RXRy9eUZIkz8+TAMyEqFuMx10\nsYNCRT8dlkxkhaIibBqKCu6p1NMLxBsySUPFz6FAQsmgr14IjAoCpYRTDx6FqZJQsSw9JdWtmAKBTRmoqQnRb2kqr\nrNQOheJlxtLq9rQWpmjC3lJUSa3oSi4RaFyVJV2IF2zAKAC2QBFIBc2hTp2fIPL6FoUthAMum3kBs8F7UaGqha",
  "JUSa3oSi4RaFyVJV2IF2zAKAC2QBFIBc2hTp2fIPL6FoUthAMum3kBs8F7UaGqhaIx5K\nSjvUYaFDJOx1rGVkwlElHeQmK593wNKBKwihAV+GLWmPwMiOimlyn6FjJpMx1zG5BEhHTugm45ZBwfUdQxScw6Vhx\n/rZtl4QcdwmLs3qrkodsaxt2XWUxHkRw65TRywLJmHcteqIZXHY8IckIZDltgxrXiaejrhVJmyVoYm5KdOg23amI/bc\nHGewX",
  "5TRywLJmHcteqIZXHY8IckIZDltgxrXiaejrhVJmyVoYm5KdOg23amI/bc\nHGewXreSonSf0qsjOgArD79zYgIaVdfTqe2N0nOae3rAh17Ixis7iXNZnehEbirNlZhs86VZeJsQUimZ1T98ah0ox\n1b1AH7EVXSCaiC9rtugRTVof923CrsuD04M7C93R8WC7qZaP/QdmEivIic1Wkw/+goiEcMez5BRF78FJuDR4E6sFLOe\nzv1tARaU9sHanHD",
  "7qZaP/QdmEivIic1Wkw/+goiEcMez5BRF78FJuDR4E6sFLOe\nzv1tARaU9sHanHDgpMEM7UubX8WSy619QRu7NpYvUVArpe+CZMWIMcRV1ZB7QM3BYckyg0LrJsLnHkKd5ISna/Kz5D\nJFa19uiZPrHqruhci109w3Kp1dBGX4cTuklwdWRoMmn0FaiCGRVjLHekjHR36uYIm5Vn895E3RacX0ZK1tD/oFo1OE\nIT0ZrNnjESMLO9yqC06nzro4s",
  "LHekjHR36uYIm5Vn895E3RacX0ZK1tD/oFo1OE\nIT0ZrNnjESMLO9yqC06nzro4shztQV3T6XqxZ+Xa0XdoascO121yVG/bS7ftcC/pAT1Zd/R2HXnIwg636mp7iD1kOdq\nDutx5XHfdhcN1mxzVO8mj03a4U9Oa/tH2iCqij0kpH+pjX8r9JmSLCovKaYJjS2xCdliUnQt+L+tvNQn8a7VhGxM2\ndTQdsaUi5fQtNyBabJdw125itrjvUdbdKeD",
  "S2xCdliUnQt+L+tvNQn8a7VhGxM2\ndTQdsaUi5fQtNyBabJdw125itrjvUdbdKeDayzCZki09IYt91E7LFGIuxUzwmWaJTQjlcWTncYTzmNlS5pLsEckcI\n4KmlGtCyVHalXTAlsZWa2NHY9ADngqrwTZoyzmeblz5glrFgs8i3dcDe9c0rAiVoU6YEsbaI15/oZzkQV2iuGY5Upy\nxiwrwnctJ1N7ExOf0FUopNcEJ0beo7pmaFnmO4ZuoepNBQ",
  "5/oZzkQV2iuGY5Upy\nxiwrwnctJ1N7ExOf0FUopNcEJ0beo7pmaFnmO4ZuoepNBQ9EQTRC0PR0kQnRp6iumuobuYFoYWmO4YuoNpZGiE6WN\nDH2MaGhpiumzoMqbKUHQihV8EQ7cxHRk6wnTf0H1MXxn6CtOnhj7F9LWhrzF9a+hbTB8a+hBTYijBdMXQFUypoejVQR\nAtGbqEaWAoevaDtWboJqaZoRmjwx9hOnQUPRUDL9nhqLjDfwGsoxXTV0",
  "poejVQR\nAtGbqEaWAoevaDtWboJqaZoRmjwx9hOnQUPRUDL9nhqLjDfwGsoxXTV0FVNmKHp+C6Lnhj7HNDE0wfSZoc8wfWPoG\n0yfGPoE09hQ9G4ATieGvsTUvAUqc0y3DN3C9MTQE/d7ATodxsA1MTdMBRuYpoamK4Zip4U4Ch6DE6T0ai3dUmb5vQ\nvhaJKXewNuOTq1HOIzHlDtbuTpOr0f4UiSkfoa6v7E5fpEBKYacfXJv29hcWH37kL/h4",
  "KXewNuOTq1HOIzHlDtbuTpOr0f4UiSkfoa6v7E5fpEBKYacfXJv29hcWH37kL/h4V7W/fmHy1b2iv9r7qfd2\n71ev3fuw96D3tbfZ2euHMLzO/zvw28/vcN3P57bndht15kp7zRe9zmdu8CfJ/qNY</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yi \u2212 f[xi,",
  "qNY</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n\ufffd\ufffd#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32\n\ufffd\n\u2212 (yi \u2212 f[xi, \u03c6])2\n2\u03c32\n#\n= argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n#\n= argmin\n\u03c6\n\" I\nX\ni=1\n(yi \u2212 f[xi, \u03c6])2\n#\n75",
  "Estimating variance\n\u2022 But we could learn it during training:\n\u2022 Do gradient descent on both model parameters, \ud835\udf19, and the variance, \n\ud835\udf0e$\n<latexit sha1_base64=\"E\n5+GWtmfcZ5Kje8LwYnATBKiZGE=\">AXS3iclZhbU9w\n2FICXtGnT9EbaKX3oi6dMZtIOMGwmafuSmQRCEgIpEK4\nJjuyV/YqyLKRZVji8U/s9Lm/o2+dPvTI9q6wjnjoziQ\nrzvfp4iNZ1jrIOMvV8vJfMzc+vjmJ5/e+uz25198+dX\nXs3e+OcjTQoZ0P0x5Ko8CklPOB",
  "iNZ1jrIOMvV8vJfMzc+vjmJ5/e+uz25198+dX\nXs3e+OcjTQoZ0P0x5Ko8CklPOBN1XTHF6lElKkoDTw+B\n0VfPDcypzlo9dZnRk4TEgkUsJApCg9k/RFRpR9E2Yh\nVC80fOYsTUr0r71feI8nMk6YGLTOQkM1rHxOI3W86OdF\nMijZoz5UWa8n6dxQ/xIkrDsV9DimVTlfT9jV2pXPh1n\nkyZq897lgC36SZCOy6g6hv7GA+iv7vbkp7oOtHGle",
  "DsV9DimVTlfT9jV2pXPh1n\nkyZq897lgC36SZCOy6g6hv7GA+iv7vbkp7oOtHGle8ni\nkTrpfg1m5eXluPhwv9tjDfaz/bgzvfDf1hGhYJFSrk\nJM+P+8uZOimJVCzktLrtFznNSHhKYnoMRUESmp+UdeYr\n7y5Ehl6USvgnlFdHr9YoSZLnl0kAZkLUKLeZDrYcaGi\n305KJrJCURE2HUF91Tq6Wn0hkzSUPFLKJBQMhirF4IZ\nFHBZN/2Bb0I0",
  "DrYcaGi\n305KJrJCURE2HUF91Tq6Wn0hkzSUPFLKJBQMhirF4IZ\nFHBZN/2Bb0I0yQhYlj6K2s7MAUBjZko6VlRT3xVdZ212\nqFQvM5YWd+btsIUTdgHihqpFd3INQKNq7KkS/GSDRgFw\nJYoAqmgObSp8xNEXt+isNA54LJZM7BSvNcValoGkNO\ntpbpEh43TcsVaRBVOZdJRdUDzvrqcBVRJmAYKX9Sag\n92MiGpST9GxkmZ65jdgyQipnUXcM",
  "3TcsVaRBVOZdJRdUDzvrqcBVRJmAYKX9Sag\n92MiGpST9GxkmZ65jdgyQipnUXcMkh4fqKuoYoOIeqY\ncf63bZeE3HaJi7N6qFKHbGsPdl1lMR5EcOuU0csCxZh3\nLXqiGVx2JaGJCGQ5bYMu4pMPB1xq0zYKkMLc1umQbfvTE\nfstTnO4H7pemslSv85sTKiA3D36W9GREi7+mo6tb1Jcs\n5rXxfo2BvBZHWrNvplU7gqtpYhc06V5aJswUhmV5",
  "iA3D36W9GREi7+mo6tb1Jcs\n5rXxfo2BvBZHWrNvplU7gqtpYhc06V5aJswUhmV50T\n0ah0oz1r1AHbBvukIyEV3RFuoSLFkd9hfgUmXB6fHi0k\nM6PimX9W2j/0PZhIbyInM1pMP/o6EhPAjt9QURe/JSbk\n0eBOrJSzns79bUEWkvbB2p5w4KTBDO1KV1+7NYdOvUEX\nuwaWKNFQK6XfgmTFiTHEVdWQe0DN/wSHcsoNC6yLC5xp\nCneSEp2vy",
  "7NYdOvUEX\nuwaWKNFQK6XfgmTFiTHEVdWQe0DN/wSHcsoNC6yLC5xp\nCneSEp2vys9QyRWtfbomT6YdXdULkWuvsG5dNaUIaHwzm\n9pnpgZTRo8hmkhRgSaSVzrKd0/M7PFdxiru/nvKm6LR\nierbR9gfjgtkpwpCeDTbs+YiRhR1utQVnKGdbHFmO/qC\nt6XK9OrJy493PaGnHDtdtctRuO0q37XCvGQE923SMdhN\n5yMIOt9pqR4g9ZDn6g7bc",
  "XK9OrJy493PaGnHDtdtctRuO0q37XCvGQE923SMdhN\n5yMIOt9pqR4g9ZDn6g7bcedx0XYXDdZsctTvJo9N2uFP\nTWv7R3ogqo9JKR/qY1/K/SZkiwqLyimCY0tsQnZYlJ\n0LfjbVnbrc3HakK2uJ2zrqYDtjSk3L6EJmSLzS3cNdu\nYrW461E23Sng2swmZIvPSWJfdROyxRiLsVM8JVlmiU0I\n5XFk53GE85jZUuaS7BnJHDOClpRrQclR2pV0",
  "IvPSWJfdROyxRiLsVM8JVlmiU0I\n5XFk53GE85jZUuaS7BnJHDOClpRrQclR2pV0wJbGVm9j\nR2cwAp4Kq8M2aMs5Xnm5c+UJaxULvIr3XR3vX9OxIlaD\nOmBLW+ge8/wt50W2CmGY5YryRmzrAwncNt2trEzOf0F\nUYlOckF0aeglpheGXmB6aOghptJQ9IsgiF4bin6dBNG5\noeYHh6gGlhaIHpvqH7mEaGRpg+M/QZpqGhIarhq5i\nqgxFJ1J",
  "sgiF4bin6dBNG5\noeYHh6gGlhaIHpvqH7mEaGRpg+M/QZpqGhIarhq5i\nqgxFJ1J4Ihi6h+nI0BGmR4YeYfrG0DeYvjD0BaZvDX2L\n6QdDP2D6xNAnmBJDCaZrhq5hSg1Frw6CaMXQFUwDQ9FvP\n7jXDN3GNDM0w/SpoU8xHRqKfhXD8xQdLyB6OhHN1Q\n9cxZYai329B9MrQV5gmhiaYvjT0JabvDX2P6XNDn2MaG\n4reDcDpxNBdTM1boDLHdM",
  "xZYai329B9MrQV5gmhiaYvjT0JabvDX2P6XNDn2MaG\n4reDcDpxNBdTM1boDLHdMfQHUzPD1zvxeg02kMXAtzy\nzSwhWlqaIrphqHolwIcJQw9RefJSLS72uRtE9rXIjHlD\ntZmfFIb5TwSU+5g7e40qY32p0hM+QgNfe1g+iIFUgo7/\nWB2vm+/hcWFg/tL/V+WHuw8mH+80r6hvdX7ofdj716v3/\nu197j3orfd2+FMw9njmeGM3Tuj7m/5/6Z",
  "/V+WHuw8mH+80r6hvdX7ofdj716v3/\nu197j3orfd2+FMw9njmeGM3Tuj7m/5/6Z+7dRb8y0db\n7tdT7f3/wPdzciRA=</latexit>\n\u02c6\u03c6, \u02c6\u03c32 = argmin\n\u03c6,\u03c32\n\"\n\u2212\nI\nX\ni=1\nlog\n\uf8ff\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yi \u2212 f[xi, \u03c6])2\n2\u03c32\n\ufffd\ufffd#\n#$\n#%  and \n#$\n#&!\n76",
  "Heteroscedastic regression\n\u2022 We were assuming that the noise \ud835\udf0e$ is the same everywhere \n(homoscedastic).\n\u2022 But we could make the noise a function of the data x.\n\u2022 Build a model with two outputs:\n<latexit sha1_base64=\"kn9X4YyDUe\nnmJKUupmvsIKEO10c=\">AWzXiclZhJU9xGFIBlZ3PIhpMKl1xUoZxKpZwphn\nKWS6psMN7AQwD2AimWpqWpk2rJbTAYGVyzU/Kb8kh1+Rv5LWkmbeaw6ZKjPt\n9329vW5JPfJTKfJiZeWvGzfefe9z+49eHCRx9/8ulni7c/P8iTMgv4I",
  "aw6ZKjPt\n9329vW5JPfJTKfJiZeWvGzfefe9z+49eHCRx9/8ulni7c/P8iTMgv4IEhkh\n35LOdSKD4oRCH5UZpxFvuSH/pn65ofXvAsF4naL65SfhKzSIlQBKyA0HBx14tL\n95tfXC/2k0kVTof9Y8PJ3fhTzoWJ6nElXGPs9cz1vwchHF7HS1U2G1U+F0db\ni4vNJbqT8uLfTbwrLTfnaGt78ceaMkKGOuikCyPD/ur6TFScWyQgSTxe8Mucp\nC",
  "i4vNJbqT8uLfTbwrLTfnaGt78ceaMkKGOuikCyPD/ur6TFScWyQgSTxe8Mucp\nC85YxI+hqFjM85OqnvUvQORkRsmGfxThVtH365RsTjPr2IfzJgV4xwzHbSx47\nIfz6phErLgqug6SgspVskrk6kOxIZDwp5BQUWZALG6gZjlrGgHQveIpfBkc\nMzWqvLWN3Wnl+TwSquLnZ36bTrbNQOh+J1xtrT/XkrouCxeMNJI7WiG7lG4N\nG0qngv6mEg",
  "N3Wnl+TwSquLnZ36bTrbNQOh+J1xtrT/XkrouCxeMNJI7WiG7lG4N\nG0qngv6mEgOADR4wQkiufQps6PH7p9RGrScBVsxtgD7gvpqRpVfAIctLRXhEN\nCqnk461TixYyrij7IHiundcDXiRwSrAUOGLozXYS5mazuoVfFJkcZXrGO4hYy\nridRcw5YBJPaOuoUopoWrQsX7F1gumztrEJWk91ExHkLWfdZ0io3lRo65TR5AF\nmzDqWnUEWRJuDCM",
  "oUopoWrQsX7F1gumztrEJWk91ExHkLWfdZ0io3lRo65TR5AF\nmzDqWnUEWRJuDCMWM8hyWx7ChGNXR+yqUFgVZGPuZInf7TvVEbw3JylcL1voy\nLpv2AoIzoAV5/+FkwFvKuvJ3PbnSXnovZ1gU/cMSxWtwrLomZas05gVm1sSs06\nV8ik2YJQlx2T0ai8pT0Z2gDuCLrsyECt/S7tYl2LI67N2FqWal5Mf937gk5\nNqRV82+g/JjSUl6mtIR3+H",
  "0Z2gDuCLrsyECt/S7tYl2LI67N2FqWal5Mf937gk5\nNqRV82+g/JjSUl6mtIR3+Hw2N4FGE9xdE8OIlEi0eBOrFSyTc39HSsQxvbB2p\n1w4KQjEpit0+YtIdevUETzYJEZjhYBuF76ZUGiRw7Ar64CW4RseqpYNFKBJBs\n0cA5nkZcbJzQ/tZ4jUur4tZkI/rLo3VKmF7n2Dy3ktKMPD4YJfU91HGfWbfPpJ\nqUYsQ8mc6CWdnHp5AZeY7eqvl7w",
  "rLo3VKmF7n2Dy3ktKMPD4YJfU91HGfWbfPpJ\nqUYsQ8mc6CWdnHp5AZeY7eqvl7wpWq2In2+2/cG4YHXKIODnw028HhGxqCNRW3\nCKsbYliWXpD9qab9e3R1Ztn5HtnZkce2mJO2o7TbFveaEfDzLctot4hHLOpI\n1FY7QuoRy9IftGXP45ZtFhbXbkrS7iyPVtvizk20/cP9MS+YPiYlcqSPfYn0mh\nAWCyoWVjGJeYTEJoTFuOxa8H+s7Onzc9",
  "PVtvizk20/cP9MS+YPiYlcqSPfYn0mh\nAWCyoWVjGJeYTEJoTFuOxa8H+s7Onzc9dqQljcyUVX0wEsjbjEU2hCWGwu4a7Z\nxrC6ZVG37CqT6RiZTQiLj1mMZ92EsBhRMbKZyxNkdiESB7HOI9jmscUS6lNwi\nuSWlaEbCnbhsrGSVfSASxNUG8TS2cwApko1GEbxHJOd15u3XkK7WJFd/HA1vHg\nmo4LhrUASxtk2vM9batF5mPUwzHLFuSU4Gsl",
  "GEbxHJOd15u3XkK7WJFd/HA1vHg\nmo4LhrUASxtk2vM9batF5mPUwzHLFuSU4GslCZwBzs71Jmd/vywIic5P7wy9I\nrS0MvKT09JDSzFDyi8APXxhKfp34YWhF5QeGHpAaWloSenA0AGloaEhpY8M\nfURpYGhA6bqh65QWhpITKTwRDN2ndGzomNIjQ48ofWnoS0qfGPqE0leGvqL0ja\nFvKH1g6ANKmaGM0g1DNyjlhpJXB364Zugapb6h5LcfX",
  "noS0qfGPqE0leGvqL0ja\nFvKH1g6ANKmaGM0g1DNyjlhpJXB364Zugapb6h5LcfXGuG7lCaGpS+tDQh5SO\nDCW/iuF5Zig53sCD0VBJ6VNDn1IqDCW/3/zwuaHPKY0NjSl9ZugzSl8b+prSx4\nY+pjQylLwbgNOJoXuUmrdAVU7prqG7lJ4bem5/L8Dny+jbNua2aWCb0sTQhNJN\nQ8kvBThKGHpGzpOhau9qs7dN5L4Wqjm3sDbjs9ok56Gacwt",
  "Nua2aWCb0sTQhNJN\nQ8kvBThKGHpGzpOhau9qs7dN5L4Wqjm3sDbjs9ok56Gacwtr706z2uT+FKo5H5\nOhbxzMX6RASuFOP1xc7uO3sLRwsNr/9i7t3tv+f5a+4b2lvOV87XzrdN3fnLu\nO0+cHWfgBM6fzt/OP86/S9tL5dJvS7836s0bZ0vnM5n6Y/AJ6x7hg=</late\nxit> \u00b5 = f1[x, \u03c6]\n\u03c32 = f2[x, \u03c6]2\n<latexit s\nha1_base64=\"KribwO0\nfDth",
  "7hg=</late\nxit> \u00b5 = f1[x, \u03c6]\n\u03c32 = f2[x, \u03c6]2\n<latexit s\nha1_base64=\"KribwO0\nfDthbO9vnjIKk9cIKug\nA=\">AXQ3iclZhb9s\n2FMedXbvulm5YXvYiLCj\nQDU0QB93lpUCbNG3TpI\nvTxEnayDEomZLZUJQiU\nYldQR9v2GfYZ9jbsNcB\nO5RkM+JhBsxAKvb8/jy\nkzjmkKHkJZ5lcW/tj4b\n3P/jwo49vfXL708+/\n+LxTtfHWVxnvq078c8\nT",
  "b8/jy\nkzjmkKHkJZ5lcW/tj4b\n3P/jwo49vfXL708+/\n+LxTtfHWVxnvq078c8\nTk8klHOBO1LJjk9SVJ\nKIo/TY+98U/HjS5pmLBa\nHcprQURCwQLmEwm4e\nLv7pjIwvWCZMxK56Hjk\njSMmBjOTC6ngTxdcbM8\nGhbsYbc8K7ZLx+VxWBM\n3SIlfdMvCzS5SWay7CX\nMjL54UQTlcPwUnkyG7X\n/sanK2XpZuycCwHzkrd\n8d50yFbmHbpGh+/",
  "5SWay7CX\nMjL54UQTlcPwUnkyG7X\n/sanK2XpZuycCwHzkrd\n8d50yFbmHbpGh+/PCug\nCTv/DY+NwuLi8trpW/Rz\nc6DaN5U7z6w3vfDNyR7\nGfR1RIn5MsO+2uJXJQk\nFQyn9PytptnNCH+OQnp\nKTQFiWg2KqIl85dsIy\ncIE7hT0insl7vUZAoy6\naRB8qIyHFmMmW0sdNcB\nr8MCiaSXFLh1wMFOXdk\n7Kj0OSOWUl/yKTSInzK\nYq+OPCY",
  "8qIyHFmMmW0sdNcB\nr8MCiaSXFLh1wMFOXdk\n7Kj0OSOWUl/yKTSInzK\nYq+OPCYRSQpJvu4Je+XE\nUETEq3I2tfUiLR0MmCn\nqRVwkvy7Zmq9JQaN6k2\nNg+nHthkbsHUVOKoly\ncoOAhmVR0NVw1QSMAmC\nrFIFY0Ax8qvh4gdM1KB\nQ4B1zUdQG14LwqkWsha\nQgxacneIBk0Ek4nLdUm\nUkEqo5bkACSOc9dRgMo\nUsgBThQs1cnCQEFHO+k",
  "Wsha\nQgxacneIBk0Ek4nLdUm\nUkEqo5bkACSOc9dRgMo\nUsgBThQs1cnCQEFHO+k\n6kWlUZMpmjpASEdJqCL\nhln3B1R2FyDmHrn5L9\naupekXEeRO4OKmiqL\noTpM2xqZ4riIUVtTWQw\nVFGHYVlUWQ8VhOxqRiE\nCUmzZsL2nkKItdyoQpZ\nagwe2nstcdOlMWszUkC\n6Wt2ypQ+C+JERFlgNW\nnrowIn7blm/Fc7cyCc1n\npVYNOnDEkq92l3",
  "OlMWszUkC\n6Wt2ypQ+C+JERFlgNW\nnrowIn7blm/Fc7cyCc1n\npVYNOnDEkq92l3levDQ\nJ31dhKrKxiZShxtMCUx\nldtpZqNRUoT1r5BZTAX\nXZ4yEVyT3a9aULK7N6\nHW01zTk9XVn+k0Gxp\naN+gdFExleWJzpMz/w\n9EIHoBmfYHFTF7MjeSB\noUpezGF/N1JHUrOwlaX\nKHTSYIJzJqbH8WSjafSq\nLOdk4MuYKBuUXroQJI8\nlB0BYrgxL",
  "F/N1JHUrOwlaX\nKHTSYIJzJqbH8WSjafSq\nLOdk4MuYKBuUXroQJI8\nlB0BYrgxLDFR7lgLyj\nZv063v0eZzlKUWbn1HP\nYKnkaltMmXpYtTdUrgT\ntfYPyeS9ow8Phkt7Q3T\nMi6tXx9OJcjEhqBHOiU\njo5czMJS8y2+quU102r\nKqQXO814MC/ITu79GK\n4Y+YjRCqs4YvODtZfXG\nksowHvublen1mxc7ZD6\ni0Q4vWruTIbzNLu9qiv\nWEG",
  "4Y+YjRCqs4YvODtZfXG\nksowHvublen1mxc7ZD6\ni0Q4vWruTIbzNLu9qiv\nWEG9GLXMtdpEMqrOG\nr2aGWIdUlvHAlz2Ou7a\n7sGjtSo78zuJoVu0c6\nVR/sHhmEqijkxH6ljX\n8zd2mQKJRZKqzCOaGgI\na5MpjPK2Cv5vSg4YPDz\naqtpkCnsZa8uUwRSNKDd\nvoTaZwnoJt5WNzZTuWq\nS7dinhydhQ1iZT+IxE5\nl3XJlMYmFoFZ6TJDGE",
  "SNKDd\nvoTaZwnoJt5WNzZTuWq\nS7dinhydhQ1iZT+IxE5\nl3XJlMYmFoFZ6TJDGE\ntQnFcWzGcYzjmJixCY\nyM5JYMoJKylZQ6Thui5\nTBFE2M0SaWwWAGPBbGg\nI3RFGe48jJr5QmjigWu\n4r5t4P4NA0tiOFQGU7S\nH1pj7lkXmWeGI5ZtiA\nnzFAlOIA9U9PDmtnpzw\nsKdJLzgqmU0yvNL3C9\nFjTY0xTdEbgRe80hS9\nnXjBpaXmB5peoRp",
  "U9PDmtnpzw\nsKdJLzgqmU0yvNL3C9\nFjTY0xTdEbgRe80hS9\nnXjBpaXmB5peoRprm\nOaV/TPqaBpgGmTzV9iq\nmvqY/pqabmEpN0YkUn\ngiaHmI61nSM6YmJ5i+\n1vQ1ps81fY7pG03fYPp\nO03eYPtb0MaZEU4LplqZ\nbmFJN0acDL9jQdANT1\nP07gdrTdMepomCaZPN\nH2C6UhT9FYMzN0fEG\nHoyacky3Nd3GlGmK3t+\n84KWmLzGNI0wfa",
  "dMepomCaZPN\nH2C6UhT9FYMzN0fEG\nHoyacky3Nd3GlGmK3t+\n84KWmLzGNI0wfaHpC0\nzfavoW02eaPsM01BR9G\n4DTiaYHmOqvQEWG6b6m\n+5heaHph/y5A52n0bIW\n5px3sYRprGmO6oyl6U4C\njhKbn6DwZiGZXm31tQv\ntaIObcwpqIz3qjmAdiz\ni2s2Z1mvdH+FIg5H6Op\nbx3NP6RASGnHy4ud82\nvsLhxtL7a/Wn1wf6D5U\ncbzRfaW",
  "s2Z1mvdH+FIg5H6Op\nbx3NP6RASGnHy4ud82\nvsLhxtL7a/Wn1wf6D5U\ncbzRfaW51vO917nW6n\nZ87jzrPO71Ov+MvrCwc\nLgLg6Xflv5c+mvp71r\n63kLT5+tO67f0z7+XIh6\nA</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"\n\u2212\nI\nX\ni=1\nlog\n\"\n1\np\n2\u21e1f2[xi, \u03c6]2\n#\n\u2212 (yi \u2212 f1[xi, \u03c6])2\n2f2[xi, \u03c6]2\n#\n77",
  "Heteroscedastic regression\n78",
  "Example 1: Univariate Regression Takeaways\n\u2022 Least squares loss is a good choice assuming normal \ndistribution\n\u2022 The best prediction is the predicted mean\n\u2022 We can also estimate global or local variance\n79",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n80",
  "Example 2: binary classification\n\u2022 Goal: predict which of two classes                     the input x belongs to\n<latexit sha1_base64=\"J3iZPT/BVwE\nf4+Q/EnM8wb0PkE=\">AWhXiclZhb9s2FIDV7tKu7Ublpe9CAsKDENn2EPX7W1t0\nvSWdHGaxunASVTMhuKUiQqsSv4Z+x1+137NzuUZLM6h3mYgdTs+T7xckhKtIJMikL3\n+/9eu/7Rx598euPmZ7c+/+Lr76+febgyIt85Dvh6lM86OAFVwKxfe10",
  "tIJMikL3\n+/9eu/7Rx598euPmZ7c+/+Lr76+febgyIt85Dvh6lM86OAFVwKxfe10JIfZTlnS\nD5YXC2bvjhBc8Lkao9Pcv4ScJiJSIRMg2h49lIqFHVvzcYzU9vr/Z7/frj08KgLax67\nWd4eue78WichmXClQ4lK4rjQT/TJxXLtQgln98alQXPWHjGYn4MRcUSXpxUdZ/n/l2I\njP0ozeFPab+OfnhFxZKimCUBmAnTkwIzE3Sx41JHv59UQmW",
  "RcUSXpxUdZ/n/l2I\njP0ozeFPab+OfnhFxZKimCUBmAnTkwIzE3Sx41JHv59UQmWl5ipsGopK6evUNwnwxyL\nnoZYzKLAwF9BXP5ywnIUa0nRrpPhlmCYJU+NqtLaxM69GAY+Fqvh5WadsPu86G7XDoX\niVsfZ8b1mL0DwR7zmpFZMJVcIPJ5XFe/FPQwEByB6nIBU8QLqNPkJIn+AKCwRCRh4k\nE6hc5H/ak6qVprHkJO9oZoUMgkn3asdWLBVC",
  "B6nIBU8QLqNPkJIn+AKCwRCRh4k\nE6hc5H/ak6qVprHkJO9oZoUMgkn3asdWLBVCYdZRcU37/rG8B1DrMAXYUvjuZgN2Nq\nvrhO86nOk6owMdxCzlTM6yZgyCGTZkRdQ5VSwqVhx/oTW6+YOmsTl2Z1V3MTQdZe3n\nV0TvOixl2njiALFmHcteoIsiRs6DFLGS5LZ/CgBPfRNyqUFgVZGEO8zTotp2ZCF6b0\nwz2S9fbqEj6LxjKiAnA7jPfgqmQ",
  "GS5LZ/CgBPfRNyqUFgVZGEO8zTotp2ZCF6b0\nwz2S9fbqEj6LxjKiAnA7jPfgqmQd/X1dGn7i+Rc1L4p8Kk/gcnqXsLyuBnWohEYVRub\nU7POFTJptiCUp5d0/TGofJMdAdoAnjTlblQ0QfavboES9aER/dgqHkp+fHPvV/59KT\nqm21j/iHZhIqKMnNVZML/o6IxPELw+oInrxUosmDQD15qYT7O5o6luOFbSL13EFBKC\naFnqHtL2LVvaO4M6m",
  "/o6IxPELw+oInrxUosmDQD15qYT7O5o6luOFbSL13EFBKC\naFnqHtL2LVvaO4M6mCeorBEy98M2EQpMcRV3ZBIwM3/AwdCygEA0ybMYyrQoc05uf\nmg9Q6TWzW0xF+Zh1b2hSiN07xtcLq+CMjwcLvgVlwco0GTzyAt1ZjlKJlTM6XTt6NC\nwxZz7f56ypui04r5+WbHvQLZqcMQ35+uonIyYWdSqC04fzroksRztQV3L5fphz6\nrNtz+RpR07XL",
  "04r5+WbHvQLZqcMQ35+uonIyYWdSqC04fzroksRztQV3L5fphz6\nrNtz+RpR07XLcpSb1tL92w72iB/x8y9HbLeIRizoS1dX2kHrEcrQHdbnzuOUahcN1m\n5LUu8ij03a4SxMt/2hvwjUzx6RUjs2xL5WjJoRFTUXtFNOEx0hsQlhMyq4F/8fKroCH\nR9dqQlgcFqKrmQCWxlziITQhLDZbuGu2MaxuOdQt8pkNkFmE8LiU5bgUTchLMZUjJ3\niG",
  "lgcFqKrmQCWxlziITQhLDZbuGu2MaxuOdQt8pkNkFmE8LiU5bgUTchLMZUjJ3\niGcsyJDYhkscJzuOE5jHDUuaS8IxkjhkhS8q1oPJ2pVMAEtT1NrU0Rj0QKYKNdgGsV\nzQlVc4V5Cq1jRVbzvanj/ioY1QxWaAJa2yR7zR9vOTRbgFMxy5XkTCArowkcYmdIn\ncXpL4gqcpILopmlM0ovLb2k9NDSQ0pzS8kvgiB6ZSn5dRJEF5ZeUHpg6QGlpa",
  "In\ncXpL4gqcpILopmlM0ovLb2k9NDSQ0pzS8kvgiB6ZSn5dRJEF5ZeUHpg6QGlpaUlpfuW\n7lMaWRpR+sTSJ5SGloaUrlu6Tqm2lJxI4Ylg6R6lE0snlB5ZekTpa0tfU/rM0meUvr\nH0DaXvLX1P6SNLH1HKLGWUbli6QSm3lLw6CKI1S9coDSwlv/1gr1k6pDSzNKP0saWPK\nR1bSn4Vw/PMUnK8gQejpZLS5Y+p1RYSn6/BdFLS19SmliaUPrC",
  "pDSzNKP0saWPK\nR1bSn4Vw/PMUnK8gQejpZLS5Y+p1RYSn6/BdFLS19SmliaUPrC0heUvrP0HaVPLX1K\naWwpeTcApxNLdym1b4GqgtIdS3coPbf03P1egC+nMXAtzG1bwTalqaUpZuWkl8KcJS\nw9IycJyPV3tUWb5vIfS1S+5gbcYXV5OcR2rJHay9Oy2uJvenSC35hHR942D5IgVSCn\nf609urA/wWlhYOfukNHvTu79xfbjWvqG96X3v/eD9",
  "JvenSC35hHR942D5IgVSCn\nf609urA/wWlhYOfukNHvTu79xfbjWvqG96X3v/eD96A2837yH3jNv6O17oZd6f3l/e\n/+s3Fj5eX+yoNGvX6tveZbr/NZ+eM/XlPSdA=</latexit>y 2 {0, 1}\n81",
  "Example 2: binary classification\n\u2022 Domain:\n\u2022 Bernoulli distribution\n\u2022 One parameter \ud835\udf06 \u2208[0,1]\n<latexit sha1_bas\ne64=\"J3iZPT/BVwEf4+Q/EnM8wb0PkE=\"\n>AWhXiclZhb9s2FIDV7tKu7Ublpe9\nCAsKDENn2EPX7W1t0vSWdHGaxunASVTM\nhuKUiQqsSv4Z+x1+137NzuUZLM6h3mYgdT\ns+T7xckhKtIJMikL3+/9eu/7Rx598euPm\nZ7c+/+Lr76+febgyIt85Dvh6lM86OAFV\nwKxfe10JIfZTlnSD",
  "9eu/7Rx598euPm\nZ7c+/+Lr76+febgyIt85Dvh6lM86OAFV\nwKxfe10JIfZTlnSD5YXC2bvjhBc8Lkao\n9Pcv4ScJiJSIRMg2h49lIqFHVvzcYzU9vr\n/Z7/frj08KgLax67Wd4eue78WichmXClQ\n4lK4rjQT/TJxXLtQgln98alQXPWHjGYn4\nMRcUSXpxUdZ/n/l2IjP0ozeFPab+OfnhFx\nZKimCUBmAnTkwIzE3Sx41JHv59UQmWl5i\npsG",
  "UdZ/n/l2IjP0ozeFPab+OfnhFx\nZKimCUBmAnTkwIzE3Sx41JHv59UQmWl5i\npsGopK6evUNwnwxyLnoZYzKLAwF9BXP5yw\nnIUa0nRrpPhlmCYJU+NqtLaxM69GAY+Fq\nvh5WadsPu86G7XDoXiVsfZ8b1mL0DwR7zm\npFZMJVcIPJ5XFe/FPQwEByB6nIBU8QLq\nNPkJIn+AKCwRCRh4kE6hc5H/ak6qVprHk\nJO9oZoUMgkn3asdWLBVCYdZR",
  "U8QLq\nNPkJIn+AKCwRCRh4kE6hc5H/ak6qVprHk\nJO9oZoUMgkn3asdWLBVCYdZRcU37/rG8B\n1DrMAXYUvjuZgN2NqvrhO86nOk6owMdxC\nzlTM6yZgyCGTZkRdQ5VSwqVhx/oTW6+YOm\nsTl2Z1V3MTQdZe3nV0TvOixl2njiALFmH\ncteoIsiRs6DFLGS5LZ/CgBPfRNyqUFgVZ\nGEO8zTotp2ZCF6b0wz2S9fbqEj6LxjKiA\nnA7jPfgqmQd",
  "5LZ/CgBPfRNyqUFgVZ\nGEO8zTotp2ZCF6b0wz2S9fbqEj6LxjKiA\nnA7jPfgqmQd/X1dGn7i+Rc1L4p8Kk/gcn\nqXsLyuBnWohEYVRubU7POFTJptiCUp5d0\n/TGofJMdAdoAnjTlblQ0QfavboES9aER/\ndgqHkp+fHPvV/59KTqm21j/iHZhIqKMnNV\nZML/o6IxPELw+oInrxUosmDQD15qYT7O\n5o6luOFbSL13EFBKCaFnqHtL2LVvaO4M6",
  "L/o6IxPELw+oInrxUosmDQD15qYT7O\n5o6luOFbSL13EFBKCaFnqHtL2LVvaO4M6\nmCeorBEy98M2EQpMcRV3ZBIwM3/AwdCyg\nEA0ybMYyrQoc05ufmg9Q6TWzW0xF+Zh1\nb2hSiN07xtcLq+CMjwcLvgVlwco0GTzyA\nt1ZjlKJlTM6XTt6NCwxZz7f56ypui04r5\n+WbHvQLZqcMQ35+uonIyYWdSqC04fzr\noksRztQV3L5fphz6rNtz+RpR",
  "pui04r5\n+WbHvQLZqcMQ35+uonIyYWdSqC04fzr\noksRztQV3L5fphz6rNtz+RpR07XLcpSb1\ntL92w72iB/x8y9HbLeIRizoS1dX2kHrEc\nrQHdbnzuOUahcN1m5LUu8ij03a4SxMt/2\nhvwjUzx6RUjs2xL5WjJoRFTUXtFNOEx0h\nsQlhMyq4F/8fKroCHR9dqQlgcFqKrmQCWx\nlziITQhLDZbuGu2MaxuOdQt8pkNkFmE8\nLiU5bgUTchLM",
  "HR9dqQlgcFqKrmQCWx\nlziITQhLDZbuGu2MaxuOdQt8pkNkFmE8\nLiU5bgUTchLMZUjJ3iGcsyJDYhkscJzuOE\n5jHDUuaS8IxkjhkhS8q1oPJ2pVMAEtT1\nNrU0Rj0QKYKNdgGsVzQlVc4V5Cq1jRVbz\nvanj/ioY1QxWaAJa2yR7zR9vOTRbgFMx\ny5XkTCArowkcYmdIncXpL4gqcpILopmlM\n0ovLb2k9NDSQ0pzS8kvgiB6ZSn5dRJEF5Z",
  "CArowkcYmdIncXpL4gqcpILopmlM\n0ovLb2k9NDSQ0pzS8kvgiB6ZSn5dRJEF5Z\neUHpg6QGlpaUlpfuW7lMaWRpR+sTSJ5SG\nloaUrlu6Tqm2lJxI4Ylg6R6lE0snlB5Zek\nTpa0tfU/rM0meUvrH0DaXvLX1P6SNLH1H\nKLGWUbli6QSm3lLw6CKI1S9coDSwlv/1gr\n1k6pDSzNKP0saWPKR1bSn4Vw/PMUnK8gQ\nejpZLS5Y+p1RYSn6/BdF",
  "DSwlv/1gr\n1k6pDSzNKP0saWPKR1bSn4Vw/PMUnK8gQ\nejpZLS5Y+p1RYSn6/BdFLS19SmliaUPr\nC0heUvrP0HaVPLX1KaWwpeTcApxNLdym1b\n4GqgtIdS3coPbf03P1egC+nMXAtzG1bwT\nalqaUpZuWkl8KcJSw9IycJyPV3tUWb5vI\nfS1S+5gbcYXV5OcR2rJHay9Oy2uJvenS\nC35hHR942D5IgVSCnf609urA/wWlhYOfuk\nNHvTu79x",
  "5OcR2rJHay9Oy2uJvenS\nC35hHR942D5IgVSCnf609urA/wWlhYOfuk\nNHvTu79xfbjWvqG96X3v/eD96A2837yH\n3jNv6O17oZd6f3l/e/+s3Fj5eX+yoNGv\nX6tveZbr/NZ+eM/XlPSdA=</latexit>y 2 {0, 1}\n<latexit sha1_base64\n=\"kRV8JplHis0yqCBAl2IsH7b+iW4=\">AW1X\niclZhJU9xGFIBlZ3PIhpMKl1xUoZxyUjbFpJzl4",
  "yqCBAl2IsH7b+iW4=\">AW1X\niclZhJU9xGFIBlZ3PIhpMKl1xUoZxyUjbFpJzl4\niobjDdwGAwD2AhTLU1L06bVElpgxsrcUrnmJ+V3\n5AfkmvyFvJY09Z7zSFUOdN539fb625tfipFXq\nyu/nXl6jvf+B9c+XPjo408+/Wzx+uf7eVJmA\nR8EiUyQ5/lXArFB4UoJD9M85iX/ID/3Rd84N\nznuUiUXvFJOXHMYuUCEXACgidL5w+9nNya+ehC\np",
  "FB4UoJD9M85iX/ID/3Rd84N\nznuUiUXvFJOXHMYuUCEXACgidL5w+9nNya+ehC\npD9q171/V8HglVBdBoPnV7t1vkfuN6ZyUbuhP37\nqreS6J3V7HlfDtubCyeLy6spq/efSQq8tLDv\ntX/k+pdDb5gEZcxVEUiW50e91bQ4rlhWiEDy6Y\nJX5jxlwSmL+BEUFYt5flzVOZi6NyAydMkg3+q\ncOvo2zUqFuf5JPbBjFkxyjHTQRs7Kovw5+NKqLQ",
  "UFYt5flzVOZi6NyAydMkg3+q\ncOvo2zUqFuf5JPbBjFkxyjHTQRs7Kovw5+NKqLQ\nsuAqajsJSukXi6oS6Q5HxoJATKLAgEzBWNxixj\nAUFpH3BU/wiSOKYQWK8tY2dadXml0PS9BJMp1n\no3Z0Hi8z1p7szVsRBY/FG04aqRXdyCUCj6ZVxVe\niFQwEByBWOAGJgkWtPJ0fP3R7iMKWk4CB+8kYB\nhe6z6ekaVXwCHLS0V4SDQqp5OtU4sWMq4o",
  "OAGJgkWtPJ0fP3R7iMKWk4CB+8kYB\nhe6z6ekaVXwCHLS0V4SDQqp5OtU4sWMq4o+yC\n4ro3XA14kcEqwFDh6M12E2Zms7qFXxcZHGV6x\njuIWMq4nUXMOWAST2jrqFKaFq0LF+wdZzpk7bx\nCVpPdRMR5C1l3WdIqN5UcOuU0eQBZsw6lp1BFnN\n0Y0ZLktn8CEY1dH7KpQWBVkY/azxO/2neoI3p\nvjFM5L19uoSPrPGcqIDsDp07+CqYB39fV",
  "n8CEY1dH7KpQWBVkY/azxO/2neoI3p\nvjFM5L19uoSPrPGcqIDsDp07+CqYB39fVkbruz5\nJzXvi7wsTuCxepWYVnUTGvWCcyqjU2pWecKmTR\nbEMqSi6pR2NReSq6E9QBfOjKTKjwLe1WXYItq8\nPeLZhqVkp+dHvlBz4+rlb1sdH/IdmEhvIytTWkw\n/+joSHckvD+ghevESixYNAvXiJhOs7WjqW4Y2\ntI/XaQUEoJkUxQcdfRKpbp47gwSYx",
  "oSHckvD+ghevESixYNAvXiJhOs7WjqW4Y2\ntI/XaQUEoJkUxQcdfRKpbp47gwSYxGisEdLvwy4\nRCixyGXVkHtAy/cHO1bKATJo5hjIJC8zTi5+\naD9DpNb1ZTET+mbVvaBKLXSvG1zOa0EZbg7n/JL\nqPsqo3+T0o1ZBlK5lgv6fiVlxdwxGynv17ypm\ni1In62fYH4LVKYOAn51s4vWIiEUdidqCpxlr\nW5JYlv6grfl2fXtk1ear78jWjiyu3",
  "In62fYH4LVKYOAn51s4vWIiEUdidqCpxlr\nW5JYlv6grfl2fXtk1ear78jWjiyu3ZSk3XaUdtv\niXjICfrZlGe0W8YhFHYnakdIPWJZ+oO27Hncs\ns3C4tpNSdqd5dFqW9y5ibZ/uDfiBdOPSYkc6se+\nRHpNCIsFQurmMQ8QmITwmJcdi34f6zsCrh5dK\n0mhMV+LrqaDmBpyCWeQhPCYnOEu2Ybw+qWRd2yq\n0ymI2Q2ISw+YjGedRPCYkTFyC",
  "mhMV+LrqaDmBpyCWeQhPCYnOEu2Ybw+qWRd2yq\n0ymI2Q2ISw+YjGedRPCYkTFyCqesjRFYhMieRzh\nPI5oHlMspTYJr0hqWRGypWwbKhslXUkHsDRGvY\n0tncEIZKJQh20Qyzndebl15ym0ixXdxQNbx4NLO\ni4YalAHsLRNzpjrbVsPmY9TrN9bLUlOBbJSmsA\n+dvrUmT39+WFnuT8cGLohNILQy8oPTD0gNLMUP\nJG4IfPDSVvJ354bug5pf",
  "SmsA\n+dvrUmT39+WFnuT8cGLohNILQy8oPTD0gNLMUP\nJG4IfPDSVvJ354bug5pfuG7lNaGlpSOjB0QGloa\nEjpQ0MfUhoYGlC6bug6pYWh5IkU7giG7lE6MnR\nE6aGh5S+MPQFpY8NfUzpS0NfUvrG0DeU3jf0Pq\nXMUEbphqEblHJDyacDP1wzdI1S31Dy7gdnzdA+\npamhKaUPDH1A6dBQ8lYM9zNDyeMN3BgNlZQ+MfQ\nJpcJQ8v7mh8MfUZ",
  "7gdnzdA+\npamhKaUPDH1A6dBQ8lYM9zNDyeMN3BgNlZQ+MfQ\nJpcJQ8v7mh8MfUZpbGhM6VNDn1L62tDXlD4y9B\nGlkaHk2wA8nRi6S6n5ClTlO4YukPpmaFn9u8C\nfL6Mvm1jbpsGtilNDE0o3TSUvCnAo4Shp+R5MlT\ntVW32tYlc10I15xbWZnxWm+Q8VHNuYe3VaVabX\nJ9CNecjMvSN/fmHFEgpXOlPFpd7+CsLex/v9L7\nceXOzp3le2vt",
  "HNuYe3VaVabX\nJ9CNecjMvSN/fmHFEgpXOlPFpd7+CsLex/v9L7\nceXOzp3le2vtF9przlfO185Np+f85NxzHjt9Z+A\nEzp/O384/zr9LB0vTpd+Wfm/Uq1faOl84nb+lP\n/4D12TvTQ=</latexit>\nPr(y|\u03bb) =\n(\n1 \u2212 \u03bb\ny = 0\n\u03bb\ny = 1\n<latexit sha1_base6\n4=\"/aVTVSs1fk/51RXaxw/ITuMfGpY=\">AWs\nHiclZhbU9tGFICV9JbSG2mnvPRF",
  "4=\"/aVTVSs1fk/51RXaxw/ITuMfGpY=\">AWs\nHiclZhbU9tGFICV9JbSG2mnvPRFUyYzaScwOJN\neXjqTQMgNUkzAQIKBWckrecNqJaQV2FH9R/pr+\ntr+g/6bnpVlb3TO8lDPJF7O9+3t7K4kK8ikKP\nTa2r83bn7w4Ucf3Lr04XPv/iy68Wb39UKRl\nHvJemMo0PwpYwaVQvKeFlvwoyzlLAskPg/MNw\n8veV6IVO3rcZPEhYrEYmQaQidLT7w",
  "emMo0PwpYwaVQvKeFlvwoyzlLAskPg/MNw\n8veV6IVO3rcZPEhYrEYmQaQidLT7wu/nd8R9\nCVUG7Af/N/9uZ2X212nVWRlP+uEg1X4TO63Gk\n4WzxeW1bX649NCpykse82ne3b720F/kIZlwpU\nOJSuK485apk8qlmsRSj5Z6JcFz1h4zmJ+DEXFE\nl6cVPX0Jv4diAz8KM3hn9J+HX2/RsWSohgnAZg\nJ08MCMxN0seNSR7+eVEJlpeYqnHYUl",
  "X0Jv4diAz8KM3hn9J+HX2/RsWSohgnAZg\nJ08MCMxN0seNSR7+eVEJlpeYqnHYUldLXqW9y\n5Q9EzkMtx1BgYS5grH4ZDkLNWR0oa/4VZgmCV\nODqr+uTup+gGPhar4RVlndzJpO5u1w6F4nbH+\nfH/eitA8Ee84aRWTCPXCDyeVBVfjVcxEByAWO\nUEpIoX0KbJTxD5HURhN0nAwIN0BIOL/FcT0rTS\nPIactLQ3RINCJvmoZW0QC5YyaSl7oPj",
  "0KbJTxD5HURhN0nAwIN0BIOL/FcT0rTS\nPIactLQ3RINCJvmoZW0QC5YyaSl7oPj+Hd8Ar\nnNYBRgqfHG0BnsZU5NZPc1HOk+qwsRwDzlTMa+\n7gCmHTJoZtQ1VSglVw5b1O7ZeMXeJC7N6qHmJ\noKs/bzt6JzmRQ3aTh1BFmzCuG3VEWRNj2nCIMt\nN+QwmnPgm4laFwqogG7Obp0G78xE8N4cZXBe\n2t5mRdJ/yVBGTABOn/kWTIW8rW+kc9uf",
  "nPgm4laFwqogG7Obp0G78xE8N4cZXBe\n2t5mRdJ/yVBGTABOn/kWTIW8rW+kc9ufJey9k\n2Bj/whLFa7Csvj6bRmncCsmtiEmnWukEmzBaE8\nvWqbZjQOlWeiPUETwIeuzIWK3tPu1SXYsibcvw\ndTzUvJj1dWf+Kjk2rNHBvzH8kmNFSUmashE/4\nfDQ3gboP3F0Tw4qUSLR4E6sVLJVzf0dKxHG9sE\n6nXDgpCMSn0GB1/Eat2nTqCB5smaKwQ",
  "oP3F0Tw4qUSLR4E6sVLJVzf0dKxHG9sE\n6nXDgpCMSn0GB1/Eat2nTqCB5smaKwQMO3CNxM\nKLXIUtWUTMDJ8w3TsYFCNMlwOsdQpkWZc3LxQ\n/sZIrVuLou5MDer9gVGqF93eByXgvKcHO45N\ndUD1BGg2k+g7RUA5ajZI7Mko5O+4WGI+Y6/fWS\nN7d2lxXzi62mPxgXrE4ZhvzibAuvR0ws6kjUFj\nyoONuSxHL0B23Nt+v7I6u2Tn8kWzt2u",
  "Xzi62mPxgXrE4ZhvzibAuvR0ws6kjUFj\nyoONuSxHL0B23Nt+v7I6u2Tn8kWzt2uG5Tknab\nUbpth3vNCPjFtmO028QjFnUkaqsZIfWI5egP2\nnLncds1C4frNiVpd5ZHp+1w5yba/tH+kGtmHpN\nSOTCPfansT0NY1FTUTjFNeIzEaQiLSdm24G+s7\nAm4ebStaQiL3UK0NRPA0oBLPIVpCIvTI9w2mxh\nWtx3qtltlMhsicxrC4lOW4FlPQ1iM",
  "bStaQiL3UK0NRPA0oBLPIVpCIvTI9w2mxh\nWtx3qtltlMhsicxrC4lOW4FlPQ1iMqRg7xXOW\nZUichkgehziPQ5rHDEuZS8IrkjlWhGwp14bKh2\nlbMgEsjVBvI0dnMAKZKtRhE8RyQXde4dx5Cu1i\nRXdxz9Vx75qONUMNmgCWdsgZ8/s7zkMW4BTDY5\nYryZlAVkYT2MVOlzqzp78gqsiTXBCNLR1TemXp\nFaWHlh5SmltKfhE0StLya+TILq0",
  "yZlAVkYT2MVOlzqzp78gqsiTXBCNLR1TemXp\nFaWHlh5SmltKfhE0StLya+TILq09JLSA0sPK\nC0tLSntWdqjNLI0ovSJpU8oDS0NKd2wdINSbSl\n5IoU7gqX7lA4tHVJ6ZOkRpa8tfU3pM0ufUfrG0\njeUvrP0HaWPLH1EKbOUbp6Sal3FLy6iCI1i1\ndpzSwlPz2g7NmaZfSzNKM0seWPqZ0YCn5VQz3\nM0vJ4w3cGC2VlD639DmlwlLy+y2IX",
  "SwlPz2g7NmaZfSzNKM0seWPqZ0YCn5VQz3\nM0vJ4w3cGC2VlD639DmlwlLy+y2IXlr6ktLE0o\nTSF5a+oPStpW8pfWrpU0pjS8m7AXg6sXSPUvsW\nqCo3bV0l9ILSy/c7wX4fBkD18bcsQ3sUJpaml\nK6ZSn5pQCPEpaek+fJSDVXtdnbJnJdi9ScO1i\nT8VltkvNIzbmDNVenW1yfYrUnA/J0DcP5i9SI\nKVwpT9bXO7gt7C0cHB/tfPz6oPdB8",
  "ltkvNIzbmDNVenW1yfYrUnA/J0DcP5i9SI\nKVwpT9bXO7gt7C0cHB/tfPz6oPdB8sP15s3tLe\n87zvbtex/vFe+g987pezwu9P72/vL+9f5buL\nxit>x0tnS2xqXrzRlPnG6/1WXr7HzfP4g0=</late\nPr(y|\u03bb) = (1 \u2212 \u03bb)1\u2212y \u00b7 \u03bby\n82",
  "Example 2: binary classification\nProblem:  \n\u2022 Output of neural network can be anything\n\u2022 Parameter \ud835\udf06 \u2208[0,1]\nSolution:\n\u2022 Pass through function that maps \n\u201canything\u201d to [0,1]\n83",
  "Example 2: binary classification\nProblem:  \n\u2022 Output of neural network can be anything\n\u2022 Parameter \ud835\udf06 \u2208[0,1]\nSolution:\n\u2022 Pass through logistic sigmoid function that \nmaps \u201canything to [0,1]:\n<latexit sha1_base64\n=\"cEBsoaGM91tCikWnUnGU0f7MPEY=\">AWoni\nclZhbU9w2FICdXlN6I+2Ul754StPptMkO20kvL5\n1JIOQGKUtgQRvGNkrexVk2fgC579B/01fW3/S\nP9Nj2zvKj5HPHRnyCrn+6zLkWRr7adS5MXa2r8\n3n3vfc/+PDmR0sf/LpZ58v3/riIE/KLODIJ",
  "PHRnyCrn+6zLkWRr7adS5MXa2r8\n3n3vfc/+PDmR0sf/LpZ58v3/riIE/KLODIJ\nFJduSznEuh+LAQheRHacZ7Et+6J9uaH54zrNcJ\nGq/uEz5KGaREqEIWAGhk+XvXC/2k2mVi2h2fDVy\nf3e9MGNB1Z9V/R89Pk2P716NZksny6trvbX649J\nCvy2sOu1ncHLrq7E3ToIy5qoIJMvz4/5aWowqlh\nUikHy25JU5T1lwyiJ+DEXFYp6PqnpAM/c",
  "cHLrq7E3ToIy5qoIJMvz4/5aWowqlh\nUikHy25JU5T1lwyiJ+DEXFYp6PqnpAM/c2RMZum\nGTwpwq3jr59RcXiPL+MfTBjVkxyzHTQxo7LIvxt\nVAmVlgVXQdNQWEq3SFydHXcsMh4U8hIKLMgE9N\nUNJgwSUkAOlzFL4IkjpkaV9765u6s8nweCVXxs\n7LO52zWdTZrh0PxOmP96f6iFlHwWFxUkmt6Equ\nEXg0qyrei3oYCA5A9DgBieI51Knz",
  "52zWdTZrh0PxOmP96f6iFlHwWFxUkmt6Equ\nEXg0qyrei3oYCA5A9DgBieI51Knz4duH1FYPxJ\nw1awMD4wXM1K1KngEOelor4gGhVTyacfaIBZMZd\nxR9kBx3duBrzIYBagq/DF0RzspUzN5tcVfFpkc\nZXrG4hYyridRMw5IBJPaKuoUop4dKgY/2BrRdM\nnbaJS9K6q5mOIGs/6zpFRvOixl2njiALFmHUteo\nIsiTs9jGLGWS5LZ/AgGNXR+",
  "nbaJS9K6q5mOIGs/6zpFRvOixl2njiALFmHUteo\nIsiTs9jGLGWS5LZ/AgGNXR+yqUFgVZGEOsTvt\np3qCF6b0xT2S9fbrEj6zxnKiA7A7tPfgqmAd/WN\nZG78+Sc174u8Kk7gcnqXsKyqBnWvBEYVRubUbP\nOFTJptiCUJRdU/fGovJUdAeoA3jTlZlQ4Vvanb\noES1aHvTsw1KyU/Phu72c+HVretvof0g2oaK8T\nG0V6fD/qGgMzxe8viCJy",
  "vanb\noES1aHvTsw1KyU/Phu72c+HVretvof0g2oaK8T\nG0V6fD/qGgMzxe8viCJy+RaPIgUE9eIuH+jqaO\nZXh60g9d1AQiklRXKLtLyLVvaO4M4mMeorBHS\n98M2EQpMchl1ZB7QM3/CktCygA0yaMYyCQvM0\n5ufmg9Q6TW9W0xE/ph1b2hSi107xtcLq6CMjwc\nzvk1l/so36Tz8p1ZhlKJlTPaXT15ewBaz7f5\n6ypui1Yr42VbHvQLZqcMA",
  "CMjwc\nzvk1l/so36Tz8p1ZhlKJlTPaXT15ewBaz7f5\n6ypui1Yr42VbHvQLZqcMAn52soXnIyIWdSqC4\n4m1roksSztQV2L5fp2z6qt1z+QpR1ZXLspSb1tL\n+2xb2mB/xs29LbeIRizoS1dX2kHrEsrQHdnz\nuG0bhcW1m5LUO8+j1ba4CxMt/3B/wgumj0mJHOt\njXyK9JoTFgoqFVUxiHiGxCWExLrsW/B8rewIeHl\n2rCWFxkIupgNYGnOJh",
  "mJHOt\njXyK9JoTFgoqFVUxiHiGxCWExLrsW/B8rewIeHl\n2rCWFxkIupgNYGnOJh9CEsNhs4a7ZxrC6bVG37\nSqT6QSZTQiLj1mMR92EsBhRMbKpyxNkdiESB4\nnOI8TmscUS6lNwjOSWmaELCnbgsomSVfSASxNUW\ntTS2PQA5ko1GAbxHJOV15uXkKrWJFV/HQ1vDwm\noYLhirUASztkD3mejvWTebjFMxy5bkVCArpQkc\nYGdAnfnpzw8rcpL",
  "V/HQ1vDwm\noYLhirUASztkD3mejvWTebjFMxy5bkVCArpQkc\nYGdAnfnpzw8rcpLzw0tDLym9MPSC0kNDynNDCW\n/CPzwhaHk14kfnht6TumBoQeUloaWlA4NHVIaGh\npS+sjQR5QGhgaUbhi6QWlhKDmRwhPB0H1KJ4ZOK\nD0y9IjSl4a+pPSJoU8ofWXoK0qvDL2i9IGhDyhl\nhjJKNw3dpJQbSl4d+OG6oeuU+oaS36w1wdUJ\noamlL60NCHl",
  "0qvDL2i9IGhDyhl\nhjJKNw3dpJQbSl4d+OG6oeuU+oaS36w1wdUJ\noamlL60NCHlI4NJb+K4XlmKDnewIPRUEnpU0OfU\nioMJb/f/PC5oc8pjQ2NKX1m6DNK3xj6htLHhj6m\nNDKUvBuA04mhe5Sat0BVTumuobuUnhl6Zn8vwBf\nT6NsW5o6pYIfSxNCE0i1DyS8FOEoYekrOk6Fq72\nrzt03kvhaqBbewNuPzq0nOQ7XgFtbeneZXk/tTq\nBZ8",
  "1DyS8FOEoYekrOk6Fq72\nrzt03kvhaqBbewNuPzq0nOQ7XgFtbeneZXk/tTq\nBZ8Qrq+ebB4kQIphTv9yfJqH7+FpYWDn3r9X3r3\ndu+t3l9v39DedL52vnG+d/rOr85954kzcIZO4P\ndTdbQ=</latexit>zp/OX87fyz8u3Ks5Xdlb1GfedGe82XTuez4v0HQ\nsig[z] =\n1\n1 + exp[\u2212z]\n84",
  "Example 2: binary classification\n<latexit sha1_base64=\"/aVTVSs1fk/51RXa\nxw/ITuMfGpY=\">AWsHiclZhbU9tGFICV9JbSG2mnvPRFUyYzaScwOJNeXjqTQMgNUkzAQIKBWc\nkrecNqJaQV2FH9R/pr+tr+g/6bnpVlb3TO8lDPJF7O9+3t7K4kK8ikKPTa2r83bn7w4Ucf3Lr0\n4XPv/iy68Wb39UKRlHvJemMo0PwpYwaVQvKeFlvwoyzlLAskPg/MNw8veV6IVO3rc",
  "XPv/iy68Wb39UKRlHvJemMo0PwpYwaVQvKeFlvwoyzlLAskPg/MNw8veV6IVO3rcZPEhYrE\nYmQaQidLT7wu/nd8R9CVUG7Af/N/9uZ2X212nVWRlP+uEg1X4TO63Gk4WzxeW1bX649NCpyks\ne82ne3b720F/kIZlwpUOJSuK485apk8qlmsRSj5Z6JcFz1h4zmJ+DEXFEl6cVPX0Jv4diAz8KM3h\nn9J+HX2/RsWSohgnAZgJ08MCMxN0seNSR",
  "h4zmJ+DEXFEl6cVPX0Jv4diAz8KM3h\nn9J+HX2/RsWSohgnAZgJ08MCMxN0seNSR7+eVEJlpeYqnHYUldLXqW9y5Q9EzkMtx1BgYS5grH4\n4ZDkLNWR0oa/4VZgmCVODqr+uTup+gGPhar4RVlndzJpO5u1w6F4nbH+fH/eitA8Ee84aRWTCP\nXCDyeVBVfjVcxEByAWOUEpIoX0KbJTxD5HURhN0nAwIN0BIOL/FcT0rTSPIactLQ3RINCJvmoZ",
  "VcxEByAWOUEpIoX0KbJTxD5HURhN0nAwIN0BIOL/FcT0rTSPIactLQ3RINCJvmoZW\n0QC5YyaSl7oPj+Hd8ArnNYBRgqfHG0BnsZU5NZPc1HOk+qwsRwDzlTMa+7gCmHTJoZtQ1VSglVw5\nb1O7ZeMXeJC7N6qHmJoKs/bzt6JzmRQ3aTh1BFmzCuG3VEWRNj2nCIMtN+QwmnPgm4laFwqogG\n7Obp0G78xE8N4cZXBe2t5mRdJ/yVBGTABOn/kW",
  "j2nCIMtN+QwmnPgm4laFwqogG\n7Obp0G78xE8N4cZXBe2t5mRdJ/yVBGTABOn/kWTIW8rW+kc9ufJey9k2Bj/whLFa7Csvj6bRmn\ncCsmtiEmnWukEmzBaE8vWqbZjQOlWeiPUETwIeuzIWK3tPu1SXYsibcvwdTzUvJj1dWf+Kjk2rN\nHBvzH8kmNFSUmashE/4fDQ3gboP3F0Tw4qUSLR4E6sVLJVzf0dKxHG9sE6nXDgpCMSn0GB1/Eat2\nnT",
  "hE/4fDQ3gboP3F0Tw4qUSLR4E6sVLJVzf0dKxHG9sE6nXDgpCMSn0GB1/Eat2\nnTqCB5smaKwQMO3CNxMKLXIUtWUTMDJ8w3TsYFCNMlwOsdQpkWZc3LxQ/sZIrVuLou5MDer9gV\nGqF93eByXgvKcHO45NdUD1BGg2k+g7RUA5ajZI7Mko5O+4WGI+Y6/fWSN7d2lxXzi62mPxgXrE4\nZhvzibAuvR0ws6kjUFjyoONuSxHL0B23Nt+v7I6u2Tn",
  "SN7d2lxXzi62mPxgXrE4\nZhvzibAuvR0ws6kjUFjyoONuSxHL0B23Nt+v7I6u2Tn8kWzt2uG5TknabUbpth3vNCPjFtmO028Q\njFnUkaqsZIfWI5egP2nLncds1C4frNiVpd5ZHp+1w5yba/tH+kGtmHpNSOTCPfansT0NY1FTUTj\nFNeIzEaQiLSdm24G+s7Am4ebStaQiL3UK0NRPA0oBLPIVpCIvTI9w2mxhWtx3qtltlMhsicxrC4l\nOW4Fl",
  "7Am4ebStaQiL3UK0NRPA0oBLPIVpCIvTI9w2mxhWtx3qtltlMhsicxrC4l\nOW4FlPQ1iMqRg7xXOWZUichkgehziPQ5rHDEuZS8IrkjlWhGwp14bKh2lbMgEsjVBvI0dnMAKZK\ntRhE8RyQXde4dx5Cu1iRXdxz9Vx75qONUMNmgCWdsgZ8/s7zkMW4BTDY5YryZlAVkYT2MVOlzqzp\n78gqsiTXBCNLR1TemXpFaWHlh5SmltKfhE0StLya+TIL",
  "YryZlAVkYT2MVOlzqzp\n78gqsiTXBCNLR1TemXpFaWHlh5SmltKfhE0StLya+TILq09JLSA0sPKC0tLSntWdqjNLI0ovSJ\npU8oDS0NKd2wdINSbSl5IoU7gqX7lA4tHVJ6ZOkRpa8tfU3pM0ufUfrG0jeUvrP0HaWPLH1EKbOU\nUbp6Sal3FLy6iCI1i1dpzSwlPz2g7NmaZfSzNKM0seWPqZ0YCn5VQz3M0vJ4w3cGC2VlD639Dm\nlwlLy+y2I",
  "zSwlPz2g7NmaZfSzNKM0seWPqZ0YCn5VQz3M0vJ4w3cGC2VlD639Dm\nlwlLy+y2IXlr6ktLE0oTSF5a+oPStpW8pfWrpU0pjS8m7AXg6sXSPUvsWqCo3bV0l9ILSy/c7wX\n4fBkD18bcsQ3sUJpamlK6ZSn5pQCPEpaek+fJSDVXtdnbJnJdi9ScO1iT8VltkvNIzbmDNVenW\npezwu9P72/vL+9f5buLx0tnS2xqXrzRlPnG6/1WXr7HzfP4g0=",
  "kvNIzbmDNVenW\npezwu9P72/vL+9f5buLx0tnS2xqXrzRlPnG6/1WXr7HzfP4g0=</latexit>1yfYrUnA/J0DcP5i9SIKVwpT9bXO7gt7C0cHB/tfPz6oPdB8sP15s3tLe87zvbtex/vFe+g987\nPr(y|\u03bb) = (1 \u2212 \u03bb)1\u2212y \u00b7 \u03bby\n<latexit sha1_base6\n4=\"cTn9peUHYNLwMDH/7yKgjvAolDo=\">AW\n3XiclZhJb9tGFICZdEvTzWlRX3ohagRIikaw",
  "eUHYNLwMDH/7yKgjvAolDo=\">AW\n3XiclZhJb9tGFICZdEvTzWlRX3ohagRIikawi\nnS5FEjsOJudWo4t24klG0NqSE08HNJcbCmsjr\n0VvfYn9Tf0R/TaXvuGpDThe2MUFeBo8r6Ps7x\nZSNFLpMjy1dU/r1x96+13n3v2vXP/jwo48/\nWbrx6X4WF6nP+34s4/TQYxmXQvF+LnLJD5OU\ns8iT/MA7Xdf84JynmYjVXj5N+DBioRKB8FkOo\nZMl",
  "4s4/TQYxmXQvF+LnLJD5OU\ns8iT/MA7Xdf84JynmYjVXj5N+DBioRKB8FkOo\nZMl1ktvTX8eMHktvuje6t7ZxB58aTMRDg7qo\nsBFABrJxmL4fD2cdm9M50N/FGcu/9pH09PlZ\nWO6vVx6WFblNYcZpP7+TG56PBKPaLiKvclyzL\njrqrST4sWZoLX/LZ9UGR8YT5pyzkR1BULOLZs\nKxyMXNvQmTkBnEKfyp3q+ibV5QsyrJp5IEZsX\nycYaDNn",
  "R8YT5pyzkR1BULOLZs\nKxyMXNvQmTkBnEKfyp3q+ibV5QsyrJp5IEZsX\nycYaDNnZU5MEPw1KopMi58uGgkK6ezqxLo\njkXI/l1MoMD8V0FfXH7OU+Tmk/pA8Qs/jiKm\nRuVgbWNnVg48HgpV8rOimorZrO1sVA6H4mXG2\npO9RS0i5F4zUklaIruUTg4awseSfsYCA4AN\nHhBMSKZ1Cnzo8XuF1EYelJwGW9GARuM9npGq\nV8xBy0tJeEg0KieS",
  "eSfsYCA4AN\nHhBMSKZ1Cnzo8XuF1EYelJwGW9GARuM9npGq\nV8xBy0tJeEg0KieSTlrVOLJjKqKXsguK6N10\nNeJ7CLEBX4YujOdhNmJrNr8v5JE+jMtMx3ELK\nVMirJmDIPpN6RG1DFVLCpX7L+glbz5k6bRIXJ\n1VXUx1B1l7advKU5kWN2k4VQRYswrBtVRFkST\ngoRixikOWmfAIDjlwdsatCYVWQhdlLY6/dqI\njeG1OEtgvbW+jJOk/Zyg",
  "VRFkST\ngoRixikOWmfAIDjlwdsatCYVWQhdlLY6/dqI\njeG1OEtgvbW+jJOk/ZygjOgC7T38Lpnze1tfj\nhe3Ok3Ne+brAJ+4YJqt9CUvDeljzRmBUTWxGz\nSpXyKTZglAaX7RN3RuLyhPRHqAO4E1XpEIFb2\nhfVyVYsjo8+BqGmhaSH93pfMsnw3JVbxv9D8k\nmVJQVia0iHf4fFY3g1oTXF0Tw5MUSTR4EqsmL\nJZzvaOpYihe2jlRzBwWhmB",
  "D8k\nmVJQVia0iHf4fFY3g1oTXF0Tw5MUSTR4EqsmL\nJZzvaOpYihe2jlRzBwWhmBT5FG1/Ear2NVUEd\nzaOUF8hoOuFbyYUmuQgaMs6oGX4hpusZQH5aJ\nB+PUZfxlmRcnL4ofUMkUrXx2Iq9M2qfaBKLbT\nPDS4XV0EZbg7n/JLPZRr86nFxdqxFKUzIm\ne0snxIMthi9l2fzXldFqhfxs2kP+gWzU/g+\nPzvZxPMREos6EtUFTzXWuiSxLO1BX",
  "0snxIMthi9l2fzXldFqhfxs2kP+gWzU/g+\nPzvZxPMREos6EtUFTzXWuiSxLO1BXYvl+mbPy\ns3jr8jSDi2u3ZSk3qaXdtviXtIDfrZl6e0W8Y\nhFHYnqanpIPWJZ2oO67Hncso3C4tpNSeqd59F\nqW9yFiZ/sDfmOdOPSbEc6ce+WA7qEBZzKuZW\nMY54iMQ6hMWoaFvwf6zsCrh5tK06hMVeJtqaD\nmBpxCUeQh3CYr2F2YTw+qWRd2yq0wmY2",
  "Q6hMWoaFvwf6zsCrh5tK06hMVeJtqaD\nmBpxCUeQh3CYr2F2YTw+qWRd2yq0wmY2TWIS\nw+YhEedR3CYkjF0CqesiRBYh0ieRzjPI5pHhM\nsJTYJz0himRGypGwLKh3HbUkHsDRBrU0sjUEP\nZKxQg0QyxldeZl15Sm0ihVdxX1bw/1LGs4Zq\nlAHsLRN9pg72LZuMg+nGB6zbElOBLISmsAedn\nrUmT/9eUFJnuS8YGrolNILQy8oPTD0gNLUP",
  "72LZuMg+nGB6zbElOBLISmsAedn\nrUmT/9eUFJnuS8YGrolNILQy8oPTD0gNLUP\nKLwAueG0p+nXjBuaHnlO4buk9pYWhBad/QPqW\nBoQGlDw19SKlvqE/puqHrlOaGkidSuCMYukfp\n2NAxpYeGHlL6wtAXlD429DGlLw19SelrQ19Te\nt/Q+5QyQxmlG4ZuUMoNJa8OvGDN0DVKPUPJbz\n/Ya4b2KE0MTSh9YOgDSkeGkl/FcD8zlDzewI3\nRU",
  "uUMoNJa8OvGDN0DVKPUPJbz\n/Ya4b2KE0MTSh9YOgDSkeGkl/FcD8zlDzewI3\nRUEnpE0OfUCoMJb/fvOCZoc8ojQyNKH1q6FNK\nXxn6itJHhj6iNDSUvBuApxNDdyk1b4HKjNIdQ\n3coPTP0zP5egC+m0bMtzG1TwTalsaExpZuGkl\n8K8Ch6Cl5ngxUc6rN3zaRcy1QC25hTcbnV5O\ncB2rBLaw5neZXk/MpUAs+Jl3f2F+8SIGUwkl/\nsrTSx",
  "N3zaRcy1QC25hTcbnV5O\ncB2rBLaw5neZXk/MpUAs+Jl3f2F+8SIGUwkl/\nsrTSxW9haWH/m073u87dnbsr9aN7TXnC+cL\n51bTtf53rnPHZ6Tt/xnT+cv5y/nX+WT5Z/W\nxit>f51+bdavXqlueYzp/VZ/v1fU/73Zg=</late\nPr(y|x) = (1 \u2212 sig[f[x|\u03c6]])1\u2212y \u00b7 sig[f[x|\u03c6]]y\n85",
  "Example 2: binary classification\n86",
  "Example 2: binary classification\n<latexit sha1_base64\n=\"r8wCtH5baT7zRf3bg6T8z25xnQ=\">AXEni\nclZhb9s2FICd7tZ1t3TD8rIXoUGBbmuCeOguLw\nXapGmbJl2SNrc2dg1KpmQ2FKVIVOJU878Y9mP2\nNux1f2D/ZQ87lGSzOocBNgOtmfN9vB2Skiw/lSL\nXKyt/z1593v/g6ofXPvr4k08/m7/+UGeF\nnA94NEJtmRz3IuheL7WmjJj9KMs9iX/NA/WTP8\nIxnuU",
  "ofXPvr4k08/m7/+UGeF\nnA94NEJtmRz3IuheL7WmjJj9KMs9iX/NA/WTP8\nIxnuUjUnr5IeT9mkRKhCJiG0GD+t63jnh+mI9H3\n7nq9vIgHpbjbnbwqNyZLt7pLF/Dn5OueTKe5K\nE+7i71Yj8Zl7mIJsd1MZyYFsYD8UvTUL+XiWik+\n96SV1W3tf973cH84srySvXxaKHbFBY7zWdncP3L\nYW+YBEXMlQ4ky/Pj7kq+yXLtAgkn1zrFTlPWXD",
  "4srySvXxaKHbFBY7zWdncP3L\nYW+YBEXMlQ4ky/Pj7kq+yXLtAgkn1zrFTlPWXD\nCIn4MRcVinvfLKoET7yZEhl6YZPBPa+Kvl2jZH\nGeX8Q+mDHToxwzE3Sx40KHP/VLodJCcxXUHYWF\n9HTimdXwhiLjgZYXUGBJmCsXjBiGQs0rNm1nuL\nnQRLHTA3L3ur67qTs+TwSquSnRbV+k0nbWa8cDs\nXLjNWNvVkrQvNYvOGkUoxjVwi8GhSlnw5Ws",
  "r67qTs+TwSquSnRbV+k0nbWa8cDs\nXLjNWNvVkrQvNYvOGkUoxjVwi8GhSlnw5WsZAc\nABimROQKJ5DmyY/fuh1EYX9KgGX9Z6AneA9m5Cm\nleYR5KSlvSQaFLJxy1rjViwlHFLeQ6K5930DO\nA6g1WAocIXR2vwPGVqMq2n+VhncZmbGO4hYyriV\nRcw5YBJM6O2oQopoWrQsn7G1jOmTprEJWk1MxE\nkLWXtR2d0byoYdupIsiCTRi1rSqCLAl",
  "JM6O2oQopoWrQsn7G1jOmTprEJWk1MxE\nkLWXtR2d0byoYdupIsiCTRi1rSqCLAlXlyGLGWS\n5KQ9gwrFnIm5VKwKsjF3sRv952aCN6b4xTOS9\ntbL0n6zxjKiAnA6TPfgqmAt/W1ZGZ70+ScVb4p\n8LE3gsVqV2FZVE9r2gnMqolNqFnlCpk0WxDKkvO\n2aUbjUHkq2hM0AXzoikyo8C3tdlWCLWvCvdsw1a\nyQ/Hhp+Xs+7pcr5tiY/0g2oaG8",
  "UbjUHkq2hM0AXzoikyo8C3tdlWCLWvCvdsw1a\nyQ/Hhp+Xs+7pcr5tiY/0g2oaG8SF0NmfD/aGgI9\nzO8vyCFy+RaPEgUC1eIuH6jpaOZXhjm0i1dlAQ\nikmhL9DxF5Fq16kieLBJjMYKAdMufDOh0CKHYV\ns2ASPDN9yZHRsoQJM6jkGMsmLjJOLH9rPEKl0c\n1nMhLlZtS+o0gjt6waXs1pQhpvDGb+kuo8y6tf5\n9JNCDVmGkjk2Szp+1cs1H",
  "0c\n1nMhLlZtS+o0gjt6waXs1pQhpvDGb+kuo8y6tf5\n9JNCDVmGkjk2Szp+1cs1HDHX6a+WvC46rYifbjb\n9wbhgdYog4KeDTbweEbGoI1Fb8CjkbEsSy9EftD\nXbrm+PrNx89Q3Z2pHDdZuStNuM0m073EtGwE+3\nHKPdIh6xqCNRW80IqUcsR3/QljuPW65ZOFy3KUm\n70zw6bYc7M9H2D/dGXDPzmJTIoXnsS2SvDmFRU1\nE7xSTmERLrEBbj",
  "5ZOFy3KUm\n70zw6bYc7M9H2D/dGXDPzmJTIoXnsS2SvDmFRU1\nE7xSTmERLrEBbjom3B31h5LuDm0bqEBZ3ctHWT\nABLQy7xFOoQFusj3DabGFa3HOqW2UyHSGzDmHx\nEYvxrOsQFiMqRk7xhKUpEusQyeMI53FE85hiKX\nVJeEVSx4qQLeXaUNkoaUsmgKUx6m3s6AxGIBOFO\nmyCWM7pzsudO0+hXazoLt53dbx/SceaoQZNAEvb\n5Ix5vW3nI",
  "x6m3s6AxGIBOFO\nmyCWM7pzsudO0+hXazoLt53dbx/SceaoQZNAEvb\n5Ix5vW3nIfNxiuExy5XkVCArpQncwc4OdaZPf35\nYkic5P7yw9ILSc0vPKT209JDSzFLyi8APn1lKfp\n34ZmlZ5QeWHpAaWFpQem+pfuUhpaGlD609CGlg\naUBpWuWrlGqLSVPpHBHsHSP0pGlI0qPLD2i9IW\nlLyh9bOljSl9a+pLSN5a+ofS+pfcpZYyStctXa\neUW0",
  "HsHSP0pGlI0qPLD2i9IW\nlLyh9bOljSl9a+pLSN5a+ofS+pfcpZYyStctXa\neUW0peHfjhqWrlPqWkt9+cNYs3aE0tTSl9IGlD\nygdWkp+FcP9zFLyeAM3RkslpRuWblAqLCW/3/zw\nqaVPKY0tjSl9YukTSl9b+prSR5Y+ojSylLwbgKc\nTS59Tat8ClTmlu5buUnpq6an7vQCfLaPv2pjbt\noFtShNLE0o3LSW/FOBRwtIT8jwZquaqNn3bRK5",
  "u5buUnpq6an7vQCfLaPv2pjbt\noFtShNLE0o3LSW/FOBRwtIT8jwZquaqNn3bRK5r\noZpxB2syPq1Nch6qGXew5uo0rU2uT6Ga8REZ+vr\nB7EUKpBSu9IP5xS5+C0sLB98td39YvrN7Z/Heav\nOG9mrnq86Nzq1Ot/Nj517ncWens98JOv/M3Zj7d\nu72wq8Lvy/8sfBnrV6Za+p80Wl9Fv76FykEC0s\n=</latexit>\nL[\u03c6] =\nI\nX\ni=1\n\u2212(1 \u2212 yi)",
  "Lvy/8sfBnrV6Za+p80Wl9Fv76FykEC0s\n=</latexit>\nL[\u03c6] =\nI\nX\ni=1\n\u2212(1 \u2212 yi) log [1 \u2212 sig[f[xi|\u03c6]]] \u2212 yi log [sig[f[xi|\u03c6]]]\n<latexit sha1_base6\n4=\"cTn9peUHYNLwMDH/7yKgjvAolDo=\">AW\n3XiclZhJb9tGFICZdEvTzWlRX3ohagRIikawi\nnS5FEjsOJudWo4t24klG0NqSE08HNJcbCmsjr\n0VvfYn9Tf0R/TaXvuGpDThe2MUFeBo8r6Ps7x\nZSN",
  "24klG0NqSE08HNJcbCmsjr\n0VvfYn9Tf0R/TaXvuGpDThe2MUFeBo8r6Ps7x\nZSNFLpMjy1dU/r1x96+13n3v2vXP/jwo48/\nWbrx6X4WF6nP+34s4/TQYxmXQvF+LnLJD5OU\ns8iT/MA7Xdf84JynmYjVXj5N+DBioRKB8FkOo\nZMl1ktvTX8eMHktvuje6t7ZxB58aTMRDg7qo\nsBFABrJxmL4fD2cdm9M50N/FGcu/9pH09PlZ\nWO6vVx6WFb",
  "t7ZxB58aTMRDg7qo\nsBFABrJxmL4fD2cdm9M50N/FGcu/9pH09PlZ\nWO6vVx6WFblNYcZpP7+TG56PBKPaLiKvclyzL\njrqrST4sWZoLX/LZ9UGR8YT5pyzkR1BULOLZs\nKxyMXNvQmTkBnEKfyp3q+ibV5QsyrJp5IEZsX\nycYaDNnZU5MEPw1KopMi58uGgkK6ezqxLo\njkXI/l1MoMD8V0FfXH7OU+Tmk/pA8Qs/jiKm\nRuVgbWNnVg48HgpV",
  "gkK6ezqxLo\njkXI/l1MoMD8V0FfXH7OU+Tmk/pA8Qs/jiKm\nRuVgbWNnVg48HgpV8rOimorZrO1sVA6H4mXG2\npO9RS0i5F4zUklaIruUTg4awseSfsYCA4AN\nHhBMSKZ1Cnzo8XuF1EYelJwGW9GARuM9npGq\nV8xBy0tJeEg0KieSTlrVOLJjKqKXsguK6N10\nNeJ7CLEBX4YujOdhNmJrNr8v5JE+jMtMx3ELK\nVMirJmDIPpN6RG1DFVLCpX",
  "N10\nNeJ7CLEBX4YujOdhNmJrNr8v5JE+jMtMx3ELK\nVMirJmDIPpN6RG1DFVLCpX7L+glbz5k6bRIXJ\n1VXUx1B1l7advKU5kWN2k4VQRYswrBtVRFkST\ngoRixikOWmfAIDjlwdsatCYVWQhdlLY6/dqI\njeG1OEtgvbW+jJOk/ZygjOgC7T38Lpnze1tfj\nhe3Ok3Ne+brAJ+4YJqt9CUvDeljzRmBUTWxGz\nSpXyKTZglAaX7RN3RuLyhPRHq",
  "he3Ok3Ne+brAJ+4YJqt9CUvDeljzRmBUTWxGz\nSpXyKTZglAaX7RN3RuLyhPRHqAO4E1XpEIFb2\nhfVyVYsjo8+BqGmhaSH93pfMsnw3JVbxv9D8k\nmVJQVia0iHf4fFY3g1oTXF0Tw5MUSTR4EqsmL\nJZzvaOpYihe2jlRzBwWhmBT5FG1/Ear2NVUEd\nzaOUF8hoOuFbyYUmuQgaMs6oGX4hpusZQH5aJ\nB+PUZfxlmRcnL4ofUMkUrXx2Iq9",
  "aOUF8hoOuFbyYUmuQgaMs6oGX4hpusZQH5aJ\nB+PUZfxlmRcnL4ofUMkUrXx2Iq9M2qfaBKLbT\nPDS4XV0EZbg7n/JLPZRr86nFxdqxFKUzIm\ne0snxIMthi9l2fzXldFqhfxs2kP+gWzU/g+\nPzvZxPMREos6EtUFTzXWuiSxLO1BXYvl+mbPy\ns3jr8jSDi2u3ZSk3qaXdtviXtIDfrZl6e0W8Y\nhFHYnqanpIPWJZ2oO67Hncso3C4tpNSeqd",
  "i2u3ZSk3qaXdtviXtIDfrZl6e0W8Y\nhFHYnqanpIPWJZ2oO67Hncso3C4tpNSeqd59F\nqW9yFiZ/sDfmOdOPSbEc6ce+WA7qEBZzKuZW\nMY54iMQ6hMWoaFvwf6zsCrh5tK06hMVeJtqaD\nmBpxCUeQh3CYr2F2YTw+qWRd2yq0wmY2TWIS\nw+YhEedR3CYkjF0CqesiRBYh0ieRzjPI5pHhM\nsJTYJz0himRGypGwLKh3HbUkHsDRBrU0sjUEP",
  "jF0CqesiRBYh0ieRzjPI5pHhM\nsJTYJz0himRGypGwLKh3HbUkHsDRBrU0sjUEP\nZKxQg0QyxldeZl15Sm0ihVdxX1bw/1LGs4Zq\nlAHsLRN9pg72LZuMg+nGB6zbElOBLISmsAedn\nrUmT/9eUFJnuS8YGrolNILQy8oPTD0gNLUP\nKLwAueG0p+nXjBuaHnlO4buk9pYWhBad/QPqW\nBoQGlDw19SKlvqE/puqHrlOaGkidSuCMYukfp\n2NAxp",
  "nlO4buk9pYWhBad/QPqW\nBoQGlDw19SKlvqE/puqHrlOaGkidSuCMYukfp\n2NAxpYeGHlL6wtAXlD429DGlLw19SelrQ19Te\nt/Q+5QyQxmlG4ZuUMoNJa8OvGDN0DVKPUPJbz\n/Ya4b2KE0MTSh9YOgDSkeGkl/FcD8zlDzewI3\nRUEnpE0OfUCoMJb/fvOCZoc8ojQyNKH1q6FNK\nXxn6itJHhj6iNDSUvBuApxNDdyk1b4HKjNIdQ\n3coPTP0",
  "CZoc8ojQyNKH1q6FNK\nXxn6itJHhj6iNDSUvBuApxNDdyk1b4HKjNIdQ\n3coPTP0zP5egC+m0bMtzG1TwTalsaExpZuGkl\n8K8Ch6Cl5ngxUc6rN3zaRcy1QC25hTcbnV5O\ncB2rBLaw5neZXk/MpUAs+Jl3f2F+8SIGUwkl/\nsrTSxW9haWH/m073u87dnbsr9aN7TXnC+cL\n51bTtf53rnPHZ6Tt/xnT+cv5y/nX+WT5Z/W\nxit>f51+bdavXq",
  "sr9aN7TXnC+cL\n51bTtf53rnPHZ6Tt/xnT+cv5y/nX+WT5Z/W\nxit>f51+bdavXqlueYzp/VZ/v1fU/73Zg=</late\nPr(y|x) = (1 \u2212 sig[f[x|\u03c6]])1\u2212y \u00b7 sig[f[x|\u03c6]]y\n*Binary cross-entropy loss*\n87",
  "Example 2: binary classification\nChoose y=1 where \ud835\udf06 is greater than 0.5, otherwise 0\nAnd we get a probability estimate!\n88",
  "Example 2: Binary Classification Takeaways\n\u2022 Binary cross entropy loss as the loss function\n\u2022 Threshold to get prediction\n\u2022 We also get a probability or \u201cconfidence value\u201d\n89",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n90",
  "Example 3: multiclass classification\nGoal: predict which of K classes                               the input x belongs to\n<latexit sha1_base64=\"GSGPcImi1ciGhH\nW2fLWnYack=\">AWkXiclZhb9s2FIDV7tZ1l6Yrmpe9CAsKDENmxEV3e2uTpreki9PESZ\no4DSiZktlQlCJRiV3BP2av2y/av9mhLJvVOczDKRmz/eJl0NSohVkUhR6be3fGzc/+fSz7+4\n9eXtr7+5ts7S3e/OyjSMg95P0xlmh8F",
  "mz/eJl0NSohVkUhR6be3fGzc/+fSz7+4\n9eXtr7+5ts7S3e/OyjSMg95P0xlmh8FrOBSKN7XQkt+lOWcJYHkh8H5huGHlzwvRKr29STjp\nwmLlYhEyDSEzpbuTwZCDaru6sPVgRymulj1twbTs6WVtc5a/fFpodsUVrzm0zu7e384GKZhmX\nClQ8mK4qS7lunTiuVahJPbw/KgmcsPGcxP4GiYgkvTqu6/1P/AUSGfpTm8Ke0X0c/vqJiSVF\nMkg",
  "nTiuVahJPbw/KgmcsPGcxP4GiYgkvTqu6/1P/AUSGfpTm8Ke0X0c/vqJiSVF\nMkgDMhOlRgZkJuthJqaPfTyuhslJzFc4aikrp69Q3yfCHIuehlhMosDAX0Fc/HLGchRpSdnug\n+FWYJglTw2qwvrk7rQYBj4Wq+EVZp286bTubtcOheJ2x/nJ/UYvQPBEfOKmkVkwl1wg8nlYV78\nQdDAQHIDqcgFTxAuo0+Qkiv4soLBcJGHiQjqFzkf9mSqpW",
  "KmkVkwl1wg8nlYV78\nQdDAQHIDqcgFTxAuo0+Qkiv4soLBcJGHiQjqFzkf9mSqpWmseQk5Z2TDQoZJKPW9YGsWAqk5a\nyB4rvP/AN4DqHWYCuwhdHc7CXMTWdX6f5WOdJVZgYbiFnKuZ1EzDkEkzorahSinh0rBl/Ymt\nN0ydN4lLs7qruYkgaz9vOzqneVHDtlNHkAWLMG5bdQRZEjb3kCUMstyUz2DAiW8iblUorAqyM\nHt5GrTbzkwEr81xBv",
  "DtlNHkAWLMG5bdQRZEjb3kCUMstyUz2DAiW8iblUorAqyM\nHt5GrTbzkwEr81xBvul7W1WJP2XDGXEBGD3mW/BVMjb+ka6sP15ci5r3xT42B/BZLUvYXk8G9a\n8ERhVE5tSs84VMm2IJSnV23T9Mah8ky0B2gCeNOVuVDR9pqXYIla8KDVRhqXkp+8nPnFz4+\nrdbMtjH/kGxCRUWZuSoy4f9R0RAeJ3h9QRPXirR5EGgnrxUwv0dTR3L8cI2kXr",
  "+\nrdbMtjH/kGxCRUWZuSoy4f9R0RAeJ3h9QRPXirR5EGgnrxUwv0dTR3L8cI2kXruoCAUk0JP0\nPYXsWpfU0dwZ9ME9RUCpl74ZkKhSY6itmwCRoZveDA6FlCIBhnOxhjKtChzTm5+aD1DpNbNbT\nEX5mHVvqFKI7TvG1wuroIyPBwu+TWXByijwSyfQVqIctRMsdmSsfvBoWGLeba/fWUz4pOK+Y\nXW0170C+YnTIM+cXZFp6PmFjUkaguOIk46",
  "tRMsdmSsfvBoWGLeba/fWUz4pOK+Y\nXW0170C+YnTIM+cXZFp6PmFjUkaguOIk465LEcrQHdS2W68c9q7be/USWduxw3aYk9Ta9dNsO9\n5oe8ItR2+3iUcs6khUV9ND6hHL0R7U5c7jtmsUDtdtSlLvPI9O2+EuTLT8o/0R18wck1I5NM\ne+VA5mISxqKmqnmCY8RuIshMWkbFvwf6zsCXh4tK1ZCIu9QrQ1E8DSkEs8hFkIi7Mt3DabGFa\n3Heq",
  "mCY8RuIshMWkbFvwf6zsCXh4tK1ZCIu9QrQ1E8DSkEs8hFkIi7Mt3DabGFa\n3Heq2W2UyGyFzFsLic5bgUc9CWIypGDvFc5ZlSJyFSB5HOI8jmscMS5lLwjOSOWaELCnXgspH\naVsyASyNUWtjR2PQA5kq1GATxHJBV17hXHkKrWJFV3Hf1XD/moY1QxWaAJZ2yB7zBzvOTRbgFM\nMxy5XkTCArownsYadHnfnpL4gqcpILomlE0qvL2i9NDSQ0p",
  "Z2yB7zBzvOTRbgFM\nMxy5XkTCArownsYadHnfnpL4gqcpILomlE0qvL2i9NDSQ0pzS8kvgiB6Yyn5dRJEl5ZeUnp\ng6QGlpaUlpX1L+5RGlkaUPrP0GaWhpSGlG5ZuUKotJSdSeCJYuk/pyNIRpUeWHlH61tK3lL6w\n9AWlx5YeU/rB0g+UPrH0CaXMUkbpqWblHJLyauDIFq3dJ3SwFLy2w/2mqU9SjNLM0qfWvqU0\nqGl5FcxPM8sJcbeDBaKi",
  "WblHJLyauDIFq3dJ3SwFLy2w/2mqU9SjNLM0qfWvqU0\nqGl5FcxPM8sJcbeDBaKil9aelLSoWl5PdbEL29DWliaUJpa8sfUXpe0vfU/rc0ueUxpaSdw\nNwOrF0j1L7FqgqKN21dJfSC0sv3O8F+GIaA9fC3LEV7FCaWpSumUp+aUARwlLz8l5MlLNXW3+\ntonc1yK14A7WZHx+Ncl5pBbcwZq70/xqcn+K1IKPSNc3DxYvUiClcKc/W1rp4rewtH",
  "nc1yK14A7WZHx+Ncl5pBbcwZq70/xqcn+K1IKPSNc3DxYvUiClcKc/W1rp4rewtHDwsNP9t\nfNo9HK4/XmDe0t73vB+9Hr+v95j32Xng9r+FXuX95f3t/bN8b/mP5cfLjXvzRnPNPa/1Wd\n76DzMS1lA=</latexit>y 2 {1, 2, . . . , K}\n91",
  "Example 3: multiclass classification \n<latexit sha1_base6\n4=\"GSGPcImi1ciGhHW2fLWnYack=\">A\nWkXiclZhb9s2FIDV7tZ1l6Yrmpe9CAsKDENm\nxEV3e2uTpreki9PESZo4DSiZktlQlCJRiV3BP\n2av2y/av9mhLJvVOczDKRmz/eJl0NSohVkU\nhR6be3fGzc/+fSz7+49eXtr7+5ts7S3e/Oy\njSMg95P0xlmh8FrOBSKN7XQkt+lOWcJYHkh8\nH5huGHlzwvRKr29ST",
  "ts7S3e/Oy\njSMg95P0xlmh8FrOBSKN7XQkt+lOWcJYHkh8\nH5huGHlzwvRKr29STjpwmLlYhEyDSEzpbuTwZ\nCDaru6sPVgRymulj1twbTs6WVtc5a/fFpodsU\nVrzm0zu7e384GKZhmXClQ8mK4qS7lunTiuVa\nhJPbw/KgmcsPGcxP4GiYgkvTqu6/1P/AUSGf\npTm8Ke0X0c/vqJiSVFMkgDMhOlRgZkJuthJqa\nPfTyuhslJzFc4aikrp69Q",
  "USGf\npTm8Ke0X0c/vqJiSVFMkgDMhOlRgZkJuthJqa\nPfTyuhslJzFc4aikrp69Q3yfCHIuehlhMosD\nAX0Fc/HLGchRpSdnug+FWYJglTw2qwvrk7rQY\nBj4Wq+EVZp286bTubtcOheJ2x/nJ/UYvQPBE\nfOKmkVkwl1wg8nlYV78QdDAQHIDqcgFTxAuo0\n+Qkiv4soLBcJGHiQjqFzkf9mSqpWmseQk5Z2T\nDQoZJKPW9YGsWAqk5ayB4rvP/",
  "+Qkiv4soLBcJGHiQjqFzkf9mSqpWmseQk5Z2T\nDQoZJKPW9YGsWAqk5ayB4rvP/AN4DqHWYCuw\nhdHc7CXMTWdX6f5WOdJVZgYbiFnKuZ1EzDkE\nkzorahSinh0rBl/YmtN0ydN4lLs7qruYkgaz\n9vOzqneVHDtlNHkAWLMG5bdQRZEjb3kCUMsty\nUz2DAiW8iblUorAqyMHt5GrTbzkwEr81xBvul\n7W1WJP2XDGXEBGD3mW/BVMjb+ka6sP",
  "AiW8iblUorAqyMHt5GrTbzkwEr81xBvul\n7W1WJP2XDGXEBGD3mW/BVMjb+ka6sP15ci5r\n3xT42B/BZLUvYXk8G9a8ERhVE5tSs84VMm2I\nJSnV23T9Mah8ky0B2gCeNOVuVDR9pqXYIla\n8KDVRhqXkp+8nPnFz4+rdbMtjH/kGxCRUWZuS\noy4f9R0RAeJ3h9QRPXirR5EGgnrxUwv0dTR3\nL8cI2kXruoCAUk0JP0PYXsWpfU0dwZ9ME9RU",
  "J3h9QRPXirR5EGgnrxUwv0dTR3\nL8cI2kXruoCAUk0JP0PYXsWpfU0dwZ9ME9RU\nCpl74ZkKhSY6itmwCRoZveDA6FlCIBhnOxhjK\ntChzTm5+aD1DpNbNbTEX5mHVvqFKI7TvG1wur\noIyPBwu+TWXByijwSyfQVqIctRMsdmSsfvB\noWGLeba/fWUz4pOK+YXW0170C+YnTIM+cXZFp\n6PmFjUkaguOIk465LEcrQHdS2W68c9q7be/U\nSWdux",
  "+YXW0170C+YnTIM+cXZFp\n6PmFjUkaguOIk465LEcrQHdS2W68c9q7be/U\nSWduxw3aYk9Ta9dNsO95oe8ItR2+3iUcs6kh\nUV9ND6hHL0R7U5c7jtmsUDtdtSlLvPI9O2+Eu\nTLT8o/0R18wck1I5NMe+VA5mISxqKmqnmCY8\nRuIshMWkbFvwf6zsCXh4tK1ZCIu9QrQ1E8DSk\nEs8hFkIi7Mt3DabGFa3Heq2W2UyGyFzFsLic\n5bgUc9CWIy",
  "K1ZCIu9QrQ1E8DSk\nEs8hFkIi7Mt3DabGFa3Heq2W2UyGyFzFsLic\n5bgUc9CWIypGDvFc5ZlSJyFSB5HOI8jmscMS5\nlLwjOSOWaELCnXgspHaVsyASyNUWtjR2PQA5k\nq1GATxHJBV17hXHkKrWJFV3Hf1XD/moY1QxW\naAJZ2yB7zBzvOTRbgFMxy5XkTCArownsYadH\nnfnpL4gqcpILomlE0qvL2i9NDSQ0pzS8kv\ngiB6Yyn5dRJEl5ZeU",
  "CArownsYadH\nnfnpL4gqcpILomlE0qvL2i9NDSQ0pzS8kv\ngiB6Yyn5dRJEl5ZeUnpg6QGlpaUlpX1L+5RGl\nkaUPrP0GaWhpSGlG5ZuUKotJSdSeCJYuk/pyN\nIRpUeWHlH61tK3lL6w9AWlx5YeU/rB0g+UPr\nH0CaXMUkbpqWblHJLyauDIFq3dJ3SwFLy2w/\n2mqU9SjNLM0qfWvqU0qGl5FcxPM8sJcbeDBa\nKil9aelLSoWl5PdbEL29DW",
  "y2w/\n2mqU9SjNLM0qfWvqU0qGl5FcxPM8sJcbeDBa\nKil9aelLSoWl5PdbEL29DWliaUJpa8sfUXp\ne0vfU/rc0ueUxpaSdwNwOrF0j1L7FqgqKN21d\nJfSC0sv3O8F+GIaA9fC3LEV7FCaWpSumUp+\naUARwlLz8l5MlLNXW3+tonc1yK14A7WZHx+Nc\nl5pBbcwZq70/xqcn+K1IKPSNc3DxYvUiClcKc\n/W1rp4rewtHDwsNP9tfNo9HK4/Xm",
  "pBbcwZq70/xqcn+K1IKPSNc3DxYvUiClcKc\n/W1rp4rewtHDwsNP9tfNo9HK4/XmDe0t73v\nvB+9Hr+v95j32Xng9r+FXuX95f3t/bN8b/mP\n5cfLjXvzRnPNPa/1Wd76DzMS1lA=</latexi\n\u2022 Domain: t>y 2 {1, 2, . . .  K}\n\u2022 Categorical distribution\n\u2022 K parameters \ud835\udf06) \u2208[0,1]\n\u2022 Sum of all parameters = 1\n<latexit sha1_\nbase64=\"wkZVy6msPhDWd+7pvcT\n9pUqMdo=\">AWkXiclZhb9s2F",
  "sha1_\nbase64=\"wkZVy6msPhDWd+7pvcT\n9pUqMdo=\">AWkXiclZhb9s2F\nIDV7tZ1t3RF87IXYUGBbuiMZOhuD\nwXapOkt6eI0cZImTgNKpmTWFKVQ\nUmJX8I/Z6/aL9m92KMtmdQ7zMAOp\n2fN9vB2SkqwgkyIvVlf/vXb9o48\n/+fSzG5/f/OLr7+ZunWtwd5Wuq\nQ98JUpvoYDmXQvFeIQrJjzLNWR\nJIfhiMNgw/vOA6F6naLyYZP01YrE",
  "Wtwd5Wuq\nQ98JUpvoYDmXQvFeIQrJjzLNWR\nJIfhiMNgw/vOA6F6naLyYZP01YrE\nQkQlZA6GzpTlfmzwc/eA/9PsSq\ng3YWTWani2trHZW649PC2tNYcVrP\nt2zW3cG/UEalglXRShZnp+srWbF\nacV0IULJpzf7Zc4zFo5YzE+gqFjC\n89OqHv/UvwuRgR+lGv5U4dfRD2t\nULMnzSRKAmbBimGNmgi52UhbR76e\nVUFlZcBXOopK6Repb5LhD",
  "+lGv5U4dfRD2t\nULMnzSRKAmbBimGNmgi52UhbR76e\nVUFlZcBXOopK6Repb5LhD4TmYS\nEnUGChFjBWPxwyzcICUnazr/hlmC\nYJU4Oqv765O636AY+Fqvh5WadvO\nm07m7XDoXiVsf5if9GKHgi3nPS\nK2YRq4QeDytKt6JOxgIDkB0OAGp\n4jm0afITRP4aorBdJGDgQTqGwUX\n+6ylpWhU8hpy0tGOiQSGTfNyNog\nFS5m0lD1QfP+ubwAvN",
  "4aorBdJGDgQTqGwUX\n+6ylpWhU8hpy0tGOiQSGTfNyNog\nFS5m0lD1QfP+ubwAvNKwCDBW+OF\nqDvYyp6bxewceFTqrcxHAPmqmY1\n3AlEMmzYzahiqlhKphy/oTW6+ZG\njWJS7N6qNpEkLWv206haV7UoO3UE\nWTBJozbVh1B1uyUJgyvDixfZ34\nJuJWhcKqIBuzq9Og3XdmInhvjM4\nL21vsyLpv2AoIyYAp898C6ZC3tY\n30oXtz5NzUfumw",
  "KqIBuzq9Og3XdmInhvjM4\nL21vsyLpv2AoIyYAp898C6ZC3tY\n30oXtz5NzUfumwMf+EBarXYXpeDa\nteScwqyY2pWadK2TSbEFIp5dt04\nzGofJMtCdoAvjQlVqo6APtfl2CLW\nvC/fswV1KfvJT5xc+Pq1WzbEx/\n5BsQkN5mbkaMuH/0dAbid4f0EL\n14q0eJBoF68VML1HS0d03hjm0i9\ndlAQiklRTNDxF7Fq16kjeLBpgsYK\nAdMufDOh0",
  "q0eJBoF68VML1HS0d03hjm0i9\ndlAQiklRTNDxF7Fq16kjeLBpgsYK\nAdMufDOh0CJHUVs2ASPDN9wYHRs\noRJMZ3MZqXmpOLH9rPEKl1c1n\nUwtys2hdUaYT2dYPLRS0ow83hgl\n9RPUAZDWb5DNJSDZhGyRybJR2/7e\ncFHDHX6a+XfFZ0WjE/32r6g3HB6\npRhyM/PtvB6xMSijkRtwZOIsy1JL\nEd/0NZiu34smr7Y9ka8cO121K\n0m4zSrf",
  "6\npRhyM/PtvB6xMSijkRtwZOIsy1JL\nEd/0NZiu34smr7Y9ka8cO121K\n0m4zSrftcK8YAT/fdox2m3jEo5E\nbTUjpB6xHP1BW+48brtm4XDdpiT\ntzvPotB3uwkTbP9of8oKZx6RUDsx\njXyr7sxAWCyoWTjFNeIzEWQiLSd\nm24P9Y2RNw82hbsxAWu7loayaApQ\nGXeAqzEBZnR7htNjGsbjvUbfKZ\nDZE5iyExWcswbOehbAYUzF2iOWZ\nUi",
  "ayaApQ\nGXeAqzEBZnR7htNjGsbjvUbfKZ\nDZE5iyExWcswbOehbAYUzF2iOWZ\nUichUgehziPQ5rHDEuZS8IrkjlW\nhGwp14bSw7QtmQCWxqi3saMzGIFM\nFeqwCWI5pzsvd+48hXaxoru45+q\n4d0XHBUMNmgCWdsgZ8/s7zkMW4BT\nDY5YryZlAVkYT2MVOlzrzp78gqs\niTXBNLJ1QemnpJaWHlh5Sqi0lvw\niC6LWl5NdJEF1YekHpgaUHlJ",
  "Olzrzp78gqs\niTXBNLJ1QemnpJaWHlh5Sqi0lvw\niC6LWl5NdJEF1YekHpgaUHlJaWl\npT2LO1RGlkaUfrU0qeUhpaGlG5Yu\nkFpYSl5IoU7gqX7lA4tHVJ6ZOkR\npW8sfUPpc0ufU3ps6TGl7y19T+l\njSx9TyixlG5aukpt5S8OgidUv\nXKQ0sJb/94KxZ2qU0szSj9ImlTy\ngdWEp+FcP9zFLyeAM3RkslpS8sfU\nGpsJT8fguiV5a+ojSxNK",
  "2qU0szSj9ImlTy\ngdWEp+FcP9zFLyeAM3RkslpS8sfU\nGpsJT8fguiV5a+ojSxNKH0paUvK\nX1n6TtKn1n6jNLYUvJuAJ5OLN2j1\nL4FqnJKdy3dpfTc0nP3ewG+WMbA\ntTF3bAM7lKaWpRuWUp+KcCjhKUj\n8jwZqeaqNn/bRK5rkVpwB2syPq9\nNch6pBXew5uo0r02uT5Fa8CEZ+ub\nB4kUKpBSu9GdLK2v4LSwtHPzcWf\nu182D3wcqj9eYN",
  "ew5uo0r02uT5Fa8CEZ+ub\nB4kUKpBSu9GdLK2v4LSwtHPzcWf\nu182D3wcqj9eYN7Q3vO+975635v\n3mPfKe12v54Ve5f3l/e39s3x7+\nY/lR8uNe/1aU+e21/osb/0HKuDWU\nA=</latexit>Pr(y = k) = \u03bbk\n92",
  "Example 3: multiclass classification \nProblem:  \n\u2022 Output of neural network can be anything\n\u2022 Parameters \ud835\udf06! \u2208[0,1], sum to one\nSolution:\n\u2022 Pass through function that maps \u201canything\u201d \nto [0,1], sum to one\n<latexit sha1_base64=\"xYlXKujSVnmf73jV6IHiusCjE6s=\">AWxniclZhb9s2FIDVX\nbvulm5YXvYiLCg6DJ0RD93lpUCbNG3apEvSxEna2DUomZLZUJQiUYkTwcB+0n7NgD1tP2WHkm1W5zAPM5CaPd8nXg5JiVaQSVHo1dW/b7z3/gcfvTxzU9ufrZ5198uXT7q",
  "tP2WHkm1W5zAPM5CaPd8nXg5JiVaQSVHo1dW/b7z3/gcfvTxzU9ufrZ5198uXT7q8MiLfOQ98JUpv\nlxwAouheI9LbTkx1nOWRJIfhScrht+dM7zQqTqQF9mfJCwWIlIhExDaLj0vJ8E6aQq0kgnbDIdVqfTk34QXQ38B34/ylY9fkO7kyYDCt+kWZQPHug+70TbU1XbC7AIdLK6ud1frj0J3Vlj\nxZp/d4e1vRv1RGpYJVzqUrChOuquZHlQs1y",
  "70TbU1XbC7AIdLK6ud1frj0J3Vlj\nxZp/d4e1vRv1RGpYJVzqUrChOuquZHlQs1yKUfHqrXxY8Y+Epi/kJFBVLeDGo6kFP/TsQGflRmsOf0n4dfeKiVFcZkEYCZMjwvMTNDFTkod/TaohMpKzVXYNBSV0tepbzLoj0TOQy0vocDC\nXEBf/XDMIFUa8nyr/hFmCYJU6Oqv7axBzkLeCxUxc/KOufTadvZqB0OxeuMtWcHi1qE5om4qSWjGVXCPweF",
  "mCYJU6Oqv7axBzkLeCxUxc/KOufTadvZqB0OxeuMtWcHi1qE5om4qSWjGVXCPweFpVvBN3MBAcgOhwAlLFC6jT5CeI/C6isMYk4KpZPrBi/JdTUrXSPIactLTXR\nINCJvmkZa0TC6YyaSn7oPj+Hd8ArnOYBegqfHE0B/sZU9P5dZpPdJ5UhYnhFnKmYl43AUMOmTQjahuqlBIuDVvW79h6ydTpLHFpVnc1NxFkHeRtR+c0L2rUduoIsmARxm2rj",
  "UMOmTQjahuqlBIuDVvW79h6ydTpLHFpVnc1NxFkHeRtR+c0L2rUduoIsmARxm2rjiBLwh1hxMwOn5\neHMODENxG3KhRWBVmYu3katNvOTASvzUkG+6XtbVQk/ecMZcQEYPeZb8FUyNv6erqw/XlyzmvfFPjEH8NktS9hedwMa94IjGoWm1KzhUyabYglKcXbdP0xqHyTLQHaAJ405W5UNE72r26BE\nvWhPv3YKh5KfnJj52f+WRQrZptY/4h2YSK",
  "dP0xqHyTLQHaAJ405W5UNE72r26BE\nvWhPv3YKh5KfnJj52f+WRQrZptY/4h2YSKijJzVWTC/6OiETyD8PqCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5E21/Eqn1NHcGdTRPUVwiYeuGbCYUmOYrasgkYGb7haepYQCEaZNi\nMZRpUeac3PzQeoZIrZvbYi7Mw6p9Q5VGaN83uFxcBWV4OJzay4PUEaDJp9BWqoRy1EyJ2ZKJ2/6hYt5tr",
  "ZvbYi7Mw6p9Q5VGaN83uFxcBWV4OJzay4PUEaDJp9BWqoRy1EyJ2ZKJ2/6hYt5tr9ZQ3RacV87OtWXvQL3MqCEN+NtzC8xETizoS1QXHF2dkliO9qCuxXJ9t2fV\n1psfyNKOHa7blKTeWS/dtsO9pgf8bNvR23iEYs6EtU16yH1iOVoD+py53HbNQqH6zYlqXeR6ftcBcmWv7RwZhrZo5JqRyZY18q+0Ii5qK2imCY+R2ISwmJRtC/6PlX0BD4+2",
  "XeR6ftcBcmWv7RwZhrZo5JqRyZY18q+0Ii5qK2imCY+R2ISwmJRtC/6PlX0BD4+21YSwuFuIt\nmYCWBpxiYfQhLDYbOG2OYthduhbrtVJrMxMpsQFp+yBI+6CWExpmLsFE9ZliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9O2ZAJYmqDWJo7GoAcyVajBWRDLBV15hXPlKbSKFV3FPVfDvW\nsa1gxVaAJY2iF7zO/vODdZgFMxyxXkjOBrIwm",
  "WRDLBV15hXPlKbSKFV3FPVfDvW\nsa1gxVaAJY2iF7zO/vODdZgFMxyxXkjOBrIwmcBc7u9SZn/6CqCInuSC6tPS0gtLyg9svSI0txS8osgiF5aSn6dBNG5peUHlp6SGlpaUlpz9IepZGlEaVPLH1CaWhpSOm6peuUakvJiRS\neCJYeUDq2dEzpsaXHlL6y9BWlm5ZuUvra0teUXl6RekjSx9RyixlG5YukEpt5S8OgiNUvXKA0sJb/9YK9ZuktpZ",
  "Wlm5ZuUvra0teUXl6RekjSx9RyixlG5YukEpt5S8OgiNUvXKA0sJb/9YK9ZuktpZmlG6WNLH1M6spT8KobnmaXkeAMPRkslpc8sfUapsJT8fguiF5a+oDSx\nzcOFy9SIKVwpx8urXTxW1haOPyp0/2lc3/v/srDtdkb2pvet953vde1/vVe+htertezwu9P72/vH+8f5c3l9VyuXzRqO/dmF3ztdf6LP/xH8vA7sU=</latexit>NKH0uaXPKX1r",
  "2/vH+8f5c3l9VyuXzRqO/dmF3ztdf6LP/xH8vA7sU=</latexit>NKH0uaXPKX1r6VtKn1r6lNLYUvJuAE4nlu5Tat8CVQWle5buUXpm6Zn7vQBfTGPgWpg7toIdSlNLU0q3LCW/FOAoYekpOU9GanZXm79tIve1SC24g80yPr+a5DxSC+5gs7vT/Gpyf4rUgo9J1\nsoftmaxk[z] =\nexp[zk]\nPK\nk0=1 exp[zk0]\n<latexit sha1_base64=\"NH7nWzSRjgRCDl",
  "go9J1\nsoftmaxk[z] =\nexp[zk]\nPK\nk0=1 exp[zk0]\n<latexit sha1_base64=\"NH7nWzSRjgRCDl\n2OVYhzjLgSho=\">AWuHiclZhbU9w2FICd9JamN9JOemLp0xm0k6A530Mp1mJoGQG6Qs\ngQUSlDZK3sVZNnYMix98/01/S1feu/6ZHtXcXniIfuDKz2fJ91OZJ8CzIpCr28/O+Vq+8\n+97H1z78PpH3/y6WcLNz7fK9IyD/kgTGWaHwSs4FIoPtBCS36Q5Zw",
  "r28/O+Vq+8\n+97H1z78PpH3/y6WcLNz7fK9IyD/kgTGWaHwSs4FIoPtBCS36Q5ZwlgeT7wcma4ftnPC9\nEqnb1RcaPEhYrEYmQaQgdL/zaz29d3D35YxhEk2/8u/4wCdJVaSRTthkelydTA+bEAh+BD/\nAuw3/srE4OjpeWFruLdcfnxZW2sKS1376xze+HA1HaVgmXOlQsqI4XFnO9FHFci1CyafXh2X\nBMxaesJgfQlGxhBdHVT3MqX8TIiM/S",
  "1HaVgmXOlQsqI4XFnO9FHFci1CyafXh2X\nBMxaesJgfQlGxhBdHVT3MqX8TIiM/SnP4U9qvo28fUbGkKC6SAMyE6XGBmQm62Gpo5+PKq\nGyUnMVNg1FpfR16puc+SOR81DLCyiwMBfQVz8cs5yFGjJ7faj4eZgmCVOjari6vj2FXPFYqI\nqflnWp9Ous147HIqXGatPdue1CM0T8YaTSmrFVHKJwONpVfFe3MNAcACixwlIFS+gTpMfmO\nwVRGFV",
  "IqXGatPdue1CM0T8YaTSmrFVHKJwONpVfFe3MNAcACixwlIFS+gTpMfmO\nwVRGFVScCVXQ7Pp6RqpXkMOeloL4kGhUzyScdaIxZMZdJRdkDx/Zu+AVznMAvQVfjiaA52M\nqams+M0n+g8qQoTwy3kTMW8bgKGHDJpRtQ1VCklHBp2rN+w9ZypkzZxaVZ3NTcRZO3mXUfnN\nC9q1HXqCLJgEcZdq4gS8I5YMTMJp6Vj2HAiW8iblUorAqyMPt5GnT",
  "RZO3mXUfnN\nC9q1HXqCLJgEcZdq4gS8I5YMTMJp6Vj2HAiW8iblUorAqyMPt5GnTbzkwEr81JBvul61XJ\nP1nDGXEBGD3mW/BVMi7+lo6t/1Zcs5q3xT4xB/DZHUPYXncDGvWCIyqjU2pWecKmTRbEMrT\n865peuNQeSa6AzQBvOnKXKjoLe12XYIla8LD2zDUvJT8LveD3xyVC2bWP+kWxCRUWZuSoy\n4f9R0QiuOnh9QRPXirR5EGgnrxUwvkd",
  "DUvJT8LveD3xyVC2bWP+kWxCRUWZuSoy\n4f9R0QiuOnh9QRPXirR5EGgnrxUwvkdTR3L8cI2kXruoCAUk0JfoO0vYtU9po7gzqYJ6isE\nTL3wzYRCkxFXdkEjAzfcP10LKAQDTJsxhjKtChzTk5+aD1DpNbNaTEX5mLVPaFKI3TPG1z\nOj4IyXBzO+CWHByijQZPIC3ViOUomRMzpZNXw0LDFnPt/nrKm6LTivnpRtse9AtmpwxDfnq\n8gecjJhZ",
  "jQZPIC3ViOUomRMzpZNXw0LDFnPt/nrKm6LTivnpRtse9AtmpwxDfnq\n8gecjJhZ1JKoLblicdUliOdqDubL9e2eVRuviVLO3a4blOSeteum2He0kP+Omo7ebxCM\nWdSq+0h9YjlaA/qcudx0zUKh+s2Jal3lken7XDnJlr+0e6Ya2Zuk1I5Mrd9qRw2ISxqKmq\nnmCY8RmITwmJSdi34jZUdARePrtWEsNgvRFczASyNuMRDaEJYbLZw12xjWN1",
  "Kmq\nnmCY8RmITwmJSdi34jZUdARePrtWEsNgvRFczASyNuMRDaEJYbLZw12xjWN10qJtulclsjM\nwmhMVHLMGjbkJYjKkYO8UTlmVIbEIkj2OcxzHNY4alzCXhGckcM0KWlGtB5eO0K5kAliaotY\nmjMeiBTBVqsA1iuaAr3CuPIVWsaKreOBqeHBJw5qhCk0AS1tkj/nDLecmC3CK4TbLleRMIC\nujCexjp0+d2d1fEFXkTi6ILiy9oPTc0nNK9",
  "0AS1tkj/nDLecmC3CK4TbLleRMIC\nujCexjp0+d2d1fEFXkTi6ILiy9oPTc0nNK9y3dpzS3lDwRBNFzS8nTSRCdWXpG6Z6le5SWl\npaUDiwdUBpZGlH60NKHlIaWhpSuWbpGqbaU3JHCFcHSXUrHlo4pPbD0gNIXlr6g9LGljyl9a\nelLSt9Y+obS+5bep5RZyihdt3SdUm4peXUQRKuWrlIaWEqe/WCvWdqnNLM0o/SBpQ8oHVlKn\norhemYpub",
  "Zyihdt3SdUm4peXUQRKuWrlIaWEqe/WCvWdqnNLM0o/SBpQ8oHVlKn\norhemYpub2BC6OlktInlj6hVFhKnt+C6JmlzyhNLE0ofWrpU0pfW/qa0keWPqI0tpS8G4C7\nE0t3KLVvgaqC0m1Ltyk9tfTU/V6Az6cxcC3MLVvBFqWpSmlG5aSJwW4lbD0hNxPRqo9q9mX\nnVNszLmDtRmfHU1yHqk5d7D27DQ7mpyfIjXnY9L19b35ixRIKZzpjxeWV",
  "o9q9mX\nnVNszLmDtRmfHU1yHqk5d7D27DQ7mpyfIjXnY9L19b35ixRIKZzpjxeWVvBbWFrY+7638mPv\nzvadpXur7Rva95X3tfeLW/F+8m75z32+t7AC70/vb+8v71/Fn9Z/H0xXhSNevVKe8wXue\nzmP8HoQjnjg=</latexit>Pr(y = k|x) = softmaxk[f[x, \u03c6]]\n93",
  "Example 3: multiclass classification\n<latexit sha1_\nbase64=\"NH7nWzSRjgRCDl2OVYhz\nzjLgSho=\">AWuHiclZhbU9w2FI\nCd9JamN9JOemLp0xm0k6A530Mp\n1mJoGQG6QsgQUSlDZK3sVZNnYMi\nx98/01/S1feu/6ZHtXcXniIfuD\nKz2fJ91OZJ8CzIpCr28/O+Vq+8+\n97H1z78PpH3/y6WcLNz7fK9IyD\n/kgTGWaHwSs4FIoPtBCS36Q5Zwlg\neT7wcma4ftnPC9Eq",
  "pH3/y6WcLNz7fK9IyD\n/kgTGWaHwSs4FIoPtBCS36Q5Zwlg\neT7wcma4ftnPC9Eqnb1RcaPEhYrE\nYmQaQgdL/zaz29d3D35YxhEk2/8u\n/4wCdJVaSRTthkelydTA+bEAh+\nBD/Auw3/srE4OjpeWFruLdcfnxZW\n2sKS1376xze+HA1HaVgmXOlQsqI4\nXFnO9FHFci1CyafXh2XBMxaesJgf\nQlGxhBdHVT3MqX8TIiM/SnP4U9qv\no28fUbG",
  "FnO9FHFci1CyafXh2XBMxaesJgf\nQlGxhBdHVT3MqX8TIiM/SnP4U9qv\no28fUbGkKC6SAMyE6XGBmQm62G\npo5+PKqGyUnMVNg1FpfR16puc+SO\nR81DLCyiwMBfQVz8cs5yFGjJ7faj\n4eZgmCVOjari6vj2FXPFYqIqflnW\nWp9Ous147HIqXGatPdue1CM0T8Ya\nTSmrFVHKJwONpVfFe3MNAcACixwl\nIFS+gTpMfmOwVRGFVScCVXQ7Pp6",
  "CM0T8Ya\nTSmrFVHKJwONpVfFe3MNAcACixwl\nIFS+gTpMfmOwVRGFVScCVXQ7Pp6\nRqpXkMOeloL4kGhUzyScdaIxZMZd\nJRdkDx/Zu+AVznMAvQVfjiaA52Mq\nams+M0n+g8qQoTwy3kTMW8bgKGHD\nJpRtQ1VCklHBp2rN+w9ZypkzZxaV\nZ3NTcRZO3mXUfnNC9q1HXqCLJgE\ncZdq4gS8I5YMTMJp6Vj2HAiW8ib\nlUorAqyMPt5GnTbzkwE",
  "UfnNC9q1HXqCLJgE\ncZdq4gS8I5YMTMJp6Vj2HAiW8ib\nlUorAqyMPt5GnTbzkwEr81JBvul6\n61XJP1nDGXEBGD3mW/BVMi7+lo6t\n/1Zcs5q3xT4xB/DZHUPYXncDGvWC\nIyqjU2pWecKmTRbEMrT865peuNQe\nSa6AzQBvOnKXKjoLe12XYIla8LD\n2zDUvJT8LveD3xyVC2bWP+kWxC\nRUWZuSoy4f9R0QiuOnh9QRPXirR\n5EGgnrxUwvkd",
  "UvJT8LveD3xyVC2bWP+kWxC\nRUWZuSoy4f9R0QiuOnh9QRPXirR\n5EGgnrxUwvkdTR3L8cI2kXruoCAU\nk0JfoO0vYtU9po7gzqYJ6isETL3w\nzYRCkxFXdkEjAzfcP10LKAQDTJ\nsxhjKtChzTk5+aD1DpNbNaTEX5mL\nVPaFKI3TPG1zOj4IyXBzO+CWHByi\njQZPIC3ViOUomRMzpZNXw0LDFnP\nt/nrKm6LTivnpRtse9AtmpwxDfnq\n8gec",
  "yi\njQZPIC3ViOUomRMzpZNXw0LDFnP\nt/nrKm6LTivnpRtse9AtmpwxDfnq\n8gecjJhZ1JKoLblicdUliOdqDub\nL9e2eVRuviVLO3a4blOSeteum\n2He0kP+Omo7ebxCMWdSq+0h9Y\njlaA/qcudx0zUKh+s2Jal3lken7X\nDnJlr+0e6Ya2Zuk1I5Mrd9qRw2IS\nxqKmqnmCY8RmITwmJSdi34jZUdAR\nePrtWEsNgvRFczASyNuMRDaEJYb\nL",
  "Rw2IS\nxqKmqnmCY8RmITwmJSdi34jZUdAR\nePrtWEsNgvRFczASyNuMRDaEJYb\nLZw12xjWN10qJtulclsjMwmhMVHL\nMGjbkJYjKkYO8UTlmVIbEIkj2Ocx\nzHNY4alzCXhGckcM0KWlGtB5eO0K\n5kAliaotYmjMeiBTBVqsA1iuaAr\n3CuPIVWsaKreOBqeHBJw5qhCk0A\nS1tkj/nDLecmC3CK4TbLleRMICuj\nCexjp0+d2d1fEFXkTi6IL",
  "qeHBJw5qhCk0A\nS1tkj/nDLecmC3CK4TbLleRMICuj\nCexjp0+d2d1fEFXkTi6ILiy9oPTc\n0nNK9y3dpzS3lDwRBNFzS8nTSRCd\nWXpG6Z6le5SWlpaUDiwdUBpZGlH6\n0NKHlIaWhpSuWbpGqbaU3JHCFcHS\nXUrHlo4pPbD0gNIXlr6g9LGljyl\n9aelLSt9Y+obS+5bep5RZyihdt3S\ndUm4peXUQRKuWrlIaWEqe/WCvWdq\nnNLM0o/SBpQ",
  "St9Y+obS+5bep5RZyihdt3S\ndUm4peXUQRKuWrlIaWEqe/WCvWdq\nnNLM0o/SBpQ8oHVlKnorhemYpub2\nBC6OlktInlj6hVFhKnt+C6Jmlzyh\nNLE0ofWrpU0pfW/qa0keWPqI0tp\nS8G4C7E0t3KLVvgaqC0m1Ltyk9tf\nTU/V6Az6cxcC3MLVvBFqWpSmlG5\naSJwW4lbD0hNxPRqo9q9mXnVNszL\nmDtRmfHU1yHqk5d7D27DQ7mpyfIj\nXn",
  "lG5\naSJwW4lbD0hNxPRqo9q9mXnVNszL\nmDtRmfHU1yHqk5d7D27DQ7mpyfIj\nXnY9L19b35ixRIKZzpjxeWVvBbWF\nrY+7638mPvzvadpXur7Rva95X3\ntfeLW/F+8m75z32+t7AC70/vb+8v\n71/Fn9Z/H0xXhSNevVKe8wXuezm\nP8HoQjnjg=</latexit>Pr(y = k|x) = softmaxk[f[x, \u03c6]]\n94",
  "Example 3:  multiclass classification\n<latexit sha1_base64\n=\"1BeKFuEjQzkdT8EhSwfnXpMSZU=\">AXlXi\nclZhtT9xGEMeP9CmlT0mrikp9YxWlqoEQZU+v\nImUQEhCIAUCBySYnNa+tW/Dem3sNRyxrH6avm0/\nT79NZ+27WzyzSO1JxJv5/T27OzO7XjvIpCj08vI\n/czfe/+Dz+6+fH8J59+9vkXt25/eVCkZR7yf\npjKND8KWMGlULyvhZb8KMs5SwLJD4PTNcMPz3le",
  "fH8J59+9vkXt25/eVCkZR7yf\npjKND8KWMGlULyvhZb8KMs5SwLJD4PTNcMPz3le\niFTt68uMnyQsViISIdNgGtye+8bOvaDKBuJE+/\n7B949vyiTQSUerNRvqo3al2nsSx7pYz8J0nFVpJ\nFO2LgeVJcgqurDLx40dQSROBuNs69nMRj/QJ\nuqhUlUnAc9+fd3XcuIxwRw6396MsXFx2rjYrH0\n+zq6OzPeTv/zAe3FpeXlpufRxsrk8Zib/LbG",
  "fd3XcuIxwRw6396MsXFx2rjYrH0\n+zq6OzPeTv/zAe3FpeXlpufRxsrk8Zib/LbGd\nz+eugP07BMuNKhZEVxvLKc6ZOK5VqEktfzflnw\njIWnLObH0FQs4cVJ1eSw9u6AZehFaQ5/SnuN9eo\ndFUuK4jIJQJkwPSowM0YXOy519NtJVRWaq7Ctq\nOolJ5OPVMQ3lDkPNTyEhoszAWM1QtHLGehrKZ9\nxW/CNMkYWpY+avruzWkmcdCVfysbEqoru",
  "MQ3lDkPNTyEhoszAWM1QtHLGehrKZ9\nxW/CNMkYWpY+avruzWkmcdCVfysbEqorua9Ub\nDoXmdYnVjf+ZFaJ6Id5w4aSTGyTUCHtdVxZfiJQ\nwEByCWOAGp4gX4NPGBOl1BFJaMBFzZSn5ZE9dK8\nxhi0pG9JjJoZJKPO6o1oJUJh3JHkg8745nANc5\nZAGCheOcrCXMVP79N8rPOkKowN95AzFfOmC5\nhyKSZUVehSinh1rCj+h2rXjJ1Oglcmj",
  "heOcrCXMVP79N8rPOkKowN95AzFfOmC5\nhyKSZUVehSinh1rCj+h2rXjJ1OglcmjVDzY0Fq\nfbzrkbnNC5q2NU0FqSCIoy7qsaCVBI2uCEzG9G0\nPYAJ56xuKVCYakghbmTp0G378xYcG2OM1gvXd\n16RcJ/zlBEjAFWn7kKpkLela+lM7U3Dc5ozcNP\nvZGkKzuLSyP2lNO4FZTWw1VTaxQkoaLTDl6UVX\naUbjkPJMdCdoDHjRlblQ0RXZ3aYF",
  "kKzuLSyP2lNO4FZTWw1VTaxQkoaLTDl6UVX\naUbjkPJMdCdoDHjRlblQ0RXZ3aYFJWvM/l2Yal5\nKfnxv6Wc+PqmWzbIx/5BogqOizFyOjPl/OBrCI\nxXF1hw8lKJkgeGJnmphP0dpY7luLCNpckdNIRi\nUuhLtPxFrLr3NBY82DRBYwWD8QtXJhRKchR1xcZ\ngxHCFw4GjgEI0ybCdYyjTosw52fxQPYOlkZtM\nRfmYdXdUKURdPcNLmd3QRseD",
  "Z\ngxHCFw4GjgEI0ybCdYyjTosw52fxQPYOlkZtM\nRfmYdXdUKURdPcNLmd3QRseDuf8mtsDFNGgjWeQ\nlmrIchTMsUnp+I1faFhirtXfpLxtOlUxP9uc9Af\njguyUYcjPBps4HzFRUY1EvuA05vQlicrRH/iale\nvVkVWb34kpR07tG6lJH4no3SrHdprRsDPthyj\n3SI6oqIaiXxNRkh1ROXoD3y547jlmoVD61ZK4nc\naR6faoZ0pUflH+yOum",
  "Pthyj\n3SI6oqIaiXxNRkh1ROXoD3y547jlmoVD61ZK4nc\naR6faoZ0pUflH+yOumTkmpXJojn2p9FsTFmoq1E\n5hmvAYCVsTFiZlVwX/x5I9AQ+Prqo1YeFOIboy\nY8CiIZd4Cq0JC9sl3FVObFi65ZBuaVMZiOkbE1\nY+JQleNatCQtjKoydwlOWZUjYmkgcRziOIxrHDI\nsylwhnJHNkhJSUq6DyUdoVGQMWjVFvY0dnMAKZK\ntThxIjFBa28w",
  "cRziOIxrHDI\nsylwhnJHNkhJSUq6DyUdoVGQMWjVFvY0dnMAKZK\ntThxIjFBa28wl5ClWxolXcd3Xcv6ZjzZBDY8C\nibLGPH/bucgCHGI4ZrmCnAmkymgAd7Bmh2qmp7\n8gqshJLoguLb2k9MLSC0oPLT2kNLeUvBE0UtLy\ndtJEJ1bek7pgaUHlJaWlpT2Le1TGlkaUfrE0ie\nUhpaGlK5ZukaptpScSOGJYOk+pSNLR5QeWXpE6S\ntLX1H6zNJ",
  "e1TGlkaUfrE0ie\nUhpaGlK5ZukaptpScSOGJYOk+pSNLR5QeWXpE6S\ntLX1H6zNJnlL629DWl7yx9R+kjSx9RyixlK5bu\nk4pt5R8OgiVUtXKQ0sJe9+sNYs3aE0szSj9LGl\njykdWkreiuF5Zik53sCD0VJ6YalG5QKS8n7Wx\nC9sPQFpYmlCaXPLX1O6VtL31L61NKnlMaWkm8Dc\nDqxdI9S+xWoKijdtXSX0jNLz9zfBfgsjYGrMLet\ng21KU",
  "L31L61NKnlMaWkm8Dc\nDqxdI9S+xWoKijdtXSX0jNLz9zfBfgsjYGrMLet\ng21KU0tTSjctJW8KcJSw9JScJyM12dXsd9oaK2\nbcwSYRn95NYh6pGXewye40vZvsT5Ga8REZ+vrB7\nEMKhBR2+sGtxRX8FZY2Dn5aWvl6f7u/cWHq5Mv\ntDd73/a+6/3QW+n92nvYe9b6fV74dwfc3/O/TX\n398LCwoOFxwtPWumNuck9X/U6v4XtfwFU4zyu<\n/",
  "92nvYe9b6fV74dwfc3/O/TX\n398LCwoOFxwtPWumNuck9X/U6v4XtfwFU4zyu<\n/latexit>\nL[\u03c6] = \u2212\nI\nX\ni=1\nlog [softmaxyi [f [xi, \u03c6]]]\n= \u2212\nI\nX\ni=1\nfyi [xi, \u03c6] \u2212 log\n\" K\nX\nk=1\nexp [ fk [xi, \u03c6]]\n#\n<latexit sha1_base64=\"xYlXKujSVnmf73jV6IHiusCjE6s=\">AWxniclZhb9s2FIDVXbvulm5YXvYiLCg6DJ0RD93l\npUCbNG3apEvSxEna2DUomZLZUJQi",
  "lZhb9s2FIDVXbvulm5YXvYiLCg6DJ0RD93l\npUCbNG3apEvSxEna2DUomZLZUJQiUYkTwcB+0n7NgD1tP2WHkm1W5zAPM5CaPd8nXg5JiVaQSVHo1dW/b7z3/gcfvTxzU9ufrZ5198uXT7q8MiLfOQ98JUpvlxwAouheI9LbTkx1nOWRJIfhScrht+dM7zQqTqQF9mfJCwWIlIhExDaLj0vJ8E6aQq0k\ngnbDIdVqfTk34QXQ38B34/ylY9fkO7k",
  "TqQF9mfJCwWIlIhExDaLj0vJ8E6aQq0k\ngnbDIdVqfTk34QXQ38B34/ylY9fkO7kyYDCt+kWZQPHug+70TbU1XbC7AIdLK6ud1frj0J3VljxZp/d4e1vRv1RGpYJVzqUrChOuquZHlQs1yKUfHqrXxY8Y+Epi/kJFBVLeDGo6kFP/TsQGflRmsOf0n4dfeKiVFcZkEYCZMjwvMTNDFTkod/Tao\nhMpKzVXYNBSV0tepbzLoj0TOQy0vocDCXEBf/",
  "VFcZkEYCZMjwvMTNDFTkod/Tao\nhMpKzVXYNBSV0tepbzLoj0TOQy0vocDCXEBf/XDMIFUa8nyr/hFmCYJU6Oqv7axBzkLeCxUxc/KOufTadvZqB0OxeuMtWcHi1qE5om4qSWjGVXCPweFpVvBN3MBAcgOhwAlLFC6jT5CeI/C6isMYk4KpZPrBi/JdTUrXSPIactLTXRINCJvmkZa0TC6\nYyaSn7oPj+Hd8ArnOYBegqfHE0B/sZU9P5dZpPdJ5",
  "PIactLTXRINCJvmkZa0TC6\nYyaSn7oPj+Hd8ArnOYBegqfHE0B/sZU9P5dZpPdJ5UhYnhFnKmYl43AUMOmTQjahuqlBIuDVvW79h6ydTpLHFpVnc1NxFkHeRtR+c0L2rUduoIsmARxm2rjiBLwh1hxMwOn5eHMODENxG3KhRWBVmYu3katNvOTASvzUkG+6XtbVQk/ecMZcQEYPeZb8FU\nyNv6erqw/XlyzmvfFPjEH8NktS9hedwMa94IjGoWm1",
  "tbVQk/ecMZcQEYPeZb8FU\nyNv6erqw/XlyzmvfFPjEH8NktS9hedwMa94IjGoWm1KzhUyabYglKcXbdP0xqHyTLQHaAJ405W5UNE72r26BEvWhPv3YKh5KfnJj52f+WRQrZptY/4h2YSKijJzVWTC/6OiETyD8PqCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5E21/Eqn1NHc\nGdTRPUVwiYeuGbCYUmOYrasgkYGb7haepYQCEaZNiMZRp",
  "QSgmhb5E21/Eqn1NHc\nGdTRPUVwiYeuGbCYUmOYrasgkYGb7haepYQCEaZNiMZRpUeac3PzQeoZIrZvbYi7Mw6p9Q5VGaN83uFxcBWV4OJzay4PUEaDJp9BWqoRy1EyJ2ZKJ2/6hYt5tr9ZQ3RacV87OtWXvQL3MqCEN+NtzC8xETizoS1QXHF2dkliO9qCuxXJ9t2fV1psf\nyNKOHa7blKTeWS/dtsO9pgf8bNvR23iEYs6EtU16yH1iOVoD+py",
  "xXJ9t2fV1psf\nyNKOHa7blKTeWS/dtsO9pgf8bNvR23iEYs6EtU16yH1iOVoD+py53HbNQqH6zYlqXeR6ftcBcmWv7RwZhrZo5JqRyZY18q+0Ii5qK2imCY+R2ISwmJRtC/6PlX0BD4+21YSwuFuItmYCWBpxiYfQhLDYbOG2OYthduhbrtVJrMxMpsQFp+yBI+6CW\nExpmLsFE9ZliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9O2ZAJYmqD",
  "BI+6CW\nExpmLsFE9ZliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9O2ZAJYmqDWJo7GoAcyVajBWRDLBV15hXPlKbSKFV3FPVfDvWsa1gxVaAJY2iF7zO/vODdZgFMxyxXkjOBrIwmcBc7u9SZn/6CqCInuSC6tPS0gtLyg9svSI0txS8osgiF5aSn6dBNG5\npeUHlp6SGlpaUlpz9IepZGlEaVPLH1CaWhpSOm6peuUakvJiRSeCJYeUDq2dE",
  "G5\npeUHlp6SGlpaUlpz9IepZGlEaVPLH1CaWhpSOm6peuUakvJiRSeCJYeUDq2dEzpsaXHlL6y9BWlm5ZuUvra0teUXl6RekjSx9RyixlG5YukEpt5S8OgiNUvXKA0sJb/9YK9ZuktpZmlG6WNLH1M6spT8KobnmaXkeAMPRkslpc8sfUapsJT8fguiF5\ntdkb2pvet953vde1/vVe+htertezwu9P72/vH+8f5c3l9VyuXzRqO/dmF3ztdf6LP/",
  "kb2pvet953vde1/vVe+htertezwu9P72/vH+8f5c3l9VyuXzRqO/dmF3ztdf6LP/xH8vA7sU=</latexit>a+oDSxNKH0uaXPKX1r6VtKn1r6lNLYUvJuAE4nlu5Tat8CVQWle5buUXpm6Zn7vQBfTGPgWpg7toIdSlNLU0q3LCW/FOAoYekpOU9GanZXm79tIve1SC24g80yPr+a5DxSC+5gs7vT/Gpyf4rUgo9J1zcOFy9SIKVwpx8urXTxW1h",
  "tIve1SC24g80yPr+a5DxSC+5gs7vT/Gpyf4rUgo9J1zcOFy9SIKVwpx8urXTxW1haOPyp0/2lc3/v/srD\nsoftmaxk[z] =\nexp[zk]\nPK\nk0=1 exp[zk0]\n*Multiclass cross-entropy loss*\n95",
  "Example 3:  multiclass classification\n0\n1.0\n1\n2\n3\nChoose the class with the largest probability\nWe also get probability or \u201cconfidence\u201d\n96",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n97",
  "Other \ndata types\n98",
  "Other Distributions\n\ud835\udc66 \u2208  \u211d Robust Regression\n\ud835\udc66 \u2208  \u211d Regression\n\ud835\udc66 \u2208  \u211d Multimodal Regression\n\ud835\udc66 \u2208 \u211d! Predict Magnitude\n\ud835\udc66 \u2208 [0,1]Predict Proportions\n\ud835\udc66 \u2208 (\u2212\ud835\udf0b, \ud835\udf0b]Predict Directions\n\ud835\udc66 \u2208 [0,1,2, \u2026 ]Predict Event Counts\nGaussian\nLaplace\nMixture of Gaussians\nGamma\nBeta\nVon Mises\nPoisson\n99",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n100",
  "Multiple outputs\n\u2022 Treat each output      as independent:\n\u2022 Negative log likelihood becomes sum of terms:\n<latexit sha1\n_base64=\"RWhPeTWEOuRIxOzWn\nR6wUxKHsnQ=\">AW1XiclZhJ\nb9w2FICVrm6OS3qSy9CjQBp4\nQ7sIl0uBRI7zmanHsce24nHGVA\nSpWFMUbIWeyaqbkWv/Un9Hf0B\nvbZ/oY8azTB6jz50AEfM+z5uj9\nTqpVLkxdraX9fevud97/o\nHNz786ONPl26+dlhnpSZzwd+I\npPs2GM5l0LxQSEKyY/T",
  "VLkxdraX9fevud97/o\nHNz786ONPl26+dlhnpSZzwd+I\npPs2GM5l0LxQSEKyY/TjLPYk/\nzIO9vU/OiCZ7lI1ExTflpzCI\nlQuGzAkKjpef97PbQC6e/DmMvm\nVRQdMP6BA6TUSXqVSikY3H69c\n/uM2SYFQFNVSYjoKOPwpojdXR\n0spab635ubSw3hZWnPbXH938I\nhgGiV/GXBW+ZHl+sr6WFqcVywr\nhS17fGJY5T5l/xiJ+AkXFYp6f\nVk",
  "PbXH938I\nhgGiV/GXBW+ZHl+sr6WFqcVywr\nhS17fGJY5T5l/xiJ+AkXFYp6f\nVk0OavcWRAI3TDL4U4XbRN+sU\nbE4z6exB2bMinGOmQ7a2ElZhD+\ndVkKlZcGVP+soLKVbJK5OqBuI\njPuFnEKB+ZmAsbr+mGXMLyDtN4\naKX/pJHDMVMONrb0aUsYjoSp\n+XjZLUNdZ6txOBSvMjYeHyxaE\nQWPxWtOGmkU3cgVAo/quK9qI\neB4ABEjxOQKJ",
  "+XjZLUNdZ6txOBSvMjYeHyxaE\nQWPxWtOGmkU3cgVAo/quK9qI\neB4ABEjxOQKJ5Dmzo/sObriMK\nWk4Arsyue1aRpVfAIctLRXhANC\nqnk461SxYyrij7IPiurdcDX\niRwSrAUOHA0Rrsp0zV83oFnxRZ\nXOU6hnvImIp40wVM2WdSz6hrq\nFJKqOp3rF+w9YypszZxSdoMNdM\nRZB1kXafIaF5U0HWaCLJgE0Zd\nq4kgS8IFImAxgy35RFMOH",
  "w9YypszZxSdoMNdM\nRZB1kXafIaF5U0HWaCLJgE0Zd\nq4kgS8IFImAxgy35RFMOHZ1x\nK4KhVBNmY/S7xu36mO4L05SeF\n86XpbFUn/BUMZ0QE4+/RMOXz\nr6ZLGx3npyLxtcFPnHsFjdKi\nyLZtOadwKzamM1NZtcIZNmC0J\nZctk19WgsKk9Fd4I6gE+6MhMqf\nENbUqwZXV4uApTzUrJT7tfc\n8np9WaPm30PySb0FBepraGdPh\n/NBTALQnv",
  "hMqf\nENbUqwZXV4uApTzUrJT7tfc\n8np9WaPm30PySb0FBepraGdPh\n/NBTALQnvL4jgxUskWjwINIuXS\nLi+o6VjGd7YOtKsHRSEYlIU3\nT6i0h16zQRPNgkRmOFgG4Xjkwo\ntMh2JV1QMtwhJurZQP5aJL+b\nI6+TPIy4+Tih/YzRBpdXxYzoW9\nW3Quq1EL3usHlohaU4eZwa+o\n7qGMerN8ekmpApahZE70k5eD\nvMCTjHb2d8s+axotSJ+v",
  "EL3usHlohaU4eZwa+o\n7qGMerN8ekmpApahZE70k5eD\nvMCTjHb2d8s+axotSJ+vt32B+O\nC1Sl9n5+PtvF6RMSijkRtwdOM\ntS1JLEt/0NZiu745smr75Tdka0\ncW125K0m47Srtca8YAT/fsYx\n2h3jEo5EbUjpB6xLP1BW/Y87\nthmYXHtpiTtzvNotS3uwkTbPz\nwY84Lpx6REBvqxL5HDWQiLBRU\nLq5jEPELiLITFuOxa8H+s7Au4e\nXSt",
  "uwkTbPz\nwY84Lpx6REBvqxL5HDWQiLBRU\nLq5jEPELiLITFuOxa8H+s7Au4e\nXStWQiL/Vx0NR3AUsAlnsIshM\nXZKdw12xhWdyzqjl1lMh0jcxbC\n4kMW41nPQliMqBhZxTOWpkich\nUgexziPY5rHFEupTcIrklpWhGw\np24bKxklX0gEsTVBvE0tnMAKZ\nKNRhG8RyTndebt15Cu1iRXfxw\nNbx4IqOC4Ya1AEs7ZJzB3uWk8\nyD6cYHrNsSU",
  "KNRhG8RyTndebt15Cu1iRXfxw\nNbx4IqOC4Ya1AEs7ZJzB3uWk8\nyD6cYHrNsSU4FslKawD52+tSZ\nP/15YUWe5OD12dApZeGXlJ6ZO\ngRpZmh5I3AC58ZSt5OvPDC0At\nKDw09pLQ0tKR0YOiA0tDQkNIHh\nj6g1DfUp3T0E1KC0PJEyncEQ\nw9oHRs6JjSY0OPKX1u6HNKHxn\n6iNIXhr6g9LWhrym9Z+g9Spmhj\nNItQ7co5YaSTwdeuGHoB",
  "SY0OPKX1u6HNKHxn\n6iNIXhr6g9LWhrym9Z+g9Spmhj\nNItQ7co5YaSTwdeuGHoBqWeoe\nTdD841Q/uUpoamlN439D6lgaHk\nrRjuZ4aSxu4MRoqKX1s6GNKh\naHk/c0Lnxr6lNLY0JjSJ4Y+ofS\nVoa8ofWjoQ0ojQ8m3AXg6MXSf\nUvMVqMop3TN0j9JzQ8/t3wX4Y\nhk928bcNQ3sUpoYmlC6bSh5U4B\nHCUPyPNkqNqrmvnmWNjwS2s\nzfi8",
  "8/t3wX4Y\nhk928bcNQ3sUpoYmlC6bSh5U4B\nHCUPyPNkqNqrmvnmWNjwS2s\nzfi8Nsl5qBbcwtqr07w2uT6Fas\nHZOhbh4sPKZBSuNKPlbW8Vd\nYWj8rf+Q+/O3p2VuxvtF9r\nzpfOV85tZ9350bnrPHL6zsDxnT\n+dv51/nH+Xj5br5d+Wf5+pb1\nexit>r63zudH7Lf/wHJyfziQ=</lat\nPr(y|f[xi, \u03c6]) =\nY\nd\nPr(yd|fd[xi, \u03c6]),\n<latexit sha1_ba",
  "63zudH7Lf/wHJyfziQ=</lat\nPr(y|f[xi, \u03c6]) =\nY\nd\nPr(yd|fd[xi, \u03c6]),\n<latexit sha1_base64=\"wBfQC1oIM\nBDR9Fil6k0OQWpPj4=\">AWfXiclZhb9s2FIDV7tZlt3ZD87IXYUGBYeiM\nZOjWPbZJ07RNujhNnKSN04CSKZkNRSkSldgV/BP2uv2/ZrtUJLN6hzmYQZSs+\nf7xMshKdEKMikKvbr6z42bH38yaef3fp86Ysv/r6m9t3vj0s0jIP+SBM",
  "mYQZSs+\nf7xMshKdEKMikKvbr6z42bH38yaef3fp86Ysv/r6m9t3vj0s0jIP+SBMZo\nfB6zgUig+0EJLfpzlnCWB5EfB+YbhR5c8L0SqDvQ046cJi5WIRMg0hPanZ6Oz\n2yurvdX649PCWltY8dpP/+zO3dFwlIZlwpUOJSuKk7XVTJ9WLNcilHy2NCwLn\nrHwnMX8BIqKJbw4req+zvx7EBn5UZrDn9J+Hf3wiolRTFNAjATpscFZiboYie\nljn",
  "HwnMX8BIqKJbw4req+zvx7EBn5UZrDn9J+Hf3wiolRTFNAjATpscFZiboYie\nljn4/rYTKSs1V2DQUldLXqW8G7o9EzkMtp1BgYS6gr34ZjkLNaRnaj4VZgm\nCVOjari+uTerhgGPhar4RVmnajbrOpu1w6F4nbH+/GBRi9A8Ee85qaRWTCXC\nDyeVRXvxT0MBAcgepyAVPEC6jT5CSJ/DVFYGhIw8CdQOci/9WMVK0jyEnHe\n0N0aCQST7pWBvE",
  "MBAcgepyAVPEC6jT5CSJ/DVFYGhIw8CdQOci/9WMVK0jyEnHe\n0N0aCQST7pWBvEgqlMOso+KL5/zeA6xmAboKXxzNwX7G1Gx+neYTnSdVYWK4\nhZypmNdNwJBDJs2IuoYqpYRLw471B7ZeMXeJi7N6q7mJoKsg7zr6JzmRY26T\nh1BFizCuGvVEWRJ2MgjljDIcls+gwEnvom4VaGwKsjC7Odp0G07MxG8NicZ7J\neut1mR9F8ylBETgN1nvgVT",
  "jDIcls+gwEnvom4VaGwKsjC7Odp0G07MxG8NicZ7J\neut1mR9F8ylBETgN1nvgVTIe/qG+nC9ufJuax9U+ATfwyT1b2E5XEzrHkjMKo\n2NqNmnStk0mxBKE+vuqbpjUPlmegO0ATwpitzoaIPtPt1CZasCQ/vw1DzUvKTn\n3u/8slptWq2jfmHZBMqKsrMVZEJ/4+KRvDowOsLInjyUokmDwL15KUS7u9o6l\niOF7aJ1HMHBaGYFHqKtr+IVfeaO",
  "EJ/4+KRvDowOsLInjyUokmDwL15KUS7u9o6l\niOF7aJ1HMHBaGYFHqKtr+IVfeaOoI7myaorxAw9cI3EwpNchR1ZRMwMnzDQ9C\nxgEI0yLAZYyjTosw5ufmh9QyRWje3xVyYh1X3hiqN0L1vcLm4CsrwcLjk1we\noIwGT6DtFQjlqNkTsyUTt4OCw1bzLX76ylvik4r5hfbXvQL5idMgz5xdk2n\no+YWNSRqC4dTjrksRytAd1LZbrhz2rt/+RJZ",
  "ylvik4r5hfbXvQL5idMgz5xdk2n\no+YWNSRqC4dTjrksRytAd1LZbrhz2rt/+RJZ27HDdpiT1tr102w73mh7wix1\nHb3eIRyzqSFRX20PqEcvRHtTlzuOaxQO121KUu8j07b4S5MtPyjgzHXzByT\nUjkyx75UDpsQFjUVtVNMEx4jsQlhMSm7FvwfK/sCHh5dqwlhsV+IrmYCWBpxi\nYfQhLDYbOGu2cawuNQd9wqk9kYmU0Ii1swaNuQliMqRg7",
  "dqwlhsV+IrmYCWBpxi\nYfQhLDYbOGu2cawuNQd9wqk9kYmU0Ii1swaNuQliMqRg7xXOWZUhsQiSPY5\nzHMc1jhqXMJeEZyRwzQpaUa0Hl47QrmQCWJqi1iaMx6IFMFWqwDWK5oCuvcK48\nhVaxoqt4Gp4cE3DmqEKTQBLu2SP+cNd5yYLcIrhmOVKciaQldE9rHTp8789\nBdEFTnJBdHU0imlV5ZeUXpk6RGluaXkF0EQvbKU/DoJoktLyk9tPSQ",
  "9rHTp8789\nBdEFTnJBdHU0imlV5ZeUXpk6RGluaXkF0EQvbKU/DoJoktLyk9tPSQ0tLSkt\nKBpQNKI0sjSp9a+pTS0NKQ0g1LNyjVlpITKTwRLD2gdGzpmNJjS48pfW3pa0q\nfWfqM0jeWvqH0vaXvKX1s6WNKmaWM0k1LNynlpJXB0G0buk6pYGl5Lcf7DVL+\n5RmlmaUPrH0CaUjS8mvYnieWUqON/BgtFRS+tzS5QKS8nvtyB6aelLShNLE0",
  "L+\n5RmlmaUPrH0CaUjS8mvYnieWUqON/BgtFRS+tzS5QKS8nvtyB6aelLShNLE0\npfWPqC0neWvqN0y9ItSmNLybsBOJ1Yuk+pfQtUFZTuWbpH6YWlF+73AnwxjYF\nrYe7aCnYpTS1NKd2lPxSgKOEpefkPBmp9q42f9tE7muRWnAHazM+v5rkPFIL\n7mDt3Wl+Nbk/RWrBx6Trm4eLFymQUrjTn91eWcNvYWnh8Jfe2m+9B3sPVh6t2\n9ob3nfe",
  "Wl+Nbk/RWrBx6Trm4eLFymQUrjTn91eWcNvYWnh8Jfe2m+9B3sPVh6t2\n9ob3nfez94P3pr3kPvkfM63sDL/Ri70/vL+/vu/8u31u+v9xr1Js32mu+8zq\nf5Yf/AYZ90HY=</latexit>yd\n<latexit sha1_base64=\"\nWeDYdl7zLaqaeYEgt57O0Kn3T1Y=\">AXIHiclZhb\nb9s2FICd7tZlt3TD8rIXYUGBbmgDe+iuQIE2adqmSRe\nnuba",
  "1Y=\">AXIHiclZhb\nb9s2FICd7tZlt3TD8rIXYUGBbmgDe+iuQIE2adqmSRe\nnubaxa1ASJbOhKEWXxKmPzPsx+xt2OP2a3Yoy2Z1D\nvMwA62Z8306JA+pi+UmUmR5t/vPwrV3n3v/Q+uf7j4\n0cefPrZ0o3PD7O4SD1+4MUyTo9dlnEpFD/IRS75cZ\nJyFrmSH7mn65ofnfM0E7Hazy8TPoxYqEQgPJZDaLT0x\n/bJwA2SsRg695w7g6yIRqW416",
  "SH7mn65ofnfM0E7Hazy8TPoxYqEQgPJZDaLT0x\n/bJwA2SsRg695w7g6yIRqW416telZvVQMbhYE2E8qS\nf3gLr8rdB5MaTEpOUOnjJiBXt5sE32g3HdI09V9+O9\n0lcL9qJRz5V6UcLa10V7v1x6GNXtNY6TSf/ujGl/7A\nj70i4ir3JMuyk143yYclS3PhSV4tDoqMJ8w7ZSE/gaZ\niEc+GZV3OyrkJEd8J4hT+qdypo28fUbIoy4jF8yI5\neM",
  "SV4tDoqMJ8w7ZSE/gaZ\niEc+GZV3OyrkJEd8J4hT+qdypo28fUbIoy4jF8yI5\neMx20sZMiD34alkIlRc6VN+0oKSTx45eG8cXKfdy\neQkN5qUCxup4Y5YyL4cVXBwofuHFUcSUXw7WNnYrqB\nsPhSr5WVGvZlW1nY3a4dC8yljb3J9nETmPxBtOktSKT\nnKFwMOqLPlquIqB4ADEKicgVjyDnLo+sPA9RGH3SsC\nl2RrPK5Ja5TyEmrS0l0SDRi",
  "FwMOqLPlquIqB4ADEKicgVjyDnLo+sPA9RGH3SsC\nl2RrPK5Ja5TyEmrS0l0SDRiL5pGWtEwuWMmope6A4zk\n1HA56nsAowVPjiaA32Eqaq2XE5n+RpVGY6hntImQp5\n3QVM2WNSz6htqEJKONRrWb9i6zlTp03h4qQeaqojyNp\nP206e0rov+3UEWTBJgzbVh1BloRrjc8iBlVu2iOYc\nOToiF0VCquCbMx+GrvtvhMdwXtzksD50vY2SlL+c4Y",
  "Vh1BloRrjc8iBlVu2iOYc\nOToiF0VCquCbMx+GrvtvhMdwXtzksD50vY2SlL+c4Yq\nogNw9ulvwZTH2/p6PLedWXHOa183+MQZw2K1D2FpOJ\n3WrBOYVROrqFnXCpm0WhBK4u2qUdjUXki2hPUAXzSF\nalQwVva7boFW1aHB7dhqmkh+cmd1e/5ZFh29Wmj/yP\nVhERZkdgS6fD/SOTD3Q3vL4jgxYslWjwI1IsXS7i+o6\nVjKd7YOlKvHTSEYlL",
  "yP\nVhERZkdgS6fD/SOTD3Q3vL4jgxYslWjwI1IsXS7i+o6\nVjKd7YOlKvHTSEYlLkl+j0F6FqH1NH8GDjCI0VAjov\nfDOh0CIHQVvWAS3DN9ynLRvIQ5P0pnP0ZJwVKScXP7S\nfIVLr+rKYCn2zal9QpRba1w0u50dBG24O5/yKw1U\nXdaTzculM9SVMyJXtLJq0GWwylmO/vrJZ82rVbIz7a\n/mBcsDqF5/Gz0RZej5BY1JEoFzwYWXNJYln6g1",
  "LJq0GWwylmO/vrJZ82rVbIz7a\n/mBcsDqF5/Gz0RZej5BY1JEoFzwYWXNJYln6g1z7f\nr2yMqtV9+SrR1aXLspSd5mlHb4l4xAn62bRntNvGIR\nR2JcjUjpB6xLP1BLnsdt2zsLh2U5K8szpabYs7N9H\n2D/bHPGf6MSmWvn7si+VgGsJiTsXcKsYRD5E4DWExKt\noW/I2VPQE3j7Y1DWGxn4m2pgNY8rnEU5iGsDg9hdtm\nE8PqtkXdtqtMJmNk",
  "ExKt\noW/I2VPQE3j7Y1DWGxn4m2pgNY8rnEU5iGsDg9hdtm\nE8PqtkXdtqtMJmNkTkNYfMwiPOtpCIshFUOreMqSBIn\nTEKnjGNdxTOuYCmxSXhFEsuKkC1l21DpOG5LOoClC\neptYukMRiBjhTpsgljO6M7LrDtPoV2s6C4+sHV8cEXH\nOUMJdQBLO+QcwY71pPMxSWGxyxbkROBrIQWsI+dPn\nVmT39uUJInOfiRbeglpReGXlB6ZOgRpamh5B",
  "71pPMxSWGxyxbkROBrIQWsI+dPn\nVmT39uUJInOfiRbeglpReGXlB6ZOgRpamh5BeBGzw3l\nPw6cYNzQ8pPT0kNLC0ILSA0MPKA0MDSh9ZOgjSj1\nDPUrXDV2nNDeUPJHCHcHQfUrHho4pPTb0mNIXhr6g9I\nmhTyh9aehLSt8Y+obSB4Y+oJQZyijdMHSDUm4oeXg\nBmuGrlHqGkp+8G5Zmif0sTQhNKHhj6k1DeU/CqG+5m\nh5PEGboyGSko3D",
  "m4oeXg\nBmuGrlHqGkp+8G5Zmif0sTQhNKHhj6k1DeU/CqG+5m\nh5PEGboyGSko3Dd2kVBhKfr+5wTNDn1EaGRpR+tTQp\n5S+NvQ1pY8NfUxpaCh5NwBPJ4buUWreApUZpbuG7lJ6\nZuiZ/b0Any+ja9uYOybBDqWxoTGlW4aSXwrwKGHoKX\nmeDFRzVTMvPitszLmFNRWfHU1qHqg5t7Dm6jQ7mlyfA\njXnYzL0jcP5ixQoKVzpR0srPfwWljYOv",
  "tszLmFNRWfHU1qHqg5t7Dm6jQ7mlyfA\njXnYzL0jcP5ixQoKVzpR0srPfwWljYOv1vt/bB6d/f\nuyv215g3t9c5Xna87tzq9zo+d+50nX7noOMtLC50F3\n5e+GX59+U/l/9a/nuqXltojvmi0/os/sfJBsOuw=\n</latexit>\nL[\u03c6] = \u2212\nI\nX\ni=1\nlog\nh\nPr(y|f[xi, \u03c6])\ni\n= \u2212\nI\nX\ni=1\nX\nd\nlog\nh\nPr(yid|fd[xi, \u03c6])\ni\n101",
  "Example 4: multivariate regression\n102",
  "Example 4: multivariate regression\n\u2022 Goal:  to predict a multivariate target\n\u2022 Solution treat each dimension independently\n\u2022 Make network with \ud835\udc37* outputs to predict means\n<latexit sha1_base64=\"BNkHogQk9/53BQqaGqLacwOs9o=\">AWlXiclZj\nZbtw2FECVdEvTLWlRF0VfhAYBiIZ2EW6vBRIvCRO7NTjZWwnHtugNJSGMUXJEmWPIwz6NX1tv6d/0tJM4zupR9qIB32niMul6REKcikKPTi4r83br73/gcfnTr\n49ufPrZ51/cufvlfpGWecgHYSrT/DBgBZdC8YE",
  "REKcikKPTi4r83br73/gcfnTr\n49ufPrZ51/cufvlfpGWecgHYSrT/DBgBZdC8YEWvLDLOcsCSQ/CM5WD+4HkhUrWnrzJ+nLBYiUiETEPo9M63wyC6Ggo1TJgeB0G1Mz2pVk+rdDo9vXNvsbdY/\n/m0sNQW7ntX/07tej4SgNy4QrHUpWFEdLi5k+rliuRSj59PawLHjGwjMW8yMoKpbw4riqxzD170Nk5EdpDv+U9uvou1dULCmKqyQA03S1wMw",
  "59PawLHjGwjMW8yMoKpbw4riqxzD170Nk5EdpDv+U9uvou1dULCmKqyQA03S1wMwEXeyo1NFvx5VQWa\nm5CpuGolL6OvVNQvyRyHmo5RUWJgL6KsfjlnOQg1puz1U/DJMk4SpUTVcXtueVsOAx0JV/LysUzidp212uFQvM5Yfr43r0Vonoi3nFRSK6aSawQeT6uK9+IeBoI\nDED1OQKp4AXW0xz5S4jCkpGAgQfpBDoX+TtTUrXSPIacdLTXRINC",
  "T6uK9+IeBoI\nDED1OQKp4AXW0xz5S4jCkpGAgQfpBDoX+TtTUrXSPIacdLTXRINCJvmkY60QC6Yy6Si7oPj+fd8ArnOYBegq/HA0B7sZU9PZdZpPdJ5UhYnhFnKmYl43AUMOmTQj\n6hqlBIuDTvWH9jaYeqsTVya1V3NTQRZe3nX0TnNixp1nTqCLFiEcdeqI8iSsMFHLGQ5bZ8CgNOfBNxq0JhVZCF2c/ToNt2ZiJ4bU4y2C9db60i6b9gKCMmA",
  "I8iSsMFHLGQ5bZ8CgNOfBNxq0JhVZCF2c/ToNt2ZiJ4bU4y2C9db60i6b9gKCMmALvP/\nAqmQt7V9K57c+Sc1H7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkXvaA/qEixZEx4+gKHmpeRHD3s/8lxtWi2jfkPySZUVJ\nSZqyIT/h8VjeCRgtcXRPDkpRJNHgTqyUsl3N/R1LEcL2wTqecOCkIxKfQV2v4iV",
  "SZqyIT/h8VjeCRgtcXRPDkpRJNHgTqyUsl3N/R1LEcL2wTqecOCkIxKfQV2v4iVt1r6gjubJqgvkLA1Au/TCg0yVHUlU3AyPALD0fHAgrRIMNmjKFMizLn5OaH1jN\nEat3cFnNhHlbdG6o0Qve+weX8KijDw+GCX3N5gDIaNPkM0lKNWI6SOTFTOjkZFhq2mGv31PeFJ1WzM832vagXzA7ZRjy89MNPB8xsagjUV1wGnHWJYnlaA/qmi/X\nd3tW",
  "mGv31PeFJ1WzM832vagXzA7ZRjy89MNPB8xsagjUV1wGnHWJYnlaA/qmi/X\nd3tWbZz8SJZ27HDdpiT1tr102w73mh7w801HbzeJRyzqSFRX20PqEcvRHtTlzuOmaxQO121KUu8sj07b4c5NtPyjvTHXzByTUjkyx75UDpsQFjUVtVNMEx4jsQlhM\nSm7Fvw/VnYFPDy6VhPCYr8QXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQFp+xBI+6",
  "/VnYFPDy6VhPCYr8QXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQFp+xBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZA\nJYmqDWJo7GoAcyVajBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGY5YryZlAVkYT2MdOnzqz018QVeQkB2/Al5RemnpJaUHlh5QmltK3gi\nCaMdS8nYSRBeWX",
  "ZlAVkYT2MdOnzqz018QVeQkB2/Al5RemnpJaUHlh5QmltK3gi\nCaMdS8nYSRBeWXlC6b+k+paWlJaUDSweURpZGlD619CmloaUhpSuWrlCqLSUnUngiWLpH6djSMaWHlh5S+srSV5SuW7pO6WtLX1P61tK3lD6x9AmlzFJG6Zqla5Ry\nS8mngyBatnSZ0sBS8u4He83SPqWZpRmlq5auUjqylLwVw/PMUnK8gQejpZLS5Y+p1RYSt7fguilpS8pT",
  "8u4He83SPqWZpRmlq5auUjqylLwVw/PMUnK8gQejpZLS5Y+p1RYSt7fguilpS8pTSxNKH1h6QtK31j6htJnlj6jNLaUfBuA04mlu5Tar0BVQ\n+pt/RL79H2o3uPl9svtLe87zvR+8Je9X7G37vW9gRd6f3p/eX97/yx8s/D7wurC0a9eaO95iuv87ew9R9L29mM</latexit>em2pduUnlt67v4uwOfTGLgW5patYIvS1NKU0g1LyZsCHCUsPSPn",
  "9mM</latexit>em2pduUnlt67v4uwOfTGLgW5patYIvS1NKU0g1LyZsCHCUsPSPnyUi1d7XZ1yZyX4vUnDtYm/HZ1STnkZpzB2vTrOryf0pUnM+Jl1f259/SIGU7tfZfwV1ha2P\ny 2 RDo\n<latexit sha1_base64=\"o+2LGl4kKW0aeanAyF5BUH\neCgUc=\">AXM3iclZhbT9xGFICXlN6I63KS1XJKkqVAliUXp5iZRAyA1SlsACYbV2B57J4zHxh7",
  "M3iclZhbT9xGFICXlN6I63KS1XJKkqVAliUXp5iZRAyA1SlsACYbV2B57J4zHxh7DEsc/q+p\nvqfpW9bX/oWds7048Z3joSsGT831zOzO+eilnuVxZ+XPuvfc/+PCj69Mv/pZ59/8eXC9a/286TIfDr0E5khx\n7JKWeCDiWTnB6mGSWx+mBd7qu+ME5zXKWiD15mdLjmESChcwnEkKjhT+cQXbT9cLt/AnLm67OYticrJ6y/nh\nnuOmWRKMyuBev",
  "D15mdLjmESChcwnEkKjhT+cQXbT9cLt/AnLm67OYticrJ6y/nh\nnuOmWRKMyuBevzopH47KpKrAvoRI9daNC3XUFVx3l7FcOM+GW/Kt38LJPlqpuytla5WlWVSyepy2koj+40JvQ\nQ3FHtB7dqA2po3c1YNJbHo4WleWV+ufgQr8tLPXa32B0/ZvADRK/iKmQPid5ftRfSeVxSTLJfE6rebfIaUr8Ux\nLRIygKEtP8uKwzXDk3IBI4YZLBPyGdOv",
  "QPid5ftRfSeVxSTLJfE6rebfIaUr8Ux\nLRIygKEtP8uKwzXDk3IBI4YZLBPyGdOvpujZLEeX4Ze2DGRI5zk6mgjR0VMvz1uGQiLSQVftNRWHBHJo5aLidgG\nfUlv4QC8TMGY3X8MYEsSVjUeVfQCz+JYyKC0l3b2IEUezRioqRnRb3AVdV1NmqHQvEqY+3p3qwVJmnM3lDUSK2o\nRq4QaFSVJV2Olk3AKAC2TBFIBM2hTZUfL3T6BoUNzQED95IJ",
  "qwVJmnM3lDUSK2o\nRq4QaFSVJV2Olk3AKAC2TBFIBM2hTZUfL3T6BoUNzQED95IJDC50XlSoaSFpBDnpaK+QBoWU0nHWkcWLGXcUX\nZBcZwbjgJUZrAKMFQ4UGMNdlMiqmk9Scyi8tcxcweMiIiWncBU/YJVzPqGqLgHKr6Hes303pBxGmbuCSth5qpi\nGHtZV1HZjgvIug6dcSwYBNGXauOGBaHy09AYgJZbsjmHDsqIhdZcJUGdqYgyzxun2nK",
  "ZV1HZjgvIug6dcSwYBNGXauOGBaHy09AYgJZbsjmHDsqIhdZcJUGdqYgyzxun2nKmLuzUkK50vX2yhR+s+J\nkREVgLNPHRkRPu3q68nMdqbJOa9VaATZwyL1a1CsqiZ1rQTmFUbq7BZ58owcbYglCUXVONxqLSlHUnqALmSVd\nkTITvaLfrEmxZFXZvw1SzgtOjO8s/0clxuaJOG/UHZRMayovU1pAK/4+GArjhmfsLIubiJdxYPAjUi5dwuL4bS",
  "tOjO8s/0clxuaJOG/UHZRMayovU1pAK/4+GArjhmfsLIubiJdxYPAjUi5dwuL4bS0\ncyc2OrSL12UGCcCYvjdOfRaJbp46Yg01iY6wQUO3CkTBhLHIYdmUVUDIc4dZt2UC+MUm/maPk7zIKLr4GfsZI\nrWuLosZUzer7gWVK6F73aB8VgvKcHM4p1dU94yMek0+vaQAcmMZE7Uk5O3FzCKWY7+slb4pWK6Jnm21/MC5\nYncL36dlo01yPCFnY4UZ",
  "k0+vaQAcmMZE7Uk5O3FzCKWY7+slb4pWK6Jnm21/MC5\nYncL36dlo01yPCFnY4UZb8KxkbYsjy9IftDXbru+OrNw8+RFt7cji2k2O2m1Habct7hUjoGdbltFuIQ9Z2OFGW+\n0IsYcsS3/Qlj2PW7ZWFy7yVG70zxabYs7M43tH+6NqSTqMSnhgXrsS7jbhExRYlFaxSmkSE2IVOMi64F/zeVX\nfWw3LWakCkOctbVMCUAsrNKTQhU2xO4a7Zxkx1",
  "RYlFaxSmkSE2IVOMi64F/zeVX\nfWw3LWakCkOctbVMCUAsrNKTQhU2xO4a7Zxkx1y6Ju2VXC07FhNiFTfExic9ZNyBQjLEZW8ZSkqSE2IZTHsZnH\nMc5jakqpTJXJLWsCNpStg2VjZOupAKmNDF6m1g6gxHwRBgdtkFTzvHOy607Txi7WOBdPLR1PLyiY0mMBlXAlLb\nROea429aTzDNTDI9ZtiSnzLBSnMCB6QywM368ISPcnBu7ml5heaHqB6Y",
  "BlXAlLb\nROea429aTzDNTDI9ZtiSnzLBSnMCB6QywM368ISPcnBu7ml5heaHqB6YGmB5hmqI3Ai98oSl6O/HCc03PM\nd3XdB/TQtMC06GmQ0xDTUNMH2n6CFNfUx/TdU3XMZWaoidSuCNouofpWNMxpoeaHmL6UtOXmD7R9AmrzR9hekb\nTd9g+kDTB5gSTQmG5puYEo1RZ8OvHBN0zVMPU3Rux+ca5oOME01TF9qOlDTAN0Vsx3M80RY83cGPUl",
  "mG5puYEo1RZ8OvHBN0zVMPU3Rux+ca5oOME01TF9qOlDTAN0Vsx3M80RY83cGPUlGP6VNO\nnmDJN0fubFz7X9DmsaYxps80fYbpa01fY/pY08eYRpqibwPwdKLpLqb6K1CZY7qj6Q6mZ5qe2b8L0NkyeraNua\n0b2MY0TBdFNT9KYAjxKanqLnyVC0V7Xp1yZ0XQvFjFtYm/FpbZTzUMy4hbVXp2ltdH0KxYyP0dA39mcfUiClc\nKUfLSz1za+wu",
  "Z0XQvFjFtYm/FpbZTzUMy4hbVXp2ltdH0KxYyP0dA39mcfUiClc\nKUfLSz1za+wuLC/utz/efnuzt2l+2vtF9prvW973/du9vq9X3r3e096g96w589N7c+tzX3fPH3xb8W/178p1H\nfm2vrfN3r/Bb/Q8UaBYX</latexit>\nPr(y|\u00b5, \u03c32) =\nDo\nY\nd=1\nPr(yd|\u00b5d, \u03c32)\n=\nDo\nY\nd=1\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yd \u2212 \u00b5d)2\n2\u03c32\n\ufffd\n<latexit sha1_base64=\"",
  "\u03c32)\n=\nDo\nY\nd=1\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yd \u2212 \u00b5d)2\n2\u03c32\n\ufffd\n<latexit sha1_base64=\"\nA2QNqhFYSuqkU4QcQafdVuO218=\">AXJ3iclZhbU9\nw2FICXtP0RtopL3xlMlM0iEMm0mbvmQmgZAbpCyBR\nIMO7ItexVk2cgyLH9gzr9MX3rtI/9Jz2yl1WsIx6M\n8HK+T7djuRrkHNWqJWVf+Y+PCjz/59Npn1z/4suv\np6/8c1ekZUypMw45k8CEhBORN0qJ",
  "7djuRrkHNWqJWVf+Y+PCjz/59Npn1z/4suv\np6/8c1ekZUypMw45k8CEhBORN0qJji9CXlKQBp/vB\nyZrm+2dUFiwTu+oip0cpSQSLWUgUhEbzf3gDecsP4ovf\n/DTIJhUvbg+hMNkCf7kY3a05BcsSclxdbe+7T3w/Fxm\n0aiKHvTr4+rxqMrq2vNjScKqX1d+cSpVdfPmalU1z6\nd5D6nsTq805q3LkbRnbDuB5Fne5uN3WgDdOAL1kyVke\nj+",
  "1d+cSpVdfPmalU1z6\nd5D6nsTq805q3LkbRnbDuB5Fne5uN3WgDdOAL1kyVke\nj+cWV5ZXm5+FCf1pY7E1/g9GN7yI/ysIypUKFnBTFYX\n8lV0cVkYqFnNbX/bKgOQlPSEIPoShISoujqslq7d2ESO\nTFmYR/QnlN9P0aFUmL4iINwEyJGhc20EXOyxV/MtRxU\nReKirCtqO45J7KPL1EXsQkDRW/gAIJYOxeuGYQN4UL\nOR1X9DzMEtTIqLKX13",
  "xU\nReKirCtqO45J7KPL1EXsQkDRW/gAIJYOxeuGYQN4UL\nOR1X9DzMEtTIqLKX13fhqQHNGioqdls6h13XWG4dC8\nSpj9fnurBWmaMreUdRIo+hGrhBoUlcVXU6WbcAoALZME\ncgELaBNnR/YdH2LwibmgCuzLV/VqGmhaAI56WhvkAaF\nnNJx1pDFixl2lF2QPG8m54GVElYBRgqHKi1Bjs5EfVl\nPUnSqZVoWN2D5KIhDZdwJRDwvWMuoYoOYe",
  "2QPG8m54GVElYBRgqHKi1Bjs5EfVl\nPUnSqZVoWN2D5KIhDZdwJRDwvWMuoYoOYeqYcf61bZ\neEXEyTVyWN0OVOmJZu7LrKInzIqKu0QsCzZh0rWaiGV\nxuOREJCWQ5Wl5BNOPR1xq0zYKkMbcyCzoNt3riP23pz\nkcL50vfUKpf+MWBnRATj79JEREdKuvpbNbO8yOWeNrw\nt04o1hsbpViEzaV12ArOaxmpsNrmyTJwtCMnsvGvq0T\nhUmrPuB",
  "bNbO8yOWeNrw\nt04o1hsbpViEzaV12ArOaxmpsNrmyTJwtCMnsvGvq0T\nhUmrPuBHXAPulKyUT8nrbUlGDL6rC/BFOVJaeHd5Z/o\npOjakWfNvoPyiY0VJS5qyEd/h8NRXCTs/cXROzFy7i1e\nBoFi/jcH23lo5Ie2PrSLN2UGCcKYurNOfJaJbp4nYg\n81Sa6wQ0O3CkTBhLXIcd2Ud0DIc4Xbt2EChNcmwnWPI\ns6KUF38rP0MkUbXl0XJ9M2q",
  "a6wQ0O3CkTBhLXIcd2Ud0DIc4Xbt2EChNcmwnWPI\ns6KUF38rP0MkUbXl0XJ9M2qe0HlWuheNyif1YIy3BzO\n6BXVAyujQZvPICtFRKSVzIle0smxXyg4xVxnf7PkbdFp\nJfR0Y9ofjAtWpwxDejrasNcjQRZ2uNUWPB852+LIcvQ\nHbc26/sjqzaOf0RbO3G4bpOjdqejdNsO94oR0Nx2g\n3kYcs7HCrekIsYcsR3/QljuPm65ZOFy3yVG7l3l0",
  "G4bpOjdqejdNsO94oR0Nx2g\n3kYcs7HCrekIsYcsR3/QljuPm65ZOFy3yVG7l3l02g\n53ZlrbP94dU0X0Y1LGI/3Yl3G/DdmiwqJyilKE0tsQ7\naYl0L/m8rO/phuWu1IVscFKyr6YAtRZTbU2hDtiewl\n1zGrPVTYe6VYJz8eW2YZs8SlJ7Vm3IVtMsJg4xROS5\n5bYhlAex3YexziPuS3lLslekdyxImhLuTaUHGdSQdsa\nWL1NnF0BiPgmbA6",
  "ROS5\n5bYhlAex3YexziPuS3lLslekdyxImhLuTaUHGdSQdsa\nWL1NnF0BiPgmbA6nAZtucA7r3DuPGHtYoF38dDV8fCK\njhWxGtQBW9pC5jnbzlPsBOMTxmuZKcM8vKcQIHtjPA\nzuXTXxBX6EkO3scNvcD03NBzTPcN3cdUGoreCIL4laHo\n7SIzw9w3TP0D1MS0NLTIeGDjGNDY0xfWLoE0xDQ0N\nM1wxdw1QZip5I4Y5g6C6mY0PHmB4YeoDp",
  "D1MS0NLTIeGDjGNDY0xfWLoE0xDQ0N\nM1wxdw1QZip5I4Y5g6C6mY0PHmB4YeoDpa0NfY/rM0Ge\nYvjH0DabvDH2H6SNDH2FKDCWYrhu6jik1FH06COJVQ1c\nxDQxF735wrhk6wDQ3NMf0saGPMY0MRW/FcD8zFD3ewI\n3RUI7pc0OfY8oMRe9vQfzS0JeYpoamL4w9AWmbw19i+\nlTQ59imhiKvg3A04mhO5iar0BVgem2oduYnhp6v4uQ\nGfLG",
  "oamL4w9AWmbw19i+\nlTQ59imhiKvg3A04mhO5iar0BVgem2oduYnhp6v4uQ\nGfLGLg25pZpYAvTzNAM0w1D0ZsCPEoYeoKeJ2MxvaqZj\n61bcy4g0zflkb5TwWM+5g06vTZW10fYrFjI/R0Nf3Z\nh9SIKVwpR/NL/btr7C4sHd3uf/z8r3te4sPV6dfaK/1\nvu/90LvV6/fu9x72nvUGvWEvnLsxd3/u4dyjhd8X/lz4\na+HvVv1gblrn217nt/D",
  "vu/90LvV6/fu9x72nvUGvWEvnLsxd3/u4dyjhd8X/lz4\na+HvVv1gblrn217nt/Dvf1f7Elc=</latexit>\nPr(y|f[x, \u03c6], \u03c32) =\nDo\nY\nd=1\n1\np\n2\u21e1\u03c32 exp\n\uf8ff\n\u2212(yd \u2212 fd[x, \u03c6])2\n2\u03c32\n\ufffd\n103",
  "Example 4: multivariate regression\n\u2022 What if the outputs vary in magnitude\n\u2022 E.g., predict weight in kilos and height in meters\n\u2022 One dimension has much bigger numbers than others\n\u2022 Could learn a separate variance for each\u2026\n\u2022 \u2026or rescale before training, and then rescale output in opposite way\n104",
  "Loss functions\n\u2022 Maximum likelihood\n\u2022 Recipe for loss functions\n\u2022 Example 1:  univariate regression\n\u2022 Example 2:  binary classification\n\u2022 Example 3:  multiclass classification\n\u2022 Other types of data\n\u2022 Multiple outputs\n\u2022 Cross entropy\n105",
  "Information Theory and Entropy\n\u2022 Claude Shannon: the \"father of information theory,\" \nwas an American mathematician, electrical engineer, \nand cryptographer\n\u2022 Theory of Communication: In his landmark 1948 \npaper, \"A Mathematical Theory of Communication,\" \nShannon introduced a formal framework for the \ntransmission, processing, and storage of information.\n\u2022 Information Theory: Quantified information, allowing \nfor the measurement of information content in \nmessages, which is crucial for data compression, error \ndetection and correction, and more.\n\u2022 Concept of Information Entropy: introduced entropy \nas a measure of the uncertainty or randomness in a \nset of possible messages, providing a limit on the best \npossible lossless compression of any communication.\n106\n\ud835\udc3b \ud835\udc65 = \u2212 3\n)\n\ud835\udc43 \ud835\udc65  log.(\ud835\udc43 \ud835\udc65 )",
  "Entropy for a Binary Event\n107\n\ud835\udc3b \ud835\udc65 = \u2212 3\n)\n\ud835\udc43 \ud835\udc65  log. \ud835\udc43 \ud835\udc65\n= \u2212\ud835\udc5d log. \ud835\udc5d \u2212 1 \u2212 \ud835\udc5d log.(1 \u2212 \ud835\udc5d)\n\ud835\udc65 \u2208 {0,1}",
  "Cross Entropy \u2013 Concept from Information Theory\n<latexit sha1_base\n64=\"JCaFxAc6FUi5Lq57CEGEpLA6QFk=\">A\nAW4XiclZhJb9w2FICVdEvTzWlRX3oRagRI\ni2ZgF+lyKZDYcTY7tR2vicxKA2lYUxRGomy\nx1bmB/RW9Nqf1F/Qn9Fre+mjpBlG79EoOoA\nj5n2fuDxS1BJkUhR6cfHPK1fevud9+79v\n71Dz786ONP5m58ulekZR7y3TCVaX4QsIJLo\nfiuFlrygyznLAk3w9OVg",
  "ud9+79v\n71Dz786ONP5m58ulekZR7y3TCVaX4QsIJLo\nfiuFlrygyznLAk3w9OVgzfP+V5IVK1o8z\nfpSwWIlIhExD6HiO95MgHVdr65PD0evX2ZH\n/k98XSh9Xt+EQ6fPJy6otjG5dfOX3ZRofmt\nLR4MK/d9q1qjHcwuLvcX659PCUltY8Nrf5v\nGNzwf9QRqWCVc6lKwoDpcWM31UsVyLUPLJ9\nX5Z8IyFJyzmh1BULOHFUVXnY+LfhMjAj9I",
  "QRqWCVc6lKwoDpcWM31UsVyLUPLJ9\nX5Z8IyFJyzmh1BULOHFUVXnY+LfhMjAj9Ic\n/pT26+ibZ1QsKYrzJAzYXpYGaCLnZY6uj\nHo0qorNRchU1DUSl9nfomuf5A5DzU8hwKLM\nwF9NUPhyxnoYpuN5X/CxMk4SpQdVfXt2aV\nP2Ax0JVfFTW0zGZdJ3V2uFQvMxYfrwzq0Vo\nnogLTiqpFVPJQKPJ1XFe3EPA8EBiB4nIFW8\ngDpNfoLIX0IUl",
  "FQvMxYfrwzq0Vo\nnogLTiqpFVPJQKPJ1XFe3EPA8EBiB4nIFW8\ngDpNfoLIX0IUlp8EXDUrqw/GswmpWmkeQ04\n62guiQSGTfNyxVogFU5l0lG1QfP+mbwDXOc\nwCdBUOHM3BdsbUZHqe5mOdJ1VhYriFnKmY1\n03AkEMmzYi6hiqlhFPDjvUztp4xdImLs3q\nruYmgqydvOvonOZFDbpOHUEWLMK4a9URZEn\nYLAYsYZDltnwMA058E3GrQmFVk",
  "q\nruYmgqydvOvonOZFDbpOHUEWLMK4a9URZEn\nYLAYsYZDltnwMA058E3GrQmFVkIW5madBt+3\nMRPDaHGdwvXS91Yqk/5ShjJgAXH3mKJgKeV\ndfSWe2P03Oae2bAh/7Q5is7iksj5thTRuBU\nbWxCTXrXCGTZgtCeXrWNU1vHCrPRHeAJoAv\nujKH3fUN7Zu6BEvWhPvfwFDzUvLD273v+Pi\noWjSXjfmHZBMqKsrMVZEJ/4+KBnB7wusLIn\nj",
  "Zu6BEvWhPvfwFDzUvLD273v+Pi\noWjSXjfmHZBMqKsrMVZEJ/4+KBnB7wusLIn\njyUokmDwL15KUS9nc0dSzHC9tE6rmDglBMC\nn2OLn8Rq+45dQR3Nk1QXyFg6oUjEwpNchR1Z\nRMwMhzhRutYQCEaZNiMZRpUeacbH5oPUOk\n1s2mAtzs+puqNI3X2Dy9lZUIabwym/5PQ\nAZTRo8hmkpRqwHCVzbKZ0/LJfaLjEXFd/Pe\nVN0WnFfLTWtgf9g",
  "ZUIabwym/5PQ\nAZTRo8hmkpRqwHCVzbKZ0/LJfaLjEXFd/Pe\nVN0WnFfLTWtgf9gtkpw5CPjtfwfMTEo5Ed\ncGTjbMuSxHe1DXbLm+2bNq7eXZGnHDtdt\nSlJv20u37XAv6QEfrTt6u048YlFHoraHlK\nPWI72oC53Htdo3C4blOSeqd5dNoOd2ai5R/\ntDLlm5jEplQPz2JfKfhPCoqaidopwmMkNi\nEsJmXgv9jZVvAzaNrNSEsbhaiq5kAlgZ",
  "lm5jEplQPz2JfKfhPCoqaidopwmMkNi\nEsJmXgv9jZVvAzaNrNSEsbhaiq5kAlgZc4\niE0ISw2l3DXbGNYXeo626VyWyIzCaExYcs\nwaNuQliMqRg7xROWZUhsQiSPQ5zHIc1jhqX\nMJeEZyRwzQpaUa0Hlw7QrmQCWxqi1saMx6I\nFMFWqwDWK5oCuvcK48hVaxoqt419Xw7iUNa\n4YqNAEsbZBrzO9vOC+yAKcYHrNcSc4EsjKaw\nE3sbFJn+v",
  "hVaxoqt419Xw7iUNa\n4YqNAEsbZBrzO9vOC+yAKcYHrNcSc4EsjKaw\nE3sbFJn+vQXRBV5kguic0vPKT2z9IzSfUv3\nKc0tJW8EQfTMUvJ2EkSnlp5SumfpHqWlpSW\nlu5buUhpZGlH6wNIHlIaWhpSuWLpCqbaUPJ\nHCHcHSHUqHlg4pPbD0gNLnlj6n9JGljyh9Y\nekLSi8svaD0nqX3KGWMkpXLV2lFtKPh0E\n0bKly5QGlpJ3P7jWLN2kNLM",
  "ljyh9Y\nekLSi8svaD0nqX3KGWMkpXLV2lFtKPh0E\n0bKly5QGlpJ3P7jWLN2kNLM0o/S+pfcpHVhK\n3orhfmYpebyBG6OlktLHlj6mVFhK3t+C6Km\nlTylNLE0ofWLpE0pfWfqK0oeWPqQ0tpR8G4\nCnE0u3KbVfgaqC0i1LtygdWTpyfxfgs2kMX\nAtzw1awQWlqaUrpmqXkTQEeJSw9Ic+TkWp3\ntenXJrKvRWrGHazN+PRskvNIzbiDtbvT9G",
  "wQWlqaUrpmqXkTQEeJSw9Ic+TkWp3\ntenXJrKvRWrGHazN+PRskvNIzbiDtbvT9Gy\nyP0Vqxoek6t7sw8pkFLY6Y/nFpbwV1ha2P\nu2t/R9787WnYW7y+0X2mveF96X3i1vyfvBu+\nqU95zOv85v/V/rqfgN</latexit>s98ja9XS/0/vD+8v72/pkP53+Z/3X+t0a9e\nKL[q||p] =\nZ 1\n\u22121\nq(z) log[q(z)]dz \u2212\nZ 1\n\u22121\nq(z) log[p(z)]dz\nKullback-Leibler Divergence -- a measure",
  "/3X+t0a9e\nKL[q||p] =\nZ 1\n\u22121\nq(z) log[q(z)]dz \u2212\nZ 1\n\u22121\nq(z) log[p(z)]dz\nKullback-Leibler Divergence -- a measure between probability distributions108\nMeasures the difference between two probability distributions: the true distribution of the labels and the \npredicted distribution of the labels by a model.",
  "Cross Entropy \u2013 Concept from Information Theory\nFor discrete distributions, the cross-entropy between two \ndistributions \ud835\udc5d and \ud835\udc5e over the same underlying set of events is \ndefined as:\n\ud835\udc3b \ud835\udc5d, \ud835\udc5e = \u2212\u2211\ud835\udc5d \ud835\udc65 \ud835\udc59\ud835\udc5c\ud835\udc54 \ud835\udc5e(\ud835\udc65)\nHere, \ud835\udc5d(\ud835\udc65) is the true probability of an event \ud835\udc65, and \ud835\udc5e(\ud835\udc65) is the \nestimated probability of the same event according to the model.\nFor instance, in binary classification:\n\ud835\udc3b \ud835\udc5d, \ud835\udc5e = \u2212[\ud835\udc66 log O\ud835\udc66 + 1 \u2212 \ud835\udc66 log(1 \u2212 O\ud835\udc66)\nHere, \ud835\udc66 is the true label (0 or 1), and O\ud835\udc66 is the predicted probability of \nthe class being 1.\n109",
  "Recap\n\u2022 Reconsidered loss functions as fitting a parametric probability model\n\u2022 Introduced Maximum Likelihood criterion for finding parameters to \nmaking the training data most probably under that model\n\u2022 Introduced a 4-step recipe for (1) picking a suitable parametric \nprobability distribution, (2) defining the model to pick one or more of \nthe parameters, (3) training the model and (4) doing inference\n\u2022 Derived loss functions for univariate regression, binary and multiclass \nclassification\n\u2022 Briefly reviewed parametric probability models for other types of data\n\u2022 Discussed how this is the same as Cross Entropy from Information \nTheory\n110",
  "Next up\n\u2022Now let\u2019s find the parameters that give the smallest \nloss\n\u2022 \u00e8 Training the model\n111",
  "Feedback?\n112",
  "Fitting Models\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Loss function\n\u2022 Training dataset of I pairs of input/output examples:\n\u2022 Loss function or cost function measures how bad model is:\nor for short:\n<latexit sha1_base64=\"aYKJ\nls3gxo4H7A6/dQhUn1+H0GU=\">ACFHicbVDLSsNAFJ3UV62vq\nEs3g0UQlJIUTdC0Y3uKtgHNDFMpN26OTBzEQsIR/hxl9x40IR\nty7c+TdO0gjaemCGc8+9l3vcSNGhTSML60N7+wuFRerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhc",
  "Rerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhcCPHEbF9NAioRzGSnL0AyuxfCSHrpfcp05C08OfcJyHVvaf\nmeltcpU6etWoGTngLDELUgUFmo7+afVDHPskJghIXqmEUk7QVx\nSzEhasWJBIoRHaEB6igbIJ8JO8qNSuKeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzC",
  "KeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzCPYpJ1iys\nSIc6p2hXiIOMJS+VhRJpjTJ8+S9lHNPK7Vr+vVxnlhRxnsgF2w\nCgCA=</latexit>D0xwAhrgEjRBC2DwAJ7AC3jVHrVn7U17n5SWtKJnG/yB9vEN+g\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnic",
  "i, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnicbVDLSsNAFJ34rPUVd\nSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc\n+g3u/BsnbRbaemCYwzn3cu89QcaZAtv+NpaWV1bX1isb1c2t7Z\n1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvn",
  "0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvnl063EagesFgodqlOjPy2LmSTaIwe+b\nNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5Yp\nmAzxgLqapjihyh9Pz5hYJ1oJrUhI/VKwpurvjFOVLGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR",
  "wxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR2CM3/yIumc1Z3zeuOuUWtelXFU0CE6RqfIQReoiW5Q\nC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</late\nxit>L [\u03c6]\nReturns a scalar that is smaller \nwhen model maps inputs to \noutputs better\n<latexit sha1_base64=\"vQjZ\nHpuyRPHfcav/lLvqCKGaH6I=\">AWyniclZhb9s2FIDVXbvul\nm5Y",
  "ase64=\"vQjZ\nHpuyRPHfcav/lLvqCKGaH6I=\">AWyniclZhb9s2FIDVXbvul\nm5YXvYiLCgwDJmRDN3lZUCbNG3TpI3TXNs4NSiZktlQlCJRiVPB\nb/tJ+zPb6/ZDdijJZnQO8zADsZnzfeLlkNQtyKQo9MrKX7fe/\n+Dz/6+PYndz797PMvly4+9VhkZ5yA/CVKb5cAKLoXiB1poy\nY+znLMkPwoOFs3/OiC54VI1b6+yvhpwmIlIhEyDaHhwov",
  "/CVKb5cAKLoXiB1poy\nY+znLMkPwoOFs3/OiC54VI1b6+yvhpwmIlIhEyDaHhwovtk0EQ\nZWOxPEiCdFJF04HkTbBybAS0+WGni4PquhK1MYTOH79Xpm2p\nzOshFPNanw4Wld5K/fFpYbUtLHntpz+8+81oMErDMuFKh5IVxc\nnqSqZPK5ZrEUo+vTMoC56x8IzF/ASKiW8OK3qgU/9exAZ+VGaw\n5/Sfh29fkTFkqK4SgIwE6bHBWYm6GInpY5+",
  "8IzF/ASKiW8OK3qgU/9exAZ+VGaw\n5/Sfh29fkTFkqK4SgIwE6bHBWYm6GInpY5+O62EykrNVdg0FJXS\n16lvsuiPRM5DLa+gwMJcQF/9cMxyFmrI9Z2B4pdhmiRMjarB2s\nbuFHLHY6Eqfl7WeZ9Ou85G7XAo3mSsbe7PaxGaJ+IdJ5XUiqnkB\noH06rivbiHgeARI8TkCpeQJ0mP0HkryIK60wCrpqVA6vCfzkl\nVSvNY8hJR3tNChk861jqxY",
  "iHgeARI8TkCpeQJ0mP0HkryIK60wCrpqVA6vCfzkl\nVSvNY8hJR3tNChk861jqxYCqTjrIHiu/f8w3gOodZgK7CD0d\nzsJcxNZ0dp/lE50lVmBhuIWcq5nUTMOSQSTOirqFKeHQsGO9wN\nZLps7axKVZ3dXcRJC1n3cdndO8qFHXqSPIgkUYd606giwJZ4URS\nxhkuS0PYcCJbyJuVSisCrIw+3kadNvOTASvzUkG+6XrbVQk/RcM\nZcQEYPeZX8",
  "S\nxhkuS0PYcCJbyJuVSisCrIw+3kadNvOTASvzUkG+6XrbVQk/RcM\nZcQEYPeZX8FUyLv6ejq3/VlyLmrfFPjEH8NkdQ9hedwMa9YIjK\nqNTalZ5wqZNFsQytPLrml641B5JroDNAG86cpcqOiatlyXYMma8\nGAZhpqXkp/82PuZT06rFbNtzBfJlRUlJmrIhP+HxWN4DqE1xdE\n8OSlEk0eBOrJSyWc39HUsRwvbBOp5w4KQjEp9BXa/iJW3WPq",
  "hP+HxWN4DqE1xdE\n8OSlEk0eBOrJSyWc39HUsRwvbBOp5w4KQjEp9BXa/iJW3WPqCO5\nsmqC+QsDUC79MKDTJUdSVTcDI8AtXVMcCtEgw2aMoUyLMufk5I\nfWM0Rq3ZwWc2EuVt0TqjRC97zB5fwoKMPF4YLfcHiAMho0+QzSU\no1YjpI5MVM6eTMoNGwx1+6vp7wpOq2Yn2+17UG/YHbKMOTnwy08\nHzGxqCNRXAL46xLEsvRHtQ1X67Xe1Ztvf",
  "6vp7wpOq2Yn2+17UG/YHbKMOTnwy08\nHzGxqCNRXAL46xLEsvRHtQ1X67Xe1ZtvfmBLO3Y4bpNSepte+\nm2He4NPeDn247ebhOPWNSRqK62h9QjlqM9qMudx23XKByu25Sk3\nlkenbDnZto+Uf7Y6ZuU1K5cjc9qVy0ISwqKmonWKa8BiJTQiL\nSdm14H+s7Am4eHStJoTFfiG6mglgacQlHkITwmKzhbtmG8PqtkP\ndqtMZmNkNiEsPmEJHnUTw",
  "Am4eHStJoTFfiG6mglgacQlHkITwmKzhbtmG8PqtkP\ndqtMZmNkNiEsPmEJHnUTwmJMxdgpnrEsQ2ITInkc4zyOaR4zLG\nUuCc9I5pgRsqRcCyofp13JBLA0Qa1NHI1BD2SqUINtEMsFXmFc\n+UptIoVXcUHroYPbmhYM1ShCWBph+wxf7Dj3GQBTjHcZrmSnAlk\nZTSBfez0qTO7+wuitzJwTOwpVeUXlp6SemRpUeU5paSJ4Igem\nkpeToJogt",
  "nAlk\nZTSBfez0qTO7+wuitzJwTOwpVeUXlp6SemRpUeU5paSJ4Igem\nkpeToJogtLyg9tPSQ0tLSktIDSw8ojSyNKH1s6WNKQ0tDStctX\nadUW0ruSOGKYOk+pWNLx5QeW3pM6StLX1H61NKnlL629DWl7yx9\nR+lDSx9SyixlG5YukEpt5S8OgiNUvXKA0sJc9+sNcs7VOaWZp\nR+sjSR5SOLCVPxXA9s5Tc3sCF0VJ6alm5QKS8nzWxA9t/Q5p",
  "Jc9+sNcs7VOaWZp\nR+sjSR5SOLCVPxXA9s5Tc3sCF0VJ6alm5QKS8nzWxA9t/Q5pY\nmlCaXPLH1G6VtL31L6xNInlMaWkncDcHdi6R6l9i1QVC6a+kup\neWnrvfC/D5NAauhbljK9ihNLU0pXTLUvKkALcSlp6R+8lItWe\n12dsmcl6L1Jw7WJvx2dEk5Gacwdrz06zo8n5KVJzPiZd3zicv0\niBlMKZfriwtIrfwtLC4U+91V9693fvLz1Ya9/Q",
  "wdrz06zo8n5KVJzPiZd3zicv0\niBlMKZfriwtIrfwtLC4U+91V9693fvLz1Ya9/Q3va+9b7zvdWv\nA2q58BM=</latexit>V+9B95Tr+8deKH3p/e394/37+L2Yr54tVg16nu32mO+9jqfxT/+\nL[\u03c6, f\n\u21e5\nxi, \u03c6], {xi, yi}I\ni=1\n\u21e4",
  "Training\n\u2022 Loss function:\n\u2022 Find the parameters that minimize the loss:\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnicbVDLSsNAFJ34rPUVd\nSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc\n+g3u/BsnbRbaemCYwzn3cu89QcaZAtv+NpaWV1bX1isb1c2t7Z\n1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYw",
  "c2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvnl063EagesFgodqlOjPy2LmSTaIwe+b\nNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5Yp\nmAzxgLqapjihyh9Pz5hYJ1oJrUhI/VKwpurvjFOVLGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEB",
  "LGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR2CM3/yIumc1Z3zeuOuUWtelXFU0CE6RqfIQReoiW5Q\nC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</late\nxit>L [\u03c6]\nReturns a scalar that is smaller \nwhen model maps inputs to \noutputs better\n<latexit sha1_base64=\"uCPd\nlhuvj1KIwIWlqS5H3qCo6ok=\">ACY3icbVBNaxRBEO2dqIk",
  "ha1_base64=\"uCPd\nlhuvj1KIwIWlqS5H3qCo6ok=\">ACY3icbVBNaxRBEO2dqIkbP\n9boTYTGRfC0zEhQL0KIFw8eVnCTwPSw1PT07DTpj6G7JrIM87fy\nXwJe46/wYM9mDrg6Yf71VRVS+vlfQYxzejaO/e/Qf7Bw/Hh4\n8eP3k6eXZ05m3juFhwq6y7yMELJY1YoEQlLmonQOdKnOeXn3v/\nEo4L635jutaZBpWRpaSAwZpOZmzCrBluVWFX+",
  "ELJY1YoEQlLmonQOdKnOeXn3v/\nEo4L635jutaZBpWRpaSAwZpOZmzCrBluVWFX+vwsbqSHf1EGbiV\nlma54zElSky/plt6xpxcVZgtJ9N4Fm9Ad0kykCkZMF9OfrPC8kY\nLg1yB92kS15i14FByJboxa7yogV/CSqSBGtDCZ+3m8o6+CUpBS+\nvCM0g36t8dLWjf7xgqNWDlt71e/J+XNlh+zFp6gaF4XeDykZRt\nLSPkRbSCY5qHQhwJ8OulFf",
  "dLWjf7xgqNWDlt71e/J+XNlh+zFp6gaF4XeDykZRt\nLSPkRbSCY5qHQhwJ8OulFfgGMIe8yM+MGt1mCKdgiya1k/xdYt\nc5oOWjcOcSXb4eySs3ez5P3s+Nvx9OR0CO6AvCSvyVuSkA/khH\nwhc7IgnFyTn+SW/BrdRofRUfTirjQaDT3PyT+IXv0BEmK97w=<\n/latexit>\u02c6\u03c6 = argmin\n\u03c6\n[L[\u03c6]]",
  "Example:  1D Linear regression loss function\n<latexit sha1_base64=\"/hMh896NPSehdG/Bg07J5FjBuSo=\">AW9HiclZhbc9Q2FI\nA3lLY0LTS07z0xdMHdpCJsvQywszkBAgJDQJuUKc7Mhe2Ssiy4vyQaP/0nfOn3t/+lLf0uPbO8Kn6M8dGfCivN9uh1JtdeIkWLy39M3Pto+sf/Lpjc9mP/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVb",
  "/i5q0v525/tZ/FR\nerzPT+WcXrosYxLofheLnLJD5OUs8iT/MA7XdH84JynmYjVbn6Z8OIhUoEwmc5hAZz40j1wuSkTh2vn/kuFkRDUrxqF+dlGvVXTfy4nEZVEdjCFb3XC+Ww+wSgtLVNe5f6vAPJw9cd\n9ZSGZRBuVT91BT6Vd3ItM5gbmFpcan+OLTQbwsLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+Oyz",
  "sLvfazNbj9zdAdxn4RcZX7kmXZUX8pyY9LlubCl7yadYuMJ8w/ZSE/gqJiEc+OyzpFlXMHIkMniFP4U7lTRz+sUbIo01MDM2L5KMNM\nB23sqMiD345LoZIi58pvOgoK6eSxo/PtDEXK/VxeQoH5qYCxOv6IpczPYVmXcUv/DiKmBqW7vLqdlW6Hg+FKvlZUa9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0q",
  "9QVXWd1drhULzKWF7bnbYich6J95w0Uiu6kSsEHlZlyRfDRQw\nEByAWOQGx4hm0qfPjBU4fUdiREnDZbCjYcM7rijStch5CTjraW6JBIZF83LFWiAVLGXWUHVAc546jAc9TWAUYKnxtAY7CVPVpF7Ox3kalZmO4R5SpkJedwFT9pnUM+oaqpASqvod63d\nsvWbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK",
  "WbqtE1cnNRDTXUEWbtp18lTmhc17Dp1BFmwCcOuVUeQJeH6MWQRgy35QFMOHJ0xK4KhVBNuZWGnvdvhMdwXtznMB56XqrJUn/OUMZ0QE4fpbMOXzr4ST21nkpz2tcFPnZGsF\njdKiwNm2lNOoFZtbGKmnWukEmzBaE0vuiaejQWlSeiO0EdwIeuSIUKPtDu1SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4",
  "SXYsjrs3oOpoXkR/cXf+bj43JHxv9D8kmNJQVia0hHf4fDQ3hjoX3F0Tw4sUSLR4E6sWLJVzf0dKxF\nG9sHanXDgpCMSnyS3T8Rai6deoIHmwcobFCQLcL30wotMhB0JV1QMvwDfdeywby0ST9Zo6+jLMi5eTih/YzRGpdXxZToW9W3Quq1EL3usHltBaU4eZwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/G",
  "Zwzq+o7qGMek0+vbhQ5aiZI71k\no5P3CyHI2Y7/fWSN0WrFfKz9bY/GBesTuH7/GywjtcjJBZ1JGoLHnasbUliWfqDtqb9cORlesnP5KtHVpcuylJu+0o7bFvWIE/GzDMtoN4hGLOhK1Y6QesSy9Adt2fO4YZuFxbWb\nkrQ7yaPVtrhTE23/YHfEc6Yfk8wTbRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu",
  "bRPCYk7F3CrGEQ+R2ISwGBVdC/6PlR0BN4+u1YSwuJWJrqYDWBpyiafQhLDYHOGu2cawumFRN+wqk8kImU0Ii89ZhGfdhLAYUjG0iqcsSZDYhEg\neRziPI5rHBEuJTcIrklhWhGwp24ZKR3FX0gEsjVFvY0tnMAIZK9RhG8RyRndeZt15Cu1iRXfxnq3jvSs6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4",
  "6zhlqUAewtEnOmONuWg+Zh1Mj1m2JCcCWQlN4BZ2tqgzefrzgpI8yXnBpaG\nXlF4YekHpgaEHlKaGkl8EXvDaUPLrxAvODT2ndN/QfUoLQwtK9wzdozQwNKD0maHPKPUN9SldMXSF0txQ8kQKdwRDdykdGTqi9NDQ0rfGPqG0heGvqD0raFvKX1v6HtKnxj6hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8M",
  "hFJmK\nN01dBVSrmh5NWBFywbukypZyj57QdnzdAtShNDE0qfGvqU0qGh5Fcx3M8MJY83cGM0VFK6ZugapcJQ8vNC14Z+orSyNCI0peGvqT0naHvKH1u6HNKQ0PJuwF4OjF0h1LzFqjMKN02d\nJvSM0P7O8F+HQZPdvG3DQNbFIaGxpTum4o+aUAjxKGnpLnyUC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfz",
  "UC1V7XJ2yZyXQvUlFtYm/FJbZLzQE25hbVXp0ltcn0K1JSPyNBX96cvUiClcKUfzC308VtYWth/sNj/ZfHh9sOFx8vtG\n9obvW973/Xu9vq9X3uPey96W729nt/7d+b6zM2ZW/Pn83/M/zn/V6Nem2nrfN3rfOb/g9NB/xE</latexit>\nL[\u03c6] =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\nLoss function:\n\u201cLeast squares loss function\u201d",
  "Example: 1D Linear regression training",
  "Example: 1D Linear regression training",
  "Example: 1D Linear regression training",
  "Example: 1D Linear regression training",
  "Example: 1D Linear regression training\nThis technique is known as gradient descent",
  "Fitting models\n\u2022 Math overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "Definitions\n\u2022 derivative\n\u2022 quantifies the sensitivity of change of a function\u2019s output with respect to its \ninput\n\u2022 a function is differentiable at a point a, if the limit\nlim!\u2192#\n$ %&! '$(%)\n!\nexists.\n\u2022 You can approximate the derivative with this limit.\n\u2022 gradient\n\u2022 the degree and direction of steepness of a graph at any point",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4\n\ud835\udf15\ud835\udc66*\n\ud835\udf15\ud835\udc65 !\n= 2 2 \u2212 4 = 0\nAlso slope, m,  of a \ntangential line \nevaluated at that point.\n\ud835\udc5a = 0",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4\n\ud835\udc5a = 2\nWhich direction (+/-) \ndo we have to go \nwhen slope > 0?\n\ud835\udf15\ud835\udc66*\n\ud835\udf15\ud835\udc65 \"\n= 2 3 \u2212 4 = 2",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4\n\ud835\udc5a = 4\nThe \nslope/steepness/gradient \ndepends on where we \nevaluate it",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4\n\ud835\udc5a = \u22122\nWhich direction (+/-) \ndo we have to go \nwhen slope < 0?",
  "<latexit sha1_base64=\"i9n/3Cv2wB79u4ThrLjkI65eg0o=\">AWoniclZhb9s2\nFIDV7tZ1l6Yblpe9CMs6DENrJEV3eRnQJk1vSZerk7RxGlAyJbOhKEWiEruC/8F+zV63P7J/s0NJNqtzmIcZSM2e7xMvh6REK8ikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl7cWbn\n91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEY",
  "91UKRlHvJ+mMo0PwpYwaVQvK+FlvwoyzlLAskPg7M1w8veF6IVO3rScZPEhYrEYmQaQidLvwiHIWVoOM5Vow6U+mtjye+r/798f+Pf/B6cLScm+5/vi0sNIWlrz2s316+5vh\nYJiGZcKVDiUriuOV5UyfVKbqUPLpzUFZ8IyFZyzmx1BULOHFSVUPaOrfgcjQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8",
  "jQj9Ic/pT26+j7V1QsKYpJEoCZMD0qMDNBFzsudfTbSVUVmquwqahqJS+Tn2THX8och5qOYEC3\nMBfXDEYMacjhzYHil2GaJEwNq8Hq+g7kKuCxUBU/L+t8TqdZ712OBSvMlaf789rEZon4h0nldSKqeQKgcfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIArRwIGHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc",
  "GHqRj6Fzk705J1\nUrzGHLS0V4TDQqZ5OtUYsmMqko+yB4vt3fAO4zmEWoKvwxdEc7GVMTWfXaT7WeVIVJoZbyJmKed0EDmEtb2LDVKCZeGHesPbO0ydYmLs3qruYmgqz9vOvonOZFDbtOHUE\nWLMK4a9URZEnY7UOWMhyWz6FASe+ibhVobAqyMLcztOg23ZmInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0Zg",
  "mInhtjPYL1vSLpv2AoIyYAu898C6ZC3tX0rntz5JzUfumwMf+CarewnL42ZYs0ZgVG1sSs06V8ik2YJQn\nl52TdMbh8oz0R2gCeBNV+ZCRe9pd+sSLFkTHtyFoeal5Mf3ej/z8Um1bLaN+YdkEyoqysxVkQn/j4qG8HzB6wsiePJSiSYPAvXkpRLu72jqWI4XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjT",
  "XtonUcwcFoZgUeoK2v4hV95o\n6gjubJqivED1wjcTCk1yFHVlEzAyfMOT0rGAQjTIsBljKNOizDm5+aH1DJFaN7fFXJiHVfeGKo3QvW9wOb8KyvBwuOBXB6gjAZNPoO0VEOWo2SOzZSO3wKDVvMtfvrKW+KTi\nvm5xte9AvmJ0yDPn56Qaej5hY1JGoLjiaOuSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+",
  "uSxHK0B3XNl+v7Pas23vxElnbscN2mJPW2vXTbDveKHvDzTUdvN4lHLOpIVFfbQ+oRy9Ee1OXO46ZrFA7XbUpS7yPTtvhzk20\n/KP9EdfMHJNSOTHvlQOmhAWNRW1U0wTHiOxCWExKbsW/B8rewIeHl2rCWFxuxBdzQSwNOQSD6EJYbHZwl2zjWF106FulUmsxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZW",
  "sxEymxAWn7IEj7oJYTGmYuwUz1iWIbEJkTyOcB\n5HNI8ZljKXhGckc8wIWVKuBZWP0q5kAlgao9bGjsagBzJVqME2iOWCrzCufIUWsWKruK+q+H+FQ1rhio0ASxtkT3mD7acmyzAKYZjlivJmUBWRhO4jZ1t6sxOf0FUkZNcE0s\nnVB6aeklpYeWHlKaW0p+EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI",
  "EQTRrqXk10kQXVh6QemBpQeUlpaWlPYt7VMaWRpR+sTSJ5SGloaUrlm6Rqm2lJxI4Ylg6T6lI0tHlB5ZekTpK0tfUfrM0meUvrb0NaXvLH1H6SNLH1H\nKLGWUrlu6Tim3lLw6CKJVS1cpDSwlv/1gr1m6TWlmaUbpY0sfUzq0lPwqhueZpeR4Aw9GSyWlzy19TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBa",
  "9TqmwlPx+C6KXlr6kNLE0ofSFpS8ofWvpW0qfWvqU0thS8m4ATieW7lFq3\nwJVBaU7lu5Qem7pufu9AJ9PY+BamFu2gi1KU0tTSjcsJb8U4Ch6Rk5T0aqvavN3jaR+1qk5tzB2ozPriY5j9ScO1h7d5pdTe5PkZrzEen6+sH8RQqkFO70pwtLK/gtLC0c3O+\nt/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqN",
  "t/NJ7sPNg6eFq+4b2hvet953o7fi/eo9J5217fC70/vb+8v71/Fr9fLG4s7jXqNevtd87XU+i4P/AC6H3T0=</latexit>@y\n@x = 2x \u2212 4\n\ud835\udc5a = \u22124\nThe \nslope/steepness/gradient \ndepends on where we \nevaluate it",
  "Gradient\nPartial derivative, e.g. rate of \nchange,  w.r.t. each input \n(independent) variable.\nGeometric Interpretation: Each variable is a unit \nvector, and then\n\u2022 gradient is the rate of change (increase) in the \ndirection of each unit vector\n\u2022 vector sum points to the overall direction of \ngreatest change (increase)",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "Gradient descent algorithm\nAlso notated as \u2207!\ud835\udc3f",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "Gradient descent\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\n<latexit sha1_base64=\"pLx+qdAbuYITIgjVh9BEOr/dc=\">AXR3iclZhbU9tGFIBNrym9kXZKH/qiKZNO2iYeO5NeXjKTQEhCIAXCN\nUHgruSVvG1ErqAiUY/sNf0F/Rt04fe1ayveic5aGeAa/P92kvZ3d18xIpsrzX+2vunXfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKc",
  "Xfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKcrmh+c8zQTsdrNLxN+HLFQiUD4LIfQYOFPZ+PI9YJkJI4d5QlzOdlr3vPj6oH6LebFdGgFA/61Um5Vjkul\nxJ+Vs53ICIkeZDfdiMvHpdBdTQeiDuTJu5e6kPcVISj/PuT8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz",
  "T8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz8augOY7+IuMp9ybLsqN9L8uOSpbnwJa/m3SLjkIJTFvIjKCoW8ey4rLNeObcgMnSCOIU/lTt19OoRJYuy7DLywIxYPsow0EbOyry4NfjU\nqikyLnym4aCQjp57OgpdIYi5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmr",
  "i5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmrtcCheZyv7c5qETmPxFtOKqkVXck1Ag+rsuTdsIuB4ABElxMQK5BnTo/XuD0EYVFLgGXzTKD9eW8rEjVKuch5KSlvSYaFBLJxy1rhVgwlVFL2QHFcW45GvA8hVmArsIXR\n3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzU",
  "3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzUXU1BFm7advJU5oXNWw7dQRZsAjDtlVHkCXhlDRkEYMsT8oDGHDk6IhdFQqrgizMrT2m0nOoLX5jiB/dL2VkuS/nOGMqIDsPv0t2DK5219JZ7ZzjQ57WvC3zsjGCy2oewNGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsH",
  "NGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsHhpoWkh/d7f7Ex8dlT28b/Y9kEyrKisRWkQ7/j4qGcBHE6wsiePJiSYPAvXkxRLO72jqWIoXto7UcwcFoZgU+SXa/iJU7WPqCO5sHKG+QkDXC9MKDTJQdCWdUDL8A2Xc8sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHM",
  "sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHMuo1+fTiQg1ZipI51lM6PnGzHLaYbfXU94UrVbIz9Yn7UG/YHYK3+dng3U8HyGxqCNRXD/ZK1LEsvSHtQ1W65Xe1aun/xAlnZoce2mJPVOem3Le41PeBnG5bebhCPWNSRqK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlh",
  "qK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlhMa\ndibhXjiIdIbEJYjIq2Bb+xsiPg4tG2mhAWtzLR1nQAS0Mu8RCaEBabLdw2JzGsbljUDbvKZDJCZhPC4lMW4VE3ISyGVAyt4ilLEiQ2IZLHEc7jiOYxwVJik/CMJYZIUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxV",
  "IUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxVqANY2iR7zHE3rZvMwymG2yxbkh\nOBrIQmcAs7W9SZ3v15QUnu5Lzg0tBLSi8MvaD0wNADSlNDyROBF7w0lDydeMG5oeU7hu6T2lhaEHpnqF7lAaGBpQ+MfQJpb6hPqUrhq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoM",
  "hq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoMZSuGrpKTeUvDrwgmVDlyn1DCXPfrDXDN2iNDE0ofSxoY8pHR\npKnorhemYoub2BC6OhktI1Q9coFYaS5zcveGHoC0ojQyNKnxv6nNI3hr6h9KmhTykNDSXvBuDuxNAdSs1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1a",
  "s1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1aZvm8h5LVAzbmGTjE+PJjkP1Ixb2OTsND2anJ8CNeMj0vXV/dmLFEgpnOkHC0t9/BaWFvbvdf\ns/d+9v3196uDx5Q3uj803n287tTr/zS+dh51lnq7PX8ef6cwdzv8+xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i",
  "xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2",
  "Gradient descent\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\n<latexit sha1_base64=\"pLx+qdAbuYITIgjVh9BEOr/dc=\">AXR3iclZhbU9tGFIBNrym9kXZKH/qiKZNO2iYeO5NeXjKTQEhCIAXCN\nUHgruSVvG1ErqAiUY/sNf0F/Rt04fe1ayveic5aGeAa/P92kvZ3d18xIpsrzX+2vunXfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKc",
  "Xfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKcrmh+c8zQTsdrNLxN+HLFQiUD4LIfQYOFPZ+PI9YJkJI4d5QlzOdlr3vPj6oH6LebFdGgFA/61Um5Vjkul\nxJ+Vs53ICIkeZDfdiMvHpdBdTQeiDuTJu5e6kPcVISj/PuT8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz",
  "T8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz8augOY7+IuMp9ybLsqN9L8uOSpbnwJa/m3SLjkIJTFvIjKCoW8ey4rLNeObcgMnSCOIU/lTt19OoRJYuy7DLywIxYPsow0EbOyry4NfjU\nqikyLnym4aCQjp57OgpdIYi5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmr",
  "i5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmrtcCheZyv7c5qETmPxFtOKqkVXck1Ag+rsuTdsIuB4ABElxMQK5BnTo/XuD0EYVFLgGXzTKD9eW8rEjVKuch5KSlvSYaFBLJxy1rhVgwlVFL2QHFcW45GvA8hVmArsIXR\n3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzU",
  "3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzUXU1BFm7advJU5oXNWw7dQRZsAjDtlVHkCXhlDRkEYMsT8oDGHDk6IhdFQqrgizMrT2m0nOoLX5jiB/dL2VkuS/nOGMqIDsPv0t2DK5219JZ7ZzjQ57WvC3zsjGCy2oewNGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsH",
  "NGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsHhpoWkh/d7f7Ex8dlT28b/Y9kEyrKisRWkQ7/j4qGcBHE6wsiePJiSYPAvXkxRLO72jqWIoXto7UcwcFoZgU+SXa/iJU7WPqCO5sHKG+QkDXC9MKDTJQdCWdUDL8A2Xc8sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHM",
  "sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHMuo1+fTiQg1ZipI51lM6PnGzHLaYbfXU94UrVbIz9Yn7UG/YHYK3+dng3U8HyGxqCNRXD/ZK1LEsvSHtQ1W65Xe1aun/xAlnZoce2mJPVOem3Le41PeBnG5bebhCPWNSRqK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlh",
  "qK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlhMa\ndibhXjiIdIbEJYjIq2Bb+xsiPg4tG2mhAWtzLR1nQAS0Mu8RCaEBabLdw2JzGsbljUDbvKZDJCZhPC4lMW4VE3ISyGVAyt4ilLEiQ2IZLHEc7jiOYxwVJik/CMJYZIUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxV",
  "IUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxVqANY2iR7zHE3rZvMwymG2yxbkh\nOBrIQmcAs7W9SZ3v15QUnu5Lzg0tBLSi8MvaD0wNADSlNDyROBF7w0lDydeMG5oeU7hu6T2lhaEHpnqF7lAaGBpQ+MfQJpb6hPqUrhq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoM",
  "hq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoMZSuGrpKTeUvDrwgmVDlyn1DCXPfrDXDN2iNDE0ofSxoY8pHR\npKnorhemYoub2BC6OhktI1Q9coFYaS5zcveGHoC0ojQyNKnxv6nNI3hr6h9KmhTykNDSXvBuDuxNAdSs1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1a",
  "s1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1aZvm8h5LVAzbmGTjE+PJjkP1Ixb2OTsND2anJ8CNeMj0vXV/dmLFEgpnOkHC0t9/BaWFvbvdf\ns/d+9v3196uDx5Q3uj803n287tTr/zS+dh51lnq7PX8ef6cwdzv8+xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i",
  "xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5",
  "lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXui",
  "NOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkV",
  "MV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf",
  "+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ",
  "VL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF",
  "SY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6R",
  "x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Nd",
  "uqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGl",
  "SwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLS",
  "K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 =",
  "7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6",
  "Gradient descent\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\n<latexit sha1_base64=\"pLx+qdAbuYITIgjVh9BEOr/dc=\">AXR3iclZhbU9tGFIBNrym9kXZKH/qiKZNO2iYeO5NeXjKTQEhCIAXCN\nUHgruSVvG1ErqAiUY/sNf0F/Rt04fe1ayveic5aGeAa/P92kvZ3d18xIpsrzX+2vunXfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKc",
  "Xfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKcrmh+c8zQTsdrNLxN+HLFQiUD4LIfQYOFPZ+PI9YJkJI4d5QlzOdlr3vPj6oH6LebFdGgFA/61Um5Vjkul\nxJ+Vs53ICIkeZDfdiMvHpdBdTQeiDuTJu5e6kPcVISj/PuT8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz",
  "T8l7lqlgVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz8augOY7+IuMp9ybLsqN9L8uOSpbnwJa/m3SLjkIJTFvIjKCoW8ey4rLNeObcgMnSCOIU/lTt19OoRJYuy7DLywIxYPsow0EbOyry4NfjU\nqikyLnym4aCQjp57OgpdIYi5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmr",
  "i5X4uL6HA/FRAXx1/xFLm5zDR867iF34cRUwNS3d5dbsqXY+HQpX8rKgnvarazmrtcCheZyv7c5qETmPxFtOKqkVXck1Ag+rsuTdsIuB4ABElxMQK5BnTo/XuD0EYVFLgGXzTKD9eW8rEjVKuch5KSlvSYaFBLJxy1rhVgwlVFL2QHFcW45GvA8hVmArsIXR\n3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzU",
  "3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4ydTpJXJzUXU1BFm7advJU5oXNWw7dQRZsAjDtlVHkCXhlDRkEYMsT8oDGHDk6IhdFQqrgizMrT2m0nOoLX5jiB/dL2VkuS/nOGMqIDsPv0t2DK5219JZ7ZzjQ57WvC3zsjGCy2oewNGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsH",
  "NGyGNW0ERjWJVdSsc4VMm\ni0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsHhpoWkh/d7f7Ex8dlT28b/Y9kEyrKisRWkQ7/j4qGcBHE6wsiePJiSYPAvXkxRLO72jqWIoXto7UcwcFoZgU+SXa/iJU7WPqCO5sHKG+QkDXC9MKDTJQdCWdUDL8A2Xc8sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHM",
  "sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7\nzB5ewoKMPF4Zxfc7iHMuo1+fTiQg1ZipI51lM6PnGzHLaYbfXU94UrVbIz9Yn7UG/YHYK3+dng3U8HyGxqCNRXD/ZK1LEsvSHtQ1W65Xe1aun/xAlnZoce2mJPVOem3Le41PeBnG5bebhCPWNSRqK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlh",
  "qK5JD6lHLEt7UJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlhMa\ndibhXjiIdIbEJYjIq2Bb+xsiPg4tG2mhAWtzLR1nQAS0Mu8RCaEBabLdw2JzGsbljUDbvKZDJCZhPC4lMW4VE3ISyGVAyt4ilLEiQ2IZLHEc7jiOYxwVJik/CMJYZIUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxV",
  "IUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5oysvs648hVaxoqt4z9bw3jUN5wxVqANY2iR7zHE3rZvMwymG2yxbkh\nOBrIQmcAs7W9SZ3v15QUnu5Lzg0tBLSi8MvaD0wNADSlNDyROBF7w0lDydeMG5oeU7hu6T2lhaEHpnqF7lAaGBpQ+MfQJpb6hPqUrhq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoM",
  "hq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0MfUcoMZSuGrpKTeUvDrwgmVDlyn1DCXPfrDXDN2iNDE0ofSxoY8pHR\npKnorhemYoub2BC6OhktI1Q9coFYaS5zcveGHoC0ojQyNKnxv6nNI3hr6h9KmhTykNDSXvBuDuxNAdSs1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1a",
  "s1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1aZvm8h5LVAzbmGTjE+PJjkP1Ixb2OTsND2anJ8CNeMj0vXV/dmLFEgpnOkHC0t9/BaWFvbvdf\ns/d+9v3196uDx5Q3uj803n287tTr/zS+dh51lnq7PX8ef6cwdzv8+xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i",
  "xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5",
  "lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXui",
  "NOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkV",
  "MV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf",
  "+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ",
  "VL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF",
  "SY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6R",
  "x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Nd",
  "uqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGl",
  "SwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLS",
  "K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 =",
  "7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3",
  "sdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1m",
  "S9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbx",
  "ZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/o",
  "OEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7",
  "xKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4",
  "fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8Y",
  "os6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0h",
  "Tw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUN",
  "0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5S",
  "RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rn",
  "FtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "Step 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\nGradient descent\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/",
  "/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZE",
  "Gtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsD",
  "IrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt",
  "xEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5",
  "qFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw",
  "6n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt",
  "tFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW",
  "8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh2",
  "4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4P",
  "ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbt",
  "sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LIT",
  "Eh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9",
  "l/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC",
  "jKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiy",
  "3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIb",
  "qLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/m",
  "U6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw8",
  "dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISm",
  "0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuU",
  "KX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJw",
  "/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212",
  "W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "<latexit sha1_base64=\"WsHIJzX3fm90AkOs9UWotK6nHbU=\">AWw3\niclZhb9s2FIDV7tZ1t3TD8rIXYUGBYeiMZOguj21SN2TLs7FSdo4DSiZktlQlCJRiVPBP2m/Zk8Dtv+yQ0k2q3OYhwXozJ3vEy+HFEUpyKQo9Orq\n37duf/DhRx9/cufTu59/sWXy3d+/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEK",
  "/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEKmIXS2tOmPgibiJFMVSx\n5pFmep1dNzP/JHzGZTZg/inIWVqOM5Vow6W/PbLlRZ2dLK6u91frPp4W1trDitX+Ds3vfjkfjNCwTrnQoWVGcrK1m+rQy1YaSz+6OyoJnLDxnMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj",
  "nMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj+thMpKzVXYNBSV0tepb9Lnj0XOQy2vocDCXEBf/XDCIDsaknx3pPhVmCYJ\nU+NqtN7fhTwFPBaq4hdlnfDZrOv0a4dD8SZj/fnBohaheSLecVJrZhKbhB4PKsq3ot7GAgOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxI",
  "gOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxIKpTDrKPi+f983gOscZgG6Cj8czcF+xtRsfp3mU50nVWFiuIWcqZjXTcCQ1jXe9hQpZRwadix/sDWHlPnbeLSrO5qbiLIO\nsi7js5pXtS469QRZMEijLtWHUGWhO1gzBIGW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7",
  "GW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7+ka6sP15ci5r3xT\n41J/AZHUvYXncDGveCIyqjc2oWecKmTRbEILNsmua3jhUnonuAE0A3RlLlT0nvagLsGSNeHRAxhqXkp+8lPvFz49rVbNbWP+Q7IJFRVl5qrIhP9HRW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5sm",
  "RW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5smqK8QMPXCLxMKTXIUdWUTMDL8wqPUsYBCNMiwGWMo06LMOdn80HqG\nSK2bTEX5mHV3VClEbr7BpeLq6AMD4dLfsPlAcpo0OQzSEs1ZjlK5tRM6fTNqIBHvfPur6e8KTqtmF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1",
  "mF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1m5LU2/bSbTvcG3rAL7Ydvd0mHrGoI1FdbQ+pRyxHe1CXO4/brlE4XLcpSb3zPDpth7sw0fKPDiZcM3NMSuXYHPtSO\nWpCWNRU1E4xTXiMxCaExaTsWvD/WNkX8PDoWk0Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOs",
  "Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOsgyJTYjkcYLz\nOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2AdzXUYBvEckFXuFceQqtYkVX8dDV8PCGhjVDFZoAlnbIPeaPdpw3WYBTDMcsV5IzgayMJn\nCAnQF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0",
  "QF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0tKh5YOKY0sjSh9aulTSkNLQ0o3LN2gVFtKTqTwRLD0gNKJp\nRNKjy09pvSVpa8ofWbpM0pfW/qa0neWvqP0saWPKWMkr7lvYp5ZaSTwdBtG7pOqWBpeTdD+41SweUZpZmlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPS",
  "mlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPStpW8p3bR0k9LYUvJtAE4nlu5Tar8CVQWlu5buUnph6YX7uwBfTGPgWpg7toIdSlNLU0q3LCVvCnCUsPScnC\nDbyhF3p/en95/3j/LveXz5fzZd2ot2+13zjdf6WZ/8B8XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+",
  "XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+4eLDymQUtjpz5ZW1vBXWFo4/Lm39mv4e7DlUfr7RfaO953vfeD96a95v3yHvm\n\u03c6  \u2212 \u03c6 \u2212 \u21b5@L\n@\u03c6\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\nStep 2:  Update parameters according to rule\n\ud835\udefc = step size or learning rate if fixed\nGradient descent\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm",
  "ha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJg",
  "86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOm",
  "5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGV",
  "LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7j",
  "An3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0",
  "VZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR0",
  "NdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUt",
  "zu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSv",
  "dnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1",
  "aWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsP",
  "3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBj",
  "PrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhc",
  "fpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxW",
  "lNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp",
  "2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2F",
  "dNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8",
  "VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9f",
  "uDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh",
  "Ec6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtD",
  "Nbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8",
  "GhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZn",
  "av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "<latexit sha1_base64=\"WsHIJzX3fm90AkOs9UWotK6nHbU=\">AWw3\niclZhb9s2FIDV7tZ1t3TD8rIXYUGBYeiMZOguj21SN2TLs7FSdo4DSiZktlQlCJRiVPBP2m/Zk8Dtv+yQ0k2q3OYhwXozJ3vEy+HFEUpyKQo9Orq\n37duf/DhRx9/cufTu59/sWXy3d+/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEK",
  "/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEKmIXS2tOmPgibiJFMVSx\n5pFmep1dNzP/JHzGZTZg/inIWVqOM5Vow6W/PbLlRZ2dLK6u91frPp4W1trDitX+Ds3vfjkfjNCwTrnQoWVGcrK1m+rQy1YaSz+6OyoJnLDxnMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj",
  "nMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj+thMpKzVXYNBSV0tepb9Lnj0XOQy2vocDCXEBf/XDCIDsaknx3pPhVmCYJ\nU+NqtN7fhTwFPBaq4hdlnfDZrOv0a4dD8SZj/fnBohaheSLecVJrZhKbhB4PKsq3ot7GAgOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxI",
  "gOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxIKpTDrKPi+f983gOscZgG6Cj8czcF+xtRsfp3mU50nVWFiuIWcqZjXTcCQ1jXe9hQpZRwadix/sDWHlPnbeLSrO5qbiLIO\nsi7js5pXtS469QRZMEijLtWHUGWhO1gzBIGW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7",
  "GW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7+ka6sP15ci5r3xT\n41J/AZHUvYXncDGveCIyqjc2oWecKmTRbEILNsmua3jhUnonuAE0A3RlLlT0nvagLsGSNeHRAxhqXkp+8lPvFz49rVbNbWP+Q7IJFRVl5qrIhP9HRW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5sm",
  "RW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5smqK8QMPXCLxMKTXIUdWUTMDL8wqPUsYBCNMiwGWMo06LMOdn80HqG\nSK2bTEX5mHV3VClEbr7BpeLq6AMD4dLfsPlAcpo0OQzSEs1ZjlK5tRM6fTNqIBHvfPur6e8KTqtmF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1",
  "mF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1m5LU2/bSbTvcG3rAL7Ydvd0mHrGoI1FdbQ+pRyxHe1CXO4/brlE4XLcpSb3zPDpth7sw0fKPDiZcM3NMSuXYHPtSO\nWpCWNRU1E4xTXiMxCaExaTsWvD/WNkX8PDoWk0Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOs",
  "Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOsgyJTYjkcYLz\nOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2AdzXUYBvEckFXuFceQqtYkVX8dDV8PCGhjVDFZoAlnbIPeaPdpw3WYBTDMcsV5IzgayMJn\nCAnQF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0",
  "QF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0tKh5YOKY0sjSh9aulTSkNLQ0o3LN2gVFtKTqTwRLD0gNKJp\nRNKjy09pvSVpa8ofWbpM0pfW/qa0neWvqP0saWPKWMkr7lvYp5ZaSTwdBtG7pOqWBpeTdD+41SweUZpZmlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPS",
  "mlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPStpW8p3bR0k9LYUvJtAE4nlu5Tar8CVQWlu5buUnph6YX7uwBfTGPgWpg7toIdSlNLU0q3LCVvCnCUsPScnC\nDbyhF3p/en95/3j/LveXz5fzZd2ot2+13zjdf6WZ/8B8XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+",
  "XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+4eLDymQUtjpz5ZW1vBXWFo4/Lm39mv4e7DlUfr7RfaO953vfeD96a95v3yHvm\n\u03c6  \u2212 \u03c6 \u2212 \u21b5@L\n@\u03c6\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\nStep 2:  Update parameters according to rule\n\ud835\udefc = step size\nGradient descent\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvY",
  "ase64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDx",
  "t4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62",
  "yLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1",
  "mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqm",
  "GLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1N",
  "o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcp",
  "Xx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYh",
  "mQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ik",
  "9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486",
  "1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyV",
  "cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIg",
  "lDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEpt",
  "EWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1L",
  "Q/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFT",
  "trlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCr",
  "k2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV",
  "A67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+",
  "SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2",
  "+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N",
  "rOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDze",
  "hjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOA",
  "OfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "Gradient descent",
  "<latexit sha1_base64=\"WsHIJzX3fm90AkOs9UWotK6nHbU=\">AWw3\niclZhb9s2FIDV7tZ1t3TD8rIXYUGBYeiMZOguj21SN2TLs7FSdo4DSiZktlQlCJRiVPBP2m/Zk8Dtv+yQ0k2q3OYhwXozJ3vEy+HFEUpyKQo9Orq\n37duf/DhRx9/cufTu59/sWXy3d+/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEK",
  "/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEKmIXS2tOmPgibiJFMVSx\n5pFmep1dNzP/JHzGZTZg/inIWVqOM5Vow6W/PbLlRZ2dLK6u91frPp4W1trDitX+Ds3vfjkfjNCwTrnQoWVGcrK1m+rQy1YaSz+6OyoJnLDxnMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj",
  "nMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj+thMpKzVXYNBSV0tepb9Lnj0XOQy2vocDCXEBf/XDCIDsaknx3pPhVmCYJ\nU+NqtN7fhTwFPBaq4hdlnfDZrOv0a4dD8SZj/fnBohaheSLecVJrZhKbhB4PKsq3ot7GAgOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxI",
  "gOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxIKpTDrKPi+f983gOscZgG6Cj8czcF+xtRsfp3mU50nVWFiuIWcqZjXTcCQ1jXe9hQpZRwadix/sDWHlPnbeLSrO5qbiLIO\nsi7js5pXtS469QRZMEijLtWHUGWhO1gzBIGW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7",
  "GW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7+ka6sP15ci5r3xT\n41J/AZHUvYXncDGveCIyqjc2oWecKmTRbEILNsmua3jhUnonuAE0A3RlLlT0nvagLsGSNeHRAxhqXkp+8lPvFz49rVbNbWP+Q7IJFRVl5qrIhP9HRW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5sm",
  "RW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5smqK8QMPXCLxMKTXIUdWUTMDL8wqPUsYBCNMiwGWMo06LMOdn80HqG\nSK2bTEX5mHV3VClEbr7BpeLq6AMD4dLfsPlAcpo0OQzSEs1ZjlK5tRM6fTNqIBHvfPur6e8KTqtmF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1",
  "mF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1m5LU2/bSbTvcG3rAL7Ydvd0mHrGoI1FdbQ+pRyxHe1CXO4/brlE4XLcpSb3zPDpth7sw0fKPDiZcM3NMSuXYHPtSO\nWpCWNRU1E4xTXiMxCaExaTsWvD/WNkX8PDoWk0Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOs",
  "Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOsgyJTYjkcYLz\nOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2AdzXUYBvEckFXuFceQqtYkVX8dDV8PCGhjVDFZoAlnbIPeaPdpw3WYBTDMcsV5IzgayMJn\nCAnQF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0",
  "QF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0tKh5YOKY0sjSh9aulTSkNLQ0o3LN2gVFtKTqTwRLD0gNKJp\nRNKjy09pvSVpa8ofWbpM0pfW/qa0neWvqP0saWPKWMkr7lvYp5ZaSTwdBtG7pOqWBpeTdD+41SweUZpZmlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPS",
  "mlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPStpW8p3bR0k9LYUvJtAE4nlu5Tar8CVQWlu5buUnph6YX7uwBfTGPgWpg7toIdSlNLU0q3LCVvCnCUsPScnC\nDbyhF3p/en95/3j/LveXz5fzZd2ot2+13zjdf6WZ/8B8XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+",
  "XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+4eLDymQUtjpz5ZW1vBXWFo4/Lm39mv4e7DlUfr7RfaO953vfeD96a95v3yHvm\n\u03c6  \u2212 \u03c6 \u2212 \u21b5@L\n@\u03c6\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\nStep 2:  Update parameters according to rule\n\ud835\udefc = step size\nGradient descent\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvY",
  "ase64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDx",
  "t4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62",
  "yLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1",
  "mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqm",
  "GLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1N",
  "o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcp",
  "Xx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYh",
  "mQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ik",
  "9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486",
  "1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyV",
  "cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIg",
  "lDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEpt",
  "EWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1L",
  "Q/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFT",
  "trlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCr",
  "k2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV",
  "A67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+",
  "SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2",
  "+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N",
  "rOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDze",
  "hjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOA",
  "OfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "<latexit sha1_base64=\"WsHIJzX3fm90AkOs9UWotK6nHbU=\">AWw3\niclZhb9s2FIDV7tZ1t3TD8rIXYUGBYeiMZOguj21SN2TLs7FSdo4DSiZktlQlCJRiVPBP2m/Zk8Dtv+yQ0k2q3OYhwXozJ3vEy+HFEUpyKQo9Orq\n37duf/DhRx9/cufTu59/sWXy3d+/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEK",
  "/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEKmIXS2tOmPgibiJFMVSx\n5pFmep1dNzP/JHzGZTZg/inIWVqOM5Vow6W/PbLlRZ2dLK6u91frPp4W1trDitX+Ds3vfjkfjNCwTrnQoWVGcrK1m+rQy1YaSz+6OyoJnLDxnMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj",
  "nMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj+thMpKzVXYNBSV0tepb9Lnj0XOQy2vocDCXEBf/XDCIDsaknx3pPhVmCYJ\nU+NqtN7fhTwFPBaq4hdlnfDZrOv0a4dD8SZj/fnBohaheSLecVJrZhKbhB4PKsq3ot7GAgOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxI",
  "gOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxIKpTDrKPi+f983gOscZgG6Cj8czcF+xtRsfp3mU50nVWFiuIWcqZjXTcCQ1jXe9hQpZRwadix/sDWHlPnbeLSrO5qbiLIO\nsi7js5pXtS469QRZMEijLtWHUGWhO1gzBIGW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7",
  "GW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7+ka6sP15ci5r3xT\n41J/AZHUvYXncDGveCIyqjc2oWecKmTRbEILNsmua3jhUnonuAE0A3RlLlT0nvagLsGSNeHRAxhqXkp+8lPvFz49rVbNbWP+Q7IJFRVl5qrIhP9HRW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5sm",
  "RW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5smqK8QMPXCLxMKTXIUdWUTMDL8wqPUsYBCNMiwGWMo06LMOdn80HqG\nSK2bTEX5mHV3VClEbr7BpeLq6AMD4dLfsPlAcpo0OQzSEs1ZjlK5tRM6fTNqIBHvfPur6e8KTqtmF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1",
  "mF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1m5LU2/bSbTvcG3rAL7Ydvd0mHrGoI1FdbQ+pRyxHe1CXO4/brlE4XLcpSb3zPDpth7sw0fKPDiZcM3NMSuXYHPtSO\nWpCWNRU1E4xTXiMxCaExaTsWvD/WNkX8PDoWk0Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOs",
  "Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOsgyJTYjkcYLz\nOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2AdzXUYBvEckFXuFceQqtYkVX8dDV8PCGhjVDFZoAlnbIPeaPdpw3WYBTDMcsV5IzgayMJn\nCAnQF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0",
  "QF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0tKh5YOKY0sjSh9aulTSkNLQ0o3LN2gVFtKTqTwRLD0gNKJp\nRNKjy09pvSVpa8ofWbpM0pfW/qa0neWvqP0saWPKWMkr7lvYp5ZaSTwdBtG7pOqWBpeTdD+41SweUZpZmlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPS",
  "mlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPStpW8p3bR0k9LYUvJtAE4nlu5Tar8CVQWlu5buUnph6YX7uwBfTGPgWpg7toIdSlNLU0q3LCVvCnCUsPScnC\nDbyhF3p/en95/3j/LveXz5fzZd2ot2+13zjdf6WZ/8B8XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+",
  "XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+4eLDymQUtjpz5ZW1vBXWFo4/Lm39mv4e7DlUfr7RfaO953vfeD96a95v3yHvm\n\u03c6  \u2212 \u03c6 \u2212 \u21b5@L\n@\u03c6\nStep 1:  Compute derivatives (slopes of function) with\nRespect to the parameters\nStep 2:  Update parameters according to rule\n\ud835\udefc = step size\nGradient descent\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvY",
  "ase64=\"Me4GYclKg9HjqA15nuwjaNhVig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD9\n3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe978MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDx",
  "t4PEplkxz7LuRSK97XQkh+nGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZ\nixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5MvQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62",
  "yLnKQvOWMRPoKhYzPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62\nEmhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14wZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1",
  "mIgOADR5QkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQ\nSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6zSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqm",
  "GLGWS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6\nLxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7WrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1N",
  "o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFB\nKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuHZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2v3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcp",
  "Xx82mPegXzE4RBPx8uInIyIWdSqCw5Dzro\nksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9HbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YFf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYh",
  "mQCWRlziIdQhLNZbuG02MaxuOdQ\nt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5jHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4xXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ik",
  "9LcpLzwytLryi9tPS0i\nNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNLQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4d+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486",
  "1ZSn4Vw/PMUn\n1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1fu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>K8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19TekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyV",
  "cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwvVnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIg",
  "lDBYLjp+jb/KXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZ\nzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+jf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfpiknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEpt",
  "EWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXD\nHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLUzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71ur3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1L",
  "Q/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn\n5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lTRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFT",
  "trlaMTeZWx8mxv3orIeSTect\nJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEISMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyGvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCr",
  "k2qoqY4gay9tO\n3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FWhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzSpXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV",
  "A67\nd2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WLy/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NEm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVl\nuDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+",
  "SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8YhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ\n/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2",
  "+WLp1CIs5FXOrGEc8RGIdwmJUtC34P1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5pHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0\n0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N",
  "rOs4ZalAHsLRJzpjbloPmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oGhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjS\ng8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDze",
  "hjJK1wxdo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD439Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduU\n4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>nhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOA",
  "OfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJwPVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9rnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#",
  "Gradient descent",
  "Gradient descent",
  "Gradient descent",
  "<latexit sha1_base64=\"WsHIJzX3fm90AkOs9UWotK6nHbU=\">AWw3\niclZhb9s2FIDV7tZ1t3TD8rIXYUGBYeiMZOguj21SN2TLs7FSdo4DSiZktlQlCJRiVPBP2m/Zk8Dtv+yQ0k2q3OYhwXozJ3vEy+HFEUpyKQo9Orq\n37duf/DhRx9/cufTu59/sWXy3d+/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEK",
  "/qwSMs85MwlWl+HLCS6H4UAst+XGWc5YEkh8F5xuGH13yvBCpOtDXGT9NWKxEJEKmIXS2tOmPgibiJFMVSx\n5pFmep1dNzP/JHzGZTZg/inIWVqOM5Vow6W/PbLlRZ2dLK6u91frPp4W1trDitX+Ds3vfjkfjNCwTrnQoWVGcrK1m+rQy1YaSz+6OyoJnLDxnMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj",
  "nMT+Bo\nmIJL06resQz/z5Exn6U5vBPab+Ovn9FxZKiuE4CMBOmJwVmJuhiJ6WOfj+thMpKzVXYNBSV0tepb9Lnj0XOQy2vocDCXEBf/XDCIDsaknx3pPhVmCYJ\nU+NqtN7fhTwFPBaq4hdlnfDZrOv0a4dD8SZj/fnBohaheSLecVJrZhKbhB4PKsq3ot7GAgOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxI",
  "gOQPQ4AaniBdRp8hNE/hqisMAkYOBOoXORf7ejFStNI\n8hJx3tNdGgkEk+7VgbxIKpTDrKPi+f983gOscZgG6Cj8czcF+xtRsfp3mU50nVWFiuIWcqZjXTcCQ1jXe9hQpZRwadix/sDWHlPnbeLSrO5qbiLIO\nsi7js5pXtS469QRZMEijLtWHUGWhO1gzBIGW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7",
  "GW7LZzDgxDcRtyoUVgVZmIM8DbptZyaC1+Y0g/ul6/Urkv5LhjJiAnD3mV/BVMi7+ka6sP15ci5r3xT\n41J/AZHUvYXncDGveCIyqjc2oWecKmTRbEILNsmua3jhUnonuAE0A3RlLlT0nvagLsGSNeHRAxhqXkp+8lPvFz49rVbNbWP+Q7IJFRVl5qrIhP9HRW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5sm",
  "RW\nN4AOH1BRE8ealEkweBevJSCfs7mjqW4VtIvXcQUEoJoW+Rre/iFX3mjqCO5smqK8QMPXCLxMKTXIUdWUTMDL8wqPUsYBCNMiwGWMo06LMOdn80HqG\nSK2bTEX5mHV3VClEbr7BpeLq6AMD4dLfsPlAcpo0OQzSEs1ZjlK5tRM6fTNqIBHvfPur6e8KTqtmF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1",
  "mF9ste1Bv2B2yjDkF2dbeD5iYlFHorg7OKsSxL\nL0R7UtViu7/es2nrzI1nascN1m5LU2/bSbTvcG3rAL7Ydvd0mHrGoI1FdbQ+pRyxHe1CXO4/brlE4XLcpSb3zPDpth7sw0fKPDiZcM3NMSuXYHPtSO\nWpCWNRU1E4xTXiMxCaExaTsWvD/WNkX8PDoWk0Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOs",
  "Ii4NCdDUTwNKYSzyEJoTF5hbum0Mq9sOdut1i8EXbMJYXGTJXjUTQiLMRVjp3jOsgyJTYjkcYLz\nOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2AdzXUYBvEckFXuFceQqtYkVX8dDV8PCGhjVDFZoAlnbIPeaPdpw3WYBTDMcsV5IzgayMJn\nCAnQF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0",
  "QF15qe/IKrIS6Iri29pvTK0itKjyw9ojS3lLwRBNGepeTtJIguLb2k9NDSQ0pLS0tKh5YOKY0sjSh9aulTSkNLQ0o3LN2gVFtKTqTwRLD0gNKJp\nRNKjy09pvSVpa8ofWbpM0pfW/qa0neWvqP0saWPKWMkr7lvYp5ZaSTwdBtG7pOqWBpeTdD+41SweUZpZmlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPS",
  "mlD6x9AmlY0vJWzE8zywlxt4MFoqKX1\nu6XNKhaXk/S2IXlr6ktLE0oTSF5a+oPStpW8p3bR0k9LYUvJtAE4nlu5Tar8CVQWlu5buUnph6YX7uwBfTGPgWpg7toIdSlNLU0q3LCVvCnCUsPScnC\nDbyhF3p/en95/3j/LveXz5fzZd2ot2+13zjdf6WZ/8B8XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+",
  "XsOg=</latexit>cj1e5q869NZF+L1I7WJvx+dUk5FacAdrd6f51WR/itSCT0jX+4eLDymQUtjpz5ZW1vBXWFo4/Lm39mv4e7DlUfr7RfaO953vfeD96a95v3yHvm\n\u03c6  \u2212 \u03c6 \u2212 \u21b5@L\n@\u03c6\nWe can also search for the optimal \nstep size at each iteration using \nLine Search\n\ud835\udefc = step size\nLine Search",
  "Line Search (bracketing)\na\nb\nc\nd\na\nb\nc\nd",
  "Line Search (bracketing)\n\u2022 For each iteration you are evaluating \nloss four times\n\u2022 Can be costly for more complex data\ntypes and loss calculations (e.g.\nimage segmentation, \u2026.)\n\u2022 Not typically used for computer \nvision\na\nb\nc\nd",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "The linear model loss function was convex.\nWe\u2019ll use a more complex (non-convex) \nmodel that we can still visualize in 2D and 3D\n\u00e8 Gabor Function",
  "Gabor Model (with Envelope)\n<latexit sha1_base6\n4=\"ONVs+tsISX5QKXsJQIVd5wdkrvk=\">A\nW83iclZhbc9NGFIAd6IVSaEM7zUtfNM0wAwU\n8DkMpL52BhHBLaBISJ4EoZFbySl6yWinSKnHQ\n+Jf0rdPX/qA+9L/0rGR70Tmbh3om8fp8397O\nrq5BJkWhe71/5i5d/uzL768tXVr69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54",
  "8tXVr69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54XIlU7+jzjhwmLlYhEyDSEj\nubP/CRIR1U0Phjd9YMoG4pD7zfPL4Q68OHUc\n+74/W6vYd+OEh1HamWxqPD+qfn81HmSx7pW/\nf8KGdhdaupc4fWuP3+/rh61O2N/VzEQ37aH\n4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5Orflnwj",
  "37aH\n4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5OrflnwjIXHLOYHUFQs4cV\nhVWdo7N2EyMCL0hz+lPbq6Kc1KpYUxXkSgJk\nwPSwM0EXOyh19OiwEiorNVdh01FUSk+nkm3\nNxA5D7U8hwILcwFj9cIhg1xpWJSrvuJnYZok\nTA0qf3l1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6",
  "qf3l1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6NuxgID\nkB0OQGp4gW0afITRN4SorAhJeCq2Wuwzbw3Y\n9K0jyGnLS0d0SDQib5qGWtEAuWMmkp26B43\nk3PAK5zWAUYKnxtAbGVPjaT3NRzpPqsLEcA\n85UzGvu4Aph0yaGbUNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIk",
  "yaGbUNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIknD6GLCEQZYn5SOYcOKZiFsVCquCbMzNPA3\nafWcmgvfmKIPjpe2tViT9pwxlxATg6DPfgqm\nQt/WVdGZ70+Sc1r4p8JE3hMVqV2F53Exr2gnM\nahIbU7POFTJptiCUp2dt04zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFs",
  "dt04zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFsQkNFmbkaMuH/0dALlh4f0EL14q0\neJBoF68VML5HS0dy/HGNpF67aAgFJNCn6PDX\n8SqXaeO4MGmCRorBEy78M2EQoscRW3ZBIwM3\n3DpdWygE0ybOYyrQoc05Ofmg/Q6TWzWkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9",
  "6TWzWkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9oOMRcR3+95E3RacX8ZG\n3SH4wLVqcMQ35ytIbXIyYWdSRqC+51nG1JYjn\n6g7Zm2/XTkVr738mWzt2uG5TknYno3TbDve\nCEfCTdcdo14lHLOpI1NZkhNQjlqM/aMudx3X\nXLByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFp",
  "3X\nXLByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFpGxb8Bsr2wIu\nHm2rCWFxsxBtzQSwNOAST6EJYbE5hNvmJIbV\ndYe67laZzIbIbEJYfM4SPOsmhMWYirFTPGZh\nsQmRPI4xHkc0jxmWMpcEl6RzLEiZEu5NlQ+T\nNuSCWBphHobOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMg",
  "BphHobOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMgCnG\nK4zXIlORPIymgCN7GzSZ3p3V8QVeROLojOLT\n2n9MzSM0r3LN2jNLeUPBE0RtLydNJEJ1aekr\nprqW7lJaWlpT2Le1TGlkaUfrM0meUhpaGlK5\nYukKptpTckcIVwdIdSoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil",
  "wdIdSoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil5dRBEy5YuUxpYSp794FizdJPSzNKM0qeW\nPqV0YCl5KobrmaXk9gYujJZKSl9a+pJSYSl5\nfgui15a+pjSxNKH0laWvKP1g6QdKn1v6nNLYU\nvJuAO5OLN2m1L4FqgpKtyzdovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZz",
  "tyzdovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZzXIjXjDjbJ+LQ2yXmkZtzBJmenaW\n1yforUjA/J0Fd3Zy9SIKVwpj+aX1zCb2FpYf\nd+d+lh98HWg8XHy5M3tFc6P3Z+6tzqLHV+7T\nzuvOhsdvqdsPv3OW5a3PXF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.",
  "XF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.06 \u00b7 \u03c61x] \u00b7 exp\n\u2713\n\u2212(\u03c60 + 0.06 \u00b7 \u03c61x)2\n8.0\n\u25c6",
  "Gabor model\n<latexit sha1_base6\n4=\"ONVs+tsISX5QKXsJQIVd5wdkrvk=\">A\nW83iclZhbc9NGFIAd6IVSaEM7zUtfNM0wAwU\n8DkMpL52BhHBLaBISJ4EoZFbySl6yWinSKnHQ\n+Jf0rdPX/qA+9L/0rGR70Tmbh3om8fp8397O\nrq5BJkWhe71/5i5d/uzL768tXVr69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54XIlU",
  "r69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54XIlU7+jzjhwmLlYhEyDSEj\nubP/CRIR1U0Phjd9YMoG4pD7zfPL4Q68OHUc\n+74/W6vYd+OEh1HamWxqPD+qfn81HmSx7pW/\nf8KGdhdaupc4fWuP3+/rh61O2N/VzEQ37aH\n4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5OrflnwjIXHL",
  "4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5OrflnwjIXHLOYHUFQs4cV\nhVWdo7N2EyMCL0hz+lPbq6Kc1KpYUxXkSgJk\nwPSwM0EXOyh19OiwEiorNVdh01FUSk+nkm3\nNxA5D7U8hwILcwFj9cIhg1xpWJSrvuJnYZok\nTA0qf3l1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6Nuxg",
  "1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6NuxgID\nkB0OQGp4gW0afITRN4SorAhJeCq2Wuwzbw3Y\n9K0jyGnLS0d0SDQib5qGWtEAuWMmkp26B43\nk3PAK5zWAUYKnxtAbGVPjaT3NRzpPqsLEcA\n85UzGvu4Aph0yaGbUNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIknD6G",
  "UNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIknD6GLCEQZYn5SOYcOKZiFsVCquCbMzNPA3\nafWcmgvfmKIPjpe2tViT9pwxlxATg6DPfgqm\nQt/WVdGZ70+Sc1r4p8JE3hMVqV2F53Exr2gnM\nahIbU7POFTJptiCUp2dt04zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFsQkNF",
  "zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFsQkNFmbkaMuH/0dALlh4f0EL14q0\neJBoF68VML5HS0dy/HGNpF67aAgFJNCn6PDX\n8SqXaeO4MGmCRorBEy78M2EQoscRW3ZBIwM3\n3DpdWygE0ybOYyrQoc05Ofmg/Q6TWzWkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9oOMR",
  "WkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9oOMRcR3+95E3RacX8ZG\n3SH4wLVqcMQ35ytIbXIyYWdSRqC+51nG1JYjn\n6g7Zm2/XTkVr738mWzt2uG5TknYno3TbDve\nCEfCTdcdo14lHLOpI1NZkhNQjlqM/aMudx3X\nXLByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFpGxb8",
  "LByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFpGxb8Bsr2wIu\nHm2rCWFxsxBtzQSwNOAST6EJYbE5hNvmJIbV\ndYe67laZzIbIbEJYfM4SPOsmhMWYirFTPGZh\nsQmRPI4xHkc0jxmWMpcEl6RzLEiZEu5NlQ+T\nNuSCWBphHobOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMgCnG",
  "obOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMgCnG\nK4zXIlORPIymgCN7GzSZ3p3V8QVeROLojOLT\n2n9MzSM0r3LN2jNLeUPBE0RtLydNJEJ1aekr\nprqW7lJaWlpT2Le1TGlkaUfrM0meUhpaGlK5\nYukKptpTckcIVwdIdSoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil5dRB",
  "SoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil5dRBEy5YuUxpYSp794FizdJPSzNKM0qeW\nPqV0YCl5KobrmaXk9gYujJZKSl9a+pJSYSl5\nfgui15a+pjSxNKH0laWvKP1g6QdKn1v6nNLYU\nvJuAO5OLN2m1L4FqgpKtyzdovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZzXIjX",
  "ovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZzXIjXjDjbJ+LQ2yXmkZtzBJmenaW\n1yforUjA/J0Fd3Zy9SIKVwpj+aX1zCb2FpYf\nd+d+lh98HWg8XHy5M3tFc6P3Z+6tzqLHV+7T\nzuvOhsdvqdsPv3OW5a3PXF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.",
  "XF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.06 \u00b7 \u03c61x] \u00b7 exp\n\u2713\n\u2212(\u03c60 + 0.06 \u00b7 \u03c61x)2\n8.0\n\u25c6\n\ud835\udf19# shifts left and right\n\ud835\udf19$ shrinks and expands the sinusoid and envelope",
  "Toy Dataset and Gabor model\n<latexit sha1_base6\n4=\"ONVs+tsISX5QKXsJQIVd5wdkrvk=\">A\nW83iclZhbc9NGFIAd6IVSaEM7zUtfNM0wAwU\n8DkMpL52BhHBLaBISJ4EoZFbySl6yWinSKnHQ\n+Jf0rdPX/qA+9L/0rGR70Tmbh3om8fp8397O\nrq5BJkWhe71/5i5d/uzL768tXVr69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54X",
  "tXVr69d/+b\n+Rvf7RZpmYe8H6YyzfcDVnApFO9roSXfz3LOk\nkDyveB4xfC9U54XIlU7+jzjhwmLlYhEyDSEj\nubP/CRIR1U0Phjd9YMoG4pD7zfPL4Q68OHUc\n+74/W6vYd+OEh1HamWxqPD+qfn81HmSx7pW/\nf8KGdhdaupc4fWuP3+/rh61O2N/VzEQ37aH\n4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5OrflnwjI",
  "7aH\n4RpPrj0cLSpLDYmXw2j278MPAHaVgmXOlQsqI\n4WOpl+rBiuRah5OrflnwjIXHLOYHUFQs4cV\nhVWdo7N2EyMCL0hz+lPbq6Kc1KpYUxXkSgJk\nwPSwM0EXOyh19OiwEiorNVdh01FUSk+nkm3\nNxA5D7U8hwILcwFj9cIhg1xpWJSrvuJnYZok\nTA0qf3l1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6N",
  "f3l1a1z5AY+FqvhJWS/QeNx2VmuHQ/Ei\nY/nlzqwVoXkiPnLSK2YRi4QeDyuKt6NuxgID\nkB0OQGp4gW0afITRN4SorAhJeCq2Wuwzbw3Y\n9K0jyGnLS0d0SDQib5qGWtEAuWMmkp26B43\nk3PAK5zWAUYKnxtAbGVPjaT3NRzpPqsLEcA\n85UzGvu4Aph0yaGbUNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIkn",
  "aGbUNVUoJVcOW9Tu23jB1PE\nlcmtVDzU0EWTt529E5zYsatJ06gizYhHbqi\nPIknD6GLCEQZYn5SOYcOKZiFsVCquCbMzNPA3\nafWcmgvfmKIPjpe2tViT9pwxlxATg6DPfgqm\nQt/WVdGZ70+Sc1r4p8JE3hMVqV2F53Exr2gnM\nahIbU7POFTJptiCUp2dt04zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFsQ",
  "t04zGofJMtCdoAvig\nK3Ohok+0u3UJtqwJ+3dhqnkp+cG97i98dFj1\nzGFj/pFsQkNFmbkaMuH/0dALlh4f0EL14q0\neJBoF68VML5HS0dy/HGNpF67aAgFJNCn6PDX\n8SqXaeO4MGmCRorBEy78M2EQoscRW3ZBIwM3\n3DpdWygE0ybOYyrQoc05Ofmg/Q6TWzWkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9o",
  "TWzWkxF+\nZi1T6hSiO0zxtczmpBGS4Op/yC6gHKaNDkM0\nhLNWA5SubILOnovV9oOMRcR3+95E3RacX8ZG\n3SH4wLVqcMQ35ytIbXIyYWdSRqC+51nG1JYjn\n6g7Zm2/XTkVr738mWzt2uG5TknYno3TbDve\nCEfCTdcdo14lHLOpI1NZkhNQjlqM/aMudx3X\nXLByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFpG",
  "X\nXLByu25Sk3WkenbDnZlo+0c7Q6ZuU1K5cDc\n9qXSb0JY1FTUTjFNeIzEJoTFpGxb8Bsr2wIu\nHm2rCWFxsxBtzQSwNOAST6EJYbE5hNvmJIbV\ndYe67laZzIbIbEJYfM4SPOsmhMWYirFTPGZh\nsQmRPI4xHkc0jxmWMpcEl6RzLEiZEu5NlQ+T\nNuSCWBphHobOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMgC",
  "phHobOTqDEchUoQ4nQSwXdOcVzp2n0C\n5WdBf3XR3L+hYM9SgCWBpgxjnr/hPMgCnG\nK4zXIlORPIymgCN7GzSZ3p3V8QVeROLojOLT\n2n9MzSM0r3LN2jNLeUPBE0RtLydNJEJ1aekr\nprqW7lJaWlpT2Le1TGlkaUfrM0meUhpaGlK5\nYukKptpTckcIVwdIdSoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil5",
  "dIdSoeWDindt3Sf0reWvqX\n0haUvKH1n6TtKP1r6kdInlj6hlFnKF21dJVS\nbil5dRBEy5YuUxpYSp794FizdJPSzNKM0qeW\nPqV0YCl5KobrmaXk9gYujJZKSl9a+pJSYSl5\nfgui15a+pjSxNKH0laWvKP1g6QdKn1v6nNLYU\nvJuAO5OLN2m1L4FqgpKtyzdovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZzX",
  "yzdovTE0hP3ewE+W\n8bAtTE3bAMblKaWpSuWUqeFOBWwtJjcj8Zq\nclZbfq2iZzXIjXjDjbJ+LQ2yXmkZtzBJmenaW\n1yforUjA/J0Fd3Zy9SIKVwpj+aX1zCb2FpYf\nd+d+lh98HWg8XHy5M3tFc6P3Z+6tzqLHV+7T\nzuvOhsdvqdsPv3OW5a3PXF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.",
  "XF8qFPxb+XPirUS/\nNTep832l9Fv7+D6C1+ew=</latexit>\nf[x, \u03c6] = sin[\u03c60 + 0.06 \u00b7 \u03c61x] \u00b7 exp\n\u2713\n\u2212(\u03c60 + 0.06 \u00b7 \u03c61x)2\n8.0\n\u25c6\n<latexit sha1_base64=\"2TnIThMg9kWPTX+3mS\n2FIP/hoRQ=\">AWwniclZhb9RGFIANvVF6IbRqXvpiNUKiFUQJopcXJEgIEBKahFwhm0Rj79g7ZDz2\n2uNkF3f/UX9N36r2x/SM7d3B50weuhLs5Hyf53LmYq+",
  "EBKahFwhm0Rj79g7ZDz2\n2uNkF3f/UX9N36r2x/SM7d3B50weuhLs5Hyf53LmYq+DTIpCLy39fe36Rx9/8ulnNz6/+cWX319a+72\nNwdFWuYh3w9TmeZHASu4FIrva6ElP8pyzpJA8sPgfNXwueFyJVe3qc8ZOExUpEImQaQmdz/zN414Q\nZQNx4j/ye0WZnFXi0fLktFqf9CSP9N1eEqSjKpocj4BM7rXy/bH5q5eLeKB/PH1wNrewtLhUf3xaW",
  "FXi0fLktFqf9CSP9N1eEqSjKpocj4BM7rXy/bH5q5eLeKB/PH1wNrewtLhUf3xaWG4L\nC172T67/V2/10/DMuFKh5IVxfHyUqZPKpZrEUo+udkrC56x8JzF/BiKiW8OKnqAU/8OxDp+1Gawz+l\n/Tr64RUVS4pinARgJkwPCsxM0MWOSx39dlIJlZWaq7BpKCqlr1PfZM/vi5yHWo6hwMJcQF/9cMByFmrI\n8c2e4pdhmiRM9aveytrOpOoFPBaq",
  "Cqlr1PfZM/vi5yHWo6hwMJcQF/9cMByFmrI\n8c2e4pdhmiRM9aveytrOpOoFPBaq4sOyzvdk0nXWaodD8SpjZX1vVovQPBHvOamkVkwlVwg8nlQVX4wX\nMRAcgFjkBKSKF1CnyU8Q+cuIwvqSgKtmZcC8F9PSNVK8xhy0tHeEg0KmeSjrVKLJjKpKPsguL7d3w\nDuM5hFqCr8MXRHOxmTE2m12k+0nlSFSaGW8iZindBAw5ZNKMqGuoUkq4NOxYv",
  "3w\nDuM5hFqCr8MXRHOxmTE2m12k+0nlSFSaGW8iZindBAw5ZNKMqGuoUkq4NOxYv2PrNVPnbeLSrO5qbiL\nI2su7js5pXlS/69QRZMEijLtWHUGWhNOgzxIGW7LZzDgxDcRtyoUVgVZmNt5GnTbzkwEr81RBvul61\nVJP0XDGXEBGD3mW/BVMi7+mo6s/1pci5q3xT4yB/AZHUvYXncDGvaCIyqjU2oWecKmTRbEMrTy65peuN\nQeSa6AzQBvOnK",
  "i5q3xT4yB/AZHUvYXncDGvaCIyqjU2oWecKmTRbEMrTy65peuN\nQeSa6AzQBvOnKXKjoA+1eXYIla8K9ezDUvJT8+P7iz3x0Ui2ZbWP+I9mEioyc1Vkwv+joj7cf/D6gi\nevFSiyYNAPXmphPMdTR3L8cI2kXruoCAUk0KP0fYXsepeU0dwZ9ME9RUCpl74ZkKhSY6irmwCRoZvuJM\n6FlCIBhk2YwxlWpQ5J4cfWs8QqXVzLObC3Ky6B6o0Qvfc",
  "kKhSY6irmwCRoZvuJM\n6FlCIBhk2YwxlWpQ5J4cfWs8QqXVzLObC3Ky6B6o0Qvfc4HJ2FZTh5nDBr7g8QBkNmnwGan6LEfJHJk\npHZ32Cg1bzLX76ylvik4r5sONtj3oF8xOGYZ8eLaB5yMmFnUkqgseXZx1SWI52oO6Zsv1w5VG6c/ka\nUdO1y3KUm9bS/dtsO9ogd8uOno7SbxiEUdiepqe0g9Yjnag7rcedx0jcLhuk1J6p3m0Wk73JmJln+0",
  "tsO9ogd8uOno7SbxiEUdiepqe0g9Yjnag7rcedx0jcLhuk1J6p3m0Wk73JmJln+0N+\nCamcekVPbNY18qe0Ii5qK2imCY+R2ISwmJRdC/7Gyq6Am0fXakJY3C5EVzMBLPW5xENoQlhstnDXbG\nNY3XSom26VyWyAzCaExecswaNuQliMqRg7xXOWZUhsQiSPA5zHAc1jhqXMJeEZyRwzQpaUa0Hlg7QrmQ\nCWRqi1kaMx6IFMFWqwDWK5oCuvcK4",
  "A5zHAc1jhqXMJeEZyRwzQpaUa0Hlg7QrmQ\nCWRqi1kaMx6IFMFWqwDWK5oCuvcK48hVaxoqt439Xw/hUNa4YqNAEsbZE95ve2nJswCmGxyxXkjOBrI\nwmcBs729SZPv0FUWe5IJobOmY0ktLyk9tPSQ0txS8osgiF5bSn6dBNGFpReUHlh6QGlpaUnpvqX7lE\naWRpQ+s/QZpaGlIaWrlq5Sqi0lT6RwR7B0j9KBpQNKjyw9ovSNpW8ofWHpC0rf",
  "E\naWRpQ+s/QZpaGlIaWrlq5Sqi0lT6RwR7B0j9KBpQNKjyw9ovSNpW8ofWHpC0rfWvqW0veWvqf0iaVPKG\nWMkrXLF2jlFtKXh0E0YqlK5QGlpLfrDXLN2mNLM0o/SpU8p7VtKfhXD/cxS8ngDN0ZLJaXrlq5TK\niwlv9+C6JWlryhNLE0ofWnpS0rfWfqO0ueWPqc0tpS8G4CnE0t3KbVvgaqC0h1LdygdWjp0vxfgs2kMX\nAtzy1awRWlqaUrp",
  "ueWPqc0tpS8G4CnE0t3KbVvgaqC0h1LdygdWjp0vxfgs2kMX\nAtzy1awRWlqaUrphqXklwI8Slh6Tp4nI9WeatO3TeRci9SMO1ib8enVJOeRmnEHa0+n6dXkfIrUjA9I1\n9cOZi9SIKVw0p/NLSzjt7C0cPBgcfmXxYc7Dxcer7RvaG943s/eHe9Ze9X7H3wtv29r3Q+9P7y/vH+\n3f+6fy7+eF80ajXr7XfOt1PvN/AdiG+tH</latexit>\nL[\u03c6",
  "tv29r3Q+9P7y/vH+\n3f+6fy7+eF80ajXr7XfOt1PvN/AdiG+tH</latexit>\nL[\u03c6] =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2",
  "",
  "\u2022 Gradient descent gets to the global \nminimum if we start in the right \n\u201cvalley\u201d\n\u2022 Otherwise, descends to a local \nminimum\n\u2022 Or get stuck near a saddle point",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "IDEA:  add noise, save \ncomputation\n\u2022 Stochastic gradient descent\n\u2022 Compute gradient based on \nonly a subset of points \u2013 a \nmini-batch\n\u2022 Work through dataset \nsampling without \nreplacement\n\u2022 One pass though the data is \ncalled an epoch",
  "Batches and Epochs \n(Ex. 30 sample dataset, batch size 5)\n[ 0  1  2  3 4 5  6  7  8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29]\n[27 15 23 17 8 9 28 24 12 0  4 16  5 13 11 22  1  2 25  3 21 26 18 29 20  7 10 14 19  6]\nEpoch # 0-----------\nStep 0, Batch # 0, Batch Range [0 1 2 3 4], Batch index: [27 15 23 17 8] \nStep 1, Batch # 1, Batch Range [5 6 7 8 9], Batch index: [ 9 28 24 12 0] \nStep 2, Batch # 2,",
  "Batch index: [27 15 23 17 8] \nStep 1, Batch # 1, Batch Range [5 6 7 8 9], Batch index: [ 9 28 24 12 0] \nStep 2, Batch # 2, Batch Range [10 11 12 13 14], Batch index: [ 4 16 5 13 11] \nStep 3, Batch # 3, Batch Range [15 16 17 18 19], Batch index: [22 1 2 25 3] \nStep 4, Batch # 4, Batch Range [20 21 22 23 24], Batch index: [21 26 18 29 20] \nStep 5, Batch # 5, Batch Range [25 26 27 28 29], Batch index: [ 7 10 14 19 6] \nEpoch # 1-----------\nStep 6, Batch # 0, Batch Range [0 1 2 3 4], Batch index: [27 15 23 17 8] \nStep 7, Batch # 1,",
  "Batch # 0, Batch Range [0 1 2 3 4], Batch index: [27 15 23 17 8] \nStep 7, Batch # 1, Batch Range [5 6 7 8 9], Batch index: [ 9 28 24 12 0] \nStep 8, Batch # 2, Batch Range [10 11 12 13 14], Batch index: [ 4 16 5 13 11] \nStep 9, Batch # 3, Batch Range [15 16 17 18 19], Batch index: [22 1 2 25 3] \n\u2026\nData Indices\nPermute\nBatch Size 5\n30/5 = 6 batches \nper epoch",
  "Stochastic gradient descent\nBefore (full batch descent)\nAfter (SGD)\nFixed learning rate \u03b1\n<latexit sha1_base64=\"HvMSYTO0RV5JmWYBrSQMdajlqQY=\">AW+3iclZhbU9w2F\nICXtOkF9I2vPTFUyYznTZhoJNeHhMIuUEKJNwSTHZkr+xVkGVjy7DE4+mP6Vunr/0x/QH9Hz2yvSt8jngoM+mq5/usy5Esyw4yKQq9vPzP3Hvf/DhRx9f+T6jU8/+/yL+Ztf7hd\npmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4KT",
  "vf/DhRx9f+T6jU8/+/yL+Ztf7hd\npmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4KTNcMPznheiFTt6ouMHycsViISIdMQGs7/7vlBlI3FsNI/rNS+TFUseaRZnqfnM1J7dz2fyWzMPL8ok2ElfKH8hOlxyGS1Whul9qOch\nZWfsVwLJj2fSwlifWRrOa4v4SZa3xnOLy4vLTd/Hi2sdIXFQfe3Pbx5a+SP0rBMuNKhZEVxtLKc6ePK1BtKXl/3y4J",
  "a3xnOLy4vLTd/Hi2sdIXFQfe3Pbx5a+SP0rBMuNKhZEVxtLKc6ePK1BtKXl/3y4JnLDxhMT+ComIJL46rJlO1dxsiIy9Kc/intNdEL19RsaQoLpI\nATDO8AjMTdLGjUke/HldCZaXmKmwbikrp6dQzafdGIuehlhdQYGEuoK9eOGaQMQ2Tc91X/DxMk4SpUeWvru9AogIeC1Xx07KZqLruO+uNw6F4lbH6dHdWi9A8Ee84qaRTCVXCDyuq\n4ovxU",
  "Wvru9AogIeC1Xx07KZqLruO+uNw6F4lbH6dHdWi9A8Ee84qaRTCVXCDyuq\n4ovxUsYCA5ALHECUsULqNPkJ4i8FURhYUrAwIN0Ap2LvBc1qVpHkNOetprokEhk3zSs9aIBVOZ9JSXoHjebc8ArnOYBegq/HA0By8zpurpdZpPdJ5UhYnhFnKmYt40d4NZkR9Q5V\nSwqVhz/oNWy+YOukSl2ZNV3MTQdZu3nd0TvOiRn2niSALFmHct5oIsiRsIyO",
  "Q5V\nSwqVhz/oNWy+YOukSl2ZNV3MTQdZu3nd0TvOiRn2niSALFmHct5oIsiRsIyOWMhyVx7CgBPRNyqUFgVZGFu52nQbzszEbw2JxncL31vSLpP2MoIyYAd5/5FUyFvK+vpTPbmybnr\nPFNgU+8MUxW/xKWx+2wpo3AqLpYTc0mV8ik2YIQbK90/TGofJM9AdoAvimK3OhokvanaYES9aE/Tsw1LyU/Oju0k98clwtm9vG/IdkEyoqysxVkQn/j4",
  "doAvimK3OhokvanaYES9aE/Tsw1LyU/Oju0k98clwtm9vG/IdkEyoqysxVkQn/j4pG8ODC6wsiePJSiSYPAs\n3kpRL2dzR1LMcL20SauYOCUEwKfYFufxGr/jVNBHc2TVBfIWDqhV8mFJrkKOrLJmBk+IVHsGMBhWiQYTvGUKZFmXOy+aH1DJFGN9tiLszDqr+hSiP09w0uZ1dBGR4OZ/yKywOU0aDN\nZ5CWasRylMyJmdLJG7+Aw4Hz7m+mvC06",
  "r+hSiP09w0uZ1dBGR4OZ/yKywOU0aDN\nZ5CWasRylMyJmdLJG7+Aw4Hz7m+mvC06rZifbnTtQb9gdsow5KfDTwfMbGoI1FdcOZx1iWJ5WgP6pot18s9qzbefE+Wduxw3aYk9Xa9dNsO94oe8NR283iUcs6khUV9dD6hHL0R\n7U5c7jpmsUDtdtSlLvNI9O2+HOTLT8o90x18wck1I5Mse+VPptCIuaitopgmPkdiGsJiUfQv+HysvBTw8+lYbwuJ",
  "HOTLT8o90x18wck1I5Mse+VPptCIuaitopgmPkdiGsJiUfQv+HysvBTw8+lYbwuJ2IfqaCWBpxCUeQhvCYnsL980uhtVNh7rpVpvXh7ZhrD4mCV4\n1G0IizEVY6d4wrIMiW2I5HGM8zimecywlLkPCOZY0bIknItqHyc9iUTwNIEtTZxNAY9gLc71GAXxHJBV17hXHkKrWJFV/Geq+G9KxrWDFVoAljaIveY5285b7IApxiOWa4kZwJZGU\n3gNna",
  "V17hXHkKrWJFV/Geq+G9KxrWDFVoAljaIveY5285b7IApxiOWa4kZwJZGU\n3gNna2qTM9/QVRU5yQXRh6QWl5aeU3pg6QGluaXkjSCIXlhK3k6C6MzSM0r3Ld2ntLS0pHTP0j1KI0sjSh9Z+ojS0NKQ0jVL1yjVlpITKTwRLN2ldGzpmNJDSw8pfWXpK0qfWPqE\n0teWvqb0naXvKH1g6QNKmaWM0nVL1ynlpJPB0G0aukqpYGl5N0P7jVLtynNL",
  "PqE\n0teWvqb0naXvKH1g6QNKmaWM0nVL1ynlpJPB0G0aukqpYGl5N0P7jVLtynNLM0ofWjpQ0pHlpK3YnieWUqON/BgtFRS+tTSp5QKS8n7WxA9t/Q5pYmlCaXPLH1G6VtL31L62NLHl\nxBX+FpYX9H5dWfl6t3Nv8f5q94X2uCbwbeD7wYrg18G9wdPBtuDvUE4+HfuxtzXc7cW6oU/Fv5c+KtV35vrvlq0Ptb+Ps/tD4Cbg=</latexit>MaWk",
  "vUE4+HfuxtzXc7cW6oU/Fv5c+KtV35vrvlq0Ptb+Ps/tD4Cbg=</latexit>MaWkm8DcDqx9CWl9itQVC6Y+kOpaeWnrq/C/DZNAauhblK9iNLU0pXTDUvKmAEcJS0/IeTJS3a42/dpE9rVIzbiDdRmfXk1yHqkZd7Bud5peTfanSM34mHR9fX/2IQVSCjv9cH5\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n,\n<latexit sha1_base64=\"lJb3qScjsYKv",
  "cH5\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n,\n<latexit sha1_base64=\"lJb3qScjsYKv7MZBcbRGdavoA=\">AW7XiclZhb9xE\nFIA3BdpSLk1B5IUXi6gSgjbKonJ5QWqTpm2alFw3SRunq7F37J1mPHbscbKptT+DN8QrPwnxY5A4/XuxOdMHohUdjfNxefGY/HDjIpCr28/M/cjQ8+/Ojmrdsf3/nk08+vzt/\n74uDIi3zkPfCVKb5UcAKLoXiPS20",
  "DjIpCr28/M/cjQ8+/Ojmrdsf3/nk08+vzt/\n74uDIi3zkPfCVKb5UcAKLoXiPS205EdZzlkSH4YnK4afnjO80Kkal9fZvwkYbESkQiZhlB/szgygbin6lv+OfZmqWPJIszxPL2Zk7D30fCazIfP8okz6lfi1O367kc5Cys\n/Y7kWTHo+lxLQ+NjWOxlfwXV0/KA/v7i8tFz/ebTQbQqLneZvu3/vq4E/SMy4UqHkhXFcXc50yeVaTeUfHzHLwues",
  "/KA/v7i8tFz/ebTQbQqLneZvu3/vq4E/SMy4UqHkhXFcXc50yeVaTeUfHzHLwuesfCUxfwYiolvDip6tyMvfsQGXhRmsM/pb06erVGxZKiuE\nwCMBOmhwVmJuhix6WOfjmphMpKzVU46SgqpadTzyTaG4ich1peQoGFuYCxeuGQcY0TMcdX/GLME0SpgaVv7K2A4kKeCxUxc/KemrG47azVjscitcZK+v7s1aE5ol4z0kjtWIauU\nbg8biq+FK8h",
  "K2A4kKeCxUxc/KemrG47azVjscitcZK+v7s1aE5ol4z0kjtWIauU\nbg8biq+FK8hIHgAMQSJyBVvIA2TX6CyOsiCktRAgYepCMYXOTtjknTSvMYctLS3hANCpnko5a1SiyYyqSl7IHiefc9A7jOYRZgqPD0RzsZUyNp/U0H+k8qQoTwz3kTMW87gIuO\nYSFvYsNVUoJVcOW9Ru2dpk6bRKXZvVQcxNB1n7ednRO86IGbaeOIAsWYdy26giyJGwcA",
  "YsNVUoJVcOW9Ru2dpk6bRKXZvVQcxNB1n7ednRO86IGbaeOIAsWYdy26giyJGwcA5YwyHJT7sMFJ56JuFWhsCrIwtzO06Dd2YieG2OMrhf2t5aRdJ/zlBGTADuPvMrmAp5W\n19NZ7Y3Tc57ZsCH3lDmKx2FZbHk8uadgJX1cTG1KxzhUyaLQjBhto2zWgcKs9E+wJNAN90ZS5UdEV7UJdgyZqw/wAuNS8lP3649CMfnVTL5rYx/yHZhIaKMnM1ZM",
  "9E+wJNAN90ZS5UdEV7UJdgyZqw/wAuNS8lP3649CMfnVTL5rYx/yHZhIaKMnM1ZML/o6EBPKrw\n+oInrxUosmDQD15qYT9HU0dy/HCNpF67qAgFJNCX6LbX8SqXaeO4MGmCRorBEy78MuEQpMcRW3ZBIwMv/DQdSygEF1kOLnGUKZFmXOy+aH1DJFaN9tiLszDqr2hSiO09w0uZ7W\ngDA+Hc35N9QBlNJjkM0hLNWA5SubITOnorV/AcB59dTPik",
  "zDqr2hSiO09w0uZ7W\ngDA+Hc35N9QBlNJjkM0hLNWA5SubITOnorV/AcB59dTPik6rZifbT9wbhgdsow5Gf9DTwfMbGoI1FbcMpxtiWJ5egP2pot16sjqzbefkeWduxw3aYk7TajdNsO95oR8LNx2g\n3iUcs6kjUVjNC6hHL0R+05c7jpusqHK7blKTdaR6dtsOdmWj5R/tDrpk5JqVyYI59qfQnISxqKmqnmCY8RuIkhMWkbFvw/1jZE/DwaFuT",
  "tsOdmWj5R/tDrpk5JqVyYI59qfQnISxqKmqnmCY8RuIkhMWkbFvw/1jZE/DwaFuTEBa3C9HWTABLAy7xJUxCWJzcwm2ziW\nF106FutX6haFtTkJYfM4SfNWTEBZjKsZO8ZRlGRInIZLHIc7jkOYxw1LmkvCMZI4ZIUvKtaDyYdqWTABLI9TbyNEZjADe51CHTRDLBV15hXPlKbSKFV3FPVfHvWs61gw1aAJY2i\nL3mOdvOW+yAKcYjlmuJGcCWRl",
  "TRDLBV15hXPlKbSKFV3FPVfHvWs61gw1aAJY2i\nL3mOdvOW+yAKcYjlmuJGcCWRlN4DZ2tqkzPf0FUVOckF0aeklpReWXlB6aOkhpbml5I0giHYtJW8nQXRu6TmlB5YeUFpaWlLas7RHaWRpROkzS59RGloaUrpq6Sql2lJyIoUng\nqX7lA4tHVJ6ZOkRpa8tfU3pC0tfUPrG0jeUvrf0PaVPLH1CKbOUbpm6Rql3FLy6SCIVixdoTSwlLz7wb1m",
  "tfU3pC0tfUPrG0jeUvrf0PaVPLH1CKbOUbpm6Rql3FLy6SCIVixdoTSwlLz7wb1m6TalmaUZpU8tfUrpwFLyVgzPM0vJ8QYejJZKStctXadUWEre34Lo\nlaWvKE0sTSh9aelLSt9Z+o7S5Y+pzS2lHwbgNOJpXuU2q9AVUHpjqU7lJ5Zeub+LsBn0xi4FuaWbWCL0tTSlNINS8mbAhwlLD0l58lINbva9GsT2dciNeMO1mR8WpvkPFIz7mDN\n7jS",
  "bWCL0tTSlNINS8mbAhwlLD0l58lINbva9GsT2dciNeMO1mR8WpvkPFIz7mDN\n7jStTfanSM34kAx97WD2IQVSCjt9f36xi7/C0sLBD0vdn5Ye7TxafLzSfKG93fm6803n206383PncedFZ7vT64Sdvzv/zt2cu7WQLvy+8MfCnxP1xlxT58tO62/hr/8As9H8eQ=\n=</latexit>\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nI\nX\ni=1\n@`i[\u03c6t]\n@\u03c6\n,",
  "",
  "Properties of SGD\n\u2022 Can escape from local minima\n\u2022 Adds noise, but still sensible updates as based on part of data\n\u2022 Still uses all data equally\n\u2022 Less computationally expensive\n\u2022 Seems to find better solutions\n\u2022 Doesn\u2019t converge in traditional sense\n\u2022 Learning rate schedule \u2013 decrease learning rate over time",
  "Simple Gradient Descent\nThink of analogy of a ball rolling \ndown a hill.\nWould it follow path like on the \nleft?\nWhy/Why not? What\u2019s missing?",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "\u2022 Weighted sum of this gradient and previous gradient\n\u2022 Not only influenced by gradient\n\u2022 Changes more slowly over time\nMomentum\n<latexit sha1\n_base64=\"75TvOUZBRiF+w+fMC\n2gmLmcN5tw=\">AXS3iclZhb\nc9w0FIA3hUIJtxSG8MCLh0yZQ\ntudLNMCj23S9JaUbJpskjZOd2R\nb9qRZceWk09/okMz/wO3hge\nOLK9q1hHeSAzZcX5Pl18JMuyvZ\nSzXK6u/rVw7YMPr3/08Y1PFj/\n97PMvly6+dV+nhSZT0d+wpPs0\nCM5UzQkWS08M",
  "SzXK6u/rVw7YMPr3/08Y1PFj/\n97PMvly6+dV+nhSZT0d+wpPs0\nCM5UzQkWS08M0oyT2OD3wTt\nYVPzijWc4SsScvUnock0iwkPl\nEQmi89KfjemE8LuWdQeX84HIaS\npJlyTmEqSO6weJbBTp3HFuD+\n7V8R8dNy+gFnOZcGMiJz7h5Vo1\nlpUbZsQv3ZRkhHuJRz0Koja\nCKdMOinOq4u4TpauSIRezRzH\ndRWeu4iHN2nDuOS7h6aQzwLrC",
  "khHuJRz0Koja\nCKdMOinOq4u4TpauSIRezRzH\ndRWeu4iHN2nDuOS7h6aQzwLrC\neGltb9a/zm4MGgLK732bzi+\nU3gBolfxFRIn5M8PxqspvK4VCP\n0Oa0W3SKnKfFPSESPoChITPj\ns585dyCSOCESQb/hHTq6OUaJY\nnz/CL2wFRpyk2mgjZ2VMjwt+O\nSibSQVPhNR2HBHZk4ahqdgGXUl\n/wCsTPGIzV8ScEci9hshdQc\n/9JI6JCEp3bWM",
  "SibSQVPhNR2HBHZk4ahqdgGXUl\n/wCsTPGIzV8ScEci9hshdQc\n/9JI6JCEp3bWMHUu7RiImSnhb\n1xFdV19moHQrFq4y153vzVpikM\nXtPUSO1ohq5QqBRVZa0H/VNwC\ngA1qcIJILm0KbKjxc6A4PCQueA\ngXvJFAYXOq8q1LSQNIKcdLQ3S\nINCyum0Y60jC6Yy7i7oDjOLUc\nBKjOYBRgq/FBjDnZTIqpZPUmn\nMovLXMXMHjIiIlp30dxV6",
  "0jC6Yy7i7oDjOLUc\nBKjOYBRgq/FBjDnZTIqpZPUmn\nMovLXMXMHjIiIlp30dxV6oq6h\nig4h6p+x/rdtF4RcdImLknroWY\nqYlh7WdeRGc6LCLpOHTEsWIR\n16ojhsVhWwpITCDLbXkMFxw7Km\nJXmTBVhbmMEu8bt+piphrc5r\nC/dL1NkqU/jNiZEQF4O5Tv4wIn\n3b19WRuO7PknNW+KtCpM4HJ6l\nYhWdRc1qwTuKo2VmGzpVh4mx\nBCPbH",
  "5Tv4wIn\n3b19WRuO7PknNW+KtCpM4HJ6l\nYhWdRc1qwTuKo2VmGzpVh4mx\nBCPbHrqlGY1FpyroXqALmTVdkT\nISXtLt1CZasCrt34VKzgtOje/\n0HdHpcrqrbRv0HZRMayovU1pAK\n/4+GAngQmusLIubkJdyYPAjUk\n5dw2N+NqSOZubBVpJ47KDBOJM\nXxu3PItGtU0fMwSaxMVYIqHbh\nlzBhTHIYdmUVUDL8wiPdsoB84\nyL95hp9nuRFRt",
  "xu3PItGtU0fMwSaxMVYIqHbh\nlzBhTHIYdmUVUDL8wiPdsoB84\nyL95hp9nuRFRtHmZ6xniNS62hY\nzph5W3Q2VK6G7b1A+rwVleDic\n0Suqe0ZGvSafXlKIgGRGMqdqSq\ndv3Rye7ta7v57ypmi1Inq62fY\nH4LZKXyfno43zfmIkIUdbrQFZ\nyhrWxZlv6grflyvTycvPtT2\nhpRxbXbnLUbjtKu21xrxgBPd2\nyjHYLecjCDjfakeIPWRZ+oO",
  "grflyvTycvPtT2\nhpRxbXbnLUbjtKu21xrxgBPd2\nyjHYLecjCDjfakeIPWRZ+oO27\nHncsl2FxbWbHLU7y6PVtrhz01\nj+4d4EDsfqmJTwQB37Eu42IVOU\nWJRWMYlpZIhNyBTjomvB/5vKL\noOHR9dqQqY4zFlXUwFTCig3L6E\nJmWJzC3fNmaqWxZ1y67Wx/+u\n2YRM8SmJzatuQqYTGyickT\nQ2xCaE8Tsw8TnAeU1NKbZI5I6l\nlRtCSsi2",
  "Wx/+u\n2YRM8SmJzatuQqYTGyickT\nQ2xCaE8Tsw8TnAeU1NKbZI5I6l\nlRtCSsi2obJ0JRUwpanR29TS\nGYyAJ8LosA2aco5Xm5decJYxQ\nKv4pGt49EVHUtiNKgCprSN7jH\nH3beZJ6Z4uY1GU8XM6wUJ3BoO\nkPszE5/Xlik5wXmh6gem5pu\neYHmh6gGmKXoj8MJXmqK3Ey8\n80/QM031N9zEtNC0wHWk6wjTUN\nMT0iaZPMPU19TFd13QdU6",
  "KXoj8MJXmqK3Ey8\n80/QM031N9zEtNC0wHWk6wjTUN\nMT0iaZPMPU19TFd13QdU6kpOp\nHCE0HTPUwnmk4wPdT0ENPXmr7G\n9JmzB9o+kbTN9r+h7TR5o+w\npRoSjDd0HQDU6op+nTghWuarmH\nqaYre/eBe03SIapiuljTR9j\nGmiK3orheaYpOt7Ag1FTjulzT\nZ9jyjRF729e+FLTl5jGmsaYvtD\n0BabvNH2H6VNn2IaYq+DcDp\nRNdTPVXo",
  "ulzT\nZ9jyjRF729e+FLTl5jGmsaYvtD\n0BabvNH2H6VNn2IaYq+DcDp\nRNdTPVXoDLHdEfTHUxPNT21fx\neg82n0bAtzWzewjWmiaYLpqb\noTQGOEpqeoPNkKNpdbfa1Ce1r\noZhzC2szPquNch6KObewdnea1U\nb7UyjmfIKGvrE/5ACKYWdfry\n0MjC/wuLC/s/9wS/9+zv3Vx6ut\nV9ob/S+63fu90b9H7tPew96w\n17o56/8GDhaCFYoMt/L",
  "LC/s/9wS/9+zv3Vx6ut\nV9ob/S+63fu90b9H7tPew96w\n17o56/8GDhaCFYoMt/LP+9/M/y\nA=</latexit>v416baGt83Wv8/ft9f8AWq0d6\nmt+1  \u03b2 \u00b7 mt + (1 \u2212 \u03b2)\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7 mt+1\nStill in batches.",
  "Without and With Momentum\nWithout Momentum, Loss = 1.31\nWith Momentum, Loss = 0.96",
  "",
  "Nesterov accelerated momentum\n\u2022 Momentum smooths out gradient of \ncurrent location\n\u2022 Alternative, smooth out gradient of \nwhere we think we will be!\n<latexit sha\n1_base64=\"tH3Etax68HZ3e\nrPlJg/UFDU2vCo=\">AXY\nniclZhbU9w2FICX9JbSG2m\n5aF98JRJ20Cw3bSy2MCITd\nIgXBNMGFkr+xVkGVjy7DE9Q\n/te6e/o0f27gqfIx7KTLrq+\nT5dfCTLsoNMikIvL/89c+O\nDz/6+JObn85+9vkX341d+\nvr/SIt85DvhalM8OAFVwKx\nfe0JIfZjl",
  "IvL/89c+O\nDz/6+JObn85+9vkX341d+\nvr/SIt85DvhalM8OAFVwKx\nfe0JIfZjlnSD5QXC6avjB\nOc8LkapdfZnx4TFSkQiZBp\nCJ3P/en4QJSeVvtevR9y\nSPN8jy9gDXzPDQaonSu3d\n8+72Fxvyk+cXJQSFL5SfMD0\nMmaxWamPVfpSzsPIzlmvBpO\ndzKUGsj6CZbCiMsugzmQ1Z0\n/qk8eP6SpXGrH2VqjIJeO7\n5/qw3rY6Gapv1Fr24e",
  "zKUGsj6CZbCiMsugzmQ1Z0\n/qk8eP6SpXGrH2VqjIJeO7\n5/qw3rY6Gapv1Fr24e64wT\n+ZW1heWm7+PFrojwsLvfHf1\nsmtbwb+IA3LhCsdSlYUR/3l\nTB9XZoCh5PWsXxY8Y+Epi/k\nRFBVLeHFcNRNSe3cgMvCiN\nId/SntN9GqNiVFcZkEYJrc\nFZiZoIsdlTr647gSKis1V2H\nbUVRKT6emV1vIHIeankJBR\nbmAsbqhUMG06FhDcz6il+E",
  "IsdlTr647gSKis1V2H\nbUVRKT6emV1vIHIeankJBR\nbmAsbqhUMG06FhDcz6il+Ea\nZIwNaj8lbVtyHjAY6EqflY\n26Gu85a43AoXmesPN+dti\nI0T8R7ThpFNPINQKP6riS\n/ESBoIDEucgFTxAto0+Qki\nr48orH8JGHiQjmBwkfeqJk0\nrzWPISUd7QzQoZJKPOtYqs\nWAqk46yA4rn3fEM4DqHWYCh\nwg9Hc7CTMVP6mk+0nlSFSa\nGe8iZi",
  "oZJKPOtYqs\nWAqk46yA4rn3fEM4DqHWYCh\nwg9Hc7CTMVP6mk+0nlSFSa\nGe8iZinTRXurmSvqGqUEq\nqGHetPbL1i6nScuDRrhpqbC\nLJ286jc5oXNeg6TQRZsAj\njrtVEkCVhtxqwhEGWx+UTuO\nDEMxG3KhRWBVmYW3kadPvOT\nASvzVEG90vXW6tI+s8ZyogJ\nwN1nfgVTIe/q+nU9ibJOW9\n8U+AjbwiT1a3C8ri9rEknc\nFXjWE3NJlfIp",
  "ZyogJ\nwN1nfgVTIe/q+nU9ibJOW9\n8U+AjbwiT1a3C8ri9rEknc\nFXjWE3NJlfIpNmCEGyPXdOM\nxqHyTHQv0ATwTVfmQkVXtPt\nNCZasCfv34VLzUvKjxaVf+e\ni4Wja3jfkPySY0VJSZqyET/\nh8NDeD5iNcXRPDkpRJNHgS\nayUsl7O9o6liOF7aJNHMHBa\nGYFPoS3f4iVt06TQPNk3QW\nCFg2oVfJhSa5CjqyiZgZPiF\nJ71jAYXoIsP2Gk",
  "Ba\nGYFPoS3f4iVt06TQPNk3QW\nCFg2oVfJhSa5CjqyiZgZPiF\nJ71jAYXoIsP2GkOZFmXOyea\nH1jNEGt1si7kwD6vuhiqN0\nN03uJzWgjI8HM75NdUDlNGg\nzWeQlmrAcpTMkZnS0Vu/gIe\n78+5vprwtOq2Yn62P+4Nxwe\nyUYcjPTtbxfMTEo5EbcHRy\ntmWJajP2hrulyvjqxaf/s\nzWdqxw3WbkrQ7HqXbdrjXjI\nCfbThGu0E8YlFHorbG",
  "tmWJajP2hrulyvjqxaf/s\nzWdqxw3WbkrQ7HqXbdrjXjI\nCfbThGu0E8YlFHorbGI6Qes\nRz9QVvuPG64rsLhuk1J2p3k\n0Wk73KmJln+0O4QTszkmpXJ\ngjn2p9NsQFjUVtVNMEx4js\nQ1hMSm7Fvw/VnYEPDy6VhvC\n4lYhupoJYGnAJb6ENoTF9hb\numuMYVjc6oZbY7/XbMNYf\nEpS/BVtyEsxlSMneIpyzIkt\niGSxyHO45DmMcNS5pLwjG",
  "MYVjc6oZbY7/XbMNYf\nEpS/BVtyEsxlSMneIpyzIkt\niGSxyHO45DmMcNS5pLwjGS\nOGSFLyrWg8mHalUwASyPU28\njRGYxApgp1OA5iuaAr3CuP\nIVWsaKreM/V8d41HWuGjQB\nLG2Se8zN503WYBT3L490+k\nSyMpoArews0WdyekviCpyk\nguiS0svKb2w9ILSA0sPKM0t\nJW8EQfTKUvJ2EkTnlp5Tum/\npPqWlpSWle5buURpZGlH6xN\nIn",
  "w9ILSA0sPKM0t\nJW8EQfTKUvJ2EkTnlp5Tum/\npPqWlpSWle5buURpZGlH6xN\nInlIaWhpSuWrpKqbaUnEjhi\nWDpLqVDS4eUHlp6SOlrS19\nT+szSZ5S+sfQNpe8tfU/pI0\nsfUcosZSuWbpGKbeUfDoIo\nhVLVygNLCXvfnCvWbpFaWZp\nRuljSx9TOrCUvBXD8xScry\nB6OlktLnlj6nVFhK3t+C6\nKWlLylNLE0ofWHpC0rfWfqO\n0qeWPqU",
  "vBXD8xScry\nB6OlktLnlj6nVFhK3t+C6\nKWlLylNLE0ofWHpC0rfWfqO\n0qeWPqU0tpR8G4DTiaU7lNq\nvQFVB6bal25SeWXrm/i7Ap9\nMYuBbmpm1gk9LU0pTSdUvJm\nwIcJSw9JefJSI13tcnXJrK\nvRWrKHWyc8UltkvNITbmDjX\nenSW2yP0Vqyodk6Gv70w8pk\nFLY6U/mFvr4Kywt7P+y1P9t\n6cH2g4WHK+MvtDd73/V+6N3\nt9Xu/9x7",
  "v70w8pk\nFLY6U/mFvr4Kywt7P+y1P9t\n6cH2g4WHK+MvtDd73/V+6N3\nt9Xu/9x72nvW2enu9cGZjJ\np+pZv769p/52flb87db9cbM\natexit>uM7tXudv/v/AI+kJZI=</l\nmt+1  \u03b2 \u00b7 mt + (1 \u2212 \u03b2)\nX\ni2Bt\n@`i[\u03c6t \u2212 \u21b5 \u00b7 mt]\n@\u03c6\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7 mt+1\n<latexit sha\n1_base64=\"75TvOUZBRiF+w\n+fMC2gmLmcN5tw=\">AXS\n3iclZhbc9w0FIA3hUIJ",
  "1_base64=\"75TvOUZBRiF+w\n+fMC2gmLmcN5tw=\">AXS\n3iclZhbc9w0FIA3hUIJtxSG\n8MCLh0yZQtudLNMCj23S9Ja\nUbJpskjZOd2Rb9qRZceWk0\n09/okMz/wO3hgeOLK9q1hHe\nSAzZcX5Pl18JMuyvZSzXK6\nu/rVw7YMPr3/08Y1PFj/97P\nMvly6+dV+nhSZT0d+wpPs0\nCM5UzQkWS08M0oyT2OD3w\nTtYVPzijWc4SsScvUnock0i",
  "+dV+nhSZT0d+wpPs0\nCM5UzQkWS08M0oyT2OD3w\nTtYVPzijWc4SsScvUnock0i\nwkPlEQmi89KfjemE8LuWdQ\neX84HIaSpJlyTmEqSO6weJ\nbBTp3HFuD+7V8R8dNy+gFnO\nZcGMiJz7h5Vo1lpUbZsQv3Z\nRkhHuJRz0KojaCKdMOinO\nq4u4TpauSIRezRzHdRWe\nu4iHN2nDuOS7h6aQzwLrCeG\nltb9a/zm4MGgLK732bzi+\nU3gBolfxF",
  "zRzHdRWe\nu4iHN2nDuOS7h6aQzwLrCeG\nltb9a/zm4MGgLK732bzi+\nU3gBolfxFRIn5M8PxqspvK4\nVCP0Oa0W3SKnKfFPSESPoCh\nITPjs585dyCSOCESQb/h\nHTq6OUaJYnz/CL2wFRpyk2m\ngjZ2VMjwt+OSibSQVPhNR2H\nBHZk4ahqdgGXUl/wCsTPGI\nzV8ScEci9hshdQc/9JI6JC\nEp3bWMHUu7RiImSnhb1xFd\nV19moHQrFq4y153",
  "GI\nzV8ScEci9hshdQc/9JI6JC\nEp3bWMHUu7RiImSnhb1xFd\nV19moHQrFq4y153vzVpikMX\ntPUSO1ohq5QqBRVZa0H/VNw\nCgA1qcIJILm0KbKjxc6A4PC\nQueAgXvJFAYXOq8q1LSQNIK\ncdLQ3SINCyum0Y60jC6Yy7\ni7oDjOLUcBKjOYBRgq/FBj\nDnZTIqpZPUmnMovLXMXMHjI\niIlp30dxV6oq6hig4h6p+x/\nrdtF4RcdImLknroWY",
  "nZTIqpZPUmnMovLXMXMHjI\niIlp30dxV6oq6hig4h6p+x/\nrdtF4RcdImLknroWYqYlh7W\ndeRGc6LCLpOHTEsWIR16o\njhsVhWwpITCDLbXkMFxw7Km\nJXmTBVhbmMEu8bt+piphrc\n5rC/dL1NkqU/jNiZEQF4O5T\nv4wIn3b19WRuO7PknNW+KtC\npM4HJ6lYhWdRc1qwTuKo2V\nmGzpVh4mxBCPbHrqlGY1Fp\nyroXqALmTVdkTISXtLt1CZ",
  "J6lYhWdRc1qwTuKo2V\nmGzpVh4mxBCPbHrqlGY1Fp\nyroXqALmTVdkTISXtLt1CZa\nsCrt34VKzgtOje/0HdHpcrq\nrbRv0HZRMayovU1pAK/4+GA\nngQmusLIubkJdyYPAjUk5d\nw2N+NqSOZubBVpJ47KDBOJ\nMXxu3PItGtU0fMwSaxMVYIq\nHbhlzBhTHIYdmUVUDL8wiPd\nsoB84yL95hp9nuRFRtHmZ6x\nniNS62hYzph5W3Q2VK6G7b\n1",
  "HIYdmUVUDL8wiPd\nsoB84yL95hp9nuRFRtHmZ6x\nniNS62hYzph5W3Q2VK6G7b\n1A+rwVleDic0Suqe0ZGvSaf\nXlKIgGRGMqdqSqdv3Rye7ta\n7v57ypmi1Inq62fYH4LZKX\nyfno43zfmIkIUdbrQFZyhrW\nxZlv6grflyvTycvPtT2h\npRxbXbnLUbjtKu21xrxgBPd\n2yjHYLecjCDjfakeIPWRZ+\noO27Hncsl2FxbWbHLU7y6PV\ntrhz01",
  "u21xrxgBPd\n2yjHYLecjCDjfakeIPWRZ+\noO27Hncsl2FxbWbHLU7y6PV\ntrhz01j+4d4EDsfqmJTwQB3\n7Eu42IVOUWJRWMYlpZIhNy\nBTjomvB/5vKLoOHR9dqQqY4\nzFlXUwFTCig3L6EJmWJzC3f\nNmaqWxZ1y67Wx/+u2YRM8S\nmJzatuQqYTGyickTQ2xC\naE8Tsw8TnAeU1NKbZI5I6l\nlRtCSsi2obJ0JRUwpanR29\nTSGYyAJ8LosA2",
  "Q2xC\naE8Tsw8TnAeU1NKbZI5I6l\nlRtCSsi2obJ0JRUwpanR29\nTSGYyAJ8LosA2aco5Xm5de\ncJYxQKv4pGt49EVHUtiNKgC\nprSN7jH3beZJ6Z4uY1GU8\nXM6wUJ3BoOkPszE5/Xlik\n5wXmh6gem5pueYHmh6gGm\nKXoj8MJXmqK3Ey80/QM031\nN9zEtNC0wHWk6wjTUNMT0ia\nZPMPU19TFd13QdU6kpOpHCE\n0HTPUwnmk4wPdT0ENPXmr",
  "NC0wHWk6wjTUNMT0ia\nZPMPU19TFd13QdU6kpOpHCE\n0HTPUwnmk4wPdT0ENPXmr7\nG9JmzB9o+kbTN9r+h7TR5\no+wpRoSjDd0HQDU6op+nTgh\nWuarmHqaYre/eBe03SIap\niuljTR9jGmiK3orheaYpOt7\nAg1FTjulzTZ9jyjRF729e+\nFLTl5jGmsaYvtD0BabvNH2H\n6VNn2IaYq+DcDpRNdTPV\nXoDLHdEfTHUxPNT21fxeg82\nn0bAtz",
  "vtD0BabvNH2H\n6VNn2IaYq+DcDpRNdTPV\nXoDLHdEfTHUxPNT21fxeg82\nn0bAtzWzewjWmiaYLpqboT\nQGOEpqeoPNkKNpdbfa1Ce1\nroZhzC2szPquNch6KObewdn\nea1Ub7UyjmfIKGvrE/5ACK\nYWdfry0MjC/wuLC/s/9wS/9\n+zv3Vx6utV9ob/S+63fu90\nb9H7tPew96w17o56/8GDha\nCFYoMt/LP+9/M/yv416baGt\natexit>83Wv",
  "3fu90\nb9H7tPew96w17o56/8GDha\nCFYoMt/LP+9/M/yv416baGt\natexit>83Wv8/ft9f8AWq0d6A=</l\nmt+1  \u03b2 \u00b7 mt + (1 \u2212 \u03b2)\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7 mt+1\nStill in batches.",
  "Nesterov Momentum\nWithout Momentum, Loss = 1.31\nWith Momentum, Loss = 0.96\nNesterov Momentum, Loss = 0.80",
  "Fitting models\n\u2022 Maths overview\n\u2022 Gradient descent algorithm\n\u2022 Linear regression example\n\u2022 Gabor model example\n\u2022 Stochastic gradient descent\n\u2022 Momentum\n\u2022 Adam",
  "The challenge with fixed step sizes\nToo small and it will \nconverge slowly, but \neventually get there.\nToo big and it will move \nquickly but might bounce \naround minimum or away.\nMoves quickly in \none dimension \nbut slowly in the \nother.",
  "Solution Part 1: Normalized gradients\n\u2022 Measure gradient \ud835\udc264&5 and pointwise squared gradient \ud835\udc2f4&5\n\u2022 Normalize:\n<latexit sha1_base6\n4=\"Y312yhN2056ikfkU4AUGd0cXIoE=\">A\nXF3icpZhbc9w0FIA3XEu4pTDkhRcPmQIDJZP\ntlMtjmzS9JSXTdLGaUb2yl41suzYcrKpZ38\nIw4/hjeGVR/4NR7Z3VZ+jPDBkpqw432djmR\nZdpBJUeiVlX/m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALf",
  "ZdpBJUeiVlX/m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALflRlnO\nWBJIfBmdrh9e8LwQqdrXVxk/SVisRCRCpiF\n0uvC75wdRclrp7/sT72tf8kizPE8vPT/KWVj\n5Gcu1YNLbPAYvGwkwJycTG2+iE1+lqkwCnu\n+P2+CF/+7ypd3TheWVpZX6j+PFvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwT",
  "TheWVpZX6j+PFvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwTMWnrGYH0NRsYQXJ1WdxYl3CyJDL0pz+Ke\n0V0fvKJiSVFcJQGYCdOjAjMTdLHjUke/nFR\nCZaXmKmwaikrp6dQzU+INRc5DLa+gwMJcQF+\n9cMQgVxombt5X/DJMk4SpYeWvru9ApgIeC1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8",
  "YeWvru9ApgIeC1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8OV7GQHAYpkTkCpeQJ0mP0H\nk9RGFRSsBAw/SMXQu8nYnpGqleQw56WgviAa\nFTPJx1ojFkxl0lH2QPG8W54BXOcwC9BV+OF\noDvYypibT6zQf6zypChPDLeRMxbxuAoYcwsr\nexYqpYRLw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIk",
  "cwsr\nexYqpYRLw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIkbDFDljDIcls\n+hQEnom4VaGwKsjC3M7ToNt2ZiJ4bY4zuF+\n63npF0n/BUEZMAO4+8yuYCnlX0tntjdNzkX\ntmwIfeyOYrO4lLI+bYU0bgVG1sQk161whk2Y\nLQrBXdk3TG4fKM9EdoAngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDaho",
  "3TG4fKM9EdoAngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDahoqLM\nXBWZ8H+oaAgPNby+InL5Vo8iBQT14qYX9H\nU8dyvLBNpJ47KAjFpNBX6PYXsepeU0dwZ9ME\n9RUCpl74ZUKhSY6irmwCRoZfeDw7FlCIBhk2\nYwxlWpQ5J5sfWs8QqXWzLebCPKy6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3j",
  "QqXWzLebCPKy6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3jSO+/+esqbotOK+flG2x70C2anDEN+\nfrqB5yMmFnUkqgvOQ86JLEc7UFds+X6Zs+q\njZfkaUdO1y3KUm9bS/dtsO9pgf8fNPR203i\nEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvN\no9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv",
  "XDdZuS1DvN\no9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv/Hyp6Ah0fXakJY3C5E\nVzMBLA25xENoQlhsbuGu2cawulQN90qk9kI\nmU0Ii49YgkfdhLAYUzF2imcsy5DYhEgeRziP\nI5rHDEuZS8IzkjlmhCwp14LKR2lXMgEsjVFr\nY0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCm",
  "Y0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCmGY5YryZlA\nVkYTuI2dbepMT39BVJGTXBdWXpF6aWl5Qe\nWnpIaW4peSMIol1LydsJvMVbekHpgaUHlJaW\nlpQOLB1QGlkaUfrQ0oeUhpaGlK5ZukaptpSc\nSOGJYOk+pSNLR5QeWXpE6XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nV",
  "SNLR5QeWXpE6XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nVUtXKQ0sJe9+cK9Zuk1pZmlG6QNLH1A6tJS8\nFcPzFJyvIEHo6WS0ieWPqFUWEre34LomaXP\nKE0sTSh9aulTSl9Z+orSR5Y+ojS2lHwbgNOJ\npXuU2q9AVUHpjqU7lJ5beu7+LsBn0xi4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1",
  "beu7+LsBn0xi4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1mZ8ejXJeaRm3MHa3Wl6NdmfIjXjI\n9L19YPZhxRIKez0pwtLfwVlhYO7iz3f1q+u\n3N36d5q+4X2Ru/L3le9b3v93s+9e73Hve3eo\nf4W/4XxCALMg=</latexit>BfOzc19M7cy1/8bfGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit",
  "zc19M7cy1/8bfGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit \nsha1_base64=\"Og9K\nKu9YcKnfWsVwlfwkg\nZEIqNM=\">AW3Xic\nlZhLc90FIDd8irl\ncKQDRsPmTIMbTMJUx\n7LNmn6SkqS5tnG6R3\nZV/ZVI8uOLSc39dwl\nO4YtP4nfwI9gC1uOb\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+k",
  "P4nfwI9gC1uOb\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+kcf\nf/LpZ3M3Pt8vs6qI+\nF6Uyaw4DFnJpVB8Tw\nst+WFecJaGkh+EJ6u\nGH5zxohSZ2tUXOT9O\nWaJELCKmITSY0EY5\nyMxqPWt5Yn/TSB5rF\nlRZOe+PyMT/4fMJm\nPmB9Ew0z7QVywqAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/",
  "mB9Ew0z7QVywqAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/vbGtz\n4chgMs6hKudKRZGV5\ntLyU6+OaFVpEk+uB\n1XJcxadsIQfQVGxlJf\nHdZOLiX8TIkM/zgr4\np7TfRN+omZpWV6kI\nZgp06MSMxN0saNKxz\n8f10LleYqahuK+n\nrzDeJ9Yei4JGWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvl",
  "rzDeJ9Yei4JGWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvlp1UzFZNJ\n31hqHQ/EyY+XJ7qwW\noXkq3nBSaOYSi4Re\nDKpa76YLGIgOACxyA\nnIFC+hTpOfMPaXEYW\nlJwED7OxWQ/+8wmp\nWmeQE562kuiQSGXf\nNyzVokFU5n2lB1QfP\n+mbwDXBcwCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJ",
  "QfP\n+mbwDXBcwCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJCZdGPesXbD\n1n6qRLXJY3XS1MBFm\n7Rd/RBc2LGvadJoIs\nWIRJ32oiyJKwUQxZy\niDLXkA059E3GrQm\nFVkIW5VWRhv+3cRPD\naHOdwv/S9tZqk/4yh\njJgA3H3mVzAV8b6+m\ns1sf5qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y",
  "V8b6+m\ns1sf5qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y3DpXnoj9AE8A3X\nVUIFb+l3W5KsGRNOL\ngNQy0qyY/uLP7Ax8f\n1krltzH9INqGispd\nFZnw/6hoCI8mvL4g\nicvk2jyINBMXiZhf0\ndTxwq8sE2kmTsoCMW\nk0Bfo9heJ6l/TRHBn\nsxT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\na",
  "eJ6l/TRHBn\nsxT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\nazxBpdLMtFsI8rPob\nqjRCf9/gcnYVlOHhc\nMYvuTxEGQ3bfIZpY\nasQMkcmykdvwpKOAE\n47/5mytui0r46XrX\nHvQLZqeKIn46WMfzk\nRCLOhLVBacaZ12SWI\n72oK7Zcn27Z/X6q+/\nI0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDu",
  "Zcn27Z/X6q+/\nI0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDutx\n53HCNwuG6TUnqnebR\naTvcmYmWf7w74pqZY\n1Imh+bYl8mgDWFRU1\nE7xSzlCRLbEBbTqm/\nB/2NlR8Do2+1ISxu\nlaKvmQCWhlziIbQhL\nLa3cN/sYljdcKgbr\nV5Q+ibQiLj1iKR92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgi",
  "5Q+ibQiLj1iKR92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgipGWV\n8yASyNUWtjR2PdC1e\n/wS6I5ZKuvNK58hRa\nxYqu4j1Xw3uXNKwZq\ntAEsLRJ7jE/2HTeZC\nFOMRyzXEnOBbJymsA\nt7GxRZ3r6C+OanOTC\n+MLSC0rPLT2n9MDSA\n0oLS8kbQRg/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRG",
  "SA\n0oLS8kbQRg/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRGlq5auUqotJ\nSdSeCJYukvpyNIRpY\neWHlL6wtIXlD629DG\nlLy19SekbS9Qet/S\n+5QySxmla5auUcotJ\nZ8OwnjF0hVKQ0vJux\n/ca5ZuUZpbmlP6wNI\nHlA4tJW/F8DyzlBxv\n4MFoqaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTS",
  "DyzlBxv\n4MFoqaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTSwl3wbgdGLpDqX2\nK1BdUrpt6Talp5aeu\nr8L8Nk0hq6FuWkr2K\nQ0szSjdN1S8qYARwl\nLT8h5Mlbdrjb92kT2\ntVjNuIN1GZ9eTXIeq\nxl3sG53ml5N9qdYzf\niIdH1tf/YhBVIKO/1\ngbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn",
  "tf/YhBVIKO/1\ngbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn\ntb3p4XeX96f3v/eP/\nOD+Z/nf9t/vdWvXql\nQ=</latexit>u+YLr/c3/8d/Qa/2o\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7\nmt+1\npvt+1 + \u270f\n\ud835\udefc is the learning rate\n\ud835\udf16 is a small constant to prevent div by 0\nSquare, sqrt and div are all pointwise",
  "Solution Part 1: Normalized gradients\n\u2022 Measure gradient \ud835\udc264&5 and pointwise squared gradient \ud835\udc2f4&5\n\u2022 Normalize:\n<latexit sha1_base6\n4=\"Y312yhN2056ikfkU4AUGd0cXIoE=\">A\nXF3icpZhbc9w0FIA3XEu4pTDkhRcPmQIDJZP\ntlMtjmzS9JSXTdLGaUb2yl41suzYcrKpZ38\nIw4/hjeGVR/4NR7Z3VZ+jPDBkpqw432djmR\nZdpBJUeiVlX/m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALf",
  "ZdpBJUeiVlX/m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALflRlnO\nWBJIfBmdrh9e8LwQqdrXVxk/SVisRCRCpiF\n0uvC75wdRclrp7/sT72tf8kizPE8vPT/KWVj\n5Gcu1YNLbPAYvGwkwJycTG2+iE1+lqkwCnu\n+P2+CF/+7ypd3TheWVpZX6j+PFvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwT",
  "TheWVpZX6j+PFvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwTMWnrGYH0NRsYQXJ1WdxYl3CyJDL0pz+Ke\n0V0fvKJiSVFcJQGYCdOjAjMTdLHjUke/nFR\nCZaXmKmwaikrp6dQzU+INRc5DLa+gwMJcQF+\n9cMQgVxombt5X/DJMk4SpYeWvru9ApgIeC1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8",
  "YeWvru9ApgIeC1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8OV7GQHAYpkTkCpeQJ0mP0H\nk9RGFRSsBAw/SMXQu8nYnpGqleQw56WgviAa\nFTPJx1ojFkxl0lH2QPG8W54BXOcwC9BV+OF\noDvYypibT6zQf6zypChPDLeRMxbxuAoYcwsr\nexYqpYRLw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIk",
  "cwsr\nexYqpYRLw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIkbDFDljDIcls\n+hQEnom4VaGwKsjC3M7ToNt2ZiJ4bY4zuF+\n63npF0n/BUEZMAO4+8yuYCnlX0tntjdNzkX\ntmwIfeyOYrO4lLI+bYU0bgVG1sQk161whk2Y\nLQrBXdk3TG4fKM9EdoAngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDaho",
  "3TG4fKM9EdoAngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDahoqLM\nXBWZ8H+oaAgPNby+InL5Vo8iBQT14qYX9H\nU8dyvLBNpJ47KAjFpNBX6PYXsepeU0dwZ9ME\n9RUCpl74ZUKhSY6irmwCRoZfeDw7FlCIBhk2\nYwxlWpQ5J5sfWs8QqXWzLebCPKy6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3j",
  "QqXWzLebCPKy6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3jSO+/+esqbotOK+flG2x70C2anDEN+\nfrqB5yMmFnUkqgvOQ86JLEc7UFds+X6Zs+q\njZfkaUdO1y3KUm9bS/dtsO9pgf8fNPR203i\nEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvN\no9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv",
  "XDdZuS1DvN\no9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv/Hyp6Ah0fXakJY3C5E\nVzMBLA25xENoQlhsbuGu2cawulQN90qk9kI\nmU0Ii49YgkfdhLAYUzF2imcsy5DYhEgeRziP\nI5rHDEuZS8IzkjlmhCwp14LKR2lXMgEsjVFr\nY0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCm",
  "Y0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCmGY5YryZlA\nVkYTuI2dbepMT39BVJGTXBdWXpF6aWl5Qe\nWnpIaW4peSMIol1LydsJvMVbekHpgaUHlJaW\nlpQOLB1QGlkaUfrQ0oeUhpaGlK5ZukaptpSc\nSOGJYOk+pSNLR5QeWXpE6XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nV",
  "SNLR5QeWXpE6XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nVUtXKQ0sJe9+cK9Zuk1pZmlG6QNLH1A6tJS8\nFcPzFJyvIEHo6WS0ieWPqFUWEre34LomaXP\nKE0sTSh9aulTSl9Z+orSR5Y+ojS2lHwbgNOJ\npXuU2q9AVUHpjqU7lJ5beu7+LsBn0xi4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1",
  "beu7+LsBn0xi4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1mZ8ejXJeaRm3MHa3Wl6NdmfIjXjI\n9L19YPZhxRIKez0pwtLfwVlhYO7iz3f1q+u\n3N36d5q+4X2Ru/L3le9b3v93s+9e73Hve3eo\nf4W/4XxCALMg=</latexit>BfOzc19M7cy1/8bfGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit",
  "zc19M7cy1/8bfGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit \nsha1_base64=\"Og9K\nKu9YcKnfWsVwlfwkg\nZEIqNM=\">AW3Xic\nlZhLc90FIDd8irl\ncKQDRsPmTIMbTMJUx\n7LNmn6SkqS5tnG6R3\nZV/ZVI8uOLSc39dwl\nO4YtP4nfwI9gC1uOb\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+k",
  "P4nfwI9gC1uOb\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+kcf\nf/LpZ3M3Pt8vs6qI+\nF6Uyaw4DFnJpVB8Tw\nst+WFecJaGkh+EJ6u\nGH5zxohSZ2tUXOT9O\nWaJELCKmITSY0EY5\nyMxqPWt5Yn/TSB5rF\nlRZOe+PyMT/4fMJm\nPmB9Ew0z7QVywqAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/",
  "mB9Ew0z7QVywqAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/vbGtz\n4chgMs6hKudKRZGV5\ntLyU6+OaFVpEk+uB\n1XJcxadsIQfQVGxlJf\nHdZOLiX8TIkM/zgr4\np7TfRN+omZpWV6kI\nZgp06MSMxN0saNKxz\n8f10LleYqahuK+n\nrzDeJ9Yei4JGWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvl",
  "rzDeJ9Yei4JGWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvlp1UzFZNJ\n31hqHQ/EyY+XJ7qwW\noXkq3nBSaOYSi4Re\nDKpa76YLGIgOACxyA\nnIFC+hTpOfMPaXEYW\nlJwED7OxWQ/+8wmp\nWmeQE562kuiQSGXf\nNyzVokFU5n2lB1QfP\n+mbwDXBcwCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJ",
  "QfP\n+mbwDXBcwCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJCZdGPesXbD\n1n6qRLXJY3XS1MBFm\n7Rd/RBc2LGvadJoIs\nWIRJ32oiyJKwUQxZy\niDLXkA059E3GrQm\nFVkIW5VWRhv+3cRPD\naHOdwv/S9tZqk/4yh\njJgA3H3mVzAV8b6+m\ns1sf5qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y",
  "V8b6+m\ns1sf5qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y3DpXnoj9AE8A3X\nVUIFb+l3W5KsGRNOL\ngNQy0qyY/uLP7Ax8f\n1krltzH9INqGispd\nFZnw/6hoCI8mvL4g\nicvk2jyINBMXiZhf0\ndTxwq8sE2kmTsoCMW\nk0Bfo9heJ6l/TRHBn\nsxT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\na",
  "eJ6l/TRHBn\nsxT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\nazxBpdLMtFsI8rPob\nqjRCf9/gcnYVlOHhc\nMYvuTxEGQ3bfIZpY\nasQMkcmykdvwpKOAE\n47/5mytui0r46XrX\nHvQLZqeKIn46WMfzk\nRCLOhLVBacaZ12SWI\n72oK7Zcn27Z/X6q+/\nI0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDu",
  "Zcn27Z/X6q+/\nI0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDutx\n53HCNwuG6TUnqnebR\naTvcmYmWf7w74pqZY\n1Imh+bYl8mgDWFRU1\nE7xSzlCRLbEBbTqm/\nB/2NlR8Do2+1ISxu\nlaKvmQCWhlziIbQhL\nLa3cN/sYljdcKgbr\nV5Q+ibQiLj1iKR92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgi",
  "5Q+ibQiLj1iKR92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgipGWV\n8yASyNUWtjR2PdC1e\n/wS6I5ZKuvNK58hRa\nxYqu4j1Xw3uXNKwZq\ntAEsLRJ7jE/2HTeZC\nFOMRyzXEnOBbJymsA\nt7GxRZ3r6C+OanOTC\n+MLSC0rPLT2n9MDSA\n0oLS8kbQRg/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRG",
  "SA\n0oLS8kbQRg/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRGlq5auUqotJ\nSdSeCJYukvpyNIRpY\neWHlL6wtIXlD629DG\nlLy19SekbS9Qet/S\n+5QySxmla5auUcotJ\nZ8OwnjF0hVKQ0vJux\n/ca5ZuUZpbmlP6wNI\nHlA4tJW/F8DyzlBxv\n4MFoqaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTS",
  "DyzlBxv\n4MFoqaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTSwl3wbgdGLpDqX2\nK1BdUrpt6Talp5aeu\nr8L8Nk0hq6FuWkr2K\nQ0szSjdN1S8qYARwl\nLT8h5Mlbdrjb92kT2\ntVjNuIN1GZ9eTXIeq\nxl3sG53ml5N9qdYzf\niIdH1tf/YhBVIKO/1\ngbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn",
  "tf/YhBVIKO/1\ngbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn\ntb3p4XeX96f3v/eP/\nOD+Z/nf9t/vdWvXql\nQ=</latexit>u+YLr/c3/8d/Qa/2o\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7\nmt+1\npvt+1 + \u270f\n\ud835\udefc is the learning rate\n\ud835\udf16 is a small constant to prevent div by 0\nSquare, sqrt and div are all pointwise\nDividing by the positive root, so normalized to 1 \nand all that is left is the sign.",
  "Solution Part 1: Normalized gradients\n\u2022 Measure mean and pointwise squared gradient\n\u2022 Normalize:\n<latexit sha1_base6\n4=\"Y312yhN2056ikfkU4AUGd0cXIoE=\">A\nXF3icpZhbc9w0FIA3XEu4pTDkhRcPmQIDJZP\ntlMtjmzS9JSXTdLGaUb2yl41suzYcrKpZ38\nIw4/hjeGVR/4NR7Z3VZ+jPDBkpqw432djmR\nZdpBJUeiVlX/m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALflRlnO\nWBJIfB",
  "m3nr7nXfe/GB/MfvTxJ58\nu3PzsoEjLPOSDMJVpfhSwgkuh+EALflRlnO\nWBJIfBmdrh9e8LwQqdrXVxk/SVisRCRCpiF\n0uvC75wdRclrp7/sT72tf8kizPE8vPT/KWVj\n5Gcu1YNLbPAYvGwkwJycTG2+iE1+lqkwCnu\n+P2+CF/+7ypd3TheWVpZX6j+PFvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwTMWnrGYH0NRsY",
  "FvptYanX/m2\nf3vxi6A/TsEy40qFkRXHcX8n0SWUqDiWfzPt\nlwTMWnrGYH0NRsYQXJ1WdxYl3CyJDL0pz+Ke\n0V0fvKJiSVFcJQGYCdOjAjMTdLHjUke/nFR\nCZaXmKmwaikrp6dQzU+INRc5DLa+gwMJcQF+\n9cMQgVxombt5X/DJMk4SpYeWvru9ApgIeC1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8OV7GQHAYpkTk",
  "C1X\nx87KexMmk6zXDofidcbqk/1ZLULzRLzmpJ\naMZVcI/B4UlV8OV7GQHAYpkTkCpeQJ0mP0H\nk9RGFRSsBAw/SMXQu8nYnpGqleQw56WgviAa\nFTPJx1ojFkxl0lH2QPG8W54BXOcwC9BV+OF\noDvYypibT6zQf6zypChPDLeRMxbxuAoYcwsr\nexYqpYRLw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIkbDFDljDIcls",
  "Lw471K7Z2mTprE5dmdVdzE0HWft5\n1dE7zoZdp4gCxZh3LXqCLIkbDFDljDIcls\n+hQEnom4VaGwKsjC3M7ToNt2ZiJ4bY4zuF+\n63npF0n/BUEZMAO4+8yuYCnlX0tntjdNzkX\ntmwIfeyOYrO4lLI+bYU0bgVG1sQk161whk2Y\nLQrBXdk3TG4fKM9EdoAngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDahoqLM\nXBWZ8H+o",
  "ngm67MhYre0G7XJViy\nJuzfhqHmpeTHPyz/yMcn1Yq5bcx/SDahoqLM\nXBWZ8H+oaAgPNby+InL5Vo8iBQT14qYX9H\nU8dyvLBNpJ47KAjFpNBX6PYXsepeU0dwZ9ME\n9RUCpl74ZUKhSY6irmwCRoZfeDw7FlCIBhk2\nYwxlWpQ5J5sfWs8QqXWzLebCPKy6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3jSO+/+esqbotO",
  "6G6o0Qnf\n4HJ2FZTh4XDBr7k8QBkNmnwGamGLEfJHJsp\nHb/0C3jSO+/+esqbotOK+flG2x70C2anDEN+\nfrqB5yMmFnUkqgvOQ86JLEc7UFds+X6Zs+q\njZfkaUdO1y3KUm9bS/dtsO9pgf8fNPR203i\nEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvN\no9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv/Hyp6Ah0fXak",
  "9N2uDMTLf9of8Q1M8ekVA7NsS+VfhPCoqai\ndopwmMkNiEsJmXgv/Hyp6Ah0fXakJY3C5E\nVzMBLA25xENoQlhsbuGu2cawulQN90qk9kI\nmU0Ii49YgkfdhLAYUzF2imcsy5DYhEgeRziP\nI5rHDEuZS8IzkjlmhCwp14LKR2lXMgEsjVFr\nY0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCmGY5YryZlA\nVk",
  "gG8RyQVde4Vx5Cq1iRVfxwNXw\n4JqGNUMVmgCWtsg95vlbzpswCmGY5YryZlA\nVkYTuI2dbepMT39BVJGTXBdWXpF6aWl5Qe\nWnpIaW4peSMIol1LydsJvMVbekHpgaUHlJaW\nlpQOLB1QGlkaUfrQ0oeUhpaGlK5ZukaptpSc\nSOGJYOk+pSNLR5QeWXpE6XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nVUtXKQ0sJe9+c",
  "XNLn1P62NLHlL6w\n9AWlry19Tel9S+9TyixlK5buk4pt5R8Ogi\nVUtXKQ0sJe9+cK9Zuk1pZmlG6QNLH1A6tJS8\nFcPzFJyvIEHo6WS0ieWPqFUWEre34LomaXP\nKE0sTSh9aulTSl9Z+orSR5Y+ojS2lHwbgNOJ\npXuU2q9AVUHpjqU7lJ5beu7+LsBn0xi4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1mZ8ejXJeaRm3",
  "4FuaW\nrWCL0tTSlNINS8mbAhwlLD0j58lItbva9GsT2\ndciNeMO1mZ8ejXJeaRm3MHa3Wl6NdmfIjXjI\n9L19YPZhxRIKez0pwtLfwVlhYO7iz3f1q+u\n3N36d5q+4X2Ru/L3le9b3v93s+9e73Hve3eo\nf4W/4XxCALMg=</latexit>BfOzc19M7cy1/8bfGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit \nsha1_base6",
  "fGPxT8X/2rUt+baz7vd\nmt+1  @L[\u03c6t]\n@\u03c6\nvt+1  @L[\u03c6t]\n@\u03c6\n2\n<latexit \nsha1_base64=\"Og9K\nKu9YcKnfWsVwlfwkg\nZEIqNM=\">AW3Xic\nlZhLc90FIDd8irl\ncKQDRsPmTIMbTMJUx\n7LNmn6SkqS5tnG6R3\nZV/ZVI8uOLSc39dwl\nO4YtP4nfwI9gC1uOb\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+kcf\nf/LpZ3M3P",
  "b\nN+r+hxlQWbKFef7rM\neRLMsOcylKvbT015W\nr7z73vsfXPvw+kcf\nf/LpZ3M3Pt8vs6qI+\nF6Uyaw4DFnJpVB8Tw\nst+WFecJaGkh+EJ6u\nGH5zxohSZ2tUXOT9O\nWaJELCKmITSY0EY5\nyMxqPWt5Yn/TSB5rF\nlRZOe+PyMT/4fMJm\nPmB9Ew0z7QVywqAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/vbGtz\n4chgMs",
  "qAae\ntdN6qA8LbSJnHWRW\nwHPSyEzNRnMLSwtLj\nV/Pi0sd4UFr/vbGtz\n4chgMs6hKudKRZGV5\ntLyU6+OaFVpEk+uB\n1XJcxadsIQfQVGxlJf\nHdZOLiX8TIkM/zgr4\np7TfRN+omZpWV6kI\nZgp06MSMxN0saNKxz\n8f10LleYqahuK+n\nrzDeJ9Yei4JGWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvlp1UzFZNJ\n31h",
  "GWF1Bg\nUSGgr340YpAoDem/H\nih+HmVpytSwDlbWti\nFlIU+Eqvlp1UzFZNJ\n31hqHQ/EyY+XJ7qwW\noXkq3nBSaOYSi4Re\nDKpa76YLGIgOACxyA\nnIFC+hTpOfMPaXEYW\nlJwED7OxWQ/+8wmp\nWmeQE562kuiQSGXf\nNyzVokFU5n2lB1QfP\n+mbwDXBcwCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJCZdGPesXbD\n1",
  "wCdBV+OJq\nDnZypyfQ6zce6SOvS\nxHALBVMJb5qAIUdMm\nhH1DVJCZdGPesXbD\n1n6qRLXJY3XS1MBFm\n7Rd/RBc2LGvadJoIs\nWIRJ32oiyJKwUQxZy\niDLXkA059E3GrQm\nFVkIW5VWRhv+3cRPD\naHOdwv/S9tZqk/4yh\njJgA3H3mVzAV8b6+m\ns1sf5qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y3DpXnoj9AE8A",
  "qcs8Y3BT72Rz\nBZ/UtYkbTDmjYCo+p\niE2o2uUImzRaEYA/t\nm6Y3DpXnoj9AE8A3X\nVUIFb+l3W5KsGRNOL\ngNQy0qyY/uLP7Ax8f\n1krltzH9INqGispd\nFZnw/6hoCI8mvL4g\nicvk2jyINBMXiZhf0\ndTxwq8sE2kmTsoCMW\nk0Bfo9heJ6l/TRHBn\nsxT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\nazxBpdLMtFsI8",
  "xT1FQKmXvhlQqFJj\nuO+bAJGhl94yDoWUI\nQGbVjGRWVgUnmx9\nazxBpdLMtFsI8rPob\nqjRCf9/gcnYVlOHhc\nMYvuTxEGQ3bfIZpY\nasQMkcmykdvwpKOAE\n47/5mytui0r46XrX\nHvQLZqeKIn46WMfzk\nRCLOhLVBacaZ12SWI\n72oK7Zcn27Z/X6q+/\nI0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDutx\n53HCNwuG6",
  "I0k4crtuUpN6ul27b\n4V7SA364ejtBvGIR\nR2J6up6SD1iOdqDutx\n53HCNwuG6TUnqnebR\naTvcmYmWf7w74pqZY\n1Imh+bYl8mgDWFRU1\nE7xSzlCRLbEBbTqm/\nB/2NlR8Do2+1ISxu\nlaKvmQCWhlziIbQhL\nLa3cN/sYljdcKgbr\nV5Q+ibQiLj1iKR92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgipGWV\n8yASyNU",
  "R92\nGsJhQMXGKJyzPkdiG\nSB5HOI8jmscS7lLw\njOSO2aELCnXgipGWV\n8yASyNUWtjR2PdC1e\n/wS6I5ZKuvNK58hRa\nxYqu4j1Xw3uXNKwZq\ntAEsLRJ7jE/2HTeZC\nFOMRyzXEnOBbJymsA\nt7GxRZ3r6C+OanOTC\n+MLSC0rPLT2n9MDSA\n0oLS8kbQRg/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRGlq5auUqotJ\nS",
  "g/t5S8nc\nA7uaVnlO5buk9pZWl\nF6Z6le5TGlsaUPrT0\nIaWRpRGlq5auUqotJ\nSdSeCJYukvpyNIRpY\neWHlL6wtIXlD629DG\nlLy19SekbS9Qet/S\n+5QySxmla5auUcotJ\nZ8OwnjF0hVKQ0vJux\n/ca5ZuUZpbmlP6wNI\nHlA4tJW/F8DyzlBxv\n4MFoqaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTSwl3wbgdGLpDq",
  "qaT0iaVPKBWk\nve3MH5m6TNKU0tTSp\n9a+pTS15a+pvSRpY8\noTSwl3wbgdGLpDqX2\nK1BdUrpt6Talp5aeu\nr8L8Nk0hq6FuWkr2K\nQ0szSjdN1S8qYARwl\nLT8h5Mlbdrjb92kT2\ntVjNuIN1GZ9eTXIeq\nxl3sG53ml5N9qdYzf\niIdH1tf/YhBVIKO/1\ngbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn\ntb3p4XeX96f",
  "gbmEZf4Wlhf3vF5d/\nXLy7fXfh3kr3hfa9\n5X3tfet+z95N3zHn\ntb3p4XeX96f3v/eP/\nOD+Z/nf9t/vdWvXql\nQ=</latexit>u+YLr/c3/8d/Qa/2o\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7\nmt+1\npvt+1 + \u270f\n<latexit sha1_base64=\"BUyZCK85q7SXMvJvVg/W/mUhw=\">AWwXiclZhbc9tEFIDVc\nivh0hSGvPCiIdMZBlqPXSjwkyb1L0lJU4TJ2nj1LOSV/I2q5",
  "iclZhbc9tEFIDVc\nivh0hSGvPCiIdMZBlqPXSjwkyb1L0lJU4TJ2nj1LOSV/I2q5WiS+JU41/Er+ER+DOclWRvdc7mgcwUL+f7tJezF128RIos73b/vnb9gw8/+viTG5+ufPb5F1/eXL31UEWF6nPh34s4/TIYxm\nXQvFhLnLJj5KUs8iT/NA73dT8JynmYjVfn6Z8JOIhUoEwmc5hMar/VHE8qkXlNF8XOY/9ubu7+7I46FQpQckFbP5yk+drjs",
  "Vfn6Z8JOIhUoEwmc5hMar/VHE8qkXlNF8XOY/9ubu7+7I46FQpQckFbP5yk+drjsauXfv1b/3O92VEVcTI6yMV9e7nW7159JCrymsO83fYHzrm8lo\nEvtFxFXuS5Zlx71ukp+ULM2FL/l8ZVRkPGH+KQv5MRQVi3h2Ulbjnbu3ITJxgziFfyp3q+j7V5QsyrLyANTjy7DTAdt7LjIg9OSqGSIufKrxsKCunmsauT505Eyv1cXkKB+amAvr+lKXMz",
  "LyANTjy7DTAdt7LjIg9OSqGSIufKrxsKCunmsauT505Eyv1cXkKB+amAvr+lKXMzy\nHFKyPFL/w4ihjkZrTR352XTSb5WVGlez5vO/3K0am8yth4tr+sReQ8Eu84qaRSdCVXCDyclyXvhB0MBAcgOpyAWPEM6qxXhtDFJaXBAzci2fQucB9OSdVq5yHkJOW9poUEgkn7WsTWLBVEY\ntZQ8U173tasBhCfputRJ9juZgL2Fqvrgu57M8jcpMx3ALK",
  "poUEgkn7WsTWLBVEY\ntZQ8U173tasBhCfputRJ9juZgL2Fqvrgu57M8jcpMx3ALKVMhr5qAIftM6hG1DVICZf6LesPbL1k6rRJXJxUXU1BFn7advJU5oXNWk7VQRZsAjDtlVFkCXhMJiwiEGWm/IYBhy5OmJXhcKqI\nAtzkMZeu+1ER/DanCWwX9pevyTpP2coIzoAu0/CqZ83tY346XtLpJzXvm6wGfuFCarfQlLw3pYi0ZgVE1sTs0qV8ik2YJ",
  "coIzoAu0/CqZ83tY346XtLpJzXvm6wGfuFCarfQlLw3pYi0ZgVE1sTs0qV8ik2YJQGl+0Td0bi8oT0R6gDuBNV6RCBe9pd6oSLFkdHt2BoaF5Md3O/\nf57KTs6m2j/0OyCRVlRWKrSIf/R0UTuP3g9QURPHmxRJMHgWryYgnO5o6luKFrSPV3EFBKCZFfom2vwhV+5oqgjsbR6ivEND1wi8TCk1yELRlHdAy/MKN1LKAfDRIvx6jL+OsSDk5/NB6",
  "vwhV+5oqgjsbR6ivEND1wi8TCk1yELRlHdAy/MKN1LKAfDRIvx6jL+OsSDk5/NB6hkil\n62MxFfpm1T5QpRba5waXy6ugDeHc37F5R7KqFfn04sLNWEpSuZMT+nszSjLYvZdn815XRaoX8bKtpD/oFs1P4Pj8b+H5CIlFHYnqgicXa12SWJb2oK7lcn2/Z+XWmx/I0g4trt2UpN6ml\n3b4l7RA362bentNvGIR2J6mp6SD1iWdqDux53LaNwuLaT",
  "Wmx/I0g4trt2UpN6ml\n3b4l7RA362bentNvGIR2J6mp6SD1iWdqDux53LaNwuLaTUnqXeTRalvcpYmWf7A/5TnTj0mxnOjHvliO6hAWcyrmVjGOeIjEOoTFqGhb8P9Y2RNw82hbdQiLg0y0NR3A0oRLPIQ6hMV6C7f\nNJobVbYu6bVeZTKbIrENYfMIiPOo6hMWQiqFVPGVJgsQ6RPI4xXmc0jwmWEpsEp6RxDIjZEnZFlQ6jduSDmBphlqbWRqD",
  "hMWQiqFVPGVJgsQ6RPI4xXmc0jwmWEpsEp6RxDIjZEnZFlQ6jduSDmBphlqbWRqDHshYoQabIJYzuvIy68pTaBUruoqHtoaHVzScM1ShDmBph+wxd7Rj3\nWQeTrF+Q7UkORHISmgCB9gZUGfx9AdvxeRJzgsuDb2k9MLQC0oPDT2kNDWUvBF4wUtDyduJF5wbek7pgaEHlBaGFpQODR1SGhgaUPrY0MeU+ob6lG4auklpbih5IoU7gqH7lE4NnVJ",
  "bek7pgaEHlBaGFpQODR1SGhgaUPrY0MeU+ob6lG4auklpbih5IoU7gqH7lE4NnVJ6ZOgRpa\n8MfUXpU0OfUvra0NeUvjP0HaUPDX1IKTOUdo3tE8pN5R8OvCDUM3KPUMJe9+sNcMHVCaGJpQ+sjQR5RODCVvxXA/M5Q83sCN0VBJ6TNDn1EqDCXvb17wtAXlEaGRpQ+N/Q5pW8NfUvpE0Of\nUBoaSr4NwNOJoXuUmq9AZUbprqG7lJ4Zemb/LsCX",
  "EaGRpQ+N/Q5pW8NfUvpE0Of\nUBoaSr4NwNOJoXuUmq9AZUbprqG7lJ4Zemb/LsCX0+jZFuaOqWCH0tjQmNItQ8mbAjxKGHpKnicD1Zxqi69N5FwL1JbWJPxdUk54FacgtrTqfF1eR8CtST0nX+wfLDymQUjpx6vrPfwVl\nhYO7nV6v3Tu7/68/mCj+UJ7w/nW+c753uk5vzoPnKfOwBk6vOn85fzj/Pv2uaWEvW0lq9fq25mun9bdW/gc2nuej</",
  "W+c753uk5vzoPnKfOwBk6vOn85fzj/Pv2uaWEvW0lq9fq25mun9bdW/gc2nuej</latexit>\nmt+1 =\n2\n4\n3.0\n\u22122.0\n5.0\n3\n5\n<latexit sha1_base64=\"xXJ7C1mbHuDFbERQLmcFRqhnZzw=\">AWwXiclZjZbtGFECZdE\nvdzWlRv/SFqBGgaBNBCpIuDwUSO8pmp7Zjy3ZiOcKQGlITD4c0F1kOoS/q1/Sx7c/0DklpwnvHDzWQanrP4Sx3Fi5e",
  "O8pmp7Zjy3ZiOcKQGlITD4c0F1kOoS/q1/Sx7c/0DklpwnvHDzWQanrP4Sx3Fi5eIkWd7t/X7v+wYcfzJjU9XPv8iy+/Wr359WEWF6nPB34s4/TYxmX\nQvFBLnLJj5OUs8iT/Mg729T8aMrTMTqIL9M+GnEQiUC4bMcQqPV/jBi+cQLyul8VOY/9ebu7+7Q46FQpQckFbP5ym+drjscuvfqn7v3O92VIVdjI6yMVte7nW7159JCrymsO83f7ujmt",
  "6FQpQckFbP5ym+drjscuvfqn7v3O92VIVdjI6yMVte7nW7159JCrymsO83f7ujmt+PhOPa\nLiKvclyzLTnrdJD8tWZoLX/L5yrDIeML8MxbyEygqFvHstKzGO3dvQWTsBnEK/1TuVtH3ryhZlGWXkQemHl2GmQ7a2EmRB7+elkIlRc6VXzcUFNLNY1cnzx2LlPu5vIQC81MBfX9CUuZn0OKV4\naKX/hxFDHIzXCjvzcvm0zy86JK93zedvqVo1N5",
  "lPu5vIQC81MBfX9CUuZn0OKV4\naKX/hxFDHIzXCjvzcvm0zy86JK93zedvqVo1N5lbHx7GBZi8h5JN5xUkml6EquEHg4L0veCTsYCA5AdDgBseIZ1FmvDLeHKCwvCRi4F8+gc4H7ck6qVjkPISct7TXRoJBIPmtZm8SCqYxayj4o\nrnvL1YDEvTdaiX6HM3BfsLUfHFdzmd5GpWZjuEWUqZCXjUBQ/aZ1CNqG6qQEi71W9Yf2HrJ1FmTuDipuprq",
  "fsLUfHFdzmd5GpWZjuEWUqZCXjUBQ/aZ1CNqG6qQEi71W9Yf2HrJ1FmTuDipuprqCLIO0raTpzQvatx2qgiyYBGbauKIEvCYTBmEYMsN+URDhydcSuCoVQRbmbhp\n7bYTHcFrc5bAfml7/ZKkf8pQRnQAdp/+FUz5vK1vxkvbXSRnWvm6wGfuBCarfQlLw3pYi0ZgVE1sTs0qV8ik2YJQGl+0Td0bi8oT0R6gDuBNV6RCBe9pt6sSLFkdHt6GoaF",
  "Yi0ZgVE1sTs0qV8ik2YJQGl+0Td0bi8oT0R6gDuBNV6RCBe9pt6sSLFkdHt6GoaF5Cd3Ovf57LTs6m\n2j/0OyCRVlRWKrSIf/R0VjuP3g9QURPHmxRJMHgWryYgnO5o6luKFrSPV3EFBKCZFfom2vwhV+5oqgjsbR6ivEND1wi8TCk1yELRlHdAy/MKN1LKAfDRIvx6jL+OsSDk5/NB6hkil62MxFfpm\n1T5QpRba5waXy6ugDeHKb/icg9l1Kvz",
  "fDRIvx6jL+OsSDk5/NB6hkil62MxFfpm\n1T5QpRba5waXy6ugDeHKb/icg9l1Kvz6cWFGrMUJXOmp3T2ZpjlsMVsu7+a8rpotUJ+vtW0B/2C2Sl8n5+PtvB8hMSijkR1wZOLtS5JLEt7UNdyub7fs3LrzY9kaYcW125KUm/TS7tca/oAT/\nftvR2m3jEo5EdTU9pB6xLO1BXfY8btGYXHtpiT1LvJotS3u0kTLPziY8Jzpx6RYjvVjXyHdQiLORV",
  "TU9pB6xLO1BXfY8btGYXHtpiT1LvJotS3u0kTLPziY8Jzpx6RYjvVjXyHdQiLORVzqxhHPERiHcJiVLQt+H+s7Au4ebStOoTF3Uy0NR3A0phLPIQ6hMV6C7fNJobVbYu6b\nVeZTCbIrENYfMIiPOo6hMWQiqFVPGNJgsQ6RPI4wXmc0DwmWEpsEp6RxDIjZEnZFlQ6iduSDmBphlqbWRqDHshYoQabIJYzuvIy68pTaBUruoHtoYHVzScM1ShDmB",
  "nZFlQ6iduSDmBphlqbWRqDHshYoQabIJYzuvIy68pTaBUruoHtoYHVzScM1ShDmBph+wxd7hj3WQeTrF+Q7Uk\nORHISmgCd7GzS53F0x+8FZMnOS+4NPS0gtDLyg9MvSI0tRQ8kbgBS8NJW8nXjA1dErpoaGHlBaGFpQODB1QGhgaUPrY0MeU+ob6lG4auklpbih5IoU7gqEHlE4MnVB6bOgxpa8MfUXpU0OfUvr\na0NeUvjP0HaUPDX1IKTOUdo",
  "pbih5IoU7gqEHlE4MnVB6bOgxpa8MfUXpU0OfUvr\na0NeUvjP0HaUPDX1IKTOUdo3tE8pN5R8OvCDUM3KPUMJe9+sNcM3aU0MTSh9JGhjygdG0reiuF+Zih5vIEbo6GS0meGPqNUGEre37zghaEvKI0MjSh9buhzSt8a+pbSJ4Y+oTQ0lHwbgKcTQ\n/cpNV+ByozSPUP3KD039Nz+XYAvp9GzLcwdU8EOpbGhMaVbhpI3BXiUMPSMPE8GqjnVFl+",
  "ByozSPUP3KD039Nz+XYAvp9GzLcwdU8EOpbGhMaVbhpI3BXiUMPSMPE8GqjnVFl+byLkWqCW3sCbji6tJzgO15BbWnE6Lq8n5FKgln5Cu9w+XH1IgpXDSj1bXe/grLC0c3u30fu7c37u3\n/mCj+UJ7w/nO+d75wek5vzgPnKfOrjNwfOdP5y/nH+ftc01sZaspbV6/VpzTdO62+t/A8geOe5</latexit>\nvt+1 =\n2\n4\n9.0\n4.0\n25.0\n3\n5\n<latexit",
  "spbV6/VpzTdO62+t/A8geOe5</latexit>\nvt+1 =\n2\n4\n9.0\n4.0\n25.0\n3\n5\n<latexit sha1_base64=\"SGq3Ss5LRy5m5lk/932q81RsPKw=\">AW6niclZhb9xEFIA3bYESbimIvP\nBiEVCtF1lEQVekNqk6S0pSZprG6fR2Dv2TjMeO/Y42dTaP8Eb4pWfxAt/hSfO2N6d+pzJAyulnp7v81zOzPgWZFIUen5n7lr1298OFHNz+e/+Tz7/YuHWl/tF",
  "hSfO2N6d+pzJAyulnp7v81zOzPgWZFIUen5n7lr1298OFHNz+e/+Tz7/YuHWl/tFWuYh3wtTmeaHASu4FIrvaElP8xyzpJA8oPg\ndNXwg3OeFyJVu/oy48cJi5WIRMg0hE4WlB/lLKz8hOlREFXJ5KTSdwaTSeUXZ7mexc+n8Ts+zwohUzXxfvX8gMdCVQFIuRhP5gf9Zc/3vXvtEQ7zPldDK8yfLCwt95frn0cLg7aw1Gt/Wye3vh76wzQsE650",
  "uRhP5gf9Zc/3vXvtEQ7zPldDK8yfLCwt95frn0cLg7aw1Gt/Wye3vh76wzQsE650KFlRHA2\nWM31csVyLUPLJvF8WPGPhKYv5ERQVS3hxXNWJmXi3ITL0ojSHP6W9Ovr+GRVLiuIyCcA0Ay0wM0EXOyp19MtxJVRWaq7CpqGolJ5OPZNlbyhyHmp5CQUW5gL6oUjBpnWMBfzvuIXYZokDHLjr6xtQ7abTPKzsp6Xya\nTrNWOSeVxsqz3VktQvNEv",
  "6oUjBpnWMBfzvuIXYZokDHLjr6xtQ7abTPKzsp6Xya\nTrNWOSeVxsqz3VktQvNEvOkloxlVwh8HhSVbwf9zEQHIDocwJSxYvJdJF4A0RhHUrAwIN0DJ2LvJcTUrXSPIacdLTXRINCJvm4Y60SC6Yy6Sg7oHjebc8ADksw9OqVGHI0BzsZU5PpeZqPdZ5UhYnhFnKmYl43\nAUMOmTQj6hqlBJODTvWb9h6ydRpm7g0q7uamwiydvOuo3OaFzXsOnU",
  "hFnKmYl43\nAUMOmTQj6hqlBJODTvWb9h6ydRpm7g0q7uamwiydvOuo3OaFzXsOnUEWbAI465VR5Al4aoxZAmDLflExhw4pmIWxUKq4IszK08DbptZyaC1+Y4g/3S9dYqkv5zhjJiArD7zFEwFfKuvprObG+anPaNwU+9kYwWd1\nTWB43w5o2AqNqYxNq1rlCJs0WhPL0omua3jhUnonuAE0Ab7oyFyp6T7tbl2DJmrB/F4al5If3evf5+PjatlsG",
  "Js0WhPL0omua3jhUnonuAE0Ab7oyFyp6T7tbl2DJmrB/F4al5If3evf5+PjatlsG/MPySZUVJSZqyIT/h8VDeE+hdcXRPDkpRJNHgTqyUslXN/R1LEcL2wTqecOCkIxKfQl2v4iVt1z6g\njubJqgvkLA1AtHJhSa5CjqyiZgZDjCHdexgEI0yLAZYyjTosw5ufih9QyRWjeXxVyYm1X3giqN0L1ucDk7C8pwczjnV5weoIwGT6DtFRDlqNkjs2Uj",
  "ufih9QyRWjeXxVyYm1X3giqN0L1ucDk7C8pwczjnV5weoIwGT6DtFRDlqNkjs2Ujt/4hYt5tr9ZQ3RacV87P1tj3oF8xOGYb87GQdz0dMLOpI\nVBc84jrksRytAd1zZbr+z2r1t98T5Z27HDdpiT1tr102w73ih7wsw1HbzeIRyzqSFRX20PqEcvRHtTlzuOGaxQO121KUu80j07b4c5MtPyj3RHXzDwmpXJoHvtS6TchLGoqaqeYJjxGYhPCYlJ",
  "GaxQO121KUu80j07b4c5MtPyj3RHXzDwmpXJoHvtS6TchLGoqaqeYJjxGYhPCYlJ2Lfg/VnYE3Dy6VhP\nC4lYhupoJYGnIJR5CE8Jis4W7ZhvD6oZD3XCrTGYjZDYhLD5hCR51E8JiTMXYKZ6yLENiEyJ5HOE8jmgeMyxlLgnPSOaYEbKkXAsqH6VdyQSwNEatjR2NtW9p3QbIJYLuvIK58pTaBUruor3XA3vXdGwZqhCE8DSJt\nljnr/p3GQBTr",
  "atjR2NtW9p3QbIJYLuvIK58pTaBUruor3XA3vXdGwZqhCE8DSJt\nljnr/p3GQBTrF5Q3UkORPIymgCt7CzRZ3p0x+8IJMnuSC6tPS0gtLyg9sPSA0txS8kYQRC8tJW8nQXRu6Tml+5buU1paWlK6Z+kepZGlEaWPLX1MaWhpSOmqpauUakvJEyncESzdpXRk6YjSQ0sPKX1l6StKn1r6\nlNLXlr6m9J2l7yh9aOlDSpmljNI1S9co5ZaSTwdBtGL",
  "6YjSQ0sPKX1l6StKn1r6\nlNLXlr6m9J2l7yh9aOlDSpmljNI1S9co5ZaSTwdBtGLpCqWBpeTdD/apVuUZpZmlD6y9BGlQ0vJWzHczywljzdwY7RUvrM0meUCkvJ+1sQvbD0BaWJpQmlzy19TulbS9S+sTSJ5TGlpJvA/B0YukOpfYrUFVQum3\npNqVnlp65vwvw2TQGroW5aSvYpDS1NKV03VLypgCPEpaekufJSLVXtenXJnJdi9SMO1ib8enZJ",
  "wvw2TQGroW5aSvYpDS1NKV03VLypgCPEpaekufJSLVXtenXJnJdi9SMO1ib8enZJOeRmnEHa69O07PJ9SlSMz4iXV/bn31IgZTClf5kYWmAv8LSwv4P/cFP/fvbPy49WGm/0N7sfdP7tvdb9D7ufeg97\nS31dvrhb2/e/OXZ+7sSgXf1/8Y/HPRr02157zVa/zW/zrPzC+CI=</latexit>\nmt+1\npvt+1 + \u270f =\n2\n4\n1.0\n\u22121.0\n1.0\n3\n5",
  "Solution Part 1: Normalized gradients\n\u2022 algorithm moves downhill a fixed \ndistance \u03b1 along each coordinate\n\u2022 makes good progress in both \ndirections \n\u2022 but will not converge unless it happens \nto land exactly at the minimum",
  "Adaptive moment estimation (Adam)\n<latexit sha1_base64=\"/rFjEWLOJfo7Y2pGRxjmVSHWthQ=\">AXaHiclZhbU9w2FICX9JbSG2\nmnZTp98YRJ2kShs2kl8cEQm6Qcl0gwYSRvbJXQZaNLcMSz60/7J/odMf0SN7d4XPEQ9hJl31fJ8l+ehi2UEmRaGXlv6ZufbRx598+tn1z2e/+PKr7+Zu/HtXpGWech7YSrT/CBgBZdC8Z4WvKDLOcs\nCSTfD05WDN8/43khUrWrLzJ+lL",
  "+Zu/HtXpGWech7YSrT/CBgBZdC8Z4WvKDLOcs\nCSTfD05WDN8/43khUrWrLzJ+lLBYiUiETEPoeO4/zw+i5LjSd7sj72df8kizPE/PIcw18/ywn+qJMvLuere792tyx/OjnIWVn7FcCya9UOwsoEw3tHIxpvoyFepKpOA57vz5rgmavNmCXJpUbPLjVaI2\njV2Lc/sO1cxAN95+2D47mFpcWl+s+jhe64sNAZ/20e3/i+7/fTsEy40qFkRXHYXcr",
  "V2Lc/sO1cxAN95+2D47mFpcWl+s+jhe64sNAZ/20e3/i+7/fTsEy40qFkRXHYXcr0UWXqDyUfzfplwTMWnrCYH0JRsYQXR1U9LCPvFkT6XpTm8E9pr45evqJiSVFcJAGYCdODAjMTdLHDUkd/HFVCZaXm\nKmwaikrp6dQzY+z1Rc5DLS+gwMJcQF+9cMAgZRpmwqyv+HmYQjZVv/KXV7cgYQGPhar4aVnPitGo7azWDofiVcbyi91pLULzRLznpJ",
  "pmwqyv+HmYQjZVv/KXV7cgYQGPhar4aVnPitGo7azWDofiVcbyi91pLULzRLznpJaMZVcIfB4VFV8MV7EQHAYpETkCpeQJ0mP0HkdRGFVSABAw/SIX\nQu8rZHpGqleQw5aWlviAaFTPJhy1ohFgxl0lJ2QPG8W54BXOcwCtBV+OFoDHYypkaT6zQf6jypChPDLeRMxbxuAm45hAm+jQ1VSgmXhi3rT2xtM3UyTlya1V3NTQRZu3nb0TnNi+q3nTq",
  "DLeRMxbxuAm45hAm+jQ1VSgmXhi3rT2xtM3UyTlya1V3NTQRZu3nb0TnNi+q3nTqCLJiEcduqI8iS\nsGf1WcIgy+PyMdxw4pmIWxUKq4JMzM08DdptZyaC5+Ywg/XS9lYrkv4zhjJiArD6zK9gKuRtfSWd2t4kOWe1bwp86A1gsNqXsDxubmvSCNzVODaiZp0rZNJsQj21rZpeuNQeSbaN2gCeNGVuVDRJe1eXY\nIpa8L+PbjVvJT8P7ir3x4VC2",
  "0rZNJsQj21rZpeuNQeSbaN2gCeNGVuVDRJe1eXY\nIpa8L+PbjVvJT8P7ir3x4VC2ZWP+Q7IJFRVl5qrIhD+goj48JfH8gevFSiwYNAPXiphP0dDR3L8cQ2kXrsoCAUk0JfoOUvYtW+po7gzqYJ6isETL3wy4RCgxFbdkEjAy/8Lx3TKAQ3WTY3GMo06LM\nOdn80HyGSK2bTEX5mHV3lClEdr7BpfTq6AMD4czfsXlAcpo0OQzSEvVZzlK5tAM6fC",
  "n80HyGSK2bTEX5mHV3lClEdr7BpfTq6AMD4czfsXlAcpo0OQzSEvVZzlK5tAM6fCtX8DJwLn6yFvik4r5qdr4/agXzA6ZRjy0+M1PB4xsagjUV1wHLWJYnlaA/qmk7Xyz2r1t7+QqZ27HDdpiT1jn\nvpth3uFT3gp+uO3q4Tj1jUkaiucQ+pRyxHe1CXO4/rtwuG5TknoneXTaDndqoukf7Q7g3GyOSansm2NfKv0mhEVNRe0U04THSGxCWEzKtg",
  "twuG5TknoneXTaDndqoukf7Q7g3GyOSansm2NfKv0mhEVNRe0U04THSGxCWEzKtgX/j5UdAQ+PtWEsLhZiLZmAljqc4lvoQlhsVnCbXMcw+q6\nQ13q0xmA2Q2ISw+M28VbEJYTGmYuwUT1iWIbEJkTwOcB4HNI8ZljKXhEckc4wImVKuCZUP0rZkAlgaotaGjsagBzJVqMFxEMsFnXmFc+YpNIsVncU9V8O9KxrWDFVoAljaIGvM8zeciyzAKW7e",
  "jsagBzJVqMFxEMsFnXmFc+YpNIsVncU9V8O9KxrWDFVoAljaIGvM8zeciyzAKW7eoelwCW\nRlNIGb2NmkzuT0F0QVOckF0YWlF5SeW3pO6b6l+5TmlpI3giDatpS8ncAbvaVnlO5ZukdpaWlJac/SHqWRpRGlTy19SmloaUjpiqUrlGpLyYkUngiW7lI6sHRA6YGlB5S+tvQ1pc8tfU7pG0vfUPre0veU\nPrb0MaXMUkbpqWrlHJLyaeDIFq2dJn",
  "A6YGlB5S+tvQ1pc8tfU7pG0vfUPre0veU\nPrb0MaXMUkbpqWrlHJLyaeDIFq2dJnSwFLy7gdrzdJNSjNLM0qfWPqE0r6l5K0YnmeWkuMNPBgtlZS+sPQFpcJS8v4WRK8sfUVpYmlC6UtLX1L6ztJ3lD6z9BmlsaXk2wCcTizdodR+BaoKSrcs3aL01N\n87yz2el1wpntmeHMXzN/Dv/Nz8/PyPjXptZnzNd53W3/zN/wG2NCWh</latexit>JT93",
  "2el1wpntmeHMXzN/Dv/Nz8/PyPjXptZnzNd53W3/zN/wG2NCWh</latexit>JT93cBPh3GwDUxN2wFG5SmlqaUrlK3hTgKGHpCTlPRmq8q02+NpF9LVJT7mDjE+uJjmP1JQ72Hh3mlxN9qdITfmAdH1b/ohBVIKO/3x3EIXf4Wlhb0Hi93fFh9uPVx4tDz+Qnu981PnZud2p9v5vfOo\nmt+1  \u03b2 \u00b7 mt + (1 \u2212 \u03b2)@L[\u03c6t]\n@\u03c6\nvt+1  \u03b3 \u00b7 vt + (1 \u2212",
  "uPVx4tDz+Qnu981PnZud2p9v5vfOo\nmt+1  \u03b2 \u00b7 mt + (1 \u2212 \u03b2)@L[\u03c6t]\n@\u03c6\nvt+1  \u03b3 \u00b7 vt + (1 \u2212 \u03b3)\n\u2713@L[\u03c6t]\n@\u03c6\n\u25c62\n<latexit sha1_base64=\"F/r4jZ8ax68fvF6zsxG3D+ju0pE=\">AW7HiclZhbc9w0FIA3hZSbi\nkMeHFQ6YMQ9tMwpTLY5s0vSUlm+baxGlG9speNbLsteVkU8/+C94YXvlJzPBfeODI9q7qc5QHMlNWnO+zLkeyLDvIpCj08vI/c9c+PD6",
  "steVkU8/+C94YXvlJzPBfeODI9q7qc5QHMlNWnO+zLkeyLDvIpCj08vI/c9c+PD6jY9ufnzrk08/+/yL+dtf7hdpmYd8L0xlmh8GrOBSKL6nhZb8\nMs5SwLJD4KzNcMPznleiFTt6suMnyQsViISIdMQOp3P/CDKhuK0ndXJt53vuSRZnmeXnjejEy8+57PZDZkfjhItR/lLKw8Xws54BVYyaS5fFL5xSjXlSXnU3LX51khZKomp/OLy0vL9Z9HC",
  "DZkfjhItR/lLKw8Xws54BVYyaS5fFL5xSjXlSXnU3LX51khZKomp/OLy0vL9Z9HCytYbHX/v\nVPb3898AdpWCZc6VCyojheWc70ScVyLULJ7f8suAZC89YzI+hqFjCi5OqTs3EuwORgRelOfxT2quj719RsaQoLpMAzITpYGZCbrYcamjX08qobJScxU2DUWl9HTqmTx7A5HzUMtLKLAwF9BXLxwyJyG\n2bjlK34RpknC1KDyV9e3IXUBj4Wq+Ki",
  "l9HTqmTx7A5HzUMtLKLAwF9BXLxwyJyG\n2bjlK34RpknC1KDyV9e3IXUBj4Wq+KisZ2Yy6TrtcOheJWx+nx3VovQPBHvOKmkVkwlVwg8nlQVX4qXMBAcgFjiBKSKF1CnyU8QeSuIwkqUgIEH6dgsCO/VhFStNI8hJx3tiGhQyCQfd6w1YsFUJh1lBx\nTPu+MZwHUOswBdhR+O5mAnY2oyvU7zsc6TqjAx3ELOVMzrJmDIZNmRF1DlVLCpWH+g1br",
  "ZwHUOswBdhR+O5mAnY2oyvU7zsc6TqjAx3ELOVMzrJmDIZNmRF1DlVLCpWH+g1br5g6axOXZnVXcxNB1m7edXRO86IGXaeOIAsWYdy16giyJOwbA5YwyHJbPoUBJ56JuFWhsCrIwuznadBtOzMR\nvDbHGdwvXW+9Iuk/ZygjJgB3n/kVTIW8q6+lM9ubJue89k2Bj70hTFb3EpbHzbCmjcCo2tiEmnWukEmzBSHYUrum6Y1D5ZnoDtAE8E1X5kJF",
  "9k2Bj70hTFb3EpbHzbCmjcCo2tiEmnWukEmzBSHYUrum6Y1D5ZnoDtAE8E1X5kJF72n36hIsWRP278FQ81Ly4/tLP/HxSbVsbhvzH5JNqKgoM1\ndFJvw/KhrAkwqvL4jgyUslmjwI1JOXStjf0dSxHC9sE6nDgpCMSn0Jbr9Ray619QR3Nk0QX2FgKkXfplQaJKjqCubgJHhF565jgUokGzRhDmRZlzsnmh9YzRGrdbIu5MA+r7oYqjdDdN7icXQ",
  "aJKjqCubgJHhF565jgUokGzRhDmRZlzsnmh9YzRGrdbIu5MA+r7oYqjdDdN7icXQVleDic\n8ysuD1BGgyafQVqActRMsdmSsdv/AIOBM67v57ypui0Yj7aNuDfsHslGHIR6cbeD5iYlFHorgkOsSxL0R7UNVu7/es2njzA1nascN1m5LU2/bSbTvcK3rAR5uO3m4Sj1jUkaiutofUI5ajPajLnc\ndN1ygcrtuUpN5pHp2w52ZaPlHu0OumTkmpXJ",
  "O3m4Sj1jUkaiutofUI5ajPajLnc\ndN1ygcrtuUpN5pHp2w52ZaPlHu0OumTkmpXJgjn2p9JsQFjUVtVNMEx4jsQlhMSm7Fvw/VnYEPDy6VhPCYr8QXc0EsDTgEg+hCWGxuYW7ZhvD6qZD3XSr9QtD12xCWHzKEjzqJoTFmIqxUzxjWYbEJkTy\nOMR5HNI8ZljKXBKekcwxI2RJuRZUPky7kglgaYxaGzsa1+4ug2QSwXdOUVzpWn0CpWdBXvuRre",
  "jKXBKekcwxI2RJuRZUPky7kglgaYxaGzsa1+4ug2QSwXdOUVzpWn0CpWdBXvuRreu6JhzVCFJoClLXKPef6W8yYLcIrhmOVKciaQldE9rHTp8709BdEFTnJBdGlpZeUXlh6QemBpQeU5p\naSN4IgemUpeTsxL+Uzek7pvqX7lJaWlpTuWbpHaWRpROkTS59QGloaUrpm6Rql2lJyIoUngqW7lA4tHVJ6aOkhpa8tfU3pM0ufUXpk6RGl7yx9R+kj",
  "GloaUrpm6Rql2lJyIoUngqW7lA4tHVJ6aOkhpa8tfU3pM0ufUXpk6RGl7yx9R+kjSx9RyixlK5buk4pt5R8OgiVUtXKQ0sJe9+cK9Z\n2qc0szSj9LGljykdWEreiuF5Zik53sCD0VJ6XNLn1MqLCXvb0H0tKXlCaWJpS+sPQFpW8tfUvpU0ufUhpbSr4NwOnE0h1K7VegqB029JtSkeWjtzfBfhsGgPXwtyFWxRmlqaUrphKXlTgKOEpWfkPB\nnb",
  "E0h1K7VegqB029JtSkeWjtzfBfhsGgPXwtyFWxRmlqaUrphKXlTgKOEpWfkPB\nnb+Fv/4DNP75Q=</latexit>mpdlebfm0i+1qkZtzB2oxPryY5j9SMO1i7O02vJvtTpGZ8SLq+vj/7kAIphZ3+dH5xBX+FpYX9H5dWfl56sP1g8eFq+4X2Zu+b3re973srvV96D3vPev3eXi/s/d37d+763I0FtfD7wh8Lfzbqtbn2mq96\n\u03c6t+1  \u03c6t \u2212 \u21b5",
  "973srvV96D3vPev3eXi/s/d37d+763I0FtfD7wh8Lfzbqtbn2mq96\n\u03c6t+1  \u03c6t \u2212 \u21b5 \u00b7\n\u02dcmt+1\np\n\u02dcvt+1 + \u270f\n\u2022 Compute mean and pointwise \nsquared gradients with momentum\n\u2022 Boost momentum near start of the \nsequence since they are initialized \nto zero\n\u2022 Update the parameters\n<latexit sha1_base64=\"GqgASwG/Rx9X51GY53Y9n0EoARA=\">AXCniclZjdUtw2FICX/qb0L2\nmnXLQ3ntJ0Om3CsJ305zKBkD9IgcACSaM7JW9CrJsbBmWePYNOn2Y3nV625foe/QBe",
  "XLQ3ntJ0Om3CsJ305zKBkD9IgcACSaM7JW9CrJsbBmWePYNOn2Y3nV625foe/QBemR7V/gc0ZnuDKz2fJ8l+UiWZQeZFIVeXv57o03r7nXevTf/gcfvTx9Ruf7BVpmYd8EKYyzQ8CVnApFB9o\noSU/yHLOkDy/eBk1fD9M54XIlW7+iLjRwmLlYhEyDSEjq/5msh7zygyiZHFf6u/7E+9qXPNIsz9Nz/OjnIU1buik6t/2A67Zy+anr1JVJgHPd+",
  "5msh7zygyiZHFf6u/7E+9qXPNIsz9Nz/OjnIU1buik6t/2A67Zy+anr1JVJgHPd+ft5Wd/XdlZ5cqi1mSTGs7vr64vLRcfzxa6LeFxV\n72Tq+8dnQH6ZhmXClQ8mK4rC/nOmjiuVahJP5v2y4BkLT1jMD6GoWMKLo6pO3MS7CZGhF6U5/Cnt1dHLR1QsKYqLJAzYXpUYGaCLnZY6ujno0qorNRchU1DUSk9nXpmFLyhyHmo5QUWJgL6KsXjhgk\nR",
  "JAzYXpUYGaCLnZY6ujno0qorNRchU1DUSk9nXpmFLyhyHmo5QUWJgL6KsXjhgk\nR8NYzfuKn4cpZEUNK39lbXsCKeOxUBU/Letxm0y6zlrtcCheZaw83p3VIjRPxGtOKqkVU8kVAo8nVcWX4iUMBAcgljgBqeIF1GnyE0ReH1GYpxIw8CAdm1nhPZuQqpXmMeSko70gGhQycda5VYMJRJR9\nkBxfNuegZwncMoQFfhi6Mx2MmYmkyP03ys86QqTA",
  "eSko70gGhQycda5VYMJRJR9\nkBxfNuegZwncMoQFfhi6Mx2MmYmkyP03ys86QqTAy3kDMV87oJOWQSXNGXUOVUsKhYcf6BVvPmDpE5dmdVdzE0HWbt51dE7zoZdp4gCyZh3LXqCLIkrCpDljDIcls+hNOPBNxq0JhVZCJuZWnQbft\nzETw3BxncL10vbWKpP+MoYyYAFx95lswFfKuvprObG+anLPaNwU+9kYwWN1DWB43pzVtBM6qjU2oWecKmT",
  "+MoYyYAFx95lswFfKuvprObG+anLPaNwU+9kYwWN1DWB43pzVtBM6qjU2oWecKmTRbEILVsWua3jhUnonuCZoAvujKXKjoknarLsGUNWH/FpxqXkp+eHvpBz4+qpbNZWP+kWxCRU\nWZuSoy4f9R0RDuY3h+QPXirR4EGgHrxUwvqOho7leGKbSD12UBCKSaEv0OUvYtU9po7gzqYJ6isETL3wzYRCgxFXdkEjAzfcEd2TKAQnWTYnGMo06LMOVn8",
  "v0OUvYtU9po7gzqYJ6isETL3wzYRCgxFXdkEjAzfcEd2TKAQnWTYnGMo06LMOVn80HyGSK2bZTEX5mbVXVClEbrBpezo6AM\nN4czfsXhAcpo0OQzSEs1ZDlK5tgM6filX8C93Xn10PeFJ1WzE/X2/agXzA6ZRjy0+N1PB4xsagjUV2wBXLWJYnlaA/qmk3Xyz2r1l9+S6Z27HDdpiT1tr102w73ih7w0w1HbzeIRyzqSFRX20PqEcvRHt\nTlzuOG",
  "z2r1l9+S6Z27HDdpiT1tr102w73ih7w0w1HbzeIRyzqSFRX20PqEcvRHt\nTlzuOG6ywcrtuUpN5pHp2w52ZaPpHuyPY15ptUiqHZtuXSr8JYVFTUTvFNOExEpsQFpOya8FvrOwIuHl0rSaExa1CdDUTwNKQS3wKTQiLzSXcNdsYVjc6oZbZTIbIbMJYfGheTroik0IizEVY6d4wrIM\niU2I5HGE8ziecywlLkPCKZY0TIlHJNqHyUdiUTwNIYtTZ",
  "roik0IizEVY6d4wrIM\niU2I5HGE8ziecywlLkPCKZY0TIlHJNqHyUdiUTwNIYtTZ2NAY9kKlCDbZBLBd05hXOmafQLFZ0Fg9cDQ+uaFgzVKEJYGmTXGOev+m8yAKcYq9+kqXDJZCV0QRuYWeLOtPdXxBVZCcXRBeWXlB6buk5pf\nuW7lOaW0qeCILomaXk6cQ8uM/oGaV7lu5RWlpaUjqwdEBpZGlE6QNLH1AaWhpSumrpKqXaUrIjhTuCpbuUji",
  "Q8uM/oGaV7lu5RWlpaUjqwdEBpZGlE6QNLH1AaWhpSumrpKqXaUrIjhTuCpbuUjiwdUXpg6QGlzy19TukjSx9R+sLSF5S+tvQ1pfcsvUcps5RumbpGqXcUvLqIhWLF2hNLCU\nPvBtWbpFqWZpRml9y29T+nQUvJUDPczS8n2Bm6MlkpKH1v6mFJhKXl+C6Knlj6lNLE0ofSJpU8ofWXpK0ofWvqQ0thS8m4AdieW7lBq3wJVBaXblm5Temrpqfu9",
  "lj6lNLE0ofSJpU8ofWXpK0ofWvqQ0thS8m4AdieW7lBq3wJVBaXblm5Temrpqfu9AJ8NY+CamJu2gk1KU0tTStctJU8KsJ\nrj3m017ns/DXv+QUBe0=</latexit>Ww9ITsJyPVrmrTt01kXYvUjDtYm/Hp0STnkZpxB2tXp+nRZH2K1IyPSNfX9mYvUiCle/V72T5+C0sLe98v9X9curN9Z/HuSvuG9lrvi96XvW96/d5Pvbu9R72t3qAX9v6Z+",
  "/V72T5+C0sLe98v9X9curN9Z/HuSvuG9lrvi96XvW96/d5Pvbu9R72t3qAX9v6Z+3xuce6rhV8Xfl/4Y+HPRn1j\n\u02dcmt+1  \nmt+1\n1 \u2212 \u03b2t+1\n\u02dcvt+1  \nvt+1\n1 \u2212 \u03b3t+1\n\ud835\udc2646# = 0\n\ud835\udc2f46# = 0",
  "Adaptive moment estimation (Adam)",
  "Other advantages of ADAM\n\u2022 Gradients can diminish or grow deep into networks. ADAM balances \nout changes across depth of layers.\n\u2022 Adam is less sensitive to the initial learning rate so it doesn\u2019t need \ncomplex learning rate schedules.",
  "Additional Hyperparameters\n\u2022 Choice of learning algorithm: SGD, Momentum, Nesterov\nMomentum, ADAM\n\u2022 Learning rate \u2013 can be fixed, on a schedule or loss dependent\n\u2022 Momentum Parameters",
  "Recap\n\u2022 Gradient Descent \u2013 Find a minimum for non-convex, complex loss \nfunctions\n\u2022 Stochastic Gradient Descent \u2013 Save compute by calculating gradients \nin batches, which adds some noise to the search\n\u2022 (Nesterov) Momentum \u2013 Add momentum to the gradient updates to \nsmooth out abrupt gradient changes\n\u2022 ADAM \u2013 Correct for inbalance between gradient components while \nproviding some momentum",
  "Next\n\u2022 Gradient of Deep Networks: Chain Rule, backpropagation and \nautomated (scalable) gradient calculations\n\u2022 Initialization\n\u2022 Measuring training performance and how to improve\n\u2022 Network regularization\n---------- End of Foundational Concepts -----------------\n\u2022 CNNs\n\u2022 Residual Networks\n\u2022 Transformers",
  "Feedback?",
  "Lecture 07a\nGradients\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "How do we efficiently compute \nthe gradient over deep \nnetworks?",
  "Loss function\n\u2022 Training dataset of I pairs of input/output examples:\n\u2022 Loss function or cost function measures how bad model is:\nor for short:\n<latexit sha1_base64=\"aYKJ\nls3gxo4H7A6/dQhUn1+H0GU=\">ACFHicbVDLSsNAFJ3UV62vq\nEs3g0UQlJIUTdC0Y3uKtgHNDFMpN26OTBzEQsIR/hxl9x40IR\nty7c+TdO0gjaemCGc8+9l3vcSNGhTSML60N7+wuFRerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhc",
  "Rerqysrq\n1v6JtbRHGHJMWDlnIuy4ShNGAtCSVjHQjTpDvMtJxRxdZvnNHu\nKBhcCPHEbF9NAioRzGSnL0AyuxfCSHrpfcp05C08OfcJyHVvaf\nmeltcpU6etWoGTngLDELUgUFmo7+afVDHPskJghIXqmEUk7QVx\nSzEhasWJBIoRHaEB6igbIJ8JO8qNSuKeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzC",
  "KeUPvRCrl4gYa7+7kiQL8\nTYd1VltrGYzmXif7leL1TO6FBFEsS4MkgL2ZQhjBzCPYpJ1iys\nSIc6p2hXiIOMJS+VhRJpjTJ8+S9lHNPK7Vr+vVxnlhRxnsgF2w\nCgCA=</latexit>D0xwAhrgEjRBC2DwAJ7AC3jVHrVn7U17n5SWtKJnG/yB9vEN+g\n{xi, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnic",
  "i, yi}I\ni=1\n<latexit sha1_base64=\"dfEN\nBjZWeBDHyQgIldnakspAK9o=\">ACBnicbVDLSsNAFJ34rPUVd\nSlCsAiuSiJFXRbduHBRwT4gCWUymTRDJ5kwcyOU0pUbf8WNC0Xc\n+g3u/BsnbRbaemCYwzn3cu89QcaZAtv+NpaWV1bX1isb1c2t7Z\n1dc2+/o0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvn",
  "0QuCW0TwYXsBVhRzlLaBgac9jJcRJw2g2G14XfaBSM\nZHewyijfoIHKYsYwaClvnl063EagesFgodqlOjPy2LmSTaIwe+b\nNbtuT2EtEqckNVSi1Te/vFCQPKEpEI6Vch07A3+MJTDC6aTq5Yp\nmAzxgLqapjihyh9Pz5hYJ1oJrUhI/VKwpurvjFOVLGirkwxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR",
  "wxG\nreK8T/PDeH6NIfszTLgaZkNijKuQXCKjKxQiYpAT7SBPJ9K4Wi\nbHEBHRyVR2CM3/yIumc1Z3zeuOuUWtelXFU0CE6RqfIQReoiW5Q\nC7URQY/oGb2iN+PJeDHejY9Z6ZJR9hygPzA+fwAZyJmL</late\nxit>L [\u03c6]\nReturns a scalar that is smaller \nwhen model maps inputs to \noutputs better\n<latexit sha1_base64=\"vQjZ\nHpuyRPHfcav/lLvqCKGaH6I=\">AWyniclZhb9s2FIDVXbvul\nm5Y",
  "ase64=\"vQjZ\nHpuyRPHfcav/lLvqCKGaH6I=\">AWyniclZhb9s2FIDVXbvul\nm5YXvYiLCgwDJmRDN3lZUCbNG3TpI3TXNs4NSiZktlQlCJRiVPB\nb/tJ+zPb6/ZDdijJZnQO8zADsZnzfeLlkNQtyKQo9MrKX7fe/\n+Dz/6+PYndz797PMvly4+9VhkZ5yA/CVKb5cAKLoXiB1poy\nY+znLMkPwoOFs3/OiC54VI1b6+yvhpwmIlIhEyDaHhwov",
  "/CVKb5cAKLoXiB1poy\nY+znLMkPwoOFs3/OiC54VI1b6+yvhpwmIlIhEyDaHhwovtk0EQ\nZWOxPEiCdFJF04HkTbBybAS0+WGni4PquhK1MYTOH79Xpm2p\nzOshFPNanw4Wld5K/fFpYbUtLHntpz+8+81oMErDMuFKh5IVxc\nnqSqZPK5ZrEUo+vTMoC56x8IzF/ASKiW8OK3qgU/9exAZ+VGaw\n5/Sfh29fkTFkqK4SgIwE6bHBWYm6GInpY5+",
  "8IzF/ASKiW8OK3qgU/9exAZ+VGaw\n5/Sfh29fkTFkqK4SgIwE6bHBWYm6GInpY5+O62EykrNVdg0FJXS\n16lvsuiPRM5DLa+gwMJcQF/9cMxyFmrI9Z2B4pdhmiRMjarB2s\nbuFHLHY6Eqfl7WeZ9Ou85G7XAo3mSsbe7PaxGaJ+IdJ5XUiqnkB\noH06rivbiHgeARI8TkCpeQJ0mP0HkryIK60wCrpqVA6vCfzkl\nVSvNY8hJR3tNChk861jqxY",
  "iHgeARI8TkCpeQJ0mP0HkryIK60wCrpqVA6vCfzkl\nVSvNY8hJR3tNChk861jqxYCqTjrIHiu/f8w3gOodZgK7CD0d\nzsJcxNZ0dp/lE50lVmBhuIWcq5nUTMOSQSTOirqFKeHQsGO9wN\nZLps7axKVZ3dXcRJC1n3cdndO8qFHXqSPIgkUYd606giwJZ4URS\nxhkuS0PYcCJbyJuVSisCrIw+3kadNvOTASvzUkG+6XrbVQk/RcM\nZcQEYPeZX8",
  "S\nxhkuS0PYcCJbyJuVSisCrIw+3kadNvOTASvzUkG+6XrbVQk/RcM\nZcQEYPeZX8FUyLv6ejq3/VlyLmrfFPjEH8NkdQ9hedwMa9YIjK\nqNTalZ5wqZNFsQytPLrml641B5JroDNAG86cpcqOiatlyXYMma8\nGAZhpqXkp/82PuZT06rFbNtzBfJlRUlJmrIhP+HxWN4DqE1xdE\n8OSlEk0eBOrJSyWc39HUsRwvbBOp5w4KQjEp9BXa/iJW3WPq",
  "hP+HxWN4DqE1xdE\n8OSlEk0eBOrJSyWc39HUsRwvbBOp5w4KQjEp9BXa/iJW3WPqCO5\nsmqC+QsDUC79MKDTJUdSVTcDI8AtXVMcCtEgw2aMoUyLMufk5I\nfWM0Rq3ZwWc2EuVt0TqjRC97zB5fwoKMPF4YLfcHiAMho0+QzSU\no1YjpI5MVM6eTMoNGwx1+6vp7wpOq2Yn2+17UG/YHbKMOTnwy08\nHzGxqCNRXAL46xLEsvRHtQ1X67Xe1Ztvf",
  "6vp7wpOq2Yn2+17UG/YHbKMOTnwy08\nHzGxqCNRXAL46xLEsvRHtQ1X67Xe1ZtvfmBLO3Y4bpNSepte+\nm2He4NPeDn247ebhOPWNSRqK62h9QjlqM9qMudx23XKByu25Sk3\nlkenbDnZto+Uf7Y6ZuU1K5cjc9qVy0ISwqKmonWKa8BiJTQiL\nSdm14H+s7Am4eHStJoTFfiG6mglgacQlHkITwmKzhbtmG8PqtkP\ndqtMZmNkNiEsPmEJHnUTw",
  "Am4eHStJoTFfiG6mglgacQlHkITwmKzhbtmG8PqtkP\ndqtMZmNkNiEsPmEJHnUTwmJMxdgpnrEsQ2ITInkc4zyOaR4zLG\nUuCc9I5pgRsqRcCyofp13JBLA0Qa1NHI1BD2SqUINtEMsFXmFc\n+UptIoVXcUHroYPbmhYM1ShCWBph+wxf7Dj3GQBTjHcZrmSnAlk\nZTSBfez0qTO7+wuitzJwTOwpVeUXlp6SemRpUeU5paSJ4Igem\nkpeToJogt",
  "nAlk\nZTSBfez0qTO7+wuitzJwTOwpVeUXlp6SemRpUeU5paSJ4Igem\nkpeToJogtLyg9tPSQ0tLSktIDSw8ojSyNKH1s6WNKQ0tDStctX\nadUW0ruSOGKYOk+pWNLx5QeW3pM6StLX1H61NKnlL629DWl7yx9\nR+lDSx9SyixlG5YukEpt5S8OgiNUvXKA0sJc9+sNcs7VOaWZp\nR+sjSR5SOLCVPxXA9s5Tc3sCF0VJ6alm5QKS8nzWxA9t/Q5p",
  "Jc9+sNcs7VOaWZp\nR+sjSR5SOLCVPxXA9s5Tc3sCF0VJ6alm5QKS8nzWxA9t/Q5pY\nmlCaXPLH1G6VtL31L6xNInlMaWkncDcHdi6R6l9i1QVC6a+kup\neWnrvfC/D5NAauhbljK9ihNLU0pXTLUvKkALcSlp6R+8lItWe\n12dsmcl6L1Jw7WJvx2dEk5Gacwdrz06zo8n5KVJzPiZd3zicv0\niBlMKZfriwtIrfwtLC4U+91V9693fvLz1Ya9/Q",
  "wdrz06zo8n5KVJzPiZd3zicv0\niBlMKZfriwtIrfwtLC4U+91V9693fvLz1Ya9/Q3va+9b7zvdWv\nA2q58BM=</latexit>V+9B95Tr+8deKH3p/e394/37+L2Yr54tVg16nu32mO+9jqfxT/+\nL[\u03c6, f\n\u21e5\nxi, \u03c6], {xi, yi}I\ni=1\n\u21e4",
  "Gradient descent algorithm\nAlso notated as \u2207!\ud835\udc3f",
  "But so far, we looked at simple models that \nwere easy to calculate gradients\n<latexit sha1_base64=\"pLx+qdAbuYITIgjVh9BEOr/dc=\">A\nXR3iclZhbU9tGFIBNrym9kXZKH/qiKZNO2iYeO5NeXjKTQEhCIAXCNUHgruSVvG1ErqAiUY/sNf0F/Rt04fe1ayveic5aGeAa/P92kvZ3d\n18xIpsrzX+2vunXfe/+D298NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKcrmh+c8z",
  "98NP/xJ59+9vnCzS/2s7hIfb7nxzJODz2WcSkU38tFLvlhknIWeZIfeKcrmh+c8zQTsdrNLxN+HLFQiUD4LIf\nQYOFPZ+PI9YJkJI4d5QlzOdlr3vPj6oH6LebFdGgFA/61Um5VjkulxJ+Vs53ICIkeZDfdiMvHpdBdTQeiDuTJu5e6kPcVISj/PuT8l7lqlg\nVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz8augOY",
  "g\nVkcdTx3Xnr68Kjh2UverHptCvxroaUtlgYanX7dUfhxb6k8JSZ/LZGtz8augOY7+IuMp9ybLsqN9L8uOSpbnwJa/m3SLjkIJTFvIjKCoW8ey\n4rLNeObcgMnSCOIU/lTt19OoRJYuy7DLywIxYPsow0EbOyry4NfjUqikyLnym4aCQjp57OgpdIYi5X4uL6HA/FRAXx1/xFLm5zDR867iF34\ncRUwNS3d5dbsqXY+HQpX8rKgnvarazmrtcChe",
  "6HA/FRAXx1/xFLm5zDR867iF34\ncRUwNS3d5dbsqXY+HQpX8rKgnvarazmrtcCheZyv7c5qETmPxFtOKqkVXck1Ag+rsuTdsIuB4ABElxMQK5BnTo/XuD0EYVFLgGXzTKD9eW\n8rEjVKuch5KSlvSYaFBLJxy1rhVgwlVFL2QHFcW45GvA8hVmArsIXR3OwkzBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4\nydTpJXJzUXU1B",
  "zBVTY/L+ThPozLTMdxCylTI6yZgyD6TekRtQxVS1nvyqvUbtl4\nydTpJXJzUXU1BFm7advJU5oXNWw7dQRZsAjDtlVHkCXhlDRkEYMsT8oDGHDk6IhdFQqrgizMrT2m0nOoLX5jiB/dL2VkuS/nOGMqIDsPv\n0t2DK5219JZ7ZzjQ57WvC3zsjGCy2oewNGyGNW0ERjWJVdSsc4VMmi0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsHhpoW",
  "GNW0ERjWJVdSsc4VMmi0IpfF29S9sag8Ee0B6gDedEUqVHBFu1OXYMnqsHsHhpoWkh/d7f7\nEx8dlT28b/Y9kEyrKisRWkQ7/j4qGcBHE6wsiePJiSYPAvXkxRLO72jqWIoXto7UcwcFoZgU+SXa/iJU7WPqCO5sHKG+QkDXC9MKDTJQdCW\ndUDL8A2Xc8sC8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7zB5ewoKMPF4Zxfc7iHMuo1",
  "8tEg/WaMvoyzIuXk5IfWM0RqXZ8WU6EvVu0TqtRC+7zB5ewoKMPF4Zxfc7iHMuo1+fTiQg1ZipI51lM6PnGzHLaYbfXU94U\nrVbIz9Yn7UG/YHYK3+dng3U8HyGxqCNRXD/ZK1LEsvSHtQ1W65Xe1aun/xAlnZoce2mJPVOem3Le41PeBnG5bebhCPWNSRqK5JD6lHLEt7\nUJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlhM",
  "5JD6lHLEt7\nUJc9jxu2UVhcuylJvdM8Wm2LOzPR8g92Rzxn+jYplkN92xdLtwlhMadibhXjiIdIbEJYjIq2Bb+xsiPg4tG2mhAWtzLR1nQAS0Mu8RCaEBab\nLdw2JzGsbljUDbvKZDJCZhPC4lMW4VE3ISyGVAyt4ilLEiQ2IZLHEc7jiOYxwVJik/CMJYZIUvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5\noysvs648hVaxoqt4z9bw3jUN5wxV",
  "UvKtqDSUdyWdABLY9Ta2NIY9EDGCjU4CWI5\noysvs648hVaxoqt4z9bw3jUN5wxVqANY2iR7zHE3rZvMwymG2yxbkhOBrIQmcAs7W9SZ3v15QUnu5Lzg0tBLSi8MvaD0wNADSlNDyROBF7w0\nlDydeMG5oeU7hu6T2lhaEHpnqF7lAaGBpQ+MfQJpb6hPqUrhq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0Mf\nUco",
  "hq5QmhtK7kjhimDoLqUjQ0eUHhp6SOkrQ19R+szQZ5S+NvQ1pW8NfUvpI0Mf\nUcoMZSuGrpKTeUvDrwgmVDlyn1DCXPfrDXDN2iNDE0ofSxoY8pHRpKnorhemYoub2BC6OhktI1Q9coFYaS5zcveGHoC0ojQyNKnxv6nNI3\nhr6h9KmhTykNDSXvBuDuxNAdSs1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1",
  "Ss1boDKjdNvQbUrPD2zvxfgs2n0bAtz01SwSWlsaEzpuqHkSQFuJQw9JfeTgZqc1aZvm8h5LVAzbmGTjE+P\nJjkP1Ixb2OTsND2anJ8CNeMj0vXV/dmLFEgpnOkHC0t9/BaWFvbvdfs/d+9v3196uDx5Q3uj803n287tTr/zS+dh51lnq7PX8ef6cwdzv8+x\nxT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`",
  "xT8W/178Z/HfRn1nbnLMl53W5+u5/wArexvo</latexit>\nL[\u03c6]\n=\nI\nX\ni=1\n`i =\nI\nX\ni=1\n(f[xi, \u03c6] \u2212 yi)2\n=\nI\nX\ni=1\n(\u03c60 + \u03c61xi \u2212 yi)2\n<latexit sha1_base64=\"Me4GYclKg9HjqA15nuwjaNhV\nig=\">AXFHiclZhb9s2FICdXbvulm5YXvYiLCswDK0RD93lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe97\n8MO5RkszqHeZiB1s",
  "3lpUCbNG3TpMvVSdo4NSiZktlQlCJRiVPBf2PYj9nbsNe97\n8MO5RkszqHeZiB1sz5PvFySEq0/FSKXK+s/LPwzrvf/Bhzc+uvnxJ59+9vnirS8O86TIAt4PEplkxz7LuRSK97XQkh+\nnGWexL/mRf7Zm+NEFz3KRqAN9lfLTmEVKhCJgGkLDxd+9QZixoBykLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5Mv\nQGXEv6srkSk3dpMpDUOF5d",
  "kLNOCSW9rasDP0zHYurdRxJVBnkRD0txvzd9VW5Mv\nQGXEv6srkSk3dpMpDUOF5dXuivVx6OFXlNY7jSfneGtr0aDURIUMVc6kCzPT3orqT4tTbWB5NObgyLnKQvOWMRPoKhYzP\nPTskrh1LsNkZEXJhn8U9qrom9fUbI4z69iH8yY6XGOmQm62Emhw19OS6HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14w\nZpAkDbN2c6D4ZDEMVOjcrC6vgt",
  "HSQnMV1A2FhfR04pn58EYi4GWV1BgQSagr14w\nZpAkDbN2c6D4ZDEMVOjcrC6vgt58nkVMnPi2oGp9O2s145HIrXGasbB/NahOaxeMNJZViKrlG4NG0LHk36mIgOADR5Q\nQkiudQp8mPH3o9RGHFSsDA/WQCnQu9vSmpWmkeQU5a2kuiQSGVfNKy1ogFUxm3lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6\nzSc6i8vcxHALGVMRr5qAIQewrvewoQo",
  "lH1QPO+2ZwDXGcwCdBW+OJqD/ZSp6ew6\nzSc6i8vcxHALGVMRr5qAIQewrvewoQop4dKgZf2KrT2mzprEJWnV1cxEkHWQtR2d0byoUdupIsiCRi1rSqCLAn3lxGLGW\nS5KQ9hwLFnIm5VKwKsjB3sRvt52aCF6bkxT2S9tbL0n6LxjKiAnA7jPfgqmAt/W1ZG57s+RcVL4p8Ik3hslqX8KyqB7\nWrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdo",
  "G57s+RcVL4p8Ik3hslqX8KyqB7\nWrBEYVRObUrPKFTJptiCUJZdt0/TGofJUtAdoAnjTFZlQ4VvanaoES9aEB3dgqFkh+cnd7o98clqumG1j/iPZhIryInVZ\nML/o6IRPNHw+oInrxEosmDQDV5iYT7O5o6luGFbSLV3EFBKCaFvkLbX0SqfU0VwZ1NYtRXCJh64ZsJhSY5DNuyCRgZvuH\nZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1Dl",
  "Jh64ZsJhSY5DNuyCRgZvuH\nZ7FhARpkUI8xkEleZJzc/NB6hkilm9tiJszDqn1DlUZo3ze4nF8FZXg4XPBrLvdRv06n35SqBHLUDInZkonrwa5hi3m2\nv3VlNdFpxXx82mPegXzE4RBPx8uInIyIWdSqCw5DzroksRztQV3z5fp2z8rNV9+TpR05XLcpSb1NL92w72mB/x8y9\nHbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt",
  "L92w72mB/x8y9\nHbLeIRizoS1dX0kHrEcrQHdbnzuOUahcN1m5LUO8uj03a4cxMt/BgzDUzx6REjsyxL5GDOoRFTUXtFJOYR0isQ1iMi7YF\nf2NlX8Do23VISzu5KtmQCWRlziIdQhLNZbuG02MaxuOdQt8pkOkZmHcLiExbjUdchLEZUjJziGUtTJNYhkscxzuOY5j\nHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5",
  "xzuOY5j\nHFUuqS8IykjhkhS8q1oLJx0pZMAEsT1NrE0Rj0QCYKNdgEsZzTlZc7V5Cq1jRVdx3Ndy/pmHNUIUmgKVtse8wbZzk/k4\nxXDMciU5FchKaQJ3sLNDndnpzw9LcpLzwytLryi9tPS0iNLjyjNLCW/CPxwz1Ly68QPLy9oPTQ0kNKC0sLSvuW9ikNL\nQ0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6",
  "Q0pfWzpY0oDSwNK1yxdo1RbSk6k8ESw9IDSsaVjSo8tPab0haUvKH1q6VNKX1r6ktI3lr6h9KGlDyljJK1y1dp5RbSl4\nd+OGqpauU+paS36w1yzdoTS1NKX0kaWPKB1ZSn4Vw/PMUnK8gQejpZLSDUs3KBWkt9vfvjc0ueUxpbGlD6z9Bmlry19T\nekTS59QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcC",
  "9QGlK3g3A6cTSfUrtW6Ayp3TX0l1Kzy09d78X4PNp9F0Lc9tWsE1pYmlC6al5JcCHCUsPSPnyVA1d7XZ2yZyXwv\nfu486Dzt7HT6naDz78K3C3cXuku/Lf2x9OfSX7X6zkJzZed1mfp7/8Al2QL4Q=</latexit>VnDtYk/HZ1STnoZpzB2vuTrOryf0pVHM+Jl1fP5y/SIGUwp1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1\n@L",
  "+Jl1fP5y/SIGUwp1+uLjcw29haeHwh27vp+693XvLD1abN7Q3Ol93vul81+l1\n@L\n@\u03c6 = @\n@\u03c6\nI\nX\ni=1\n`i =\nI\nX\ni=1\n@`i\n@\u03c6\n<latexit sha1_base64=\"IO9SyW0GPrsxQlDBYLjp+jb/\nKXU=\">AXpXiclZjZbtw2FEDH6ZamW9Ki8ENfhBpB0zYZzBjp8lIgseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+\njf9FLSDK17aTQ1EJu53C7JLV",
  "gseNsdup1bCeWM6A0lIYxRcla7HGE+YZ+TV/b7+\njf9FLSDK17aTQ1EJu53C7JLV5iRZ3uv9s3Dtvfc/+PCj6x/f+OTz7/4uatL/ezuEh9PvBjGaeHsu4FIoPcpFLfp\niknEWe5AfeyarmB2c8zUSs9vKLhB9HLFQiED7LITS8tXDHDVLml27C0lw6bhcymEptNLIS9IxmLq/AYlHgpVehHLU\nzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXY",
  "ptNLIS9IxmLq/AYlHgpVehHLU\nzGZvkNVqDgse9Op67pnWcJ8Xt7rdZf9SAfesXYfanM1mvdKx7F8Z9bPj7MqE93UvQv9+3vXdUjnznJl/EfFy90Oby71u\nr3qx6GFflNY6jQ/W8NbX4/cUewXEVe5L1mWHfV7SX5c6sn5k9vuEXGYVQnLORHUFQs4tlxWa3p1LkNkZETxCn8U7lT\nRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhX",
  "p1LkNkZETxCn8U7lT\nRS/XKFmUZReRByYMcJxhpoM2dlTkwa/HpVBJkXPl1x0FhXTy2NEbxBmJlPu5vIAC81MBY3X8MYOlymEb3XAVP/fjKGKQ\nGXdlbRtWq14LflpUW2o6bTtrlaMTeZWx8mxv3orIeSTectJIpehGrhB4OC1L3g27GAgOQHQ5AbHiGbSp8+MFTh9ROEIS\nMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgM",
  "+MFTh9ROEIS\nMHAvnsDgAmdnSpWOQ8hJy3tFdGgkEg+aVmrxIKljFrKLiOc9vRgMG9J1qH/ocrcFuwtR0Vi/nkzyNykzHcA8pUyG\nvuoAp+3C6drChCimrY3LZ+h1bO0ydNImLk2qoqY4gay9tO3lK86JGbaeKIAs2Ydi2qgiyJFzwRixikOWmPIQJR46O2FW\nhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nW",
  "W\nhsCrIxtxKY6/d6IjeG9OEjgvbW+tJOk/YygjOgCnT/8VTPm8ra/Gc9uZJes8nWBT5wxLFa7CkvDelqzTmBWTWxKzS\npXyKTZglAan7dNPRqLyhPRnqAO4ENXpEIFl7S7VQm2rA67d2GqaSH50b3uT3xyXPb0sdG/SDahoaxIbA3p8P9oaAS3WL\ny/IXL5Zo8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixw",
  "o8SBQLV4s4fqOlo6leGPrSLV2UBCKSZFfoOMvQtWuU0XwYOMIjRUCul34y4RCixwEbVkHtAx/4WHBsoF8NE\nm/nqMv46xIObn4of0MkUrXl8VU6JtV+4IqtdC+bnA5rwVluDmc8SuqeyijXp1PLy7UiKUomRO9pJPXbpbDEbOd/mrJ6\n6LVCvnpetMfjAtWp/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXj",
  "/B9fjpcx+sREos6ErUFT2fWtiSxLP1BW/Ptenlk5frH8jWDi2u3ZSk3WaUdtviXjECfrphGe0G8\nYhFHYnakZIPWJZ+oO27HncsM3C4tpNSdqd5dFqW9y5ibZ/sDfmOdOPSbEc6ce+WLp1CIs5FXOrGEc8RGIdwmJUtC34P\n1Z2Bdw82lYdwuJWJtqaDmBpxCWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI",
  "CWeQh3CYn2E2YTw+qGRd2wq0wmY2TWISw+YRGedR3CYkjF0CqesCRBYh0ieRzjPI5p\nHhMsJTYJr0hiWRGypWwbKh3HbUkHsDRBvU0sncEIZKxQh0QyxndeZl15ym0ixXdxQNbx4MrOs4ZalAHsLRJzpjbloP\nmYdTrN9xLUlOBLISmsAt7GxRZ/b05wUleZLzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oG\nhA0oDQwNKH",
  "LzgtDLyg9N/Sc0gNDyhNDSVvBF6wYyh5O/GCM0PKN03dJ/SwtC0oG\nhA0oDQwNKHxv6mFLfUJ/SVUNXKc0NJU+kcEcwdI/SsaFjSg8NPaT0paEvKX1q6FNKXxn6itK3hr6l9KGhDylhjJK1wx\ndo5QbSj4deMGKoSuUeoaSdz84a4ZuUZoYmlD6yNBHlI4MJW/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD4\n39Dmlbwx9Q+kTQ59Q",
  "W/FcD8zlDzewI3RUEnpM0OfUSoMJe9vXvDC0BeURoZGlD4\n39Dmlbwx9Q+kTQ59QGhpKvg3A04mhu5Sar0BlRum2oduUnhp6av8uwOfL6Nk25qZpYJPS2NCY0nVDyZsCPEoYekKeJw\nrnW863budPqdXzoPOk87W51Bx1/4Y+HPhb8W/l78bvHF4t7ifq1eW2jqfNVp/SwO/wVGa0LO</latexit>PVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcn",
  "2jqfNVp/SwO/wVGa0LO</latexit>PVXNVmX5vIdS1Qc25hTcZntUnOAzXnFtZcnWa1yfUpUHM+JkNf259/SIGUwpV+eHOpj7/C0sL+crf/c/f+9v2lByvNF9\n@`i\n@\u03c6 =\n2\n4\n@`i\n@\u03c60\n@`i\n@\u03c61\n3\n5 =\n\" 2(\u03c60 + \u03c61xi \u2212 yi)\n2xi(\u03c60 + \u03c61xi \u2212 yi)\n#\nFor example, linear, 1-\nlayer models.\nLeast squares loss for \nlinear regression\nPartial derivative w.r.t. \neach parameter",
  "What about deep learning models?\n<latexit sha1_base64=\"k/qwGEiv4MxbNO+/hsHXDcOCr9\ng=\">AXeniclZhbU9w2FMeXlN6I+20PTFEyadtkl3WJeXjqTQMgNUiCwQMJudmSv7FWQZeMLPH4C/S1/XL9Lp1Oj2\nzvKj5HPHRnYLXn9eRdM6RLduNpUiz1dW/F95973P/jw2keLH3/y6WefL13/4jCN8sTjfS+SUXLspRLoXg/E5nkx3HC\nWehKfuSebmh+dM6TVETqILuM+T",
  "efL13/4jCN8sTjfS+SUXLspRLoXg/E5nkx3HC\nWehKfuSebmh+dM6TVETqILuM+TBkgRK+8FgGptHSv87A9Sejolc63/6m2+wE/rk8Y6NitXRuwY+dkAfVL2hPhwMVqTx0e\nIMBotN5zVL516rc6+cDTN05h6MgzsWB2stB2vlbCjsIHSjaQHQ8UvdeXob/sUTMWw81s7utJzdKWfDjpZWVrur1cehjV7TW\nOk0n93R9a/Gg3Hk5SFXmSdZmp70V",
  "UTMWw81s7utJzdKWfDjpZWVrur1cehjV7TW\nOk0n93R9a/Gg3Hk5SFXmSdZmp70VuNsWLAkE57k5eIgT3nMvFMW8BNoKhbydFhUiSqdm2AZO36UwJ/KnMr6do+ChWl6Gbq\ngDFk2STHTRhs7yTP/12EhVJxnXHn1QH4unSxydNadsUi4l8lLaDAvETBXx5uwhHkZ1MbiQPELwpDpsbFYH1zr4R48kCog\np/lVZ2UZVuzWk4NK9SrD85mHsRGQ/FG",
  "hHkZ1MbiQPELwpDpsbFYH1zr4R48kCog\np/lVZ2UZVuzWk4NK9SrD85mHsRGQ/FG06cVBLt5AoBD8qi4N2gi4HgAESXExApnoJPHR8oiB6isC8k4MKUzPOSuFYZDyA\nmLdlLIoNGLPm0pdogKkhl2JLsg8Rxbjoa8CyBLMBU4YujHOzHTJWzfhmfZklYpNqGR0iYCng1BCzZY1KvqK1QuZTQ1Wupfs\neq50ydNoGL4mqibYg1UHS1mQJjYsatzWV",
  "R0iYCng1BCzZY1KvqK1QuZTQ1Wupfs\neq50ydNoGL4mqibYg1UHS1mQJjYsatzWVBamgCIO2qrIglYSr2JiFDKLctEew4NDRFrtUKCwVpDB3k8htjx1rC67NaQz7\npa3bLEj4zxmKiDbA7tPfgimPt+Ub0VztzIJzXul1g0+dCSr3YUlQb2s2SCwqsZWUmUVK6Sk0QJTEl20lXo2FimPRXuB2o\nA3XZ4I5b8lu121oGS1eXAblprkp/82P2JT4fF",
  "K6Sk0QJTEl20lXo2FimPRXuB2o\nA3XZ4I5b8lu121oGS1eXAblprkp/82P2JT4fFqt42+h+JjhK89jmSJv/h6Mx3DdxfYEFJy+SKHlgqJIXSbi+o9SxBe2\ntlS5g4ZQTIrsEm1/Eah2n8qCJxuFaK5g0H7hmwmFkuz7bE2aDF8wnAUkAeWqRXr9GTUZonFz8UD2DpZLry2Ii9M2qfUG\nVWtC+bnA57wVtuDmc8yu6uyibh1PN8rVmCUomFOd0umr",
  "UD2DpZLry2Ii9M2qfUG\nVWtC+bnA57wVtuDmc8yu6uyibh1PN8rVmCUomFOd0umrQZrBFrPt/irldOqCvjZVjMezAuyk3sePxt4XwEREU1EvmCI\n5fVlyQqy3jga16ub8+s2Hr1AyntwK1KyXx28zSrZor5gBP9u2zHab6IiKaiTy1cyQ6ojKMh74sdx27YKi9aulMTvLI5\nWtU7V6Ly9w8mcDjVx6RIjvWxL5KD2oSFGRVmVmGkD7htYW3CwjBv",
  "i9aulMTvLI5\nWtU7V6Ly9w8mcDjVx6RIjvWxL5KD2oSFGRVmVmGkD7htYW3CwjBvq+A3luwLuHm0VbUJC3dT0ZpAxaNucRLqE1YWG/htr\nKxYem2RbptlzIZT5CyNmHhIxbiVdcmLAyoMLAKT1kcI2FtInGc4DhOaBxjLIptIpyR2JIRUlK2gkomUVukDVg0RaNLYPB\nDGSk0ICNEYtTWnmptfIUqmJFq7hvG7h/xcAZQw61AYt2yB5zBjvWTebi",
  "RaNLYPB\nDGSk0ICNEYtTWnmptfIUqmJFq7hvG7h/xcAZQw61AYt2yB5zBjvWTebiEMxyxbkWCBVTAO4izW7VDM7/bl+QU5yrn9p6C\nWlF4ZeUHpk6BGliaHkicD1nxtKnk5c/9zQc0oPDT2kNDc0p7RvaJ9S31Cf0oeGPqTUM9SjdMPQDUozQ8mJFO4Ih5QOjF0\nQumxoceUvjD0BaWPDX1M6UtDX1L6xtA3lN439D6lzFBG6ahm5RyQ8mrA9dfN",
  "jF0\nQumxoceUvjD0BaWPDX1M6UtDX1L6xtA3lN439D6lzFBG6ahm5RyQ8mrA9dfN3SdUtdQ8uwHe83QXUpjQ2NKHxj6gNKxoeS\npGO5nhpLjDdwYDZWUPjH0CaXCUPL85vrPDH1GaWhoSOlTQ59S+trQ15Q+MvQRpYGh5N0AnE4M3afUvAUqUkr3DN2j9MzQM\n/t7AT5Po2srzB3jYIfSyNCI0i1DyZMCHCUMPSXnSV81VzXzQrTEijm3sCbis94",
  "M\n/t7AT5Po2srzB3jYIfSyNCI0i1DyZMCHCUMPSXnSV81VzXzQrTEijm3sCbis94k5r6acwtrk6z3uT65Ks5n5Cpbx7OX6R\nfNlp/VZvsfj1EqVw=</latexit>ASOFKP1pa6eG3sLRxuNbt/dy9u3d35d5684b2Wuebzo3Od51e5fOvc7jzm6n3/EWxgt/LPy58NfX/yzfWP5+VYtfWeh6\nh1 = a[\u03b20 + \u23260x]\nh2 = a[\u03b21 + \u23261h1]\nh3 = a[\u03b22 +",
  "t/LPy58NfX/yzfWP5+VYtfWeh6\nh1 = a[\u03b20 + \u23260x]\nh2 = a[\u03b21 + \u23261h1]\nh3 = a[\u03b22 + \u23262h2]\nf[x, \u03c6] = \u03b23 + \u23263h3",
  "We need to compute partial derivatives w.r.t.\nevery parameter!\n<latexit sha1_base64=\"JE7fQ4+WZVE\nSdJEBwEOhglbuk/Q=\">AW+niclZhbU9w2FICXlN6I03LS18ZTLTaZMdtpNeHh\nMIuUEKBZIMNmRvbJXQZaNL7DEdX9M3zp97Z/pe39Ij2zvCp8jHspMur5PutyJMu\nyvUSKLF9d/WfhnXfe/+D298tPjxJ59+9vnSzS8OsrhIfT70YxmnRx7LuBSKD3ORS\n36UpJxFnuSH3um65ofnPM1E",
  "PjxJ59+9vnSzS8OsrhIfT70YxmnRx7LuBSKD3ORS\n36UpJxFnuSH3um65ofnPM1ErPbzy4SfRCxUIhA+yE0Wvrdcb0gmYhRmX8/qFwZq1\nDyIGdpGl/MSeXcdVwmkwlz3KyIRqVwhXIjlk98Jsu1SiuVG6TML92Epblg0nG5lCB\nWx6aWk+oKrqPVaGltb9a/zm0MGgLK732b2d086uxO479IuIq9yXLsuPBapKflLpa\nX/Jq0S0ynjD/lIX8GIq",
  "/zm0MGgLK732b2d086uxO479IuIq9yXLsuPBapKflLpa\nX/Jq0S0ynjD/lIX8GIqKRTw7KetEVc5tiIydIE7hn8qdOnr1ipJFWXYZeWDq0WY6\naCNHRd58MtJKVRS5Fz5TUNBIZ08dnTWnbFIuZ/LSygwPxXQV8efMEhYDnOz6Cp+4cd\nRxNS4dNc2diFPHg+FKvlZUc9TVXWdjdrhULzOWHu6P69F5DwSbzmpFZ0JdcIPKzK\nkvfDPgaCAxB9Tk",
  "+FKvlZUc9TVXWdjdrhULzOWHu6P69F5DwSbzmpFZ0JdcIPKzK\nkvfDPgaCAxB9TkCseAZ16vx4gTNAFNalBAzci6fQucB5UZGqVc5DyElHe0U0KCST\nzvWOrFgKqOsgeK49x2NOB5CrMAXYUfjuZgL2Gqml2X82meRmWmY7iFlKmQ10N4\nMeUdQhZRwqd+xfsXWC6ZO28TFSd3VEeQtZ92nTyleVHjrlNHkAWLMOxadQRZEna\nRMYsYZLktj2DAk",
  "sXWC6ZO28TFSd3VEeQtZ92nTyleVHjrlNHkAWLMOxadQRZEna\nRMYsYZLktj2DAkaMjdlUorAqyMHfS2Ou2negIXpvTBO6XrdRkvSfM5QRHYC7T/8Kp\nnze1dfjue3MknNe+7rAp84EJqt7CUvDZlizRmBUbayiZp0rZNJsQh216pe2NReS\nK6A9QBfNMVqVDBFe1OXYIlq8PuHRhqWkh+fLf/I5+elKv6tH/IdmEirIisVWkw/+\njojE8t/D6gi",
  "DBFe1OXYIlq8PuHRhqWkh+fLf/I5+elKv6tH/IdmEirIisVWkw/+\njojE8t/D6gievFiyYNAPXmxhP0dTR1L8cLWkXruoCAUkyK/RLe/CFX3mjqCOxtH\nqK8Q0PXCLxMKTXIQdGUd0DL8whPYsoB8NEi/GaMv46xIOdn80HqGSK3rbTEV+mHV3\nVClFr7Bpfzq6AMD4dzfs3lHsqo1+Tiws1ZilK5lRP6fS1m8HZwHr31PeFK1WyM\n82/agXzA7h",
  "zq6AMD4dzfs3lHsqo1+Tiws1ZilK5lRP6fS1m8HZwHr31PeFK1WyM\n82/agXzA7he/zs9Emno+QWNSRqC48ljrksSytAd1zZfr1Z6Vm6+/I0s7tLh2U5J6\n217abYt7TQ/42Zalt1vEIxZ1JKqr7SH1iGVpD+qy53HLNgqLazclqXeWR6tcecmW\nv7B/oTnTB+TYjnWx75Yuk0IizkVc6sYRzxEYhPCYlR0Lfh/rOwJeHh0rSaExZ1MdD\nUdwNKY",
  "B+TYjnWx75Yuk0IizkVc6sYRzxEYhPCYlR0Lfh/rOwJeHh0rSaExZ1MdD\nUdwNKYSzyEJoTF5hbum0Mq1sWdcu1m8PXbMJYfExi/ComxAWQyqGVvGUJQkSmxD\nJ4wTncULzmGApsUl4RhLjJAlZVtQ6STuSjqApSlqbWpDHoAL3eowTaI5YyuvMy6\n8hRaxYqu4qGt4eE1DecMVagDWNom95jbltvMg+nGI5ZtiQnAlkJTeAOdnaoMzv9eU\nFJTn",
  "qu4qGt4eE1DecMVagDWNom95jbltvMg+nGI5ZtiQnAlkJTeAOdnaoMzv9eU\nFJTnJecGnoJaUXhl5QemjoIaWpoeSNwAteGEreTrzg3NBzSg8MPaC0MLSgdGjokNL\nA0IDSR4Y+otQ31Kd03dB1SnNDyYkUngiG7lM6MXRC6ZGhR5S+NPQlpU8MfULpK0Nf\nUfrW0LeUPjD0AaXMUEbphqEblHJDyacDL1gzdI1Sz1Dy7gf3mqE7lCaGJpQ+NPQh",
  "UfrW0LeUPjD0AaXMUEbphqEblHJDyacDL1gzdI1Sz1Dy7gf3mqE7lCaGJpQ+NPQhp\nWNDyVsxPM8MJcbeDAaKil9auhTSoWh5P3NC54b+pzSyNCI0meGPqP0jaFvKH1s6G\nNKQ0PJtwE4nRi6R6n5ClRmlO4aukvpmaFn9u8CfD6Nnm1hbpsKtimNDY0p3TSUvCnA\nUcLQU3KeDFS7q82+NpF9LVBzbmFtxmdXk5wHas4trN2dZleT/SlQcz4hXd",
  "UvCnA\nUcLQU3KeDFS7q82+NpF9LVBzbmFtxmdXk5wHas4trN2dZleT/SlQcz4hXd84mH9Ig\nt3YXHh1sKXy78t/7H85/JfjfrOQnvNrV7nb/nv/wB7kgI4</latexit>ZTCTj9aWhngr7C0cPBDf/BT/97uvZX7a+0X2hu9r3vf9L7tDXo/9+73nvR2esOe3/\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n<latexit sha1_base64=\"CgqealUvUgO\nNHeX",
  "/\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n<latexit sha1_base64=\"CgqealUvUgO\nNHeXUaVLIFyDX/KE=\">AW3HiclZhJU9xGFIBlZ3PIYpxUuOSiCuWqVIpMQcpZLq\n6ywdjG4DAYBrAZIC1NS9Om1RJaYLBKt9xSueYn5T/kP+SanPNa0kxb7zWHTBVMz/s\n+9fK6W5uXSJHly8t/3bj5zrvf/BrQ/nPvr4k09vz9/5bD+Li9TnAz+WcXrosYxLo\nf",
  "fK6W5uXSJHly8t/3bj5zrvf/BrQ/nPvr4k09vz9/5bD+Li9TnAz+WcXrosYxLo\nfgF7nkh0nKWeRJfuCdrWl+cMHTMRqL79K+HEQiUC4bMcQqfzv7hbR0MvSMbi+L\n47zIrotBT3V6qTcqNyh1xK+Fm5lERePCldQUgkrXMdHuUlvZknulfx7Pnc4vLve\nW649LCytYdFpP/3TO1+MhqPYLyKucl+yLDtaWU7y45KlufAlr+aGRcYT5p+xkB9B\nU",
  "49LCytYdFpP/3TO1+MhqPYLyKucl+yLDtaWU7y45KlufAlr+aGRcYT5p+xkB9B\nUbGIZ8dlnYrKvQuRkRvEKfyp3K2jbx9RsijLriIPzIjl4wzHbSxoyIPfjouhUqKn\nCu/aSgopJvHrs6rOxIp93N5BQXmpwL6vpjljI/h+zPDRW/9OMoYmpUDlfXd6py6PF\nQqJKfF/VMVFXWa8dDsXrjNWNvVktIueReMNJbWiK7lG4GFVlrwX9jAQHIDocQJ",
  "F\nQqJKfF/VMVFXWa8dDsXrjNWNvVktIueReMNJbWiK7lG4GFVlrwX9jAQHIDocQJi\nxTOoU+fHC9wVRGHlScBlsyxgKbgvKlK1ynkIOelor4gGhUTyScdaIxZMZdRdkFx3\nbuBjxPYRagq/DF0RzsJkxV0+NyPsnTqMx0DLeQMhXyugkYs+kHlHXUIWUcKjfsX\n7G1gumztrExUnd1VRHkLWXdp08pXlRo65TR5AFizDsWnUEWRLOEyMWMchyWz6",
  "sX\n7G1gumztrExUnd1VRHkLWXdp08pXlRo65TR5AFizDsWnUEWRLOEyMWMchyWz6FAUe\nujthVobAqyMLsp7HXbTvREbw2Jwnsl63XpL0XzCUER2A3ae/BVM+7+pr8cx2p8m5q\nH1d4BN3DJPVPYSlYTOsaSMwqjZWUbPOFTJptiCUxpdU/fGovJEdAeoA3jTFalQwV\nvaUl2CJavDwyUYalpIfvRt73s+OS6X9bR/0g2oaKsSGwV6fD/qGgEVya",
  "TFalQwV\nvaUl2CJavDwyUYalpIfvRt73s+OS6X9bR/0g2oaKsSGwV6fD/qGgEVya8viCJy+\nWaPIgUE9eLOH8jqaOpXh60g9d1AQikmRX6HtL0LVPaO4M7GEeorBHS98M2EQpMc\nBF1ZB7QM3CNtSwgHw3Sb8boyzgrUk5Ofmg9Q6TW9WkxFfpi1T2hSi10zxtczo6CM\nlwcLvg1h3so16Ty8u1IilKJkTPaWTk2GWwxaz7f56ypui1Qr5+WbHvQ",
  "xtczo6CM\nlwcLvg1h3so16Ty8u1IilKJkTPaWTk2GWwxaz7f56ypui1Qr5+WbHvQLZqfwfX\n5+uonIyQWdSqC25qrHVJYlnag7pmy/XtnpWbJ9+QpR1aXLspSb1tL+2xb2mB/x8\ny9LbLeIRizoS1dX2kHrEsrQHdnzuGUbhcW1m5LUO82j1ba4MxMt/2BvzHOmb5NiO\ndK3fbEcNiEs5lTMrWIc8RCJTQiLUdG14DdWdgVcPLpWE8JiPxNdTQew",
  "HOmb5NiO\ndK3fbEcNiEs5lTMrWIc8RCJTQiLUdG14DdWdgVcPLpWE8JiPxNdTQewNOISD6EJYb\nHZwl2zjWF1y6Ju2VUmkzEymxAWn7AIj7oJYTGkYmgVz1iSILEJkTyOcR7HNI8JlhK\nbhGckscwIWVK2BZWO46kA1iaoNYmlsagBzJWqME2iOWMrzMuvIUWsWKruKBreHB\nNQ3nDFWoA1jaJnvMHW5bN5mHUwy3WbYkJwJZCU1gHzt96kzv/ry",
  "WsWKruKBreHB\nNQ3nDFWoA1jaJnvMHW5bN5mHUwy3WbYkJwJZCU1gHzt96kzv/rygJHdyXnBl6BWl4\nZeUnpg6AGlqaHkicALXhKnk684MLQC0r3Dd2ntDC0oHRg6IDSwNCA0seGPqbUN9S\nndM3QNUpzQ8kdKVwRDN2jdGzomNJDQw8pfWnoS0qfGvqU0leGvqL0jaFvKH1o6ENK\nmaGM0nVD1ynlhpJXB16waugqpZ6h5NkP9pqhfUoTQxNKHx",
  "GvqL0jaFvKH1o6ENK\nmaGM0nVD1ynlhpJXB16waugqpZ6h5NkP9pqhfUoTQxNKHxn6iNKRoeSpGK5nhpLbG\n7gwGiop3TB0g1JhKHl+84Lnhj6nNDI0ovSZoc8ofW3oa0qfGPqE0tBQ8m4A7k4M3a\nXUvAUqM0p3DN2h9NzQc/t7AT6bRs+2MLdNBduUxobGlG4aSp4U4FbC0DNyPxmo9qw2\nfdtEzmuBmnELazM+PZrkPFAzbmHt2Wl6NDk/BWr",
  "lG4aSp4U4FbC0DNyPxmo9qw2\nfdtEzmuBmnELazM+PZrkPFAzbmHt2Wl6NDk/BWrGx6Tr6/uzFymQUjTn84vruC3s\nLSw/1v5YfevZ17iw9W2ze0t5wvna+cr50V50fngfPU6TsDx3f+dP52/nH+XThZ+H\nXht4XfG/XmjfaYz53OZ+GP/wBpavXY</latexit>\nL[\u03c6] =\nI\nX\ni=1\n`i =\nI\nX\ni=1\nl[f[xi, \u03c6], yi]\n<latexit sha1_base64=\"Z8pXI1y37l",
  "it>\nL[\u03c6] =\nI\nX\ni=1\n`i =\nI\nX\ni=1\nl[f[xi, \u03c6], yi]\n<latexit sha1_base64=\"Z8pXI1y37lX\nEeGWnMsH2tezVrhY=\">AW+niclZhb9s2FICd7tZlt3bd8rIXYUWBYeiMZOguj2\n3S9JZ0ce5po9SgZEpmQ1GKRCVONe/H7G3Y6/7M3vdDdijJZnUOA2wGrPn+8TLISn\nRCjIpCr28/PfCtXfefe/9D65/uPjRx598+tmNm58fFGmZh3w/TGWaHwWs4FIov",
  "n\nRCjIpCr28/PfCtXfefe/9D65/uPjRx598+tmNm58fFGmZh3w/TGWaHwWs4FIovq+Fl\nvwoyzlLAskPg9M1w/PeV6IVO3py4yfJCxWIhIh0xAa3vjN86OchZWfsVwLJj2fSz\nmsxHQ6D/lBFHDNhtXpdOr5ZyUb1X8PwnScXUqBv9D9VtJTxu6hveuL3cX64/Hi2\nstIXbvfYzGN78cuSP0rBMuNKhZEVxvLKc6ZPK1B1KPl30y4JnLDxlMT+Go",
  "4/Hi2\nstIXbvfYzGN78cuSP0rBMuNKhZEVxvLKc6ZPK1B1KPl30y4JnLDxlMT+GomIJL06q\nOlFT7w5ERl6U5vBPa+Ovn1FxZKiuEwCMBOmxwVmJuhix6WOfj6phMpKzVXYNBSV0\ntOpZ7LujUTOQy0vocDCXEBfvXDMIFMa5mbRV/wiTJMEcln5q+vbkKyAx0JVHFJq5mk\n67TrtcOheJWx+nRvXovQPBFvOKmkVkwlVwg8nlYV78d9DAQHIPqc",
  "x0JVHFJq5mk\n67TrtcOheJWx+nRvXovQPBFvOKmkVkwlVwg8nlYV78d9DAQHIPqcgFTxAuo0+Qki\nbwVRWJcScNUsHFgH3s6UVK0jyEnHe0l0aCQST7pWGvEgqlMOsouKJ53xzOA6xmA\nboKXxzNwW7G1HR2neYTnSdVYWK4hZypmNdNwJBDWO872FClHBp2LF+wdYOU6dt4t\nKs7mpuIsjay7uOzmleYDt2nDqCLFiEcdeqI8iScBcZsYRBltvy",
  "LF+wdYOU6dt4t\nKs7mpuIsjay7uOzmleYDt2nDqCLFiEcdeqI8iScBcZsYRBltvyEAaceCbiVoXCqiA\nLc5CnQbftzETw2pxksF+63npF0n/OUEZMAHaf+RZMhbyr6Vz25sl57z2TYFPvDFMV\nvcSlsfNsGaNwKja2JSada6QSbMFoTy96JqmNw6VZ6I7QBPAm67MhYre0u7WJViyJu\nzfhaHmpeTH3/V/4JOTatlsG/OHZBMqKsrMVZEJ/4+KRv",
  "Am67MhYre0u7WJViyJu\nzfhaHmpeTH3/V/4JOTatlsG/OHZBMqKsrMVZEJ/4+KRvDcwusLInjyUokmDwL15KU\nS7u9o6liOF7aJ1HMHBaGYFPoSbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiG\nJ7BjAYVokGEzxlCmRZlzcvND6xkitW5ui7kwD6vuDVUaoXvf4HJ+FZTh4XDOr7g8Q\nBkNmnwGalGLEfJnJgpnbzyCw1bzLX76ylvik4r",
  "DVUaoXvf4HJ+FZTh4XDOr7g8Q\nBkNmnwGalGLEfJnJgpnbzyCw1bzLX76ylvik4r5mcbXvQL5idMgz52XADz0dMLO\npIVBceZx1SWI52oO65sv17Z5VG6+JUs7drhuU5J6216bYd7RQ/42ajt5vEIxZ1\nJKqr7SH1iOVoD+py53HTNQqH6zYlqXeWR6ftcOcmWv7R3hjOteaYlMqROfal0m9CW\nNRU1E4xNSfbrtiEsJiUXQv+j5VdAQ+PrtWEsD",
  "mWv7R3hjOteaYlMqROfal0m9CW\nNRU1E4xNSfbrtiEsJiUXQv+j5VdAQ+PrtWEsDgoRFczASyNuMRDaEJYbLZw12xjWN\n10qJtulclsjMwmhMXHLMGjbkJYjKkYO8VTlmVIbEIkj2OcxzHNY4alzCXhGckcM0K\nWlGtB5eO0K5kAliaotYmjMeiBTBVqsA1iuaAr3CuPIVWsaKreN/V8P4VDWuGKjQB\nLG2RPeb5W85NFuAUwzHLleRMICujCRxg",
  "uaAr3CuPIVWsaKreN/V8P4VDWuGKjQB\nLG2RPeb5W85NFuAUwzHLleRMICujCRxgZ0Cd2ekviCpykguiS0svKb2w9ILSQ0sPKc\n0tJb8IgmjHUvLrJIjOLT2n9MDSA0pLS0tK9y3dpzSyNKL0kaWPKA0tDSlds3SNUm0\npOZHCE8HSPUrHlo4pPbL0iNIXlr6g9ImlTyh9aelLSt9Y+obSB5Y+oJRZyihdt3Sd\nUm4peXUQRKuWrlIaWEp+8Fes3",
  "6g9ImlTyh9aelLSt9Y+obSB5Y+oJRZyihdt3Sd\nUm4peXUQRKuWrlIaWEp+8Fes3RAaWZpRulDSx9SOrKU/CqG5ml5HgD0ZLJaVPL\nX1KqbCU/H4LoueWPqc0sTSh9Jmlzyh9belrSh9b+pjS2FLybgBOJ5buUmrfAlUFpd\nuWblN6ZumZ+70An09j4FqYW7aCLUpTS1NKNywlvxTgKGHpKTlPRq9q83eNpH7WqTm\n3MHajM+uJjmP1Jw7WHt3ml",
  "aCLUpTS1NKNywlvxTgKGHpKTlPRq9q83eNpH7WqTm\n3MHajM+uJjmP1Jw7WHt3ml1N7k+RmvMx6fr6wfxFCqT0oH4vu4LfwtLCwf9lR/79\n7bv3b6/2r6hvd7qvd175veSu+n3v3ek96gt98Le/8sLC7cWvhi6del35f+WPqzUa\n8tNfc6nU+S3/9C4EkAnw=</latexit> @`i\n@\u03b2k\nand\n@`i\n@\u2326k\nLoss: sum of individual terms:\nSGD Algorithm:\nMillions and even billions of \nparameters:\nWe need the partial derivative with \nrespect to every weight and bias",
  "=</latexit> @`i\n@\u03b2k\nand\n@`i\n@\u2326k\nLoss: sum of individual terms:\nSGD Algorithm:\nMillions and even billions of \nparameters:\nWe need the partial derivative with \nrespect to every weight and bias we \nwant to update for every sample in \nthe batch.\n\ud835\udf19 = {\ud835\udefd!, \u03a9!, \ud835\udefd\", \u03a9\", \ud835\udefd#, \u03a9#, \u2026 }",
  "Network equation gets unwieldy even for \nsmall models\n\u2022 Model equation for 2 hidden layers of 3 units each:\n<latexit \nsha1_base64=\"yFXKW\nFAp5/LfGhFiO4YiKSG\nGOJI=\">AY1XiclZj\nbts2GIDd7NRlp3bDg\nAC7ERa0HdbO8KHdjO\ngTZqeki5Jc2wjN6BkS\nmZDUYpEJU4F3w273SP\ntOfYAu91eYT8l2Yz4M\n0NjoDXzf594+ElKtLy\nEs0x2On9dmXv/Q8+/\nOjqx/OfPrZ519cu/\n7lbhbnqU93/JjH6b5H\nMsqZ",
  "lKtLy\nEs0x2On9dmXv/Q8+/\nOjqx/OfPrZ519cu/\n7lbhbnqU93/JjH6b5H\nMsqZoDuSU73k5SyO\nN0ztaVnzvhKYZi8W2\nPEvoICKhYAHziYTQ4f\nW5/bNbzs1fHDcZsVuH\nRWdyuy51J27kxeOCTF\nxOA3ngJhmDaGfi3Hbq\nslYOXDmiksz49K/uZD\nzQfg/7vYbfU/5M72O9\n39D7SndTFo7kwBWxy\nCOPpo7rzt90R1lCfFp\n02vfu+dFsT",
  "fg/7vYbfU/5M72O9\n39D7SndTFo7kwBWxy\nCOPpo7rzt90R1lCfFp\n02vfu+dFsTOc7ULbRO\nzec3iWH07vcHrvNpy\nBMxvJ/wykbw6kf24g/\nUsOpH+5gfTfcSCH1xY\n7U75cXChWxcW/Vn4\n/D610N3GPt5RIX0Ocm\nyg24nkYOCpJL5nE7m3\nTyjkI0jEtIDKAoS0Wx\nQlHtg4tyAyNAJ4hT+C\nemU0fNXFCTKsrPIAz\nMicpSZTAVt7CXwc+",
  "tIDKAoS0Wx\nQlHtg4tyAyNAJ4hT+C\nemU0fNXFCTKsrPIAz\nMicpSZTAVt7CXwc+D\ngokl1T4VUNBzh0ZO2\npDOUOWUl/yMygQP2XQ\nV8cfkZT4ErbdvCvoqR\n9HERHDwl1a2ZwUrkdD\nJgp6nJdbcDJpOiulQ6\nF4kbH0dHtWC5M0Ym8p\nqRUVCUXCDScFAVth2\n0TMAqAtSkCsaAZ1Kny\n4wVO16Bwy+GAi2plu\nGC8mKCqhaQh5KShvUI\naFB",
  "Vth2\n0TMAqAtSkCsaAZ1Kny\n4wVO16Bwy+GAi2plu\nGC8mKCqhaQh5KShvUI\naFBJOxw1rGVkwlVFD2\nQLFcW4ClCZwixAV+G\nLGnOwlRAxmV4n6VimU\nZGpmNlCSkRIyZgyD7\nhakRNQ+Scl9vzvPWra\nb0g4qhOXJyUXU1VxLC\n206YjU5wXMWw6ZcSwY\nBGTauMGBaHB8SQRAS\nyXJcPYcCRoyJ2lQlTZ\nWhbqSx12w7URFzbY\n4T2C9Nb6VA",
  "BGTauMGBaHB8SQRAS\nyXJcPYcCRoyJ2lQlTZ\nWhbqSx12w7URFzbY\n4T2C9Nb6VA6T8hRkZU\nAHaf+mZE+LSpL8cz25\nkm56T0VYGOnRFMVvMS\nkobVsKaNwKjq2ASbZa\n4ME2cLQml82jRVbywq\nTVhzgCpgbro8ZSI4p9\n0pS7BkVdi9A0Nc04P\nfmjfo+NB0VHbRv2Hsg\nkVZXliq0iFL1HREI4k\n5vqCiDl5MTcmDwLl5\nMUc7u/G1JHUXNg",
  "NB0VHbRv2Hsg\nkVZXliq0iFL1HREI4k\n5vqCiDl5MTcmDwLl5\nMUc7u/G1JHUXNgqUs4\ndFJgnMkzY/uzUDSvK\nSNmZ+PI6CsEVL3wTZg\nwJjkImrIKBm+4XBlW\nUC+MUi/GqP4yxPKbr\n5GesZIqWubospUw+r5\ng2VK6F536B8dhWU4eF\nwQi+43DMy6lX59OJcD\nElqJHOspnT82s0kbDH\nb7i+nvCparZAer9btQ\nb9gdnLfp8eHq+Z8hM\nj",
  "59OJcD\nElqJHOspnT82s0kbDH\nb7i+nvCparZAer9btQ\nb9gdnLfp8eHq+Z8hM\njCDjfqgtOstS6OLEt7\nUNdsuZ7vWbH6+nu0tE\nOLazc5qrfupd2uBf0\ngB6vWXq7hjxkYcbd\nU9xB6yLO1BXfY8rtlG\nYXHtJkf1TvNotS3uzD\nSWf7CtjqLqmBTzoTr2\nxdytQqYosSitYhzR0B\nCrkClGedOCv01li8HD\no2lVIVPcyFhTUwFTG\nlJuDqEK",
  "xdytQqYosSitYhzR0B\nCrkClGedOCv01li8HD\no2lVIVPcyFhTUwFTG\nlJuDqEKmWK1hZtmHTP\nVNYu6ZlcJT0aGWYVM8\nTGJzFXIVMsRhaxSO\nSJIZYhVAeR2YeRziPi\nSklNsmckcQyI2hJ2RZ\nUOoqbkgqY0thobWxpD\nHrAY2E0WAdNOcMrL7O\nuPGsYoFX8Y6t4Z0LG\npbEqFAFTGkd7THXbd\nuMs9MRyzbElOmGElO\nIEbprOBnenpzws",
  "YoFX8Y6t4Z0LG\npbEqFAFTGkd7THXbd\nuMs9MRyzbElOmGElO\nIEbprOBnenpzwsKdJ\nLzgjNzA91fQU0z1N\n9zBNUW/CLzghabo14\nkXnGh6gumupruY5prm\nmO5ouoNpoGmA6SNH2\nHqa+pjuqzpMqZSU3Qi\nhSeCptuYjQdYbqv6T\n6mLzV9iekTZ9g+krT\nV5i+1fQtpg80fYAp0Z\nRguqLpCqZU/TqwAuW\nNF3C1NMU/faDvabpB\nqaJpgm",
  "rT\nV5i+1fQtpg80fYAp0Z\nRguqLpCqZU/TqwAuW\nNF3C1NMU/faDvabpB\nqaJpgmDzV9iOlQU/S\nrGJ5nmqLjDTwYNeWYP\ntX0KaZMU/T7zQuea/o\nc0jTCNnmj7D9I2mb\nzB9rOljTEN0bsBOJ1\nouoWpfgtUZJhuarqJ6\nbGmx/b3AnQ2jZ5tYa7\nrCtYxjTWNMV3VFP1Sg\nKOEpkfoPBmI+q42fdu\nE7muBmHELqzM+vRrlP\nBAzbmH13Wl6N",
  "jTWNMV3VFP1Sg\nKOEpkfoPBmI+q42fdu\nE7muBmHELqzM+vRrlP\nBAzbmH13Wl6Nbo/BW\nLGR6jrK7uzFymQUrjT\nH15b7JpvYXFht9fu/t\ni+u3l38f5S/Yb2aub\n1ret71rd1k+t+60nrY\n3WTsuf+3Pu7l/5v5d\n2FuYLPy28Hulzl2pr/\n</latexit>mq1fgs/PEfuP6qZQ=\ny0 = \u03c60\n0 + \u03c60\n1a [ 10 +  11a[\u271310 + \u271311x] +  12a[\u271320 + \u271321x] +",
  "mq1fgs/PEfuP6qZQ=\ny0 = \u03c60\n0 + \u03c60\n1a [ 10 +  11a[\u271310 + \u271311x] +  12a[\u271320 + \u271321x] +  13a[\u271330 + \u271331x]]\n+ \u03c60\n2a[ 20 +  21a[\u271310 + \u271311x] +  22a[\u271320 + \u271321x] +  23a[\u271330 + \u271331x]]\n+ \u03c60\n3a[ 30 +  31a[\u271310 + \u271311x] +  32a[\u271320 + \u271321x] +  33a[\u271330 + \u271331x]]",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "<latexit sha1_base64=\"lgMrqedG/dF\nQKzS8m4ZszuxYJuE=\">AW7HiclZhb9s2FICVbu267pZuWF72IiwoMAydYbfd5W\nVAmzS9JV2SJk7SxqlByZTMhqIUXRKngv/F3oa97icN2H/Zw4l2bTOYR5moDVzvk+\n8HJISLS+RIsu73X+Wrn3w4fUbH938+NYn372+RfLt78yOIi9Xnfj2WcHnks41Io3\ns9FLvlRknIWeZIfeqfrmh+e8zQTsdrPLxN",
  "2+RfLt78yOIi9Xnfj2WcHnks41Io3\ns9FLvlRknIWeZIfeqfrmh+e8zQTsdrPLxN+ErFQiUD4LIfQcDkZeEyFu6v7qCEos\ndzNiy707tQ3o54OP+jBr1F0FsA9xbBvQVwfxHcnw6mw+XVbqdbfVxa6DWFVaf57Ax\nvfz0ajGK/iLjKfcmy7LjXTfKTkqW58CWf3hoUGU+Yf8pCfgxFxSKenZRVaqbuHYiM\n3CBO4Z/K3Sq6eEXJoiy7jDwI5aPM8",
  "Wf3hoUGU+Yf8pCfgxFxSKenZRVaqbuHYiM\n3CBO4Z/K3Sq6eEXJoiy7jDwI5aPM8x0MaOiz45aQUKilyrvy6oaCQbh67Os/uS\nKTcz+UlFJifCuir649ZyvwcZuPWQPELP4ipkblYG1jdwrJ56FQJT8rqpmZTtvORuV\nwKF5lrD3fn9cich6J95xUim6kisEHk7LknfCDgaCAxAdTkCseAZ16vx4gdtDFai\nBAzciyd6fbmvpqRqlfMQctLS3hAN",
  "Hk7LknfCDgaCAxAdTkCseAZ16vx4gdtDFai\nBAzciyd6fbmvpqRqlfMQctLS3hANConk5a1TiyYyqil7IHiundcDXiewixAV+GLo\nznYS5iazq7L+SRPozLTMdxCylTIqyZgyD6TekRtQxVSwqV+y/oNW6+YOm0SFydV1\nMdQdZ+2nbylOZFjdpOFUEWLMKwbVURZEm4b4xYxCDLTXkIA45cHbGrQmFVkIW5k8Z\neu+1ER/DanCSwX9reRknSf85",
  "bVURZEm4b4xYxCDLTXkIA45cHbGrQmFVkIW5k8Z\neu+1ER/DanCSwX9reRknSf85QRnQAdp/+Fkz5vK2vx3PbnSXnvPJ1gU/cMUxW+xKWh\nvWwZo3AqJrYlJpVrpBJswWhNL5om7o3FpUnoj1AHcCbrkiFCha0u1UJlqwOD+7CUN\nNC8uMfOj/yUnZ1dtG/0eyCRVlRWKrSIf/R0UjeFLh9QURPHmxRJMHgWryYgn3dzR\n1LMULW0equYOCUEyK/",
  "eyCRVlRWKrSIf/R0UjeFLh9QURPHmxRJMHgWryYgn3dzR\n1LMULW0equYOCUEyK/BJtfxGq9jVBHc2jlBfIaDrhW8mFJrkIGjLOqBl+IZnrmUB\n+WiQfj1GX8ZkXJy80PrGSKVrm+LqdAPq/YNVWqhfd/gcn4VlOHhcM6vuNxDGfXqf\nHpxoUYsRcmc6CmdvB1kOWwx2+6vprwuWq2Qn2027UG/YHYK3+dnw08HyGxqCNRX\nDIsdYliWVpD+qaL9",
  "B1kOWwx2+6vprwuWq2Qn2027UG/YHYK3+dnw08HyGxqCNRX\nDIsdYliWVpD+qaL9fFnpWb78nSzu0uHZTknqbXtpti3tFD/jZlqW3W8QjFnUkqvp\nIfWIZWkP6rLncs2CotrNyWpd5ZHq21x5yZa/sH+GE6l+pgUy5E+9sVyUIewmFMxt\n4qxPtq2xTqExahoW/A3VvYEPDzaVh3C4k4m2poOYGnEJR5CHcJivYXbZhPD6pZF3b\nKrTCZjZNYhLD",
  "oW/A3VvYEPDzaVh3C4k4m2poOYGnEJR5CHcJivYXbZhPD6pZF3b\nKrTCZjZNYhLD5lER51HcJiSMXQKp6yJEFiHSJ5HOM8jmkeEywlNgnPSGKZEbKkbAs\nqHcdtSQewNEGtTSyNQ9krFCDTRDLGV15mXlKbSKFV3FfVvD/SsazhmqUAewtE32\nmDvYtm4yD6cYjlm2JCcCWQlN4A52dqgzO/15QUlOcl5waeglpReGXlB6aOghpamh5B\neBF7wyl",
  "cYjlm2JCcCWQlN4A52dqgzO/15QUlOcl5waeglpReGXlB6aOghpamh5B\neBF7wylPw68YJzQ8pPTD0gNLC0ILSvqF9SgNDA0qfGPqEUt9Qn9J1Q9cpzQ0lJ1J\n4Ihi6T+nY0DGlR4YeUfra0NeUPjP0GaVvDH1D6XtD31P6yNBHlDJDGaUbhm5Qyg0l\nrw68YM3QNUo9Q8lvP9hrhu5QmhiaUPrY0MeUjgwlv4rheWYoOd7Ag9FQSelzQ59TK\ngw",
  "YM3QNUo9Q8lvP9hrhu5QmhiaUPrY0MeUjgwlv4rheWYoOd7Ag9FQSelzQ59TK\ngwlv9+84KWhLymNDI0ofWHoC0rfGfqO0qeGPqU0NJS8G4DTiaF7lJq3QGVG6a6hu5\nSeGXpmfy/A59Po2Rbmtqlgm9LY0JjSTUPJLwU4Sh6Ss6TgWruarO3TeS+Fqg5t7Am\n47OrSc4DNecW1tydZleT+1Og5nxMur5xMH+RAimFO/1webWH38LSwsG9Tu+nzo",
  "m\n47OrSc4DNecW1tydZleT+1Og5nxMur5xMH+RAimFO/1webWH38LSwsG9Tu+nzoPdB\n6sP15o3tDedb5xvne+cnvOz89B5uw4fcd3/nb+Xbq+dGNFrfy+8sfKn7V6bam5i\nun9Vn56z+G+/nu</latexit>\u03c6 = {\u03b20, \u23260, \u03b21, \u23261, \u03b22, \u23262, \u03b23, \u23263}\nProblem 1:  Computing gradients\n<latexit sha1_base64=\"JE7fQ4+WZVE\nSdJEBwEOhglbuk/Q=\">AW+niclZh",
  "1:  Computing gradients\n<latexit sha1_base64=\"JE7fQ4+WZVE\nSdJEBwEOhglbuk/Q=\">AW+niclZhbU9w2FICXlN6I03LS18ZTLTaZMdtpNeHh\nMIuUEKBZIMNmRvbJXQZaNL7DEdX9M3zp97Z/pe39Ij2zvCp8jHspMur5PutyJMu\nyvUSKLF9d/WfhnXfe/+D298tPjxJ59+9vnSzS8OsrhIfT70YxmnRx7LuBSKD3ORS\n36UpJxFnuSH3um65ofnPM1ErPbzy4",
  "+9vnSzS8OsrhIfT70YxmnRx7LuBSKD3ORS\n36UpJxFnuSH3um65ofnPM1ErPbzy4SfRCxUIhA+yE0Wvrdcb0gmYhRmX8/qFwZq1\nDyIGdpGl/MSeXcdVwmkwlz3KyIRqVwhXIjlk98Jsu1SiuVG6TML92Epblg0nG5lCB\nWx6aWk+oKrqPVaGltb9a/zm0MGgLK732b2d086uxO479IuIq9yXLsuPBapKflLpa\nX/Jq0S0ynjD/lIX8GIqKRTw7K",
  "gLK732b2d086uxO479IuIq9yXLsuPBapKflLpa\nX/Jq0S0ynjD/lIX8GIqKRTw7KetEVc5tiIydIE7hn8qdOnr1ipJFWXYZeWDq0WY6\naCNHRd58MtJKVRS5Fz5TUNBIZ08dnTWnbFIuZ/LSygwPxXQV8efMEhYDnOz6Cp+4cd\nRxNS4dNc2diFPHg+FKvlZUc9TVXWdjdrhULzOWHu6P69F5DwSbzmpFZ0JdcIPKzK\nkvfDPgaCAxB9TkCseAZ1",
  "Uc9TVXWdjdrhULzOWHu6P69F5DwSbzmpFZ0JdcIPKzK\nkvfDPgaCAxB9TkCseAZ16vx4gTNAFNalBAzci6fQucB5UZGqVc5DyElHe0U0KCST\nzvWOrFgKqOsgeK49x2NOB5CrMAXYUfjuZgL2Gqml2X82meRmWmY7iFlKmQ10N4\nMeUdQhZRwqd+xfsXWC6ZO28TFSd3VEeQtZ92nTyleVHjrlNHkAWLMOxadQRZEna\nRMYsYZLktj2DAkaMjdlU",
  "O28TFSd3VEeQtZ92nTyleVHjrlNHkAWLMOxadQRZEna\nRMYsYZLktj2DAkaMjdlUorAqyMHfS2Ou2negIXpvTBO6XrdRkvSfM5QRHYC7T/8Kp\nnze1dfjue3MknNe+7rAp84EJqt7CUvDZlizRmBUbayiZp0rZNJsQh216pe2NReS\nK6A9QBfNMVqVDBFe1OXYIlq8PuHRhqWkh+fLf/I5+elKv6tH/IdmEirIisVWkw/+\njojE8t/D6gievFiyY",
  "XYIlq8PuHRhqWkh+fLf/I5+elKv6tH/IdmEirIisVWkw/+\njojE8t/D6gievFiyYNAPXmxhP0dTR1L8cLWkXruoCAUkyK/RLe/CFX3mjqCOxtH\nqK8Q0PXCLxMKTXIQdGUd0DL8whPYsoB8NEi/GaMv46xIOdn80HqGSK3rbTEV+mHV3\nVClFr7Bpfzq6AMD4dzfs3lHsqo1+Tiws1ZilK5lRP6fS1m8HZwHr31PeFK1WyM\n82/agXzA7he/zs9E",
  "4dzfs3lHsqo1+Tiws1ZilK5lRP6fS1m8HZwHr31PeFK1WyM\n82/agXzA7he/zs9Emno+QWNSRqC48ljrksSytAd1zZfr1Z6Vm6+/I0s7tLh2U5J6\n217abYt7TQ/42Zalt1vEIxZ1JKqr7SH1iGVpD+qy53HLNgqLazclqXeWR6tcecmW\nv7B/oTnTB+TYjnWx75Yuk0IizkVc6sYRzxEYhPCYlR0Lfh/rOwJeHh0rSaExZ1MdD\nUdwNKYSzyEJo",
  "Wx75Yuk0IizkVc6sYRzxEYhPCYlR0Lfh/rOwJeHh0rSaExZ1MdD\nUdwNKYSzyEJoTF5hbum0Mq1sWdcu1m8PXbMJYfExi/ComxAWQyqGVvGUJQkSmxD\nJ4wTncULzmGApsUl4RhLjJAlZVtQ6STuSjqApSlqbWpDHoAL3eowTaI5YyuvMy6\n8hRaxYqu4qGt4eE1DecMVagDWNom95jbltvMg+nGI5ZtiQnAlkJTeAOdnaoMzv9eU\nFJTnJecGno",
  "4eE1DecMVagDWNom95jbltvMg+nGI5ZtiQnAlkJTeAOdnaoMzv9eU\nFJTnJecGnoJaUXhl5QemjoIaWpoeSNwAteGEreTrzg3NBzSg8MPaC0MLSgdGjokNL\nA0IDSR4Y+otQ31Kd03dB1SnNDyYkUngiG7lM6MXRC6ZGhR5S+NPQlpU8MfULpK0Nf\nUfrW0LeUPjD0AaXMUEbphqEblHJDyacDL1gzdI1Sz1Dy7gf3mqE7lCaGJpQ+NPQhp\nWNDy",
  "eUPjD0AaXMUEbphqEblHJDyacDL1gzdI1Sz1Dy7gf3mqE7lCaGJpQ+NPQhp\nWNDyVsxPM8MJcbeDAaKil9auhTSoWh5P3NC54b+pzSyNCI0meGPqP0jaFvKH1s6G\nNKQ0PJtwE4nRi6R6n5ClRmlO4aukvpmaFn9u8CfD6Nnm1hbpsKtimNDY0p3TSUvCnA\nUcLQU3KeDFS7q82+NpF9LVBzbmFtxmdXk5wHas4trN2dZleT/SlQcz4hXd84mH9I",
  "UcLQU3KeDFS7q82+NpF9LVBzbmFtxmdXk5wHas4trN2dZleT/SlQcz4hXd84mH9Ig\nt3YXHh1sKXy78t/7H85/JfjfrOQnvNrV7nb/nv/wB7kgI4</latexit>ZTCTj9aWhngr7C0cPBDf/BT/97uvZX7a+0X2hu9r3vf9L7tDXo/9+73nvR2esOe3/\n\u03c6t+1  \u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n<latexit sha1_base64=\"CgqealUvUgO\nNHeXUaVLIF",
  "\u2212 \u03c6t \u2212 \u21b5\nX\ni2Bt\n@`i[\u03c6t]\n@\u03c6\n<latexit sha1_base64=\"CgqealUvUgO\nNHeXUaVLIFyDX/KE=\">AW3HiclZhJU9xGFIBlZ3PIYpxUuOSiCuWqVIpMQcpZLq\n6ywdjG4DAYBrAZIC1NS9Om1RJaYLBKt9xSueYn5T/kP+SanPNa0kxb7zWHTBVMz/s\n+9fK6W5uXSJHly8t/3bj5zrvf/BrQ/nPvr4k09vz9/5bD+Li9TnAz+WcXrosYxLo\nfgF7nkh",
  "XSJHly8t/3bj5zrvf/BrQ/nPvr4k09vz9/5bD+Li9TnAz+WcXrosYxLo\nfgF7nkh0nKWeRJfuCdrWl+cMHTMRqL79K+HEQiUC4bMcQqfzv7hbR0MvSMbi+L\n47zIrotBT3V6qTcqNyh1xK+Fm5lERePCldQUgkrXMdHuUlvZknulfx7Pnc4vLve\nW649LCytYdFpP/3TO1+MhqPYLyKucl+yLDtaWU7y45KlufAlr+aGRcYT5p+xkB9B\nUbGIZ8d",
  "YdFpP/3TO1+MhqPYLyKucl+yLDtaWU7y45KlufAlr+aGRcYT5p+xkB9B\nUbGIZ8dlnYrKvQuRkRvEKfyp3K2jbx9RsijLriIPzIjl4wzHbSxoyIPfjouhUqKn\nCu/aSgopJvHrs6rOxIp93N5BQXmpwL6vpjljI/h+zPDRW/9OMoYmpUDlfXd6py6PF\nQqJKfF/VMVFXWa8dDsXrjNWNvVktIueReMNJbWiK7lG4GFVlrwX9jAQHIDocQJi\nxTOo",
  "fF/VMVFXWa8dDsXrjNWNvVktIueReMNJbWiK7lG4GFVlrwX9jAQHIDocQJi\nxTOoU+fHC9wVRGHlScBlsyxgKbgvKlK1ynkIOelor4gGhUTyScdaIxZMZdRdkFx3\nbuBjxPYRagq/DF0RzsJkxV0+NyPsnTqMx0DLeQMhXyugkYs+kHlHXUIWUcKjfsX\n7G1gumztrExUnd1VRHkLWXdp08pXlRo65TR5AFizDsWnUEWRLOEyMWMchyWz6FAUe\nu",
  "gumztrExUnd1VRHkLWXdp08pXlRo65TR5AFizDsWnUEWRLOEyMWMchyWz6FAUe\nujthVobAqyMLsp7HXbTvREbw2Jwnsl63XpL0XzCUER2A3ae/BVM+7+pr8cx2p8m5q\nH1d4BN3DJPVPYSlYTOsaSMwqjZWUbPOFTJptiCUxpdU/fGovJEdAeoA3jTFalQwV\nvaUl2CJavDwyUYalpIfvRt73s+OS6X9bR/0g2oaKsSGwV6fD/qGgEVya8viCJy",
  "V\nvaUl2CJavDwyUYalpIfvRt73s+OS6X9bR/0g2oaKsSGwV6fD/qGgEVya8viCJy+\nWaPIgUE9eLOH8jqaOpXh60g9d1AQikmRX6HtL0LVPaO4M7GEeorBHS98M2EQpMc\nBF1ZB7QM3CNtSwgHw3Sb8boyzgrUk5Ofmg9Q6TW9WkxFfpi1T2hSi10zxtczo6CM\nlwcLvg1h3so16Ty8u1IilKJkTPaWTk2GWwxaz7f56ypui1Qr5+WbHvQLZqfwf",
  "CM\nlwcLvg1h3so16Ty8u1IilKJkTPaWTk2GWwxaz7f56ypui1Qr5+WbHvQLZqfwfX\n5+uonIyQWdSqC25qrHVJYlnag7pmy/XtnpWbJ9+QpR1aXLspSb1tL+2xb2mB/x8\ny9LbLeIRizoS1dX2kHrEsrQHdnzuGUbhcW1m5LUO82j1ba4MxMt/2BvzHOmb5NiO\ndK3fbEcNiEs5lTMrWIc8RCJTQiLUdG14DdWdgVcPLpWE8JiPxNdTQewNOISD6",
  "iO\ndK3fbEcNiEs5lTMrWIc8RCJTQiLUdG14DdWdgVcPLpWE8JiPxNdTQewNOISD6EJYb\nHZwl2zjWF1y6Ju2VUmkzEymxAWn7AIj7oJYTGkYmgVz1iSILEJkTyOcR7HNI8JlhK\nbhGckscwIWVK2BZWO46kA1iaoNYmlsagBzJWqME2iOWMrzMuvIUWsWKruKBreHB\nNQ3nDFWoA1jaJnvMHW5bN5mHUwy3WbYkJwJZCU1gHzt96kzv/rygJHdyX",
  "KBreHB\nNQ3nDFWoA1jaJnvMHW5bN5mHUwy3WbYkJwJZCU1gHzt96kzv/rygJHdyXnBl6BWl4\nZeUnpg6AGlqaHkicALXhKnk684MLQC0r3Dd2ntDC0oHRg6IDSwNCA0seGPqbUN9S\nndM3QNUpzQ8kdKVwRDN2jdGzomNJDQw8pfWnoS0qfGvqU0leGvqL0jaFvKH1o6ENK\nmaGM0nVD1ynlhpJXB16waugqpZ6h5NkP9pqhfUoTQxNKHxn6iNKR",
  "aFvKH1o6ENK\nmaGM0nVD1ynlhpJXB16waugqpZ6h5NkP9pqhfUoTQxNKHxn6iNKRoeSpGK5nhpLbG\n7gwGiop3TB0g1JhKHl+84Lnhj6nNDI0ovSZoc8ofW3oa0qfGPqE0tBQ8m4A7k4M3a\nXUvAUqM0p3DN2h9NzQc/t7AT6bRs+2MLdNBduUxobGlG4aSp4U4FbC0DNyPxmo9qw2\nfdtEzmuBmnELazM+PZrkPFAzbmHt2Wl6NDk/BWrGx6Tr6",
  "4U4FbC0DNyPxmo9qw2\nfdtEzmuBmnELazM+PZrkPFAzbmHt2Wl6NDk/BWrGx6Tr6/uzFymQUjTn84vruC3s\nLSw/1v5YfevZ17iw9W2ze0t5wvna+cr50V50fngfPU6TsDx3f+dP52/nH+XThZ+H\nXht4XfG/XmjfaYz53OZ+GP/wBpavXY</latexit>\nL[\u03c6] =\nI\nX\ni=1\n`i =\nI\nX\ni=1\nl[f[xi, \u03c6], yi]\n<latexit sha1_base64=\"Z8pXI1y37lX\nEeGW",
  "\u03c6] =\nI\nX\ni=1\n`i =\nI\nX\ni=1\nl[f[xi, \u03c6], yi]\n<latexit sha1_base64=\"Z8pXI1y37lX\nEeGWnMsH2tezVrhY=\">AW+niclZhb9s2FICd7tZlt3bd8rIXYUWBYeiMZOguj2\n3S9JZ0ce5po9SgZEpmQ1GKRCVONe/H7G3Y6/7M3vdDdijJZnUOA2wGrPn+8TLISn\nRCjIpCr28/PfCtXfefe/9D65/uPjRx598+tmNm58fFGmZh3w/TGWaHwWs4FIovq+Fl\nv",
  "pCr28/PfCtXfefe/9D65/uPjRx598+tmNm58fFGmZh3w/TGWaHwWs4FIovq+Fl\nvwoyzlLAskPg9M1w/PeV6IVO3py4yfJCxWIhIh0xAa3vjN86OchZWfsVwLJj2fSz\nmsxHQ6D/lBFHDNhtXpdOr5ZyUb1X8PwnScXUqBv9D9VtJTxu6hveuL3cX64/Hi2\nstIXbvfYzGN78cuSP0rBMuNKhZEVxvLKc6ZPK1B1KPl30y4JnLDxlMT+GomIJL06",
  "stIXbvfYzGN78cuSP0rBMuNKhZEVxvLKc6ZPK1B1KPl30y4JnLDxlMT+GomIJL06q\nOlFT7w5ERl6U5vBPa+Ovn1FxZKiuEwCMBOmxwVmJuhix6WOfj6phMpKzVXYNBSV0\ntOpZ7LujUTOQy0vocDCXEBfvXDMIFMa5mbRV/wiTJMEcln5q+vbkKyAx0JVHFJq5mk\n67TrtcOheJWx+nRvXovQPBFvOKmkVkwlVwg8nlYV78d9DAQHIPqcgFTxAu",
  "Jq5mk\n67TrtcOheJWx+nRvXovQPBFvOKmkVkwlVwg8nlYV78d9DAQHIPqcgFTxAuo0+Qki\nbwVRWJcScNUsHFgH3s6UVK0jyEnHe0l0aCQST7pWGvEgqlMOsouKJ53xzOA6xmA\nboKXxzNwW7G1HR2neYTnSdVYWK4hZypmNdNwJBDWO872FClHBp2LF+wdYOU6dt4t\nKs7mpuIsjay7uOzmleYDt2nDqCLFiEcdeqI8iScBcZsYRBltvyEAaceC",
  "OU6dt4t\nKs7mpuIsjay7uOzmleYDt2nDqCLFiEcdeqI8iScBcZsYRBltvyEAaceCbiVoXCqiA\nLc5CnQbftzETw2pxksF+63npF0n/OUEZMAHaf+RZMhbyr6Vz25sl57z2TYFPvDFMV\nvcSlsfNsGaNwKja2JSada6QSbMFoTy96JqmNw6VZ6I7QBPAm67MhYre0u7WJViyJu\nzfhaHmpeTH3/V/4JOTatlsG/OHZBMqKsrMVZEJ/4+KRvDcwusL",
  "Yre0u7WJViyJu\nzfhaHmpeTH3/V/4JOTatlsG/OHZBMqKsrMVZEJ/4+KRvDcwusLInjyUokmDwL15KU\nS7u9o6liOF7aJ1HMHBaGYFPoSbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiG\nJ7BjAYVokGEzxlCmRZlzcvND6xkitW5ui7kwD6vuDVUaoXvf4HJ+FZTh4XDOr7g8Q\nBkNmnwGalGLEfJnJgpnbzyCw1bzLX76ylvik4r5mcbXv",
  "vf4HJ+FZTh4XDOr7g8Q\nBkNmnwGalGLEfJnJgpnbzyCw1bzLX76ylvik4r5mcbXvQL5idMgz52XADz0dMLO\npIVBceZx1SWI52oO65sv17Z5VG6+JUs7drhuU5J6216bYd7RQ/42ajt5vEIxZ1\nJKqr7SH1iOVoD+py53HTNQqH6zYlqXeWR6ftcOcmWv7R3hjOteaYlMqROfal0m9CW\nNRU1E4xNSfbrtiEsJiUXQv+j5VdAQ+PrtWEsDgoRFcz",
  "hjOteaYlMqROfal0m9CW\nNRU1E4xNSfbrtiEsJiUXQv+j5VdAQ+PrtWEsDgoRFczASyNuMRDaEJYbLZw12xjWN\n10qJtulclsjMwmhMXHLMGjbkJYjKkYO8VTlmVIbEIkj2OcxzHNY4alzCXhGckcM0K\nWlGtB5eO0K5kAliaotYmjMeiBTBVqsA1iuaAr3CuPIVWsaKreN/V8P4VDWuGKjQB\nLG2RPeb5W85NFuAUwzHLleRMICujCRxgZ0Cd2e",
  "uPIVWsaKreN/V8P4VDWuGKjQB\nLG2RPeb5W85NFuAUwzHLleRMICujCRxgZ0Cd2ekviCpykguiS0svKb2w9ILSQ0sPKc\n0tJb8IgmjHUvLrJIjOLT2n9MDSA0pLS0tK9y3dpzSyNKL0kaWPKA0tDSlds3SNUm0\npOZHCE8HSPUrHlo4pPbL0iNIXlr6g9ImlTyh9aelLSt9Y+obSB5Y+oJRZyihdt3Sd\nUm4peXUQRKuWrlIaWEp+8Fes3RAaWZp",
  "Tyh9aelLSt9Y+obSB5Y+oJRZyihdt3Sd\nUm4peXUQRKuWrlIaWEp+8Fes3RAaWZpRulDSx9SOrKU/CqG5ml5HgD0ZLJaVPL\nX1KqbCU/H4LoueWPqc0sTSh9Jmlzyh9belrSh9b+pjS2FLybgBOJ5buUmrfAlUFpd\nuWblN6ZumZ+70An09j4FqYW7aCLUpTS1NKNywlvxTgKGHpKTlPRq9q83eNpH7WqTm\n3MHajM+uJjmP1Jw7WHt3ml1N7k+R",
  "S1NKNywlvxTgKGHpKTlPRq9q83eNpH7WqTm\n3MHajM+uJjmP1Jw7WHt3ml1N7k+RmvMx6fr6wfxFCqT0oH4vu4LfwtLCwf9lR/79\n7bv3b6/2r6hvd7qvd175veSu+n3v3ek96gt98Le/8sLC7cWvhi6del35f+WPqzUa\n8tNfc6nU+S3/9C4EkAnw=</latexit> @`i\n@\u03b2k\nand\n@`i\n@\u2326k\nLoss: sum of individual terms:\nSGD Algorithm:\nParameters:\nNeed to compute gradients",
  "Algorithm to compute gradient efficiently\n\u2022 \u201cBackpropagation algorithm\u201d\n\u2022 Rumelhart, Hinton, and Williams (1986)",
  "BackProp intuition #1:  the forward pass\n\u2022\nThe weight on the orange arrow multiplies activation (ReLU output) of previous layer  \n\u2022\nWe want to know how change in orange weight affects loss\n\u2022\nIf we double activation in previous layer, weight will have twice the effect\n\u2022\nConclusion: we need to know the activations at each layer.\nRemember! There\u2019s an implied weight \non every arrow in the diagram",
  "BackProp intuition #2: the backward pass\nTo calculate how a small change in a weight or bias feeding into hidden layer \ud835\udc21!modifies \nthe loss, we need to know:\n\u2022 how a change in layer \ud835\udc21! changes the model output \ud835\udc1f\n\u2022 how a change in the model output changes the loss \ud835\udc59",
  "BackProp intuition #2: the backward pass\nTo calculate how a small change in a weight or bias feeding into hidden layer \ud835\udc21\"modifies \nthe loss, we need to know:\n\u2022 how a change in layer \ud835\udc21\" affects \ud835\udc21! \n\u2022 how \ud835\udc21! changes the model output \ud835\udc1f\n\u2022 how a change in the model output \ud835\udc1f changes the loss \ud835\udc59\nWe know this from the \nprevious step",
  "BackProp intuition #2: the backward pass\nTo calculate how a small change in a weight or bias feeding into hidden layer \ud835\udc21#modifies \nthe loss, we need to know:\n\u2022 how a change in layer \ud835\udc21# affects \ud835\udc21\" \n\u2022 how a change in layer \ud835\udc21\" affects \ud835\udc21! \n\u2022 how \ud835\udc21! changes the model output \ud835\udc1f\n\u2022 how a change in the model output \ud835\udc1f changes the loss \ud835\udc59\nWe know these from the \nprevious steps",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "Toy Network\nf \ud835\udc65(, \ud835\udf19 = \ud835\udefd) + \ud835\udf14) \u22c5 a \ud835\udefd# + \ud835\udf14# \u22c5 a \ud835\udefd\" + \ud835\udf14\" \u22c5 a \ud835\udefd! + \ud835\udf14! \u22c5 \ud835\udc65(\n\u2113( = \ud835\udc53 \ud835\udc65(, \ud835\udf19 \u2212 \ud835\udc66(\n#\n\ud835\udc65!\n\u210e\"\n\u210e#\n\u210e$\n\ud835\udc53\n\u2113!\n\ud835\udc66!\n1 input\n3 layers, 1 hidden unit each",
  "Gradients of toy function\nWe want to calculate:\nTells us how a small \nchange in \ud835\udefd& or \ud835\udf14& change \nthe loss \u2113& for the ith \nexample\n<latexit sha1_base64=\n\"xKo8apd0WvpzoMJQmihazt3aosY=\">AYFXiclZ\nhbc9tEFICdcCvh1sIweBFQ6czDFOMXSjw2CZN2zQ\npcS5O0sapZyWv5G1WK0WXxKnGvwP+DG8Mrza+Cs\nJHujczZDk5ni5XzfXnR2V1rJjaVIs07n4XFd9597\n/0Pbny49NHn3z62c1bn+nUZ54vO9FMkoOXZ",
  "5XzfXnR2V1rJjaVIs07n4XFd9597\n/0Pbny49NHn3z62c1bn+nUZ54vO9FMkoOXZyKR\nTvZyKT/DBOAtdyQ/ck1XND854kopI7WUXMT8OWaC\nELzyWQWh4a+H3gZ8wrxjELMkEk86ASzkU0sBl2d\nsWHSm07vO4DRnI+d/a0QhD9iw8/YVqi61+6iO6uw\n9JZd3Lt2F9eqUnXyw7zGIHSjScHUaHrNTqGJ4c3bn\nXan/HNoVsXbrfqv97w1pejwS",
  "Zd3Lt2F9eqUnXyw7zGIHSjScHUaHrNTqGJ4c3bn\nXan/HNoVsXbrfqv97w1pejwSjy8pCrzJMsTY+6nT\ng7LnTnuTpUGe8ph5JyzgR1BULOTpcVGuoalzByI\njx48S+Kcyp4xerlGwME0vQhfMkGXjFDMdtLGjPN/\nOS6EivOMK6/qyM+lk0WOXpDOSCTcy+QFJiXCBir4\n40ZpCmDZbs0UPzci8IQclgMVta2IVcuD4QqORTL+\nHptOmslQ6H4l",
  "QFJiXCBir4\n40ZpCmDZbs0UPzci8IQclgMVta2IVcuD4QqORTL+\nHptOmslQ6H4lXGyvrevBWR8VC84aSRUtGNXCHwYFo\nUvB20MRAcgGhzAiLFU2hT58f1nS6isGUl4KJaMAMw\ndqakaZXxAHLS0F4SDQqx5JOGtUosmMqwoeyC4jh3H\nA14lsAswFDh6M52I2Zms7qZXySJWGR6hjuIWEq4G\nUXcMkerO0dbKhcSqjqNaxfsbXD1EmduCguh5r",
  "2I2Zms7qZXySJWGR6hjuIWEq4G\nUXcMkerO0dbKhcSqjqNaxfsbXD1EmduCguh5roCL\n2kqaTJTQvsA0bThlBFizCoGmVEWRJuMGOWMgy3V\n5CBcOjpiV4XCqiALs5dEbrPvWEfw2pzEsF+a3lpB\n0n/GUEZ0AHaf/hVMebypr0Zz25kl56z0dYFPnDFMV\nrMKS4LqsmadwFXVsSk1y1whk2YLQkl03jT1aCwqj0\nXzAnUAb7o8Ecq/pN0tS7Bkd",
  "MKS4LqsmadwFXVsSk1y1whk2YLQkl03jT1aCwqj0\nXzAnUAb7o8Ecq/pN0tS7BkdXhwFy41ySU/+q59n0+\nOi47eNvo/JvQUJrHtoZ0+BoNjeCRjtcXRPDkRJN\nHgTKyYsk3N/R1LEL2wdKecOCkIxKbILtP1FoJp1y\ngebBSisUJAtwu/TCg0yb7flHVAy/ALhxPLAvLQRX\nrVNXoySvOEk5sfWs8QKXV9W0yEflg1b6hSC837Bpf\nzWlCGh8MZ",
  "ALhxPLAvLQRX\nrVNXoySvOEk5sfWs8QKXV9W0yEflg1b6hSC837Bpf\nzWlCGh8MZv6K6izLqVvl0o1yNWIKSOdFTOnk1SDPY\nYrbdX05VbRaAT/dqPuDcHs5J7HT4cbeD4CYlFHo\nrbgNGhtSxL0h+0NV+ul0dWbLz6liztwOLaTUnarU\ndpty3uFSPgp5uW0W4Sj1jUkaiteoTUI5alP2jLnsd\nN21VYXLspSbuzPFptizs30fL398ZwlNXHpEi",
  "0W4Sj1jUkaiteoTUI5alP2jLnsd\nN21VYXLspSbuzPFptizs30fL398ZwlNXHpEiO9LE\nvkoMqhMWMiplVLE+2TbEKYTHMmxb8P1Z2BTw8mlYV\nwmIvFU1NB7A04hJfQhXCYrWFm2Ydw+qmRd20q0zGY\n2RWISw+YSG+6iqExYCKgVU8YXGMxCpE8jGeRzTPM\nZYim0SnpHYMiNkSdkWVDKOmpIOYGmCeptYOoMRyEi\nhDusglO68lLrylNoFSu6",
  "M\nZYim0SnpHYMiNkSdkWVDKOmpIOYGmCeptYOoMRyEi\nhDusglO68lLrylNoFSu6ivu2jvtXdJwx1KAOYGmL\n7DFnsGXdZC5OcfXySqdLICumCexhp0ed2enP9Qtyk\nnP9C0MvKD039JzSA0MPKE0MJW8Er9jKHk7cf0zQ8\n8o3Td0n9Lc0JzSvqF9Sn1DfUofG/qYUs9Qj9JVQ1c\npzQwlJ1J4Ihi6R+nY0DGlh4YeUvrC0BeUPjX0KaUv\nDX1J",
  "ofG/qYUs9Qj9JVQ1c\npzQwlJ1J4Ihi6R+nY0DGlh4YeUvrC0BeUPjX0KaUv\nDX1J6RtD31D60NCHlDJDGaVrhq5Ryg0lnw5cf8XQF\nUpdQ8m7H+w1Q3uUxobGlD4y9BGlI0PJWzE8zwlx\nt4MBoqKV03dJ1SYSh5f3P954Y+pzQ0NKT0maHPKH1\nt6GtKnxj6hNLAUPJtAE4nhu5Sar4CFSml24ZuU3p\nq6Kn9uwCfT6NrW5hbpoEtSiNDI0o3DC",
  "6hNLAUPJtAE4nhu5Sar4CFSml24ZuU3p\nq6Kn9uwCfT6NrW5hbpoEtSiNDI0o3DCVvCnCUMPSE\nnCd9Vd/VZl+byH3NV3NuYXGZ7VJzn015xZW351mt\ncn9yVdzPiZDX9uf0iBlO6X32W7+CsLezfa3d/at\n/f/vH2g5X6C+2N1letr1vftLqtn1sPWk9bvVa/5S3\n8u3hnsb34/fJvy38s/7n8V6UuLtR1vmg1/pb/g/T\nU214</latexit>@`i",
  "a/5S3\n8u3hnsb34/fJvy38s/7n8V6UuLtR1vmg1/pb/g/T\nU214</latexit>@`i\n@\u03b20\n,\n@`i\n@!0\n,\n@`i\n@\u03b21\n,\n@`i\n@!1\n,\n@`i\n@\u03b22\n,\n@`i\n@!2\n,\n@`i\n@\u03b23\n,\nand\n@`i\n@!3\nf \ud835\udc65(, \ud835\udf19 = \ud835\udefd) + \ud835\udf14) \u22c5 a \ud835\udefd# + \ud835\udf14# \u22c5 a \ud835\udefd\" + \ud835\udf14\" \u22c5 a \ud835\udefd! + \ud835\udf14! \u22c5 \ud835\udc65(\n\u2113( = \ud835\udc53 \ud835\udc65(, \ud835\udf19 \u2212 \ud835\udc66( #",
  "Toy function\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc65!\n\u210e\"\n\u210e#\n\u210e$\n\ud835\udc53$\n\u2113!\n\ud835\udc66!\n\ud835\udc53%\n\ud835\udc53\"\n\ud835\udc53#\nActivations\n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* ) \nIntermediate values",
  "Refresher: The Chain Rule\nFor \u210e \ud835\udc65 = \ud835\udc54 \ud835\udc53 \ud835\udc65\nthen \u210e! \ud835\udc65 = \ud835\udc54! \ud835\udc53 \ud835\udc65\n\ud835\udc53! \ud835\udc65 , where \u210e! \ud835\udc65 is the derivative of \u210e \ud835\udc65 .\nOr can be written as\n\ud835\udf15\u210e\n\ud835\udf15\ud835\udc53 = \ud835\udf15\u210e\n\ud835\udf15\ud835\udc54\n\ud835\udf15\ud835\udc54\n\ud835\udf15\ud835\udc53\n\ud835\udc53\n\ud835\udc54\n\u210e\n\ud835\udc65",
  "Forward pass\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\nf \ud835\udc65(, \ud835\udf19 = \ud835\udefd) + \ud835\udf14) \u22c5 a \ud835\udefd# + \ud835\udf14# \u22c5 a \ud835\udefd\" + \ud835\udf14\" \u22c5 a \ud835\udefd! + \ud835\udf14! \u22c5 \ud835\udc65(\n\u2113( = \ud835\udc53 \ud835\udc65(, \ud835\udf19 \u2212 \ud835\udc66(\n#\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65& \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the derivatives \nof the loss with respect to \nthese intermediate \nquantities, but in reverse \norder.\n<latexit sha1_base64=\"rq4C8K+oC31Y0b9WL/\nbB5Has2CA=\">AXu3iclZhb9s2FIDt7tZlt3TDkIe9CAsKDENn2O267WVDmzS9JV2ci5O0cWpQMiW\nzoShFl8Sp4J+zX7PX7WH/ZoeSbFbnMGhrIDV7vk8kdXgRLTeWIs263f/a1z748KOP7n+6dJn3/x5\nVfLN74+SKM8fjAi2SUHLks5VIoPshEJvlRnH",
  "Is263f/a1z748KOP7n+6dJn3/x5\nVfLN74+SKM8fjAi2SUHLks5VIoPshEJvlRnHAWupIfuqfrmh+e8yQVkdrPLmN+ErJACV94LIPQ6Eb7\nj6GfMK8YxizJBJPOkEs5EjMT8EfFndnsljM8y9nYeZs9Gd15Z9cf3Z67S2+v9/Z71Nt7j/4u3GHoRtO\nCqfHsndvpzkbLq91Ot/w4tNCrC6ut+tMf3fh2PBxHXh5ylXmSpelxrxtnJ4Wu05N8tjTM",
  "ndvpzkbLq91Ot/w4tNCrC6ut+tMf3fh2PBxHXh5ylXmSpelxrxtnJ4Wu05N8tjTMUx4z75QF/\nBiKioU8PSnKkZ45NyEydvwogT+VOWX0zSsKFqbpZeiCGbJskmKmgzZ2nGf+byeFUHGeceVDfm5dLI\n0dPGYuEe5m8hALzEgF9dbwJg/xkMLmWhopfeFEYQvK4drGDiTJ5YFQBYdE6ok2mzWdjdLhULzKWHu\nyv6hFZDwUrzmpFR0JVcIPJgVBe",
  "vK4drGDiTJ5YFQBYdE6ok2mzWdjdLhULzKWHu\nyv6hFZDwUrzmpFR0JVcIPJgVBe8EHQwEByA6nIBI8RTq1PlxfaeHKCwsCbioZsoQjN0ZqVplPICcN\nLQXRINCLPm0Ya0TC4YybCh7oDjOTUcDniUwCtBV+OJoDPZipmbz6zI+zZKwSHUMt5AwFfCyCbhlDyb1\nLjZULiVc6jWsP7G1y9RpnbgoLrua6Aiy9pOmkyU0L7D+Gk4ZQRZMwqBplRFk",
  "yb1\nLjZULiVc6jWsP7G1y9RpnbgoLrua6Aiy9pOmkyU0L7D+Gk4ZQRZMwqBplRFkSdgGxyxkOW6PIbDh0\ndsatCYVWQidlPIrfZdqwjeG5OY1gvTW+jIOk/ZygjOgCrT38Lpjze1Nejhe3Mk3Ne+rAp84EBqt5C\nUuC6rbmjcBd1bEZNctcIZNmC0JdNE0dW8sKo9F8wZ1AC+6PBHKf0O7VZgyurw8BbcapJLfvxT5y6f\nnhRdvWz0PySbUFGax7",
  "8sKo9F8wZ1AC+6PBHKf0O7VZgyurw8BbcapJLfvxT5y6f\nnhRdvWz0PySbUFGax7aKdPg9KhrDgxfPL4jgwYskGjwIlIMXSdjf0dCxBE9sHSnHDgpCMSmyS7T8RaC\na15QR3NkoRH2FgK4XvplQaJB9vynrgJbhG4QlgnkoZv0qnv0ZJTmCSebH5rPECl1vS0mQj+smhuq1\nEJz3+BycRWU4eFwzq+43EUZdat8ulGuxixByZzqIZ2+HKYZLDHb6",
  "0mQj+smhuq1\nEJz3+BycRWU4eFwzq+43EUZdat8ulGuxixByZzqIZ2+HKYZLDHb6i+HvCparYCfbdbtQb9gdHLP42ej\nTweAbGoI1FdcGaz1iWJZWkP6lpM1zd7Vmy+/JFM7cDi2k1J6q17abct7hU94Gdblt5uEY9Y1JGorq\nH1COWpT2oy57HLdtdWFy7KUm98zxabYu7MNH09/cnPGP6mBTJsT72RXJYhbCYUTGzilHIAyRWISyGe\ndOC/2Nl",
  "Um98zxabYu7MNH09/cnPGP6mBTJsT72RXJYhbCYUTGzilHIAyRWISyGe\ndOC/2NlT8Do2lVISz2U9HUdABLYy7xLVQhLFZLuGnWMaxuWdQtu8pkPEFmFcLiIxbiu65CWAyoGFjF\nUxbHSKxCJI8TnMcJzWOMpdgm4RGJLSNCpRtQiWTqCnpAJamqLWpTHogYwUarAOYjmlMy+1zjyFZrG\nis3hga3hwRcMZQxXqAJa2yRpzhtvWRebiFMxy5bkWC",
  "wUarAOYjmlMy+1zjyFZrG\nis3hga3hwRcMZQxXqAJa2yRpzhtvWRebiFMxy5bkWCArpgnsY6dPnfnpz/ULcpJz/UtDLym9MPSC0k\nNDylNDCW/CFx/1Dy68T1zw09p/TA0ANKc0NzSgeGDij1DfUpfWjoQ0o9Qz1K1w1dpzQzlJxI4Ylg\n6D6lE0MnlB4ZekTpc0OfU/rY0MeUvjD0BaWvDX1N6X1D71PKDGWUbhi6QSk3lLw6cP01Q9codQ0lv/",
  "Tpc0OfU/rY0MeUvjD0BaWvDX1N6X1D71PKDGWUbhi6QSk3lLw6cP01Q9codQ0lv/1\ngrRnapzQ2NKb0gaEPKB0bSn4Vw/PMUHK8gQejoZLSJ4Y+oVQYSn6/uf4zQ59RGhoaUvrU0KeUvjL0Fa\nWPDH1EaWAoeTcApxND9yg1b4GKlNIdQ3coPTP0zP5egC+G0bVNzG1TwTalkaERpZuGkl8KcJQw9JSc\nJ31V72rzt01kX/PVgltYnfH51STnvlp",
  "0bVNzG1TwTalkaERpZuGkl8KcJQw9JSc\nJ31V72rzt01kX/PVgltYnfH51STnvlpwC6t3p/nVZH/y1YJPSNc3DhYvUiClsNOPld7+C0sLRzc7vR\n+6dzd+Xn13lr9hvZ67vW960fWr3Wr617rcetfmvQ8tp/tf9u/9P+d+X3FW/l1Yqs1Gvt+pvWo3PSv\n4/wvxJWA=</latexit>@`i\n@f3\n,\n@`i\n@h3\n,\n@`i\n@f2\n,\n@`i\n@h2\n,",
  "o3PSv\n4/wvxJWA=</latexit>@`i\n@f3\n,\n@`i\n@h3\n,\n@`i\n@f2\n,\n@`i\n@h2\n,\n@`i\n@f1\n,\n@`i\n@h1\n,\nand\n@`i\n@f0\nf \ud835\udc65(, \ud835\udf19 = \ud835\udefd) + \ud835\udf14) \u22c5 a \ud835\udefd# + \ud835\udf14# \u22c5 a \ud835\udefd\" + \ud835\udf14\" \u22c5 a \ud835\udefd! + \ud835\udf14! \u22c5 \ud835\udc65(\n\u2113( = \ud835\udc53 \ud835\udc65(, \ud835\udf19 \u2212 \ud835\udc66( #",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n<latexit sha1_base64=\"rq4C8K+oC31Y0b9WL/\nbB5Has2CA=\">AXu3iclZhb9s2FIDt7tZlt3TDkIe9CAsKDENn2O267WVDmzS9JV2ci5O0cWpQMiW\nzoShFl8Sp4J+zX7PX7WH/ZoeSbFbnMGhrIDV7vk8kdXgRLTeWIs263f/a1z748KOP7n+6dJn3/x5\nVfLN74+SKM8fjAi2SUHLks5VIoPshEJvlRn",
  "WIs263f/a1z748KOP7n+6dJn3/x5\nVfLN74+SKM8fjAi2SUHLks5VIoPshEJvlRnHAWupIfuqfrmh+e8yQVkdrPLmN+ErJACV94LIPQ6Eb7\nj6GfMK8YxizJBJPOkEs5EjMT8EfFndnsljM8y9nYeZs9Gd15Z9cf3Z67S2+v9/Z71Nt7j/4u3GHoRtO\nCqfHsndvpzkbLq91Ot/w4tNCrC6ut+tMf3fh2PBxHXh5ylXmSpelxrxtnJ4Wu05N8tjT",
  "sndvpzkbLq91Ot/w4tNCrC6ut+tMf3fh2PBxHXh5ylXmSpelxrxtnJ4Wu05N8tjTMUx4z75QF/\nBiKioU8PSnKkZ45NyEydvwogT+VOWX0zSsKFqbpZeiCGbJskmKmgzZ2nGf+byeFUHGeceVDfm5dLI\n0dPGYuEe5m8hALzEgF9dbwJg/xkMLmWhopfeFEYQvK4drGDiTJ5YFQBYdE6ok2mzWdjdLhULzKWHu\nyv6hFZDwUrzmpFR0JVcIPJgVB",
  "QvK4drGDiTJ5YFQBYdE6ok2mzWdjdLhULzKWHu\nyv6hFZDwUrzmpFR0JVcIPJgVBe8EHQwEByA6nIBI8RTq1PlxfaeHKCwsCbioZsoQjN0ZqVplPICcN\nLQXRINCLPm0Ya0TC4YybCh7oDjOTUcDniUwCtBV+OJoDPZipmbz6zI+zZKwSHUMt5AwFfCyCbhlDyb1\nLjZULiVc6jWsP7G1y9RpnbgoLrua6Aiy9pOmkyU0L7D+Gk4ZQRZMwqBplRF",
  "Dyb1\nLjZULiVc6jWsP7G1y9RpnbgoLrua6Aiy9pOmkyU0L7D+Gk4ZQRZMwqBplRFkSdgGxyxkOW6PIbDh0\ndsatCYVWQidlPIrfZdqwjeG5OY1gvTW+jIOk/ZygjOgCrT38Lpjze1Nejhe3Mk3Ne+rAp84EBqt5C\nUuC6rbmjcBd1bEZNctcIZNmC0JdNE0dW8sKo9F8wZ1AC+6PBHKf0O7VZgyurw8BbcapJLfvxT5y6f\nnhRdvWz0PySbUFGax",
  "W8sKo9F8wZ1AC+6PBHKf0O7VZgyurw8BbcapJLfvxT5y6f\nnhRdvWz0PySbUFGax7aKdPg9KhrDgxfPL4jgwYskGjwIlIMXSdjf0dCxBE9sHSnHDgpCMSmyS7T8RaC\na15QR3NkoRH2FgK4XvplQaJB9vynrgJbhG4QlgnkoZv0qnv0ZJTmCSebH5rPECl1vS0mQj+smhuq1\nEJz3+BycRWU4eFwzq+43EUZdat8ulGuxixByZzqIZ2+HKYZLDHb",
  "S0mQj+smhuq1\nEJz3+BycRWU4eFwzq+43EUZdat8ulGuxixByZzqIZ2+HKYZLDHb6i+HvCparYCfbdbtQb9gdHLP42ej\nTweAbGoI1FdcGaz1iWJZWkP6lpM1zd7Vmy+/JFM7cDi2k1J6q17abct7hU94Gdblt5uEY9Y1JGorq\nH1COWpT2oy57HLdtdWFy7KUm98zxabYu7MNH09/cnPGP6mBTJsT72RXJYhbCYUTGzilHIAyRWISyGe\ndOC/2N",
  "KUm98zxabYu7MNH09/cnPGP6mBTJsT72RXJYhbCYUTGzilHIAyRWISyGe\ndOC/2NlT8Do2lVISz2U9HUdABLYy7xLVQhLFZLuGnWMaxuWdQtu8pkPEFmFcLiIxbiu65CWAyoGFjF\nUxbHSKxCJI8TnMcJzWOMpdgm4RGJLSNCpRtQiWTqCnpAJamqLWpTHogYwUarAOYjmlMy+1zjyFZrG\nis3hga3hwRcMZQxXqAJa2yRpzhtvWRebiFMxy5bkW",
  "YwUarAOYjmlMy+1zjyFZrG\nis3hga3hwRcMZQxXqAJa2yRpzhtvWRebiFMxy5bkWCArpgnsY6dPnfnpz/ULcpJz/UtDLym9MPSC0k\nNDylNDCW/CFx/1Dy68T1zw09p/TA0ANKc0NzSgeGDij1DfUpfWjoQ0o9Qz1K1w1dpzQzlJxI4Ylg\n6D6lE0MnlB4ZekTpc0OfU/rY0MeUvjD0BaWvDX1N6X1D71PKDGWUbhi6QSk3lLw6cP01Q9codQ0lv",
  "kTpc0OfU/rY0MeUvjD0BaWvDX1N6X1D71PKDGWUbhi6QSk3lLw6cP01Q9codQ0lv/1\ngrRnapzQ2NKb0gaEPKB0bSn4Vw/PMUHK8gQejoZLSJ4Y+oVQYSn6/uf4zQ59RGhoaUvrU0KeUvjL0Fa\nWPDH1EaWAoeTcApxND9yg1b4GKlNIdQ3coPTP0zP5egC+G0bVNzG1TwTalkaERpZuGkl8KcJQw9JSc\nJ31V72rzt01kX/PVgltYnfH51STnvl",
  "G0bVNzG1TwTalkaERpZuGkl8KcJQw9JSc\nJ31V72rzt01kX/PVgltYnfH51STnvlpwC6t3p/nVZH/y1YJPSNc3DhYvUiClsNOPld7+C0sLRzc7vR\n+6dzd+Xn13lr9hvZ67vW960fWr3Wr617rcetfmvQ8tp/tf9u/9P+d+X3FW/l1Yqs1Gvt+pvWo3PSv\n4/wvxJWA=</latexit>@`i\n@f3\n,\n@`i\n@h3\n,\n@`i\n@f2\n,\n@`i\n@h2\n,",
  "o3PSv\n4/wvxJWA=</latexit>@`i\n@f3\n,\n@`i\n@h3\n,\n@`i\n@f2\n,\n@`i\n@h2\n,\n@`i\n@f1\n,\n@`i\n@h1\n,\nand\n@`i\n@f0\nf \ud835\udc65(, \ud835\udf19 = \ud835\udefd) + \ud835\udf14) \u22c5 a \ud835\udefd# + \ud835\udf14# \u22c5 a \ud835\udefd\" + \ud835\udf14\" \u22c5 a \ud835\udefd! + \ud835\udf14! \u22c5 \ud835\udc65(\n\u2113( = \ud835\udc53 \ud835\udc65(, \ud835\udf19 \u2212 \ud835\udc66(\n#",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n<latexit sha1_base64=\"4RB5KDkz\noRu21HTpGhVUhQYZU=\">AWr3iclZjZcts2FECZdEvTzWmnfukLp57MpJ\n1EYyVN25fOJHaczU4tL7IdW4KUiCFGARpLrYUj6kX9PX9hP6N70gKSG8F\n36oZhwh9xiuQBIiF4iRZavrv57foH3708Sc3Pr352edfPnV0q2vD7K4\nSH3e92MZp0cey7gUivdzkUt+lKSc",
  "avrv57foH3708Sc3Pr352edfPnV0q2vD7K4\nSH3e92MZp0cey7gUivdzkUt+lKScRZ7kh97ZuaHFzNRKz282nCTyMWKhEI\nn+UQGi49GAQp8tBwtJcMOkOuJRDMTOBYFg+mM3c39z7d4Lhg3vTofhuLS\ny2lmtPi4tdJvCitN8esNb34Go9gvIq5yX7IsO+muJvlpqZvwJZ/dHBQZT5h\n/xkJ+AkXFIp6dltXoZu5tiIzcIE7hT+VuFX3/ipJFWT",
  "uJvlpqZvwJZ/dHBQZT5h\n/xkJ+AkXFIp6dltXoZu5tiIzcIE7hT+VuFX3/ipJFWTaNPDAjlo8zHTQxk\n6KPj1tBQqKXKu/LqhoJBuHrs6Ve5IpNzP5RQKzE8F9NX1xwzSlUNCbw4Uv\n/TjKGJqVA7WNnYgZx4PhSr5eVEldzZrOxuVw6F4lbH2Yn9Ri8h5JN5xUkml6\nEquEHg4K0veCTsYCA5AdDgBseIZ1Knz4wVuF1FYTBIwcC+eQOcCd3dGq",
  "5xUkml6\nEquEHg4K0veCTsYCA5AdDgBseIZ1Knz4wVuF1FYTBIwcC+eQOcCd3dGqlY5\nDyEnLe2YaFBIJ+0rHViwVRGLWUPFNe97WrA8xRmAboKXxzNwV7C1Gx+Xc4n\neRqVmY7hFlKmQl41AUP2Y3vYkMVUsKlfsv6HVu7TJ01iYuTqupjiBrP20\n7eUrzokZtp4ogCxZh2LaqCLIkbP0RixhkuSkPYcCRqyN2VSisCrIwe2nstd\ntOdASvzUkC",
  "kZtp4ogCxZh2LaqCLIkbP0RixhkuSkPYcCRqyN2VSisCrIwe2nstd\ntOdASvzUkC+6XtbZQk/RcMZUQHYPfpb8GUz9v6eryw3XlyLipfF/jEHcNktS\n9haVgPa94IjKqJzahZ5QqZNFsQSuPLtql7Y1F5ItoD1AG86YpUqOA97W5Vg\niWrw4O7MNS0kPzkXuchn5yWq3rb6H9INqGirEhsFenw/6hoBA8bvL4gicvl\nmjyIFBNXizh/o6mjqV4Ye",
  "chn5yWq3rb6H9INqGirEhsFenw/6hoBA8bvL4gicvl\nmjyIFBNXizh/o6mjqV4YetINXdQEIpJkU/R9hehal9TRXBn4wj1FQK6Xvhm\nQqFJDoK2rANahm94bFoWkI8G6dj9GWcFSknNz+0niFS6fq2mAr9sGrfUKU\nW2vcNLhdXQRkeDhf8is9lFGvzqcXF2rEUpTMiZ7SyZtBlsMWs+3+asrotU\nK+flm0x70C2an8H1+PtzE8xESizoS1QXnFG",
  "rEUpTMiZ7SyZtBlsMWs+3+asrotU\nK+flm0x70C2an8H1+PtzE8xESizoS1QXnFGtdkliW9qCuxXJ9v2fl5psfyd\nIOLa7dlKTepd2+Je0QN+vmXp7RbxiEUdiepqekg9Ylnag7rsedyjcLi2k\n1J6p3n0Wpb3IWJln+wP+Y508ekWI70sS+WgzqExZyKuVWMIx4isQ5hMSraF\nvwfK3sCHh5tqw5hsZeJtqYDWBpxiYdQh7BYb+G2cSwumVRt+w",
  "MIx4isQ5hMSraF\nvwfK3sCHh5tqw5hsZeJtqYDWBpxiYdQh7BYb+G2cSwumVRt+wqk8kYmXUI\ni89YhEdh7AYUjG0imcsSZBYh0gexziPY5rHBEuJTcIzklhmhCwp24JKx3Fb\n0gEsTVBrE0tj0AMZK9RgE8RyRldeZl15Cq1iRVdx39Zw/4qGc4Yq1AEsbZM\n95g62rZvMwymGY5YtyYlAVkIT2MNOjzrz058XlOQk5wVTQ6eUXhp6SemhoY\neU",
  "M\n95g62rZvMwymGY5YtyYlAVkIT2MNOjzrz058XlOQk5wVTQ6eUXhp6SemhoY\neUpoaSXwResGso+XiBReGXlB6YOgBpYWhBaV9Q/uUBoYGlD419CmlvqE+pe\nuGrlOaG0pOpPBEMHSf0rGhY0qPD2i9LWhryl9buhzSo8NPab0naHvKH1s6\nGNKmaGM0g1DNyjlhpJXB16wZugapZ6h5Lcf7DVDe5QmhiaUPjH0CaUjQ8mvY\nnieGUqON/BgNFR",
  "NyjlhpJXB16wZugapZ6h5Lcf7DVDe5QmhiaUPjH0CaUjQ8mvY\nnieGUqON/BgNFRS+sLQF5QKQ8nvNy94ZegrSiNDI0pfGvqS0reGvqX0maHP\nKA0NJe8G4HRi6B6l5i1QmVG6Y+gOpeGntvfC/DFNHq2hbltKtimNDY0pnT\nTUPJLAY4Shp6R82Sgmrva/G0Tua8FasEtrMn4/GqS80AtuIU1d6f51eT+FKg\nFH5OubxwsXqRASuFOP1xa6eK3s",
  "0Tua8FasEtrMn4/GqS80AtuIU1d6f51eT+FKg\nFH5OubxwsXqRASuFOP1xa6eK3sLRwcL/T/bnzcOenlUdrzRvaG853zvfOHa\nfr/OI8cp47Pafv+M6fzl/O384/y93lw+U3y3/U6vVrzTXfOK3PsvgPRinivg\n=</latexit>@`i\n@f3\n= 2(f3 \u2212 yi)\n\u2022 The first of these \nderivatives is trivial\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) =",
  "these \nderivatives is trivial\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc53* \u2212 \ud835\udc66& )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022 The second of these \nderivatives is computed \nvia the chain rule\n<latexit sha1_base64=\"sudt\nOYgkBeUti2xjR/l8LmPz9oM=\">AW8HiclZhb9s2FICV7tZml\n7Yblpe9CsKDENnJNu67WVAmzS9JV2uTtLGqUHJlMyGohSJSpwK\n/h97G/a6f7T9mh1Kslmdw2CYgdTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaH",
  "YgdTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwWs4FIo3tdCS\n36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEYmQaQgNb+lBlLOw\nGmQs14Jf8ClHIqpDYyH1Q/Tqf8r8qI6TLT/aq2pNrx1Z7m3XH9\n8WlhpC3e89rM9vP3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6dy",
  "9vP3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJ\nkyPC8xM0MWOSx39clIJlZWaq7DpKCqlr1Pf5NofiZyHWl5CgYW5\ngLH64ZhBhjSsyOJA8YswTRKmRtVgdX0H0hTwWKiKn5X16kynXW\ne9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0Hk",
  "XW\ne9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeST\njrVGLFjKpKPsgeL7d30DuM5hFWCo8MXRGuxlTE1n9TSf6DypChP\nDPeRMxbzuAqYcwrbexYqpYSqYcf6DVu7TJ2iUuzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlB",
  "iUuzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYF\nWRjbudp0O07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc\n9mfJOa9U+ATfwyL1a3C8riZ1qwTmFUbm1KzhUyabYglKcXd\nOMxqHyTHQnaAL40JW5UNE72r26BFvWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLF",
  "BFvWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0C9eKmE6zta\nOpbjW0i9dpBQSgmhb5Ex1/EqlunjuDBpgkaKwRMu/DNhEKLHEV\nd2QSMDN9w3VsoBNMmzmGMq0KHNOLn5oP0Ok1s1lMRfmZtW9oE\nojdK8bXM5rQRluDuf8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5C",
  "Duf8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5CfDTfwesTEo5EbcGDjrMtSxH\nf9DWfLu+O7Jq4/W3ZGvHDtdtStJuO0q37XCvGAE/23SMdpN4xK\nKORG21I6QesRz9QVvuPG6ZuFw3aYk7c7y6LQd7txE2z/aH3PNz\nGNSKkfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xFN",
  "SKkfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xFNoQlhsjnDXbGNY3XSom26VyWyMzCaExScswbN\nuQliMqRg7xVOWZUhsQiSPY5zHMc1jhqXMJeEVyRwrQraUa0Pl47\nQrmQCWJqi3iaMzGIFMFeqwDWK5oDuvcO48hXaxoru47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJEn",
  "ru47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJEn\nuSC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oL\nS0tKS0b2mf0sjSiNLHlj6mNLQ0pHTN0jVKtaXkiRTuCJbuUzq2d\nEzpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9B",
  "R+tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGlI0vJr2K4n1l\nKHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6R\nNLn1AaW0reDcDTiaV7lNq3QFVB6Y6lO5SeWXrmfi/A58sYuDbml\nm1gi9LU0pTSDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nxM",
  "DUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nxMhr5+MH+RAik9qN/LruC3sLRw8H\n1v5afe/Z0f7zxYbd/QXve+8r72vFWvJ+9B95Tb9vre6H3z4K3c\nGNhcSlf+n3pj6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nHow does a small \nchange in \u210e! change \u2113$?",
  "/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nHow does a small \nchange in \u210e! change \u2113$?\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022\nThe second derivative \nis computed via the \nchain rule\n<latexit sha1_base64=\"sudt\nOYgkBeUti2xjR/l8LmPz9oM=\">AW8HiclZhb9s2FICV7tZml\n7Yblpe9CsKDENnJNu67WVAmzS9JV2uTtLGqUHJlMyGohSJSpwK\n/h97G/a6f7T9mh1Kslmdw2CYgdTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwW",
  "dTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwWs4FIo3tdCS\n36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEYmQaQgNb+lBlLOw\nGmQs14Jf8ClHIqpDYyH1Q/Tqf8r8qI6TLT/aq2pNrx1Z7m3XH9\n8WlhpC3e89rM9vP3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6dyEy",
  "P3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJ\nkyPC8xM0MWOSx39clIJlZWaq7DpKCqlr1Pf5NofiZyHWl5CgYW5\ngLH64ZhBhjSsyOJA8YswTRKmRtVgdX0H0hTwWKiKn5X16kynXW\ne9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0Hkry",
  "e9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeST\njrVGLFjKpKPsgeL7d30DuM5hFWCo8MXRGuxlTE1n9TSf6DypChP\nDPeRMxbzuAqYcwrbexYqpYSqYcf6DVu7TJ2iUuzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYF",
  "uzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYF\nWRjbudp0O07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc\n9mfJOa9U+ATfwyL1a3C8riZ1qwTmFUbm1KzhUyabYglKcXd\nOMxqHyTHQnaAL40JW5UNE72r26BFvWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0",
  "vWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0C9eKmE6zta\nOpbjW0i9dpBQSgmhb5Ex1/EqlunjuDBpgkaKwRMu/DNhEKLHEV\nd2QSMDN9w3VsoBNMmzmGMq0KHNOLn5oP0Ok1s1lMRfmZtW9oE\nojdK8bXM5rQRluDuf8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5CfD",
  "f8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5CfDTfwesTEo5EbcGDjrMtSxH\nf9DWfLu+O7Jq4/W3ZGvHDtdtStJuO0q37XCvGAE/23SMdpN4xK\nKORG21I6QesRz9QVvuPG6ZuFw3aYk7c7y6LQd7txE2z/aH3PNz\nGNSKkfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xFNoQ",
  "kfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xFNoQlhsjnDXbGNY3XSom26VyWyMzCaExScswbN\nuQliMqRg7xVOWZUhsQiSPY5zHMc1jhqXMJeEVyRwrQraUa0Pl47\nQrmQCWJqi3iaMzGIFMFeqwDWK5oDuvcO48hXaxoru47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJEn\nu",
  "47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJEn\nuSC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oL\nS0tKS0b2mf0sjSiNLHlj6mNLQ0pHTN0jVKtaXkiRTuCJbuUzq2d\nEzpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGl",
  "tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGlI0vJr2K4n1l\nKHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6R\nNLn1AaW0reDcDTiaV7lNq3QFVB6Y6lO5SeWXrmfi/A58sYuDbml\nm1gi9LU0pTSDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nxMhr",
  "vJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nxMhr5+MH+RAik9qN/LruC3sLRw8H\n1v5afe/Z0f7zxYbd/QXve+8r72vFWvJ+9B95Tb9vre6H3z4K3c\nGNhcSlf+n3pj6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nHow does a small \nchange in \ud835\udc53$ change \u2113!?\nHow does a small \nchange in \u210e$ change f3?",
  "/latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nHow does a small \nchange in \ud835\udc53$ change \u2113!?\nHow does a small \nchange in \u210e$ change f3?\nHow does a small \nchange in \u210e$ change \u2113!?\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022\nThe second of these \nderivatives is computed \nvia the chain rule\n<latexit sha1_base64=\"sudt\nOYgkBeUti2xjR/l8LmPz9oM=\">AW8HiclZhb9s2FICV7tZml\n7Yblpe9CsKDENnJNu67WVAmzS9JV2uTtLGqUHJlMyGohSJSpwK\n/h97G/a6f7T9mh1Kslmdw2CYgdTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWa",
  "CYgdTs+T7eDklJVpBJUejl5b8Xr\n3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwWs4FIo3tdCS\n36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEYmQaQgNb+lBlLOw\nGmQs14Jf8ClHIqpDYyH1Q/Tqf8r8qI6TLT/aq2pNrx1Z7m3XH9\n8WlhpC3e89rM9vP3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6d",
  "M9vP3laDBKwzLhSoeSFcXxynKmTyrTaij5dHFQFj\nxj4SmL+TEUFUt4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJ\nkyPC8xM0MWOSx39clIJlZWaq7DpKCqlr1Pf5NofiZyHWl5CgYW5\ngLH64ZhBhjSsyOJA8YswTRKmRtVgdX0H0hTwWKiKn5X16kynXW\ne9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0H",
  "nXW\ne9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8Tk\nCpeQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeST\njrVGLFjKpKPsgeL7d30DuM5hFWCo8MXRGuxlTE1n9TSf6DypChP\nDPeRMxbzuAqYcwrbexYqpYSqYcf6DVu7TJ2iUuzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30Tcql",
  "2iUuzeqi5iSBrP+\n86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYF\nWRjbudp0O07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc\n9mfJOa9U+ATfwyL1a3C8riZ1qwTmFUbm1KzhUyabYglKcXd\nOMxqHyTHQnaAL40JW5UNE72r26BFvWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKL",
  "6BFvWhAf3YKp5Kfnxd737fHJSL\nZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0C9eKmE6zta\nOpbjW0i9dpBQSgmhb5Ex1/EqlunjuDBpgkaKwRMu/DNhEKLHEV\nd2QSMDN9w3VsoBNMmzmGMq0KHNOLn5oP0Ok1s1lMRfmZtW9oE\nojdK8bXM5rQRluDuf8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5",
  "uDuf8iuoBymjQ5DNISzViOUrmxCzp5PWg0HDEX\nKe/XvKm6LRifrbR9gfjgtUpw5CfDTfwesTEo5EbcGDjrMtSxH\nf9DWfLu+O7Jq4/W3ZGvHDtdtStJuO0q37XCvGAE/23SMdpN4xK\nKORG21I6QesRz9QVvuPG6ZuFw3aYk7c7y6LQd7txE2z/aH3PNz\nGNSKkfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xF",
  "NSKkfmsS+VgyaERU1F7RThMdIbEJYTMquBf/Hyp6Am0fXakJY\n3C5EVzMBLI24xFNoQlhsjnDXbGNY3XSom26VyWyMzCaExScswbN\nuQliMqRg7xVOWZUhsQiSPY5zHMc1jhqXMJeEVyRwrQraUa0Pl47\nQrmQCWJqi3iaMzGIFMFeqwDWK5oDuvcO48hXaxoru47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJE",
  "oru47+q4f0XHm\nqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnVmT39BVJEn\nuSC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oL\nS0tKS0b2mf0sjSiNLHlj6mNLQ0pHTN0jVKtaXkiRTuCJbuUzq2d\nEzpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9",
  "9R+tbSt5Q+tPQhpcxSRum6peuU\nckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGlI0vJr2K4n1l\nKHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6R\nNLn1AaW0reDcDTiaV7lNq3QFVB6Y6lO5SeWXrmfi/A58sYuDbml\nm1gi9LU0pTSDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nx",
  "SDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9\nqk5xHas4drL06zWqT61Ok5nxMhr5+MH+RAik9qN/LruC3sLRw8H\n1v5afe/Z0f7zxYbd/QXve+8r72vFWvJ+9B95Tb9vre6H3z4K3c\nGNhcSlf+n3pj6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nAlready computed!",
  "6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\nAlready computed!\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022 The remaining \nderivatives also \ncalculated by further \nuse of chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh",
  "kiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPW",
  "uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VA",
  "vJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUt",
  "AQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvY",
  "6bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwY",
  "MNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytU",
  "eV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmg",
  "h7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+g",
  "G9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaE",
  "m6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb",
  "qNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53']",
  "= \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022 The remaining \nderivatives also \ncalculated by further \nuse of chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh",
  "kiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPW",
  "uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VA",
  "vJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUt",
  "AQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvY",
  "6bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwY",
  "MNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytU",
  "eV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmg",
  "h7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+g",
  "G9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaE",
  "m6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb",
  "qNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\nAlready computed!",
  "@h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\nAlready computed!\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022\nThe remaining \nderivatives also \ncalculated by further \nuse of chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5r",
  "EkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluP",
  "2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4V",
  "FvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGU",
  "DAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSv",
  "06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04Vw",
  "pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/Guyt",
  "teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYm",
  "Vh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+",
  "BG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9Qmlka",
  "Vm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Md",
  "zqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53']",
  "= \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022\nThe remaining \nderivatives also \ncalculated by further \nuse of chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5r",
  "EkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluP",
  "2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4V",
  "FvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGU",
  "DAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSv",
  "06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04Vw",
  "pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/Guyt",
  "teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYm",
  "Vh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+",
  "BG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9Qmlka",
  "Vm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Md",
  "zqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53']",
  "= \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022\nThe remaining \nderivatives also \ncalculated by further \nuse of chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5r",
  "EkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluP",
  "2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4V",
  "FvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGU",
  "DAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSv",
  "06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04Vw",
  "pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/Guyt",
  "teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYm",
  "Vh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+",
  "BG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9Qmlka",
  "Vm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Md",
  "zqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"sudtOYgkBeUti2xj",
  "sha1_base64=\"sudtOYgkBeUti2xjR/l8LmPz9oM=\">A\nAW8HiclZhb9s2FICV7tZml7Yblpe9CsKDENnJNu67WVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/h97G/a6f7T9mh1Kslmdw2CYgdTs+T7e\nDklJVpBJUejl5b8Xr3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwWs4FIo3tdCS36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEY\nm",
  "KNIyD3k/TGWaHwWs4FIo3tdCS36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEY\nmQaQgNb+lBlLOwGmQs14Jf8ClHIqpDYyH1Q/Tqf8r8qI6TLT/aq2pNrx1Z7m3XH98WlhpC3e89rM9vP3laDBKwzLhSoeSFcXxynKmTyrTa\nij5dHFQFjxj4SmL+TEUFUt4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJkyPC8xM0MWOSx39clIJlZWaq7D",
  "4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJkyPC8xM0MWOSx39clIJlZWaq7DpKCqlr1Pf5NofiZyHWl5\nCgYW5gLH64ZhBhjSsyOJA8YswTRKmRtVgdX0H0hTwWKiKn5X16kynXWe9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8TkC\npeQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeSTjrVGLFjKpKPsgeL",
  "eQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeSTjrVGLFjKpKPsgeL7d30DuM5hFWCo8MXRGuxlTE1n9TSf6DypChPDPeRM\nxbzuAqYcwrbexYqpYSqYcf6DVu7TJ2iUuzeqi5iSBrP+86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYFWRjbudp0O\n07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc9mfJ",
  "30TcqlBYFWRjbudp0O\n07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc9mfJOa9U+ATfwyL1a3C8riZ1qwTmFUbm1KzhUyabYglKcXdOMxqHyTHQnaAL40\nJW5UNE72r26BFvWhAf3YKp5Kfnxd737fHJSLZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0C9eKmE6ztaOpbjW0i9dpBQSgmhb5\nEx1/EqlunjuDBpgkaKwRMu/DNhEK",
  "8VKLFg0C9eKmE6ztaOpbjW0i9dpBQSgmhb5\nEx1/EqlunjuDBpgkaKwRMu/DNhEKLHEVd2QSMDN9w3VsoBNMmzmGMq0KHNOLn5oP0Ok1s1lMRfmZtW9oEojdK8bXM5rQRluDuf8iuoBym\njQ5DNISzViOUrmxCzp5PWg0HDEXKe/XvKm6LRifrbR9gfjgtUpw5CfDTfwesTEo5EbcGDjrMtSxHf9DWfLu+O7Jq4/W3ZGvHDtdtStJu\nO0q37XCvG",
  "w5CfDTfwesTEo5EbcGDjrMtSxHf9DWfLu+O7Jq4/W3ZGvHDtdtStJu\nO0q37XCvGAE/23SMdpN4xKORG21I6QesRz9QVvuPG6ZuFw3aYk7c7y6LQd7txE2z/aH3PNzGNSKkfmsS+VgyaERU1F7RThMdIbEJYTM\nquBf/Hyp6Am0fXakJY3C5EVzMBLI24xFNoQlhsjnDXbGNY3XSom26VyWyMzCaExScswbNuQliMqRg7xVOWZUhsQiSPY5zHMc1",
  "oQlhsjnDXbGNY3XSom26VyWyMzCaExScswbNuQliMqRg7xVOWZUhsQiSPY5zHMc1jhqXMJeEVy\nRwrQraUa0Pl47QrmQCWJqi3iaMzGIFMFeqwDWK5oDuvcO48hXaxoru47+q4f0XHmqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnV\nmT39BVJEnuSC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oLS0tKS0b2mf0sjSiNL",
  "SC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oLS0tKS0b2mf0sjSiNLHlj6mNLQ0pHTN0jVKtaXkiRTuCJbuU\nzq2dEzpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum6peuUckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGlI0vJr2K4n1lK\nHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6RNLn1",
  "vJr2K4n1lK\nHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6RNLn1AaW0reDcDTiaV7lNq3QFVB6Y6lO5SeWXrmfi/A58sYuDbmlm1gi9\nLU0pTSDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9qk5xHas4drL06zWqT61Ok5nxMhr5+MH+RAik9qN/LruC3sLRw8H1v5afe/Z0f7\nzxYbd/QXve+8r72vFWvJ+9B95Tb9vre6",
  "+RAik9qN/LruC3sLRw8H1v5afe/Z0f7\nzxYbd/QXve+8r72vFWvJ+9B95Tb9vre6H3z4K3cGNhcSlf+n3pj6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\n<latexit sha1_base64=\"4RB5KDkzoRu21HTpGhVUhQYZU=\">A\nAWr3iclZjZcts2FECZdEvTzWmnfukLp57MpJ1EYyVN25fOJHaczU4tL7IdW4KUiCF",
  "Wr3iclZjZcts2FECZdEvTzWmnfukLp57MpJ1EYyVN25fOJHaczU4tL7IdW4KUiCFGARpLrYUj6kX9PX9hP6N70gKSG8F36oZhwh9xi\nuQBIiF4iRZavrv57foH3708Sc3Pr352edfPnV0q2vD7K4SH3e92MZp0cey7gUivdzkUt+lKScRZ7kh97ZuaHFzNRKz282nCTyMWKh\nEIn+UQGi49GAQp8tBwtJcMOkOuJRDMTOBYFg+mM3c39z7d4Lhg3",
  "z282nCTyMWKh\nEIn+UQGi49GAQp8tBwtJcMOkOuJRDMTOBYFg+mM3c39z7d4Lhg3vTofhuLSy2lmtPi4tdJvCitN8esNb34Go9gvIq5yX7IsO+muJvlpq\nZvwJZ/dHBQZT5h/xkJ+AkXFIp6dltXoZu5tiIzcIE7hT+VuFX3/ipJFWTaNPDAjlo8zHTQxk6KPj1tBQqKXKu/LqhoJBuHrs6Ve5IpNz\nP5RQKzE8F9NX1xwzSlUNCbw4Uv/TjKGJq",
  "Pj1tBQqKXKu/LqhoJBuHrs6Ve5IpNz\nP5RQKzE8F9NX1xwzSlUNCbw4Uv/TjKGJqVA7WNnYgZx4PhSr5eVEldzZrOxuVw6F4lbH2Yn9Ri8h5JN5xUkml6EquEHg4K0veCTsYCA5AdD\ngBseIZ1Knz4wVuF1FYTBIwcC+eQOcCd3dGqlY5DyEnLe2YaFBIJ+0rHViwVRGLWUPFNe97WrA8xRmAboKXxzNwV7C1Gx+Xc4neRqVmY7h\nFlKmQl41AUP",
  "rHViwVRGLWUPFNe97WrA8xRmAboKXxzNwV7C1Gx+Xc4neRqVmY7h\nFlKmQl41AUP2Y3vYkMVUsKlfsv6HVu7TJ01iYuTqupjiBrP207eUrzokZtp4ogCxZh2LaqCLIkbP0RixhkuSkPYcCRqyN2VSisCrIwe2\nnstdtOdASvzUkC+6XtbZQk/RcMZUQHYPfpb8GUz9v6eryw3XlyLipfF/jEHcNktS9haVgPa94IjKqJzahZ5QqZNFsQSuPLtql7",
  "b8GUz9v6eryw3XlyLipfF/jEHcNktS9haVgPa94IjKqJzahZ5QqZNFsQSuPLtql7Y1F5ItoD1\nAG86YpUqOA97W5VgiWrw4O7MNS0kPzkXuchn5yWq3rb6H9INqGirEhsFenw/6hoBA8bvL4gicvlmjyIFBNXizh/o6mjqV4YetINXdQEIp\nJkU/R9hehal9TRXBn4wj1FQK6XvhmQqFJDoK2rANahm94bFoWkI8G6dj9GWcFSknNz+0niFS6fq2",
  "TRXBn4wj1FQK6XvhmQqFJDoK2rANahm94bFoWkI8G6dj9GWcFSknNz+0niFS6fq2mAr9sGrfUKUW2vcNLhdXQRkeDhf8is\ns9lFGvzqcXF2rEUpTMiZ7SyZtBlsMWs+3+asrotUK+flm0x70C2an8H1+PtzE8xESizoS1QXnFGtdkliW9qCuxXJ9v2fl5psfydIOLa7d\nlKTepd2+Je0QN+vmXp7RbxiEUdiepqekg9Ylnag7rsedyjcLi2k1J6p",
  "ydIOLa7d\nlKTepd2+Je0QN+vmXp7RbxiEUdiepqekg9Ylnag7rsedyjcLi2k1J6p3n0Wpb3IWJln+wP+Y508ekWI70sS+WgzqExZyKuVWMIx4isQ\n5hMSraFvwfK3sCHh5tqw5hsZeJtqYDWBpxiYdQh7BYb+G2cSwumVRt+wqk8kYmXUIi89YhEdh7AYUjG0imcsSZBYh0gexziPY5rHBEuJT\ncIzklhmhCwp24JKx3Fb0gEsTVBrE0tj0AMZK9",
  "0imcsSZBYh0gexziPY5rHBEuJT\ncIzklhmhCwp24JKx3Fb0gEsTVBrE0tj0AMZK9RgE8RyRldeZl15Cq1iRVdx39Zw/4qGc4Yq1AEsbZM95g62rZvMwymGY5YtyYlAVkIT2MN\nOjzrz058XlOQk5wVTQ6eUXhp6SemhoYeUpoaSXwResGso+XiBReGXlB6YOgBpYWhBaV9Q/uUBoYGlD419CmlvqE+peuGrlOaG0pOpPBEM\nHSf0rGhY0qPD2i9L",
  "gBpYWhBaV9Q/uUBoYGlD419CmlvqE+peuGrlOaG0pOpPBEM\nHSf0rGhY0qPD2i9LWhryl9buhzSo8NPab0naHvKH1s6GNKmaGM0g1DNyjlhpJXB16wZugapZ6h5Lcf7DVDe5QmhiaUPjH0CaUjQ8mvYnie\nGUqON/BgNFRS+sLQF5QKQ8nvNy94ZegrSiNDI0pfGvqS0reGvqX0maHPKA0NJe8G4HRi6B6l5i1QmVG6Y+gOpeGntvfC/DFNHq2hb",
  "0pfGvqS0reGvqX0maHPKA0NJe8G4HRi6B6l5i1QmVG6Y+gOpeGntvfC/DFNHq2hbltKt\nimNDY0pnTUPJLAY4Shp6R82Sgmrva/G0Tua8FasEtrMn4/GqS80AtuIU1d6f51eT+FKgFH5OubxwsXqRASuFOP1xa6eK3sLRwcL/T/bnzc\nOenlUdrzRvaG853zvfOHafr/OI8cp47Pafv+M6fzl/O384/y93lw+U3y3/U6vVrzTXfOK3PsvgPRiniv",
  "vfOHafr/OI8cp47Pafv+M6fzl/O384/y93lw+U3y3/U6vVrzTXfOK3PsvgPRinivg=</latexit>@`i\n@f3\n= 2(f3 \u2212 yi)",
  "Backward pass\n1. Compute the \nderivatives of the loss \nwith respect to these \nintermediate quantities, \nbut in reverse order.\n\u2022 The remaining \nderivatives also \ncalculated by further use \nof chain rule\n<latexit sha1_base64=\"jJl+XpmdZFW1B/yC01sXVjP7mZg=\">\nAb3ic3ZjrbtxEFIC3pYESbi0IREJWURFLSpRtlDgD1KbNL0lJfdLG6ersXfsnWY8dnxJNrVW/IVH5DHgCThje3fqc2alFiEkiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh",
  "kiFR2\nON83tzPj8cVLpMjyxcXfL1x869LM2+9cfnf2vfc/+PCjK1c/3sviIvX5rh/LOD3wWMalUHw3F7nkB0nKWeRJvu8dL2u+f8rTMRqJz9P\n+FHEQiUC4bMcQr2rl/50g5T5pZuwNBdMOi6XsidGJhD0ylujkfPVTw4yB73y2xERXcmD/PoscgPk1nVfo2uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPW",
  "uwHDcV4SC/MeuqWBWRx1P\nHdXEHpO5g2rDrYWKxHvbrzfCfm1ozszeaGNTsTluPWyMi1hN7zST8y9N3/s78B9PmX8Xi9MW1pKq/1GSoOritE3SHRHRmqQp+fzvZ7J\n3ZX5xYbH6c2ih2xTmO83fRu/qp323H/tFxFXuS5Zlh93FJD8qdeu+5KNZt8h4wvxjFvJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VA",
  "vJDKCoW8eyorE7nkXMNIn0niFP4p3Knir5ao2R\nRlp1HpgRywcZjpoY4dFHvx4VAqVFDlXft1RUEgnjx191Dt9kXI/l+dQYH4qYKyOP2CQqRxuCHCa8jM/jiKm+qW7tLIJ6fJ4KFTJT4r\nq5jAatZ2VyuFQnGYsPdqZtCJyHomXnDRSKbqRKQIPR2XJF8IFDAQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUt",
  "AQHIBY4AbHiGbSp8+MFThdRuBlKwMC9eAiDC5ytEWla5TyEnLS0Z0\nSDQiL5sGUtEwuWMmop26A4zjVHA56nsAowVPjhaA2E6ZG43o5H+ZpVGY6hntImQp51QVM2YftvYUNVUgJVf2W9TO2tpg6bhIXJ9VQUx\n1B1k7advKU5kX1204VQRZswrBtVRFkSXh06bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvY",
  "6bOIQZabcg8mHDk6YleFwqogG3Mjb1234mO4L05TOB6aXsrJUn/KUMZ0QG4+vSvYMrnbX\n05ntjODmnla8LfOgMYLHaVga1tMadwKzamIjala5QibNFoTS+Kxt6tFYVJ6I9gR1AF90RSpU8Ip2syrBltVh9yZMNS0kP/xm4TYfH\npWL+rLR/yHZhIayIrE1pMNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwY",
  "MNv0FAfHpbx/oIXrxYosWDQLV4sYTzHS0dS/HG1pFq7aAgFJMiP0eXvwhVu04VwYONIzRWCOh24ZcJhRY5C\nNqyDmgZfuGx37KBfDRJv56jL+OsSDk5/NB+hkil62MxFfpm1T5QpRba5waXk1pQhpvDKZ9S3UMZ9ep8enGh+ixFyRzqJR0+d7McLjHb1\nV8teV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytU",
  "eV20WiE/W36g3HB6hS+z096q3g9QmJR6K24D3L2pYklqU/aGuyXV8dWbn6/GuytUOLazclabcZpd2uFNGwE/WLKNdIx6xqCNR\nW80IqUcsS3/Qlj2Pa7ZWFy7KUm74zxabYs7MdH2D3YGPGf6MSmWf3YF0u3DmExp2JuFeOIh0isQ1iMirYF/4+VbQE3j7ZVh7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmg",
  "h7C4kYm2\npgNY6nOJp1CHsFhfwm2ziWF1zaKu2VUmkwEy6xAWH7AIz7oOYTGkYmgVj1mSILEOkTwOcB4HNI8JlhKbhFcksawI2VK2DZUO4rakA1ga\not6Gls5gBDJWqMmiOWM7rzMuvMU2sWK7uJdW8e7UzrOGWpQB7C0Tq4x123XmQeTjE8ZtmSnAhkJTSBG9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+g",
  "G9jZoM746c8LSvIk5wXnhp5T\nemboGaX7hu5TmhpK3gi8YMtQ8nbiBaeGnlK6Z+gepYWhBaW7hu5SGhgaUHrf0PuU+ob6lC4bukxpbih5IoU7gqE7lA4MHVB6YOgBpU8\nNfUrpQ0MfUvrM0GeUvjT0JaV3Db1LKTOUbpi6Aql3FDy6cALlgxdotQzlLz7wbVm6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaE",
  "m6AaliaEJpfcMvUdp31DyVgz3M0PJ4w3cGA2VlD4\ny9BGlwlDy/uYFTwx9QmlkaETpY0MfU/rC0BeUPjD0AaWhoeTbADydGLpNqfkKVGaUbhq6SemJoSf27wJ8soyebWOumwbWKY0NjSldNZS\n8KcCjhKH5HkyUM2pNv7aRM61QE24hTUZH9cmOQ/UhFtYczqNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb",
  "qNa5PzKVATPiBDX9mbfEiBlMJ37sy38VfYWlh79ZC9/uF25vfzd9Zar\n7QXu7Mdb7sXO90Oz907nQedjY6ux1/5unMLzO/zvz2+R9zn819MefU6sULTZ1POq2/uRt/AXQSuV4=</latexit> @`i\n@f2\n= @h3\n@f2\n\u2713 @f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h2\n= @f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713",
  "@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f1\n= @h2\n@f1\n\u2713 @f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@h1\n= @f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n\u2713 @f1\n@h1\n@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"sudtOYgkBeUti2xj",
  "sha1_base64=\"sudtOYgkBeUti2xjR/l8LmPz9oM=\">A\nAW8HiclZhb9s2FICV7tZml7Yblpe9CsKDENnJNu67WVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/h97G/a6f7T9mh1Kslmdw2CYgdTs+T7e\nDklJVpBJUejl5b8Xr3/gcfnT9xuLHn3z62c1btz8/KNIyD3k/TGWaHwWs4FIo3tdCS36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEY\nm",
  "KNIyD3k/TGWaHwWs4FIo3tdCS36U5ZwlgeSHwema4YfnPC9Eqvb1ZcZPEhYrEY\nmQaQgNb+lBlLOwGmQs14Jf8ClHIqpDYyH1Q/Tqf8r8qI6TLT/aq2pNrx1Z7m3XH98WlhpC3e89rM9vP3laDBKwzLhSoeSFcXxynKmTyrTa\nij5dHFQFjxj4SmL+TEUFUt4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJkyPC8xM0MWOSx39clIJlZWaq7D",
  "4cVLV6Zn6dyEy8qM0hz+l/Tr6bo2KJUVxmQRgJkyPC8xM0MWOSx39clIJlZWaq7DpKCqlr1Pf5NofiZyHWl5\nCgYW5gLH64ZhBhjSsyOJA8YswTRKmRtVgdX0H0hTwWKiKn5X16kynXWe9djgUrzJWn+3PWxGaJ+ItJ43UimnkCoH06rivbiHgeARI8TkC\npeQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeSTjrVGLFjKpKPsgeL",
  "eQJsmP0HkryAKu1ECBh6kExhc5O9OSdNK8xhy0tFeEQ0KmeSTjrVGLFjKpKPsgeL7d30DuM5hFWCo8MXRGuxlTE1n9TSf6DypChPDPeRM\nxbzuAqYcwrbexYqpYSqYcf6DVu7TJ2iUuzeqi5iSBrP+86Oqd5UaOuU0eQBZsw7lp1BFkSrh0jljDIclsewoQT30TcqlBYFWRjbudp0O\n07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc9mfJ",
  "30TcqlBYFWRjbudp0O\n07MxG8NycZnJeut16R9J8zlBETgNnvgVTIe/qa+nc9mfJOa9U+ATfwyL1a3C8riZ1qwTmFUbm1KzhUyabYglKcXdOMxqHyTHQnaAL40\nJW5UNE72r26BFvWhAf3YKp5Kfnxd737fHJSLZtjY/4h2YSGijJzNWTC/6OhEdyt8P6CF68VKLFg0C9eKmE6ztaOpbjW0i9dpBQSgmhb5\nEx1/EqlunjuDBpgkaKwRMu/DNhEK",
  "8VKLFg0C9eKmE6ztaOpbjW0i9dpBQSgmhb5\nEx1/EqlunjuDBpgkaKwRMu/DNhEKLHEVd2QSMDN9w3VsoBNMmzmGMq0KHNOLn5oP0Ok1s1lMRfmZtW9oEojdK8bXM5rQRluDuf8iuoBym\njQ5DNISzViOUrmxCzp5PWg0HDEXKe/XvKm6LRifrbR9gfjgtUpw5CfDTfwesTEo5EbcGDjrMtSxHf9DWfLu+O7Jq4/W3ZGvHDtdtStJu\nO0q37XCvG",
  "w5CfDTfwesTEo5EbcGDjrMtSxHf9DWfLu+O7Jq4/W3ZGvHDtdtStJu\nO0q37XCvGAE/23SMdpN4xKORG21I6QesRz9QVvuPG6ZuFw3aYk7c7y6LQd7txE2z/aH3PNzGNSKkfmsS+VgyaERU1F7RThMdIbEJYTM\nquBf/Hyp6Am0fXakJY3C5EVzMBLI24xFNoQlhsjnDXbGNY3XSom26VyWyMzCaExScswbNuQliMqRg7xVOWZUhsQiSPY5zHMc1",
  "oQlhsjnDXbGNY3XSom26VyWyMzCaExScswbNuQliMqRg7xVOWZUhsQiSPY5zHMc1jhqXMJeEVy\nRwrQraUa0Pl47QrmQCWJqi3iaMzGIFMFeqwDWK5oDuvcO48hXaxoru47+q4f0XHmqEGTQBLW+SM+YMt5yELcIrhMcuV5EwgK6MJ3MbONnV\nmT39BVJEnuSC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oLS0tKS0b2mf0sjSiNL",
  "SC6tPS0gtLyg9tPSQ0txS8osgiHYtJb9Ogujc0nNKDyw9oLS0tKS0b2mf0sjSiNLHlj6mNLQ0pHTN0jVKtaXkiRTuCJbuU\nzq2dEzpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum6peuUckvJq4MgWrV0ldLAUvLbD86apduUZpZmlD6y9BGlI0vJr2K4n1lK\nHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6RNLn1",
  "vJr2K4n1lK\nHm/gxmipPSZpc8oFZaS329B9MLSF5QmliaUPrf0OaVvLH1D6RNLn1AaW0reDcDTiaV7lNq3QFVB6Y6lO5SeWXrmfi/A58sYuDbmlm1gi9\nLU0pTSDUvJLwV4lLD0lDxPRq9qs3eNpHrWqTm3MHajM9qk5xHas4drL06zWqT61Ok5nxMhr5+MH+RAik9qN/LruC3sLRw8H1v5afe/Z0f7\nzxYbd/QXve+8r72vFWvJ+9B95Tb9vre6",
  "+RAik9qN/LruC3sLRw8H1v5afe/Z0f7\nzxYbd/QXve+8r72vFWvJ+9B95Tb9vre6H3z4K3cGNhcSlf+n3pj6U/G/XaQlvnC6/zWfrX9Rz/Qs=</latexit>@`i\n@h3\n= @f3\n@h3\n@`i\n@f3\n<latexit sha1_base64=\"4RB5KDkzoRu21HTpGhVUhQYZU=\">A\nAWr3iclZjZcts2FECZdEvTzWmnfukLp57MpJ1EYyVN25fOJHaczU4tL7IdW4KUiCF",
  "Wr3iclZjZcts2FECZdEvTzWmnfukLp57MpJ1EYyVN25fOJHaczU4tL7IdW4KUiCFGARpLrYUj6kX9PX9hP6N70gKSG8F36oZhwh9xi\nuQBIiF4iRZavrv57foH3708Sc3Pr352edfPnV0q2vD7K4SH3e92MZp0cey7gUivdzkUt+lKScRZ7kh97ZuaHFzNRKz282nCTyMWKh\nEIn+UQGi49GAQp8tBwtJcMOkOuJRDMTOBYFg+mM3c39z7d4Lhg3",
  "z282nCTyMWKh\nEIn+UQGi49GAQp8tBwtJcMOkOuJRDMTOBYFg+mM3c39z7d4Lhg3vTofhuLSy2lmtPi4tdJvCitN8esNb34Go9gvIq5yX7IsO+muJvlpq\nZvwJZ/dHBQZT5h/xkJ+AkXFIp6dltXoZu5tiIzcIE7hT+VuFX3/ipJFWTaNPDAjlo8zHTQxk6KPj1tBQqKXKu/LqhoJBuHrs6Ve5IpNz\nP5RQKzE8F9NX1xwzSlUNCbw4Uv/TjKGJq",
  "Pj1tBQqKXKu/LqhoJBuHrs6Ve5IpNz\nP5RQKzE8F9NX1xwzSlUNCbw4Uv/TjKGJqVA7WNnYgZx4PhSr5eVEldzZrOxuVw6F4lbH2Yn9Ri8h5JN5xUkml6EquEHg4K0veCTsYCA5AdD\ngBseIZ1Knz4wVuF1FYTBIwcC+eQOcCd3dGqlY5DyEnLe2YaFBIJ+0rHViwVRGLWUPFNe97WrA8xRmAboKXxzNwV7C1Gx+Xc4neRqVmY7h\nFlKmQl41AUP",
  "rHViwVRGLWUPFNe97WrA8xRmAboKXxzNwV7C1Gx+Xc4neRqVmY7h\nFlKmQl41AUP2Y3vYkMVUsKlfsv6HVu7TJ01iYuTqupjiBrP207eUrzokZtp4ogCxZh2LaqCLIkbP0RixhkuSkPYcCRqyN2VSisCrIwe2\nnstdtOdASvzUkC+6XtbZQk/RcMZUQHYPfpb8GUz9v6eryw3XlyLipfF/jEHcNktS9haVgPa94IjKqJzahZ5QqZNFsQSuPLtql7",
  "b8GUz9v6eryw3XlyLipfF/jEHcNktS9haVgPa94IjKqJzahZ5QqZNFsQSuPLtql7Y1F5ItoD1\nAG86YpUqOA97W5VgiWrw4O7MNS0kPzkXuchn5yWq3rb6H9INqGirEhsFenw/6hoBA8bvL4gicvlmjyIFBNXizh/o6mjqV4YetINXdQEIp\nJkU/R9hehal9TRXBn4wj1FQK6XvhmQqFJDoK2rANahm94bFoWkI8G6dj9GWcFSknNz+0niFS6fq2",
  "TRXBn4wj1FQK6XvhmQqFJDoK2rANahm94bFoWkI8G6dj9GWcFSknNz+0niFS6fq2mAr9sGrfUKUW2vcNLhdXQRkeDhf8is\ns9lFGvzqcXF2rEUpTMiZ7SyZtBlsMWs+3+asrotUK+flm0x70C2an8H1+PtzE8xESizoS1QXnFGtdkliW9qCuxXJ9v2fl5psfydIOLa7d\nlKTepd2+Je0QN+vmXp7RbxiEUdiepqekg9Ylnag7rsedyjcLi2k1J6p",
  "ydIOLa7d\nlKTepd2+Je0QN+vmXp7RbxiEUdiepqekg9Ylnag7rsedyjcLi2k1J6p3n0Wpb3IWJln+wP+Y508ekWI70sS+WgzqExZyKuVWMIx4isQ\n5hMSraFvwfK3sCHh5tqw5hsZeJtqYDWBpxiYdQh7BYb+G2cSwumVRt+wqk8kYmXUIi89YhEdh7AYUjG0imcsSZBYh0gexziPY5rHBEuJT\ncIzklhmhCwp24JKx3Fb0gEsTVBrE0tj0AMZK9",
  "0imcsSZBYh0gexziPY5rHBEuJT\ncIzklhmhCwp24JKx3Fb0gEsTVBrE0tj0AMZK9RgE8RyRldeZl15Cq1iRVdx39Zw/4qGc4Yq1AEsbZM95g62rZvMwymGY5YtyYlAVkIT2MN\nOjzrz058XlOQk5wVTQ6eUXhp6SemhoYeUpoaSXwResGso+XiBReGXlB6YOgBpYWhBaV9Q/uUBoYGlD419CmlvqE+peuGrlOaG0pOpPBEM\nHSf0rGhY0qPD2i9L",
  "gBpYWhBaV9Q/uUBoYGlD419CmlvqE+peuGrlOaG0pOpPBEM\nHSf0rGhY0qPD2i9LWhryl9buhzSo8NPab0naHvKH1s6GNKmaGM0g1DNyjlhpJXB16wZugapZ6h5Lcf7DVDe5QmhiaUPjH0CaUjQ8mvYnie\nGUqON/BgNFRS+sLQF5QKQ8nvNy94ZegrSiNDI0pfGvqS0reGvqX0maHPKA0NJe8G4HRi6B6l5i1QmVG6Y+gOpeGntvfC/DFNHq2hb",
  "0pfGvqS0reGvqX0maHPKA0NJe8G4HRi6B6l5i1QmVG6Y+gOpeGntvfC/DFNHq2hbltKt\nimNDY0pnTUPJLAY4Shp6R82Sgmrva/G0Tua8FasEtrMn4/GqS80AtuIU1d6f51eT+FKgFH5OubxwsXqRASuFOP1xa6eK3sLRwcL/T/bnzc\nOenlUdrzRvaG853zvfOHafr/OI8cp47Pafv+M6fzl/O384/y93lw+U3y3/U6vVrzTXfOK3PsvgPRiniv",
  "vfOHafr/OI8cp47Pafv+M6fzl/O384/y93lw+U3y3/U6vVrzTXfOK3PsvgPRinivg=</latexit>@`i\n@f3\n= 2(f3 \u2212 yi)",
  "We extend this to get to the \nparameters \ud835\udf14\u2019s and \ud835\udefd\u2019s",
  "Backward pass\n2. Find how the loss \nchanges as a function of \nthe parameters b and w.\n<latexit sha1_base64=\"8HDOzmvyHV\nVJjcXRYBy6FB0LVX0=\">AXhniclZhb9s2FIDt7tZmt3bDkIe9Cs6DENrJMO6\n7mVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/ht73f7W/s0OJdmMzmHQLkBn7nwfLzok\nJUp+KkWul5b+7V76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gn",
  "76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gnq4YfnPEsF4na0xcpP45ZpEQoAqYhNLzVvTEIMxaUg5RlWjD\npDbiUQzG9FPC5ZsPyZDr1fvzDQ3ZYAZf8vmbrmgOVqCL2eYNBgvHUkS8+jDh2L\ntDxvL8Obtpd5S9efRwnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHP",
  "wnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHPj8tq0qbeHYiMvDJ4J/SXhW9XKNkcZ5fxD6YMdPjHDMTdL\nGjQoe/H5dCpYXmKqg7Cgvp6cQzK8AbiYwHWl5AgQWZgLF6wZhBhjSsk4WB4udBEs\ndMjcrByto2pMnkVAlPy2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4",
  "Py2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4Dm2a/Piht4wo7BEJGLifTGBwobczJU0rzSPISUt7TQop\nJPWtYqsWAq45ayC4rn3fEM4DqDWYChwg9Hc7CbMjWd1dN8orO4zE0M95AxFfGqC\n7jkAJb1DjZUISVUDVrWn9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC4",
  "n9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC49kzErQqFVUEW5laW+O2+UxPBa3OSwn5pe2slSf8ZQxkx\nAdh95lcwFfC2vprMbW+WnLPKNwU+8cYwWe0qLIvqy5p1AlfVxKbUrHKFTJotCGXJ\neds0o3GoPBXtCzQBvOmKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0Aieo",
  "mKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0AieoXh9QRPXiLR5EGgmrxEwv0dTR3L8MI2kWruoCAUk0JfoO0\nvItWuU0XwYJMYjRUCpl34ZUKhSQ7DtmwCRoZfOA04FlCALjKorzGQSV5knNz80Hq\nGSKWb2ImzMOqfUOVRmjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vek",
  "mjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vekPxgWzUwQBPx2u4/mIiEUdidqC45ezLUksR3/Q1ny5Xh\n5Zuf7mZ7K0I4frNiVptxml23a4V4yAn24RrtBPGJR6K2mhFSj1iO/qAtdx43XF\nfhcN2mJO3O8ui0He7cRMs/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtq",
  "/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtqaCWBpxCW+hDqExXoLt80mhtUNh7rhVplMx8isQ1h8y\nmJ81XUIixEVI6d4wtIUiXWI5HGM8zimeUyxlLokPCOpY0bIknItqGyctCUTwNIE9\nTZxdAYjkIlCHTZBLOd05eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Z",
  "eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Zqc/PyzJSc4PLy9oPTc0nNKDyw9oDSzlLwR+OGOpeTt\nxA/PLD2jdN/SfUoLSwtK+5b2KQ0tDSl9YukTSgNLA0pXLV2lVFtKTqTwRLB0j9Kx\npWNKDy09pPSVpa8ofWbpM0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaU",
  "M0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaUvrY0seUjiwlb8XwPLOUHG/gwWipPS5pc8pFZaS9zc/\nfGnpS0pjS2NKX1j6gtK3lr6l9KmlTymNLCXfBuB0YukupfYrUJlTum3pNqWnlp6\n6vwvw+T6roW5aRvYpDSxNKF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I1",
  "KF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I152My9LX9+YcUSOl+9V12GX+FpYX9X3rLv/Xub/96+F\nK84X2euf7zg+dnzrLnQedh51na1OvxN0+5f3b+7/yxeX+wt3l98UKvXuk2dbz\nutv8WH/wEsBTb4</latexit> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022 Another application \nof the chain rule\nHow does a small \nchange in",
  "t> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022 Another application \nof the chain rule\nHow does a small \nchange in \ud835\udc53& change \ud835\udc59!?\nHow does a small \nchange in wk change \ud835\udc53&?\nHow does a small \nchange in wk change \ud835\udc59!?\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n2. Find how the loss \nchanges as a function of \nthe parameters b and w.\n<latexit sha1_base64=\"8HDOzmvyHV\nVJjcXRYBy6FB0LVX0=\">AXhniclZhb9s2FIDt7tZmt3bDkIe9Cs6DENrJMO6\n7mVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/ht73f7W/s0OJdmMzmHQLkBn7nwfLzok\nJUp+KkWul5b+7V76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gn",
  "76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gnq4YfnPEsF4na0xcpP45ZpEQoAqYhNLzVvTEIMxaUg5RlWjD\npDbiUQzG9FPC5ZsPyZDr1fvzDQ3ZYAZf8vmbrmgOVqCL2eYNBgvHUkS8+jDh2L\ntDxvL8Obtpd5S9efRwnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHP",
  "wnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHPj8tq0qbeHYiMvDJ4J/SXhW9XKNkcZ5fxD6YMdPjHDMTdL\nGjQoe/H5dCpYXmKqg7Cgvp6cQzK8AbiYwHWl5AgQWZgLF6wZhBhjSsk4WB4udBEs\ndMjcrByto2pMnkVAlPy2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4",
  "Py2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4Dm2a/Piht4wo7BEJGLifTGBwobczJU0rzSPISUt7TQop\nJPWtYqsWAq45ayC4rn3fEM4DqDWYChwg9Hc7CbMjWd1dN8orO4zE0M95AxFfGqC\n7jkAJb1DjZUISVUDVrWn9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC4",
  "n9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC49kzErQqFVUEW5laW+O2+UxPBa3OSwn5pe2slSf8ZQxkx\nAdh95lcwFfC2vprMbW+WnLPKNwU+8cYwWe0qLIvqy5p1AlfVxKbUrHKFTJotCGXJ\neds0o3GoPBXtCzQBvOmKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0Aieo",
  "mKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0AieoXh9QRPXiLR5EGgmrxEwv0dTR3L8MI2kWruoCAUk0JfoO0\nvItWuU0XwYJMYjRUCpl34ZUKhSQ7DtmwCRoZfOA04FlCALjKorzGQSV5knNz80Hq\nGSKWb2ImzMOqfUOVRmjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vek",
  "mjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vekPxgWzUwQBPx2u4/mIiEUdidqC45ezLUksR3/Q1ny5Xh\n5Zuf7mZ7K0I4frNiVptxml23a4V4yAn24RrtBPGJR6K2mhFSj1iO/qAtdx43XF\nfhcN2mJO3O8ui0He7cRMs/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtq",
  "/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtqaCWBpxCW+hDqExXoLt80mhtUNh7rhVplMx8isQ1h8y\nmJ81XUIixEVI6d4wtIUiXWI5HGM8zimeUyxlLokPCOpY0bIknItqGyctCUTwNIE9\nTZxdAYjkIlCHTZBLOd05eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Z",
  "eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Zqc/PyzJSc4PLy9oPTc0nNKDyw9oDSzlLwR+OGOpeTt\nxA/PLD2jdN/SfUoLSwtK+5b2KQ0tDSl9YukTSgNLA0pXLV2lVFtKTqTwRLB0j9Kx\npWNKDy09pPSVpa8ofWbpM0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaU",
  "M0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaUvrY0seUjiwlb8XwPLOUHG/gwWipPS5pc8pFZaS9zc/\nfGnpS0pjS2NKX1j6gtK3lr6l9KmlTymNLCXfBuB0YukupfYrUJlTum3pNqWnlp6\n6vwvw+T6roW5aRvYpDSxNKF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I1",
  "KF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I152My9LX9+YcUSOl+9V12GX+FpYX9X3rLv/Xub/96+F\nK84X2euf7zg+dnzrLnQedh51na1OvxN0+5f3b+7/yxeX+wt3l98UKvXuk2dbz\nutv8WH/wEsBTb4</latexit> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022 Another application \nof the chain rule\nAlready calculated in \npart 1.",
  "> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022 Another application \nof the chain rule\nAlready calculated in \npart 1.\n./!\n.0! = \u210e1 \nHow does a small \nchange in wk change \ud835\udc59$?\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Backward pass\n2. Find how the loss \nchanges as a function of \nthe parameters b and w.\n<latexit sha1_base64=\"8HDOzmvyHV\nVJjcXRYBy6FB0LVX0=\">AXhniclZhb9s2FIDt7tZmt3bDkIe9Cs6DENrJMO6\n7mVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/ht73f7W/s0OJdmMzmHQLkBn7nwfLzok\nJUp+KkWul5b+7V76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gn",
  "76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gnq4YfnPEsF4na0xcpP45ZpEQoAqYhNLzVvTEIMxaUg5RlWjD\npDbiUQzG9FPC5ZsPyZDr1fvzDQ3ZYAZf8vmbrmgOVqCL2eYNBgvHUkS8+jDh2L\ntDxvL8Obtpd5S9efRwnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHP",
  "wnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHPj8tq0qbeHYiMvDJ4J/SXhW9XKNkcZ5fxD6YMdPjHDMTdL\nGjQoe/H5dCpYXmKqg7Cgvp6cQzK8AbiYwHWl5AgQWZgLF6wZhBhjSsk4WB4udBEs\ndMjcrByto2pMnkVAlPy2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4",
  "Py2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4Dm2a/Piht4wo7BEJGLifTGBwobczJU0rzSPISUt7TQop\nJPWtYqsWAq45ayC4rn3fEM4DqDWYChwg9Hc7CbMjWd1dN8orO4zE0M95AxFfGqC\n7jkAJb1DjZUISVUDVrWn9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC4",
  "n9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC49kzErQqFVUEW5laW+O2+UxPBa3OSwn5pe2slSf8ZQxkx\nAdh95lcwFfC2vprMbW+WnLPKNwU+8cYwWe0qLIvqy5p1AlfVxKbUrHKFTJotCGXJ\neds0o3GoPBXtCzQBvOmKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0Aieo",
  "mKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0AieoXh9QRPXiLR5EGgmrxEwv0dTR3L8MI2kWruoCAUk0JfoO0\nvItWuU0XwYJMYjRUCpl34ZUKhSQ7DtmwCRoZfOA04FlCALjKorzGQSV5knNz80Hq\nGSKWb2ImzMOqfUOVRmjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vek",
  "mjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vekPxgWzUwQBPx2u4/mIiEUdidqC45ezLUksR3/Q1ny5Xh\n5Zuf7mZ7K0I4frNiVptxml23a4V4yAn24RrtBPGJR6K2mhFSj1iO/qAtdx43XF\nfhcN2mJO3O8ui0He7cRMs/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtq",
  "/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtqaCWBpxCW+hDqExXoLt80mhtUNh7rhVplMx8isQ1h8y\nmJ81XUIixEVI6d4wtIUiXWI5HGM8zimeUyxlLokPCOpY0bIknItqGyctCUTwNIE9\nTZxdAYjkIlCHTZBLOd05eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Z",
  "eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Zqc/PyzJSc4PLy9oPTc0nNKDyw9oDSzlLwR+OGOpeTt\nxA/PLD2jdN/SfUoLSwtK+5b2KQ0tDSl9YukTSgNLA0pXLV2lVFtKTqTwRLB0j9Kx\npWNKDy09pPSVpa8ofWbpM0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaU",
  "M0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaUvrY0seUjiwlb8XwPLOUHG/gwWipPS5pc8pFZaS9zc/\nfGnpS0pjS2NKX1j6gtK3lr6l9KmlTymNLCXfBuB0YukupfYrUJlTum3pNqWnlp6\n6vwvw+T6roW5aRvYpDSxNKF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I1",
  "KF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I152My9LX9+YcUSOl+9V12GX+FpYX9X3rLv/Xub/96+F\nK84X2euf7zg+dnzrLnQedh51na1OvxN0+5f3b+7/yxeX+wt3l98UKvXuk2dbz\nutv8WH/wEsBTb4</latexit> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022\nAnother application of \nthe chain rule\n\u2022\nSimilarly for b",
  "it> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\u2022\nAnother application of \nthe chain rule\n\u2022\nSimilarly for b \nparameters\n<latexit sha1_base64=\"8HDOzmvyHV\nVJjcXRYBy6FB0LVX0=\">AXhniclZhb9s2FIDt7tZmt3bDkIe9Cs6DENrJMO6\n7mVAmzS9JV2uTtLGqUHJlMyGohSJSpwK/ht73f7W/s0OJdmMzmHQLkBn7nwfLzok\nJUp+KkWul5b+7V76ONPv3s+o2Fz7/48quvb",
  "3f7W/s0OJdmMzmHQLkBn7nwfLzok\nJUp+KkWul5b+7V76ONPv3s+o2Fz7/48quvb976Zj9Pizg/SCRSXbos5xLoXh\nfCy35YZpxFvuSH/gnq4YfnPEsF4na0xcpP45ZpEQoAqYhNLzVvTEIMxaUg5RlWjD\npDbiUQzG9FPC5ZsPyZDr1fvzDQ3ZYAZf8vmbrmgOVqCL2eYNBgvHUkS8+jDh2L\ntDxvL8Obtpd5S9efRwnJTuN1p/raGt74bDUZJU",
  "mgOVqCL2eYNBgvHUkS8+jDh2L\ntDxvL8Obtpd5S9efRwnJTuN1p/raGt74bDUZJUMRc6UCyPD9aXkr1cWlaDSfLg\nyKnKcsOGERP4KiYjHPj8tq0qbeHYiMvDJ4J/SXhW9XKNkcZ5fxD6YMdPjHDMTdL\nGjQoe/H5dCpYXmKqg7Cgvp6cQzK8AbiYwHWl5AgQWZgLF6wZhBhjSsk4WB4udBEs\ndMjcrByto2pMnkVAlPy2qNTOdtp21yuFQvMpYe",
  "QWZgLF6wZhBhjSsk4WB4udBEs\ndMjcrByto2pMnkVAlPy2qNTOdtp21yuFQvMpYeb43b0VoHot3nDRSKaRKwQeTc\nuS96IeBoIDED1OQKJ4Dm2a/Piht4wo7BEJGLifTGBwobczJU0rzSPISUt7TQop\nJPWtYqsWAq45ayC4rn3fEM4DqDWYChwg9Hc7CbMjWd1dN8orO4zE0M95AxFfGqC\n7jkAJb1DjZUISVUDVrWn9jaYeqkSVySVkPNTARZe",
  "Wd1dN8orO4zE0M95AxFfGqC\n7jkAJb1DjZUISVUDVrWn9jaYeqkSVySVkPNTARZe1nb0RnNixq1nSqCLFiEUduqI\nsiScEcbsZhBlpvyEC49kzErQqFVUEW5laW+O2+UxPBa3OSwn5pe2slSf8ZQxkx\nAdh95lcwFfC2vprMbW+WnLPKNwU+8cYwWe0qLIvqy5p1AlfVxKbUrHKFTJotCGXJ\neds0o3GoPBXtCzQBvOmKTKjwkna3KsGSNeHBXbj",
  "y5p1AlfVxKbUrHKFTJotCGXJ\neds0o3GoPBXtCzQBvOmKTKjwkna3KsGSNeHBXbjUrJD86F7vPp8cl0tm25j/kGxC\nQ3mRuhoy4f/R0AieoXh9QRPXiLR5EGgmrxEwv0dTR3L8MI2kWruoCAUk0JfoO0\nvItWuU0XwYJMYjRUCpl34ZUKhSQ7DtmwCRoZfOA04FlCALjKorzGQSV5knNz80Hq\nGSKWb2ImzMOqfUOVRmjfN7ic14IyPBzO+BXVfZ",
  "04FlCALjKorzGQSV5knNz80Hq\nGSKWb2ImzMOqfUOVRmjfN7ic14IyPBzO+BXVfZRv86nxRqxDKUzImZ0smbQa5\nhi7l2fzXldFpRfx0vekPxgWzUwQBPx2u4/mIiEUdidqC45ezLUksR3/Q1ny5Xh\n5Zuf7mZ7K0I4frNiVptxml23a4V4yAn24RrtBPGJR6K2mhFSj1iO/qAtdx43XF\nfhcN2mJO3O8ui0He7cRMs/3BvDydkckxI5Mse+RA7q",
  "6K2mhFSj1iO/qAtdx43XF\nfhcN2mJO3O8ui0He7cRMs/3BvDydkckxI5Mse+RA7qEBY1FbVTrM6/bEOYTEu2h\nb8P1Z2BTw82lYdwuJWLtqaCWBpxCW+hDqExXoLt80mhtUNh7rhVplMx8isQ1h8y\nmJ81XUIixEVI6d4wtIUiXWI5HGM8zimeUyxlLokPCOpY0bIknItqGyctCUTwNIE9\nTZxdAYjkIlCHTZBLOd05eXOlafQKlZ0FfdHfev6Fg",
  "pY0bIknItqGyctCUTwNIE9\nTZxdAYjkIlCHTZBLOd05eXOlafQKlZ0FfdHfev6Fgz1KAJYGmT7DFvsOncZD5Oc\nf2uTKdLICulCdzCzhZ1Zqc/PyzJSc4PLy9oPTc0nNKDyw9oDSzlLwR+OGOpeTt\nxA/PLD2jdN/SfUoLSwtK+5b2KQ0tDSl9YukTSgNLA0pXLV2lVFtKTqTwRLB0j9Kx\npWNKDy09pPSVpa8ofWbpM0pfW/qa0neWvqP0kaWPK",
  "pXLV2lVFtKTqTwRLB0j9Kx\npWNKDy09pPSVpa8ofWbpM0pfW/qa0neWvqP0kaWPKGWMkrXLF2jlFtKPh34Yql\nK5T6lpJ3P9hrlm5RmlqaUvrY0seUjiwlb8XwPLOUHG/gwWipPS5pc8pFZaS9zc/\nfGnpS0pjS2NKX1j6gtK3lr6l9KmlTymNLCXfBuB0YukupfYrUJlTum3pNqWnlp6\n6vwvw+T6roW5aRvYpDSxNKF03VLypgBHCUtPyHkyVM",
  "kupfYrUJlTum3pNqWnlp6\n6vwvw+T6roW5aRvYpDSxNKF03VLypgBHCUtPyHkyVM1dbfa1idzXQjXnDtZkfFa\nb5DxUc+5gzd1pVpvcn0I152My9LX9+YcUSOl+9V12GX+FpYX9X3rLv/Xub/96+F\nK84X2euf7zg+dnzrLnQedh51na1OvxN0+5f3b+7/yxeX+wt3l98UKvXuk2dbz\nutv8WH/wEsBTb4</latexit> @`i\n@\u03b2k\n=",
  "0+5f3b+7/yxeX+wt3l98UKvXuk2dbz\nutv8WH/wEsBTb4</latexit> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n@`i\n@!k\n= @fk\n@!k\n@`i\n@fk\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* ) \n1",
  "Backward pass\n2. Find how the loss \nchanges as a function of \nthe parameters b and w.\n\ud835\udc53' = \ud835\udefd' + \ud835\udf14' \u22c5 \ud835\udc65 \n\u210e( = a[\ud835\udc53'] \n\ud835\udc53( = \ud835\udefd( + \ud835\udf14( \u22c5 \u210e( \n\u210e) = a[\ud835\udc53(] \n\ud835\udc53) = \ud835\udefd) + \ud835\udf14) \u22c5 \u210e) \n\u210e* = a[\ud835\udc53)] \n\ud835\udc53* = \ud835\udefd* + \ud835\udf14* \u22c5 \u210e* \n\u2113& = \ud835\udc66& \u2212 \ud835\udc53* )",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "Jupyter Notebook Example\n7_Backprop_with_Micrograd_lite_pt1.ipynb",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "Matrix calculus\n<latexit sha1\n_base64=\"mfEFVtCS3NwDcfGp1\nx/Vcf8tZys=\">AWuXiclZhb\nb9s2FIDVXbvulm5YXvYiLCgwD\nJ0Rd90FGAq0SdNb0iVp4iRtnBq\nUTMlsKEqRqMSp4F+zX7PX7Wn/\nZoeSbFbnMA8zkIo93ydeDkndgk\nyKQq+u/nvtvfc/+PCj69/cuP\nTz7/4sulm18dFGmZh3wQpjLNj\nwJWcCkUH2ihJT/Kcs6SQPLD4H\nTd8MNznhci",
  "P\nTz7/4sulm18dFGmZh3wQpjLNj\nwJWcCkUH2ihJT/Kcs6SQPLD4H\nTd8MNznhciVfv6MuMnCYuViET\nINIRGS/eGQcT8e/4w4LFQVZAwn\nYvpjI2q/mw4hMOd5vBTc7g7G3\nI1XmijpZXV3mr982mh3xZWvPa3\nM7r5zXg4TsMy4UqHkhXFcX810\nycVy7UIJZ/dGJYFz1h4ymJ+DEX\nFEl6cVPU4Z/4tiIz9KM3hT2m/\njr57RsWSorhMAjChg",
  "IJZ/dGJYFz1h4ymJ+DEX\nFEl6cVPU4Z/4tiIz9KM3hT2m/\njr57RsWSorhMAjChg5MCMxN0s\neNSR7+dVEJlpeYqbBqKSunr1Dd\nJ8ci56GWl1BgYS6gr34YTkL\nNaT2xlDxizBNEgaZGa5t7M6qNp\nv8rKzTPJt1nY3aMYm8ylh7ur+\noRWieiLecVFIrpIrB7Pqor34\nh4GgMQPU5AqngBdZr8BJHfRx\nSWlQMPEin0LnIfzEjVSvNY8h\nJR3t",
  "rB7Pqor34\nh4GgMQPU5AqngBdZr8BJHfRx\nSWlQMPEin0LnIfzEjVSvNY8h\nJR3tFNChk871jqxYCqTjrIHi\nu/f8g3gsABDv16HIUdzsJcxNZ\nufp/lU50lVmBhuIWcq5nUTMOSQ\nSTOirqFKeHUsGP9ga0XTJ2i\nUuzuqu5iSBrP+86Oqd5UeOuU0e\nQBYsw7lp1BFkSLgJjljDIclse\nwYAT30TcqlBYFWRh7uRp0G07M\nxG8NqcZ7Jeut1GR",
  "sw7lp1BFkSLgJjljDIclse\nwYAT30TcqlBYFWRh7uRp0G07M\nxG8NqcZ7Jeut1GR9J8zlBETgN1\nnjoKpkHf19XRh+/PknNe+KfCp\nP4HJ6p7C8rgZ1rwRGFUbm1Gzh\nUyabYglKcXdP0xqHyTHQHaAJ\n405W5UNE72u26BEvWhIe3Yah5K\nfnxj72f+fSkWjXbxvxDsgkVFW\nXmqsiE/0dFY7jt4PUFETx5qUS\nTB4F68lIJ13c0dSzHC9tE6rm",
  "jXbxvxDsgkVFW\nXmqsiE/0dFY7jt4PUFETx5qUS\nTB4F68lIJ13c0dSzHC9tE6rmDg\nlBMCn2Jtr+IVfecOoI7myaorx\nAw9cKRCYUmOYq6sgkYGY5wA3Us\noBANMmzGMq0KHNOLn5oPUOk1\ns1lMRfmZtW9oEojdK8bXC7OgjL\ncHM75FacHKNBk8gLdWY5SiZ\nUzOl09fDQsMWc+3+esqbotOK+\ndlm2x70C2anDEN+NtrE8xETizo\nS1QVPLM",
  "SiZ\nUzOl09fDQsMWc+3+esqbotOK+\ndlm2x70C2anDEN+NtrE8xETizo\nS1QVPLM6JLEc7UFdi+X6bs+q\nzdc/kKUdO1y3KUm9bS/dtsO9og\nf8bMvR2y3iEYs6EtXV9pB6xHK\n0B3W587jlGoXDdZuS1DvPo9N2u\nAsTLf9of8I1M49JqRybx75UDp\nsQFjUVtVNMEx4jsQlhMSm7Fvw\nfK3sCbh5dqwlhcacQXc0EsDTmE\ng+hCWGx2cJds41h",
  "UVtVNMEx4jsQlhMSm7Fvw\nfK3sCbh5dqwlhcacQXc0EsDTmE\ng+hCWGx2cJds41hdcuhbrlVJr\nMJMpsQFh+zBI+6CWExpmLsFE9Z\nliGxCZE8TnAeJzSPGZYyl4RnJ\nHPMCFlSrgWVT9KuZAJYmqLWpo7\nGoAcyVajBNojlgq68wrnyFrF\niq7igavhwRUNa4YqNAEsbZM95\ng+3nZswCk2b6mOJGcCWRlN4A5\n2dqgzf/oLo8yQXRpaWXlF5Y",
  "a4YqNAEsbZM95\ng+3nZswCk2b6mOJGcCWRlN4A5\n2dqgzf/oLo8yQXRpaWXlF5Y\nekHpoaWHlOaWkjeCIHphKXk7Ca\nJzS8pPbD0gNLS0pLSgaUDSiN\nLI0ofWfqI0tDSkNJ1S9cp1ZaSJ\n1K4I1i6T+nE0gmlR5YeUfrS0p\neUPrH0CaWvLH1F6VtL31L6wNI\nHlDJLGaUblm5Qyi0lnw6CaM3SN\nUoDS8m7H+w1S3cozSzNKH1o6U\nNKx5aSt2",
  "NI\nHlDJLGaUblm5Qyi0lnw6CaM3SN\nUoDS8m7H+w1S3cozSzNKH1o6U\nNKx5aSt2K4n1lKHm/gxmipPSp\npU8pFZaS97cgem7pc0oTSxNKn\n1n6jNI3lr6h9LGljymNLSXfBuD\npxNI9Su1XoKqgdNfSXUrPLD1z\nfxfgi2kMXAtz21awTWlqaUrp\nqXkTQEeJSw9Jc+TkWqvavOvTeS\n6FqkFd7A24/OzSc4jteAO1l6d\n5meT61OkFnxCur5xs",
  "eJSw9Jc+TkWqvavOvTeS\n6FqkFd7A24/OzSc4jteAO1l6d\n5meT61OkFnxCur5xsPiQAimFK/\n1oaWPv8LSwsGdXv+X3t3duyv\n319ovtNe9b73vO+9verd97\n4u14Ay/0/vT+8v72/ln+fZktT5\nbfNOp719pzvY6v+XiP1hA6FY\n=</latexit>\na =\n2\n664\na1\na2\na3\na4\n3\n775\n<latexit sha1_base64=\"sBVI\n6wX86kODo9XRrTkpDSEFUDI=\">",
  "sha1_base64=\"sBVI\n6wX86kODo9XRrTkpDSEFUDI=\">AXaXiclZhb9s2FICd7tZ1t\n3TDlmF7ERYUGIbOsLvu8jKgTZreki7OxUnaODUomZLZUJSi+NU\n8Mv2K/cbhv2HUqyGZ3DdGiAztz5PpHU4aFubixFmnU6fy9de+\nfd97/4PqHNz76+JNP1u+flBGuWJx/teJKPkyGUpl0LxfiYy\nY/ihLPQlfzQPV3X/HDCk1REaj+7iPlJyAIlfOGx",
  "GuWJx/teJKPkyGUpl0LxfiYy\nY/ihLPQlfzQPV3X/HDCk1REaj+7iPlJyAIlfOGxDELD5X8HfsK8\nYhCzJBNMOv7MtAeuz2bO7/DLA6EKN2RZIqYz5w2HsGF3NpikMfN\n40Wnf8UKwB2/07yD/f/Sf3k6/OxtwNVpMfbi82ml3yj+HNrp1Y7\nV/WGN78cDUaRl4dcZ5kaXrc7cTZSaEH8CSf3RjkKYfpnLKAH\n0NTsZCnJ0W5LjPnFkRGjh8l8E9",
  "Rl4dcZ5kaXrc7cTZSaEH8CSf3RjkKYfpnLKAH\n0NTsZCnJ0W5LjPnFkRGjh8l8E9lThm9fETBwjS9CF0wYLjFDMd\ntLHjPN/OymEivOMK68ayM+lk0WOXmRnJBLuZfICGsxLBMzV8c\nYM8pRBKdwYKH7uRWHIDODtY0dyFi1wvwsL8tiNms6G6WjE3mVs\nfZkf9GLyHgoXnPSanoTq4QeDArCt4O2hgIDkC0OQGR4in0qfPj\n+k4XUdgGEjBwN5r",
  "kf9GLyHgoXnPSanoTq4QeDArCt4O2hgIDkC0OQGR4in0qfPj\n+k4XUdgGEjBwN5rC5Hxnd0a6VhkPICcN7QXRoBFLPm1Y68SCpQw\nbyh4ojnPL0YBDAXpOWYceR2uwFzM1mx+X8WmWhEWqY3iEhKmAl0\nPAKXtQ4bvYULmU5f64bP2BrV2mTuvERXE51URHkLWfNJ0soXlRo\n6ZTRpAFRg0rTKCLAkXrRELGWS5bg/hENHR+yqUFgVpDB7SeQ2\nx",
  "LWfNJ0soXlRo\n6ZTRpAFRg0rTKCLAkXrRELGWS5bg/hENHR+yqUFgVpDB7SeQ2\nx451BNfmNIb90vQ2CpL+CUMZ0QHYfpXMOXxpr4eLWxnpxJ6e\nsGnzpjWKzmISwJqtOaDwJnVcdm1CxzhUyaLQgl0XnT1LOxqDwWz\nRPUAbzp8kQo/5J2u2xByerw4DacapJLfvxj+2c+PSk6etvo/5Bs\nQkdpHts60uG36GgEt0lcXxDBixdJtHgQKBcvknB9",
  "pJLfvxj+2c+PSk6etvo/5Bs\nQkdpHts60uG36GgEt0lcXxDBixdJtHgQKBcvknB9R0vHElzYOlK\nuHTSEYlJkF2j7i0A1jykjeLJRiOYKAd0v/DKh0CL7flPWAS3DL9\nzwLQXkoZP0qnP0ZJTmCScXP1TPECl1fVlMhL5ZNS+oUgvN6waXi\n6OgDTeHCb/icBdl1K3y6Ua5GrEJXOql3T6cpBmsMVsu79c8qp\ntQJ+tlmPB/OC1ck9j58N/F6BMS",
  "Bdl1K3y6Ua5GrEJXOql3T6cpBmsMVsu79c8qp\ntQJ+tlmPB/OC1ck9j58N/F6BMSijkR9wROWtS9JLMt40NeiXC\n/PrNh8+QMp7cDi2k1J+q1nabct7hUz4GdbltluEY9Y1JGor3qG1\nCOWZTzoy57HLdtZWFy7KUm/8zxabYu7MFH5+/tjnjH9mBTJkX7s\ni+SgCmExo2JmFaOQB0isQlgM86YF/4+VPQE3j6ZVhbDYS0VT0wE\nsjbjEp1CFsFh",
  "i+SgCmExo2JmFaOQB0isQlgM86YF/4+VPQE3j6ZVhbDYS0VT0wE\nsjbjEp1CFsFht4aZx7C6ZVG37CqT8RiZVQiLj1iIz7oKYTGgYm\nAVT1kcI7EKkTyOcR7HNI8xlmKbhFcktqwIKSlbQSXjqCnpAJama\nLSpZTCYgYwUGrAOYjmlZdaK0+hKla0ivu2gftXDJwx1KEOYGmb\n7DFnsG3dZC5OsX5ztiQ5FsiKaQJ72OlRZ/705/oFeZJz/QtDLy",
  "XDJwx1KEOYGmb\n7DFnsG3dZC5OsX5ztiQ5FsiKaQJ72OlRZ/705/oFeZJz/QtDLy\ng9N/Sc0kNDylNDCVvBK6/ayh5O3H9iaETSg8MPaA0NzSntG9on\n1LfUJ/Sh4Y+pNQz1KN03dB1SjNDyRMp3BEM3ad0bOiY0iNDjyh9\nbuhzSh8b+pjSF4a+oPS1oa8pvW/ofUqZoYzSDUM3KOWGk8Hr9\nm6BqlrqHk3Q/2mqE9SmNDY0ofGPqA0pGh5K0Y7",
  "vW/ofUqZoYzSDUM3KOWGk8Hr9\nm6BqlrqHk3Q/2mqE9SmNDY0ofGPqA0pGh5K0Y7meGkscbuDEaKi\nl9YugTSoWh5P3N9Z8Z+ozS0NCQ0qeGPqX0laGvKH1k6CNKA0PJt\nwF4OjF0j1LzFahIKd0xdIfSM0P7N8F+GIZXVthbpsOtimNDI0\no3TSUvCnAo4Shp+R50lf1VW3+tYlc13y14BZWZ3x+NMm5rxbcwu\nqr0/xocn3y1YKPydQ3DhYfUi",
  "hp+R50lf1VW3+tYlc13y14BZWZ3x+NMm5rxbcwu\nqr0/xocn3y1YKPydQ3DhYfUiClcKUfLq928VdY2ji40+7+0r67c\n3f13lr9hfZ69vWd63vW93Wr617rcetXqvf8pb2li6W/lz6t/\nVm6ufL3yTaVeW6qP+aLV+FtZ/Q87kyrv</latexit>\n@f\n@a =\n2\n6666664\n@f\n@a1\n@f\n@a2\n@f\n@a3\n@f\n@a4\n3\n7777775\nScalar function \ud835\udc53 \u22c5  of a vector \ud835\udc1a\nThe derivative is a vector of shape \ud835\udc1a",
  "Matrix calculus\n<latexit \nsha1_base64=\"M1r\nHZ1aEGtiEuf+2w5K\noKtvAJD4=\">AW9X\niclZhbU9tGFMdN2r\nRp2iaknfLSF02ZdD\nqdxIOBXl46k0DIDV\nJMwEDAxLOSV/KG1Ur\noAnY0/ih96/S1n6d\nP/So9K8ne6JzloZ4\nh2vx/72dPbu6uLEU\nabay8s/CjY8+vnJ\np7c+u/35F1/eubt4\n76vDNMoTj/e8SEbJ\nsctSLoXivUxkh/HC\nWehK/mRe76p",
  "J\np7c+u/35F1/eubt4\n76vDNMoTj/e8SEbJ\nsctSLoXivUxkh/HC\nWehK/mRe76p+dElT\n1IRqYNsEvOzkAVK+\nMJjGUiDxUnf9R87vz\nl9lwdCFW7IskSMp2\nxQdDrT7/VltbqsTf\nt9uK5W6mqlrtbqWq\nWuVepara5X6nqlroP\nK1XDexWBxeaW9Uv4\ncWujUheVW/esO7n0\nz7A8jLw+5yjzJ0vS0\nsxJnZwVLMuFJPr3d\nz1MeM+cBfwUio",
  "WujUheVW/esO7n0\nz7A8jLw+5yjzJ0vS0\nsxJnZwVLMuFJPr3d\nz1MeM+cBfwUioqF\nPD0ryhNnfugDB0/\nSuBPZU6pflijYGaT\nkIXnDAUYqZFm3sN\nM/8X8KoeI848qrO\nvJz6WSRowPuDEXCvU\nxOoMC8RMBYHW/Eu\nZlsCy3+4pfeVEYMo\nhMf2Nrb1rUK8Ev8n\nKJptOmZ6v06EBe59h\n4cTBvRWQ8FO85aS\n06EauMfBgWhS8HbQ\nxEByA",
  "UK8Ev8n\nKJptOmZ6v06EBe59h\n4cTBvRWQ8FO85aS\n06EauMfBgWhS8HbQ\nxEByAaHMCIsVTaFPH\nx/WdDqKQkhIwcDca\nw+B85/WUNK0yHkBM\nGrYTYoNCLPm4dok\nLljKsGHZB4vj3Hc04\nJCAnlPmocfRGuzHT\nE1n9TI+zpKwSLWGe\n0iYCnjZBUzZY1LPqO\nlQuZRQ1Wu4fseu10\nyd14GL4nKoiVaQ6y\nBperKExkUNm5SQS\n5IwqDpKhX",
  "LPqO\nlQuZRQ1Wu4fseu10\nyd14GL4nKoiVaQ6y\nBperKExkUNm5SQS\n5IwqDpKhXknCADFn\nIMp1eQATDh2t2K1\nCYasgidlNIrfZd6w\nVnJvjGPZL07dVkPBf\nMhQRLcDu01fBlMeb\n9s1o7nZmwbks/brA\nx84IFqtZhSVBNa1ZJ\nzCrWptSZxkr5KTRA\nimJrpOPRqLlceiO\nUEt4E2XJ0L5H9gel\nCVIWS3H8BUk1zy04\nftn/j4rFjR20b/",
  "imJrpOPRqLlceiO\nUEt4E2XJ0L5H9gel\nCVIWS3H8BUk1zy04\nftn/j4rFjR20b/Q6\nIJDaV5bGtIy/+joS\nHcsnB+gYIXL5Jo8UA\noFy+ScL6jpWMJTmy\ntlGsHBaGYFNkEbX8\nRqGadUsGDjUI0VhB\n0u3BlQqF9v2mWQva\nDFe4+VoSyEOT9Ko5\nejJK84STw/lMyil\nXR+LidA3q+aBKrWhe\nW5wOa8FZbg5XPJrq\nrsom4VTzfK1ZAlK\nJh",
  "4STw/lMyil\nXR+LidA3q+aBKrWhe\nW5wOa8FZbg5XPJrq\nrsom4VTzfK1ZAlK\nJhjvaTjt/0gy1m2\n/3lkldFqyvgF9t1fz\nAuWJ3c8/jFYBuvR0\nBc1CNRW/C0Y21LEp\nelP2hrnq4fjqzYfvs\njSe3A4rU7JWm3HqX\ndbfFeMwJ+sWMZ7Q7\nxERf1SNRWPULqIy5\nLf9CWPY47tlYvHan\nJO3O4mh1W7xzJ0p/\n/2DEM6YfkyI51I9\nkexXEjZ",
  "ULqIy5\nLf9CWPY47tlYvHan\nJO3O4mh1W7xzJ0p/\n/2DEM6YfkyI51I9\nkexXEjZm1JhZjVHIA\n2SsJGwM86YL/o8t+\nwJuHk1XJWFjNxVNm\nxawacglnkIlYWO1h\nZvOWsPWHYt1x25lMh\n4hZyVh4zMW4lXEj\nYG1BhYjecsjpGxk\ngcRziOIxrHGJtimwm\nvSGxZEZJStoRKRlH\nTpAVsGqPexpbOYAQ\nyUqjDWsTmlGZeas0\n8hbJY0Szu2T",
  "m\nvSGxZEZJStoRKRlH\nTpAVsGqPexpbOYAQ\nyUqjDWsTmlGZeas0\n8hbJY0Szu2TruXdNx\nxlCDWsCmXbLHnP6u\ndZO5OMT6DdcS5Fg\nV0wD2MWeLvXMnv5cv\nyBPcq4/MXRC6ZWhV\n5QeGXpEaWIoeSNw/\ndeGkrcT17809JLSQ\n0MPKc0NzSntGdqj1D\nfUp/SpoU8p9Qz1KN\n0dJPSzFDyRAp3BE\nMPKB0ZOqL02NBjSt8\nY+obS54Y+p/TE0",
  "Up/SpoU8p9Qz1KN\n0dJPSzFDyRAp3BE\nMPKB0ZOqL02NBjSt8\nY+obS54Y+p/TE0BN\nK3xv6ntLHhj6mlBn\nKN0ydItSbij5dOD\n6G4ZuUOoaSt79YK8Z\n2qU0NjSm9ImhTygd\nGkreiuF+Zih5vIEb\no6GS0heGvqBUGEre3\n1z/laGvKA0NDSl9a\nehLSt8Z+o7SZ4Y+o\nzQwlHwbgKcTQ/cpN\nV+BipTSPUP3KL0w9M\nL+XYDPl9G1JeauaW",
  "8Z+o7SZ4Y+o\nzQwlHwbgKcTQ/cpN\nV+BipTSPUP3KL0w9M\nL+XYDPl9G1JeauaW\nCX0sjQiNJtQ8mbAj\nxKGHpOnid9VZ9qs69\nN5Fz1ZxbWB3xW0\nSc1/NuYXVp9OsNjm\nfDXnIzL0rcP5hxQI\nKZz0g8XlDv4KSwuH\nq+3Oz+31vfXlRxv1\nF9pbrW9b37V+aHVa\nv7QetZ63uq1ey2v9u\n3Bz4c7C3aWrpT+W/\nlz6q7LeWKjrfN1q/\nJb+/g",
  "V+aHVa\nv7QetZ63uq1ey2v9u\n3Bz4c7C3aWrpT+W/\nlz6q7LeWKjrfN1q/\nJb+/g+TfPpt</late\nxit>\nA =\n2\n664\na11\na12\na13\na21\na22\na23\na31\na32\na33\na41\na42\na43\n3\n775\n<latexit sha1_base64=\"IO/lnioei3wiE\nDVhi/74wEu7W8=\">AYn3iclZjrbts2FIDd7NZ1t3bDgAD7ozXoMAydESfZ5c+ANmat\nkb540TgNKpmQ2FKXokjgV/AB7mv3dH",
  "d7NZ1t3bDgAD7ozXoMAydESfZ5c+ANmat\nkb540TgNKpmQ2FKXokjgV/AB7mv3dHmVvs0NJNqNzmBk1kJo93yce6pCUZLmxFGk2P/\nvrZkPvzo409uf3rns8+/+PKru/e+3k+jPH4nhfJKDl0WcqlUHwvE5nkh3HCWehKfuCer\nWh+cMGTVERqN7uK+UnIAiV84bEMQqf3Zu73/IR5RS9mSaYdPyRafdc/HI+QO+eSBU4Y\nsS8Rw5PzPIey06HRG",
  "V84bEMQqf3Zu73/IR5RS9mSaYdPyRafdc/HI+QO+eSBU4Y\nsS8Rw5PzPIey06HRGox+mKAvTlcXRqHeRxszjxXx7wQshbW9K5oXpmRemZ14gmacmXpye\nHF64sX3T7w0PfHS9MRLNDFX/cl8n96dm2/Plx+HNjp1Y65Vf7qn97t9/qRl4dcZ5kaXr\ncmY+zk0Jn9CQf3enlKYdsZyzgx9BULOTpSVGu5pHzACJ9x48S+FOZU0avH1GwME2vQhdMG",
  "0Jn9CQf3enlKYdsZyzgx9BULOTpSVGu5pHzACJ9x48S+FOZU0avH1GwME2vQhdMG\nOAgxUwHbew4z/zfTwqh4jzjyqsS+bl0sjRW8Ppi4R7mbyCBvMSAWN1vAGDwmWwge70FL/\n0ojBkUJne8uoWlLDaFvw8LzfTaNR0VktHF/ImY/n57qQXkfFQvOk1LRndwg8GBUFLwdt\nDEQHIBocwIixVPoU9fH9Z0OonDxkICBu9EQBuc72yPStcp4ADVpa",
  "wg8GBUFLwdt\nDEQHIBocwIixVPoU9fH9Z0OonDxkICBu9EQBuc72yPStcp4ADVpaK+JBo1Y8mHDWiEWTGX\nYUHZAcZwHjgYcFqDnlOvQ42gOdmKmRuPjMj7MkrBIdQxnSJgKeJkCTtmDJb+NDZVLWS7/6\n9YrbG0zdVYXLorLoSY6gqzdpOlkCa2L6jedMoIsWIRB0yojyJwqe+zkEGV6/YpnHDo6Ih\ndFQqrgizMbhK5zdyxjuC1OYxhvzS91YK",
  "sWIRB0yojyJwqe+zkEGV6/YpnHDo6Ih\ndFQqrgizMbhK5zdyxjuC1OYxhvzS91YKU/4KhiugA7D79LZjyeFNfiSa2My7ORenrBh86A\n5is5iEsCarTGieBs6pjI2qWtUImrRaEkuiyaerRWFQei+YJ6gDedHkilH9Ne1i2YMnqcO8\nhnGqS378c/sXPjwp5vW20f+QakJHaR7bOtLh9+ioDw8XeH1BE9eJNHkQaCcvEjC9R1NH\nUvwtaRcu6gIRS",
  "W20f+QakJHaR7bOtLh9+ioDw8XeH1BE9eJNHkQaCcvEjC9R1NH\nUvwtaRcu6gIRSTIrtC218EqnlMGcGDjUI0VgjofuGbCYUm2febsg5oGb7hMcmygDx0kl5\n1jp6M0jzh5OKH1jNESl1fFhOhb1bNC6rUQvO6weXkKGjDzeGC3C4iyrqVvV0o1z1WYKO\ndRTOnzTSzPYrbdX051bRaAT9fr/PBuGB2cs/j56freD4CYlFHor7gudTalySWJR/0N",
  "TOnzTSzPYrbdX051bRaAT9fr/PBuGB2cs/j56freD4CYlFHor7gudTalySWJR/0NV\nmu10dWrL/5iSztwOLaTUn6rUdpty3uDSPg5xuW0W4Qj1jUkaiveoTUI5YlH/Rlr+OG7Sws\nrt2UpN9xHa2xZ2YaPn7uwOeMf2YFMm+fuyLZK8KYTGjYmYVo5AHSKxCWAzpgX/x8qOgJ\ntH06pCWOymoqnpAJb6XOJTqEJYrLZw06xjWN2wqBt2lcl4gMwq",
  "WAzpgX/x8qOgJ\ntH06pCWOymoqnpAJb6XOJTqEJYrLZw06xjWN2wqBt2lcl4gMwqhMU1FuKzrkJYDKgYWMUz\nFsdIrEKkjgNcxwGtY4yl2CbhGYktM0KWlG1BJYOoKekAloYo29CSDEYgI4US1kEsp3Tlpd\naVp9AqVnQV79kS792QOGOoQx3A0ibZY05v07rJXFxi/brBUuRYICumBexip0ud8dOf6xfk\nSc71rwy9ovTS0EtKDw9oDQxlPwicP",
  "rJXFxi/brBUuRYICumBexip0ud8dOf6xfk\nSc71rwy9ovTS0EtKDw9oDQxlPwicP1tQ8mvE9e/MPSC0n1D9ynNDc0p3TN0j1LfUJ/Sp4\nY+pdQz1KN0xdAVSjNDyRMp3BEM3aV0YOiA0kNDyk9MvSI0meGPqP0taGvKX1n6DtKHxv6\nmFJmKN01dBVSrmh5NWB6y8bukypayj57Qd7zdAupbGhMaVPDH1Cad9Q8qsY7meGkscbuD\nEaKil9buhzS",
  "WB6y8bukypayj57Qd7zdAupbGhMaVPDH1Cad9Q8qsY7meGkscbuD\nEaKil9buhzSoWh5Peb67809CWloaEhpS8MfUHpW0PfUrpm6BqlgaHk3QA8nRi6Q6l5C1Sk\nlG4ZukXpuaHn9vcCfDKNrm1hbpoONimNDI0oXTeU/FKARwlDz8jzpK/q9r4bRO5rvlqwi\n2srvj4aFJzX024hdVXp/HR5PrkqwkfkKGv7k9epEBJ4Up/eneug9/C0sb+Qrvz",
  "i\n2srvj4aFJzX024hdVXp/HR5PrkqwkfkKGv7k9epEBJ4Up/eneug9/C0sb+Qrvza3tpa2nu\n0XL9hvZ267vW/daPrU7rt9aj1rNWt7X8mb+nPlr5u+Zf2a/n12bfTXbrdSZW/Ux37Qan9\nmj/wAlkaBJ</latexit>\n@f\n@A =\n2\n66666664\n@f\n@a11\n@f\n@a12\n@f\n@a13\n@f\n@a21\n@f\n@a22\n@f\n@a23\n@f\n@a31\n@f\n@a32\n@f\n@a33\n@f\n@a41\n@f\n@a42\n@f\n@a43\n3\n77777775\nScalar function",
  "function \ud835\udc53 \u22c5  of a matrix \ud835\udc1a\nThe derivative is a matrix of shape \ud835\udc1a",
  "Matrix calculus\n<latexit sha1\n_base64=\"mfEFVtCS3NwDcfGp1\nx/Vcf8tZys=\">AWuXiclZhb\nb9s2FIDVXbvulm5YXvYiLCgwD\nJ0Rd90FGAq0SdNb0iVp4iRtnBq\nUTMlsKEqRqMSp4F+zX7PX7Wn/\nZoeSbFbnMA8zkIo93ydeDkndgk\nyKQq+u/nvtvfc/+PCj69/cuP\nTz7/4sulm18dFGmZh3wQpjLNj\nwJWcCkUH2ihJT/Kcs6SQPLD4H\nTd8MNznhci",
  "P\nTz7/4sulm18dFGmZh3wQpjLNj\nwJWcCkUH2ihJT/Kcs6SQPLD4H\nTd8MNznhciVfv6MuMnCYuViET\nINIRGS/eGQcT8e/4w4LFQVZAwn\nYvpjI2q/mw4hMOd5vBTc7g7G3\nI1XmijpZXV3mr982mh3xZWvPa3\nM7r5zXg4TsMy4UqHkhXFcX810\nycVy7UIJZ/dGJYFz1h4ymJ+DEX\nFEl6cVPU4Z/4tiIz9KM3hT2m/\njr57RsWSorhMAjChg",
  "IJZ/dGJYFz1h4ymJ+DEX\nFEl6cVPU4Z/4tiIz9KM3hT2m/\njr57RsWSorhMAjChg5MCMxN0s\neNSR7+dVEJlpeYqbBqKSunr1Dd\nJ8ci56GWl1BgYS6gr34YTkL\nNaT2xlDxizBNEgaZGa5t7M6qNp\nv8rKzTPJt1nY3aMYm8ylh7ur+\noRWieiLecVFIrpIrB7Pqor34\nh4GgMQPU5AqngBdZr8BJHfRx\nSWlQMPEin0LnIfzEjVSvNY8h\nJR3t",
  "rB7Pqor34\nh4GgMQPU5AqngBdZr8BJHfRx\nSWlQMPEin0LnIfzEjVSvNY8h\nJR3tFNChk871jqxYCqTjrIHi\nu/f8g3gsABDv16HIUdzsJcxNZ\nufp/lU50lVmBhuIWcq5nUTMOSQ\nSTOirqFKeHUsGP9ga0XTJ2i\nUuzuqu5iSBrP+86Oqd5UeOuU0e\nQBYsw7lp1BFkSLgJjljDIclse\nwYAT30TcqlBYFWRh7uRp0G07M\nxG8NqcZ7Jeut1GR",
  "sw7lp1BFkSLgJjljDIclse\nwYAT30TcqlBYFWRh7uRp0G07M\nxG8NqcZ7Jeut1GR9J8zlBETgN1\nnjoKpkHf19XRh+/PknNe+KfCp\nP4HJ6p7C8rgZ1rwRGFUbm1Gzh\nUyabYglKcXdP0xqHyTHQHaAJ\n405W5UNE72u26BEvWhIe3Yah5K\nfnxj72f+fSkWjXbxvxDsgkVFW\nXmqsiE/0dFY7jt4PUFETx5qUS\nTB4F68lIJ13c0dSzHC9tE6rm",
  "jXbxvxDsgkVFW\nXmqsiE/0dFY7jt4PUFETx5qUS\nTB4F68lIJ13c0dSzHC9tE6rmDg\nlBMCn2Jtr+IVfecOoI7myaorx\nAw9cKRCYUmOYq6sgkYGY5wA3Us\noBANMmzGMq0KHNOLn5oPUOk1\ns1lMRfmZtW9oEojdK8bXC7OgjL\ncHM75FacHKNBk8gLdWY5SiZ\nUzOl09fDQsMWc+3+esqbotOK+\ndlm2x70C2anDEN+NtrE8xETizo\nS1QVPLM",
  "SiZ\nUzOl09fDQsMWc+3+esqbotOK+\ndlm2x70C2anDEN+NtrE8xETizo\nS1QVPLM6JLEc7UFdi+X6bs+q\nzdc/kKUdO1y3KUm9bS/dtsO9og\nf8bMvR2y3iEYs6EtXV9pB6xHK\n0B3W587jlGoXDdZuS1DvPo9N2u\nAsTLf9of8I1M49JqRybx75UDp\nsQFjUVtVNMEx4jsQlhMSm7Fvw\nfK3sCbh5dqwlhcacQXc0EsDTmE\ng+hCWGx2cJds41h",
  "UVtVNMEx4jsQlhMSm7Fvw\nfK3sCbh5dqwlhcacQXc0EsDTmE\ng+hCWGx2cJds41hdcuhbrlVJr\nMJMpsQFh+zBI+6CWExpmLsFE9Z\nliGxCZE8TnAeJzSPGZYyl4RnJ\nHPMCFlSrgWVT9KuZAJYmqLWpo7\nGoAcyVajBNojlgq68wrnyFrF\niq7igavhwRUNa4YqNAEsbZM95\ng+3nZswCk2b6mOJGcCWRlN4A5\n2dqgzf/oLo8yQXRpaWXlF5Y",
  "a4YqNAEsbZM95\ng+3nZswCk2b6mOJGcCWRlN4A5\n2dqgzf/oLo8yQXRpaWXlF5Y\nekHpoaWHlOaWkjeCIHphKXk7Ca\nJzS8pPbD0gNLS0pLSgaUDSiN\nLI0ofWfqI0tDSkNJ1S9cp1ZaSJ\n1K4I1i6T+nE0gmlR5YeUfrS0p\neUPrH0CaWvLH1F6VtL31L6wNI\nHlDJLGaUblm5Qyi0lnw6CaM3SN\nUoDS8m7H+w1S3cozSzNKH1o6U\nNKx5aSt2",
  "NI\nHlDJLGaUblm5Qyi0lnw6CaM3SN\nUoDS8m7H+w1S3cozSzNKH1o6U\nNKx5aSt2K4n1lKHm/gxmipPSp\npU8pFZaS97cgem7pc0oTSxNKn\n1n6jNI3lr6h9LGljymNLSXfBuD\npxNI9Su1XoKqgdNfSXUrPLD1z\nfxfgi2kMXAtz21awTWlqaUrp\nqXkTQEeJSw9Jc+TkWqvavOvTeS\n6FqkFd7A24/OzSc4jteAO1l6d\n5meT61OkFnxCur5xs",
  "eJSw9Jc+TkWqvavOvTeS\n6FqkFd7A24/OzSc4jteAO1l6d\n5meT61OkFnxCur5xsPiQAimFK/\n1oaWPv8LSwsGdXv+X3t3duyv\n319ovtNe9b73vO+9verd97\n4u14Ay/0/vT+8v72/ln+fZktT5\nbfNOp719pzvY6v+XiP1hA6FY\n=</latexit>\na =\n2\n664\na1\na2\na3\na4\n3\n775\n<latexi\nt sha1_base64\n=\"0X5PXmVPHj\nvZgxYItzfAjtf\nWv9",
  "sha1_base64\n=\"0X5PXmVPHj\nvZgxYItzfAjtf\nWv9c=\">AWsn\niclZhbT9xGFI\nCd9JamN9KqvPT\nFKopUVemKTWna\nl0oJhNwgZQksE\nDBZjb1j74Tx2\nPgCS6z9J/01fW\n3/QP9Nz9jenfi\nc4aErEU/O93k\nuZ2Z81Mp8mJ1\n9d8bNz/48KOP\n7n16e3Pv/iy6\n+W7nx9kCdlFv\nBhkMgkO/JZzqV\nQfFiIQvKjNOMs\n9iU/9M82ND+8\n4F",
  "3Pv/iy6\n+W7nx9kCdlFv\nBhkMgkO/JZzqV\nQfFiIQvKjNOMs\n9iU/9M82ND+8\n4FkuErVfXKX8N\nGaREqEIWAGh0d\nIDzw9D93fX83k\nkVOXHrMjEdBa\nOqv7M8+Bwvzn8\nPO4Gi/4aGlt\nbda/1xa6LeF\naf9DUZ3vh174y\nQoY6KQLI8P+m\nvpsVpxbJCBJLP\nbntlzlMWnLGI\nn0BRsZjnp1U9w\nJl7FyJjN0wy+F\nOFW0fP6NicZ\n5fxT6Y0",
  "JLP\nbntlzlMWnLGI\nn0BRsZjnp1U9w\nJl7FyJjN0wy+F\nOFW0fP6NicZ\n5fxT6Y0MFJjpk\nO2thJWYS/nVZC\npWXBVdA0FJbSL\nRJXZ8sdi4wHh\nbyCAgsyAX1gw\nnLWFBATm97il8\nGSRwzyIy3vrk\n7q9o08vOyzu9s\n1nU2a0cn8jpj/\nfn+ohZR8Fi846\nSWtGVXCPwaF\nZVvBf1MBAcgOh\nxAhLFc6hT58cP\n3T6isJ4kYOB+\nMoXOhe6rG",
  "6\nSWtGVXCPwaF\nZVvBf1MBAcgOh\nxAhLFc6hT58cP\n3T6isJ4kYOB+\nMoXOhe6rGalaF\nTyCnHS0Y6JBIZ\nV82rE2iAVTGXe\nUPVBc96rAYc\nFGLj1Ogw4moO9\nlKnZ/LyCT4sr\nnIdwy1kTEW8b\ngKGHDCpR9Q1VC\nklnBp0rD+w9Yq\npszZxSVp3NdMR\nZO1nXafIaF7U\nuOvUEWTBIoy6V\nh1BloTdP2Yxgy\ny35REMOHZ1xK\n4KhVBFuYgS/",
  "O1nXafIaF7U\nuOvUEWTBIoy6V\nh1BloTdP2Yxgy\ny35REMOHZ1xK\n4KhVBFuYgS/x\nu26mO4LU5TWG/\ndL3NiqT/gqGM6\nADsPn0UTAW8q\n28kC9udJ+ei9n\nWBT90JTFb3FJZ\nFzbDmjcCo2ti\nMmnWukEmzBaEs\nueyaujcWlaeiO\n0AdwJuzIQK39\nPu1SVYsjrs3Y\nOhZqXkJz/1fuH\nT02pVbxv9D8km\nVJSXqa0iHf4f\nFY3hfoPXF0Tw5",
  "SVYsjrs3Y\nOhZqXkJz/1fuH\nT02pVbxv9D8km\nVJSXqa0iHf4f\nFY3hfoPXF0Tw5\nCUSTR4E6slLJF\nzf0dSxDC9sHan\nnDgpCMSmK7T\n9RaS659QR3Nk\nRn2FgK4XjkwoN\nMlh2JV1QMtwh\nDunZQEFaJBM8\nZAJnmZcXLxQ+s\nZIrWuL4uZ0Der\n7gVaqF73eBy\ncRaU4eZwa853\nUcZ9Zt8+kmpxi\nxDyZzqKZ2+8f\nICtpht9dT3hS\ntVsTP",
  "73eBy\ncRaU4eZwa853\nUcZ9Zt8+kmpxi\nxDyZzqKZ2+8f\nICtpht9dT3hS\ntVsTPt9r2oF8w\nO2UQ8PRFp6P\niFjUkagueFSx1\niWJZWkP6los1/\nd7Vm29+ZEs7cj\ni2k1J6m17abc\nt7jU94Ofblt5u\nE49Y1JGoraH1\nCOWpT2oy57Hb\ndsoLK7dlKTeR\n6tsVdmGj5h/s\nTXjD9mJTIsX7s\nS6TXhLBYULGw\niknMIyQ2ISzGZ\ndeC/2NlT8",
  "6tsVdmGj5h/s\nTXjD9mJTIsX7s\nS6TXhLBYULGw\niknMIyQ2ISzGZ\ndeC/2NlT8DNo2\ns1ISwOctHVdA\nBLYy7xEJoQFps\nt3DXbGFa3Leq2\nXWUynSCzCWHxK\nYvxqJsQFiMqR\nlbxjKUpEpsQye\nME53FC85hiKbV\nJeEZSy4yQJWV\nbUNk6Uo6gKUp\nam1qaQx6IBOFG\nmyDWM7pysutK0\n+hVazoKh7aGh\n5e03DBUIU6gKU\ndsdcb8e6yX",
  "m1qaQx6IBOFG\nmyDWM7pysutK0\n+hVazoKh7aGh\n5e03DBUIU6gKU\ndsdcb8e6yXyc\nYv16aklyKpCV\n0gQOsDOgzvzpz\nw8r8iTnh1eGXl\nF6aeglpYeGHlK\naGUreCPzwlaH\nk7cQPLwy9oPTA\n0ANKS0NLSoeGD\nikNDQ0pfWLoE\n0oDQwNKNwzdoL\nQwlDyRwh3B0H1\nKJ4ZOKD0y9IjS\n14a+pvSZoc8o\nPTb0mNJ3hr6j9\nJGhjyhlhjJK",
  "lDyRwh3B0H1\nKJ4ZOKD0y9IjS\n14a+pvSZoc8o\nPTb0mNJ3hr6j9\nJGhjyhlhjJKNw\n3dpJQbSj4d+O\nG6oeuU+oaSdz/\nYa4YOKE0NTSl9\nbOhjSseGkrdiu\nJ8ZSh5v4MZoq\nKT0uaHPKRWGkv\nc3P3xp6EtKY0N\njSl8Y+oLSt4a\n+pfSpoU8pjQwl\n3wbg6cTQPUrNV\n6Aqp3TX0F1Kzw\n09t38X4Itp9G\n0Lc8dUsENpYmh\nC6Zah5E0BHi",
  "bg6cTQPUrNV\n6Aqp3TX0F1Kzw\n09t38X4Itp9G\n0Lc8dUsENpYmh\nC6Zah5E0BHiUM\nPSPk6Fqr2rz\nr03kuhaqBbewN\nuPzs0nOQ7XgFt\nZeneZnk+tTqBZ\n8Qrq+ebD4kAI\nphSv9aGmlj7/C\n0sLB/V7/QW9td\n23l4Xr7hfaW8\n53zvfOD03d+dR\n46z5yBM3QC50/\nnL+dv5/lteXj\nZbYcNOrNG+05\n3zid37L8D4+A5\nYA=</latexit>",
  "5yBM3QC50/\nnL+dv5/lteXj\nZbYcNOrNG+05\n3zid37L8D4+A5\nYA=</latexit>\nf =\n2\n4\nf1\nf2\nf3\n3\n5\n<latexit sha1_base64=\"BFfEM\nn6FpZwTaPuSgXnrJTcRZno=\">AYniclZhb9s2FIDd7NZlt3b\nDgAB70RZ0GIbOiNPs8jKgTZreki5XJ2nj1KBkSmZDUYouiVPB7/s1\ne93+yv7NDiXZjM6hAS9Aa/p8n0jq8CJabixFmq2s/Htr4b",
  "kSmZDUYouiVPB7/s1\ne93+yv7NDiXZjM6hAS9Aa/p8n0jq8CJabixFmq2s/Htr4b3P/jwo\n9sfL37y6Wef3Hn7pdHaZQnHu96kYySE5elXArFu5nIJD+JE85CV\n/Jj93xD8+NLnqQiUofZdczPQhYo4QuPZRDq3134tucnzCt6MUsywa\nTc31/3PjKxs7v8MkDoQo3ZFkiRuNFdJXfLzrjG5ex8rvzvUO91Tm\n9B9Tr9eZod3XOdmd5uN3VOdt9M",
  "kDoQo3ZFkiRuNFdJXfLzrjG5ex8rvzvUO91Tm\n9B9Tr9eZod3XOdmd5uN3VOdt9MGe7szc7oM5212bs91Z3hr1Fnt\ncDaZj3b+zvNJeKf8cWujUheVW/bfbv/v1oDeIvDzkKvMkS9PTzkqc\nnRW6DU9yqD5Pecy8cxbwUygqFvL0rCgn89i5B5GB40cJ/FOZU0ZvX\nlGwME2vQxdM6OAwxUwHbew0z/zfzgqh4jzjyqsa8nPpZJGjV4YzE\nAn3Mn",
  "U0ZvX\nlGwME2vQxdM6OAwxUwHbew0z/zfzgqh4jzjyqsa8nPpZJGjV4YzE\nAn3MnkNBeYlAvrqeEMG6cpg/Sz2FL/yojBkJne+uYeJK1aEvwiL9\neSTtNZ7N0dCJnGevPD6e1iIyH4h0nlZSKrmSGwINxUfB20MZAcAC\nizQmIFE+hTp0f13c6iMLeIQEDd6MRdM539sekapXxAHLS0F4TDQq\nx5KOGtUEsGMqwoRyA4j3HA04TEDPKehx9EYHMR",
  "MRdM539sekapXxAHLS0F4TDQq\nx5KOGtUEsGMqwoRyA4j3HA04TEDPKehx9EYHMRMjSfXZXyUJWGR\n6huIWEq4GUTcMseTPJ9bKhcSrjUa1h/YGufqfM6cVFcdjXREWQdJ\nk0nS2he1KDplBFkwSQMmlYZQZaEnX7AQgZrst9uOHQ0RG7KhRWB\nZmYu0nkNtuOdQTPzVEM6XpbRYk/ZcMZUQHYPXpT8GUx5v6RjS1nU\nlyLktfF/jIGcJgNS9hSV",
  "uOdQTPzVEM6XpbRYk/ZcMZUQHYPXpT8GUx5v6RjS1nU\nlyLktfF/jIGcJgNS9hSVDd1qQRuKs6NqZmStk0mxBKImumqbujU\nXlsWjeoA7gRZcnQvk3tPtlCasDvfuw60mueSnP7V/5qOzYkUvG/0\nfySZUlOaxrSId/h8VDeBsgecXRPDgRINHgTKwYsk7O9o6FiCJ7aO\nlGMHBaGYFNk1Wv4iUM1rygjubBSivkJA1wufTCg0yL7flHVAy/AJ\np",
  "o6FiCJ7aO\nlGMHBaGYFNk1Wv4iUM1rygjubBSivkJA1wufTCg0yL7flHVAy/AJ\npyTLBPLQTXrVPXoySvOEk80PzWeIlLreFhOhH1bNDVqoblvcDm9C\nsrwcLjkMy53UbdKp9ulKsBS1AyR3pIR296aQZLzLb6yGvilYr4B\ndbdXvQLxid3P4RX8Lj0dALOpIVBcS61SWJZ2oO6ptP1Zs+KrT\nc/kqkdWFy7KUm9dS/tsWd0QN+sW3p7TbxiEUdi",
  "IVBcS61SWJZ2oO6ptP1Zs+KrT\nc/kqkdWFy7KUm9dS/tsWd0QN+sW3p7TbxiEUdieqe0g9Ylnag7r\nsedy23YXFtZuS1DvJo9W2uFMTX/cMgzpo9JkRzoY18ke1UIixkV\nM6sYhTxAYhXCYpg3LfiOlQMBD4+mVYWwuJuKpqYDWBpwiW+hCmGx\nWsJNs45hduibtVJuMhMqsQFp+yEN91FcJiQMXAKp6zOEZiFSJ5H\nOI8DmkeYyzFNgmPSGwZET",
  "uibtVJuMhMqsQFp+yEN91FcJiQMXAKp6zOEZiFSJ5H\nOI8DmkeYyzFNgmPSGwZETKlbBMqGUZNSQewNEKtjSyNQ9kpFCDdR\nDLKZ15qXmKTSLFZ3FXVvD3RkNZwxVqANY2iFrzOntWBeZi1OsXz\nVYkhwLZMU0gbvY2aXO5PTn+gU5ybn+taHXlF4ZekXpsaHlCaGkl8\nEr9vKPl14vqXhl5SemToEaW5oTmlXUO7lPqG+pQ+MfQJpZ6hHqUb\nhm",
  "HlCaGkl8\nEr9vKPl14vqXhl5SemToEaW5oTmlXUO7lPqG+pQ+MfQJpZ6hHqUb\nhm5QmhlKTqTwRD0kNKhoUNKTw9ofSVoa8ofWboM0pfG/qa0neG\nvqP0kaGPKGWGMko3Dd2klBtKXh24/rqh65S6hpLfrDWDN2lNDY0p\nvSxoY8pHRhKfhXD8xQcryB6OhktLnhj6nVBhKfr+5/ktDX1IaG\nhpS+sLQF5S+NfQtpU8NfUpYCh5NwCnE0MPKDVvg",
  "tLnhj6nVBhKfr+5/ktDX1IaG\nhpS+sLQF5S+NfQtpU8NfUpYCh5NwCnE0MPKDVvgYqU0j1D9yi9MP\nTC/l6AT4fRtU3MHVPBDqWRoRGlW4aSXwpwlD0nJwnfVXvapO3TWR\nf89WUW1id8cnVJOe+mnILq3enydVkf/LVlA9J1zePpi9SIKWw0/f\nvLHfwW1haOFptd35pr+2tLT9cr9/Q3m590/qu9UOr0/q19bD1rLXb\n6ra8hT8X/lr4e+GfJW",
  "aOFptd35pr+2tLT9cr9/Q3m590/qu9UOr0/q19bD1rLXb\n6ra8hT8X/lr4e+GfJWfpydLpZ1KXbhVX/NVq/G3dPIfNi+eUg=<\n/latexit>\n@f\n@a =\n2\n6664\n@f1\n@a1\n@f2\n@a1\n@f3\n@a1\n@f1\n@a2\n@f2\n@a2\n@f3\n@a2\n@f1\n@a3\n@f2\n@a3\n@f3\n@a3\n@f1\n@a4\n@f2\n@a4\n@f4\n@a4\n3\n7775\nVector function \ud835\udc1f \u22c5  of a vector \ud835\udc1a\nVector of scalar \nvalued functions\nColumns are each \nelement function\nRows are each \nvariable element",
  "Comparing vector and matrix\nf0 = \u03b20 + !0x\nh1 = sin[f0]\nf1 = \u03b21 + !1h1\nh2 = exp[f1]\nf2 = \u03b22 + !2h2\nh3 = cos[f2]\nf3 = \u03b23 + !3h3\nh4 = log[f3]\ny = \u03b24 + !4h4\nScalar derivatives:\n<latexit sha1_base64=\"UL2kuIrH0fw\ncv78EJSVz1qpcjbU=\">AW53iclZjLbtw2FEDl9JWmL6dFvelGqBEgbVPDbtLHp\nkBix3nZqZ9jO/E4A0pDaRhTlCxR9jCfEN3Rbf9pC7Jd32UtIMo3vpRQewh3PEU\nldkhKlIJOi0",
  "jO/E4A0pDaRhTlCxR9jCfEN3Rbf9pC7Jd32UtIMo3vpRQewh3PEU\nldkhKlIJOi0MvL/8xde+fd97/4PqHNz76+JNP5u/+flBkZ5yHthKtP8KGAFl0\nLxnhZa8qMs5ywJD8MTtcMPzneSFSta8vM36SsFiJSIRMQ2gwf9qPchZW/YzlWj\nDpR4O7E/trNKjuTib+rz6yiHK7H3DNBne/6cJj6HQxL8xhzYR+DWYX1xeWq4/Pi2\nstIVFr/1sD25+",
  "Tib+rz6yiHK7H3DNBne/6cJj6HQxL8xhzYR+DWYX1xeWq4/Pi2\nstIVFr/1sD25+OewP07BMuNKhZEVxvLKc6ZPKNB1KPrnRLwuesfCUxfwYiolvDi\np6qxM/FsQGfpRmsOf0n4dfuIiVFcZkEYCZMjwrMTNDFjksd/XJSCZWVmquwaSgq\npa9T36TYH4qch1peQoGFuYC+uGIQI1DMSNvuIXYZokTA2r/ur6DuQy4LFQFT8r\n60GZTLrOeu1wKF",
  "1peQoGFuYC+uGIQI1DMSNvuIXYZokTA2r/ur6DuQy4LFQFT8r\n60GZTLrOeu1wKF5lrD7dn9UiNE/EG04qRVTyRUCjydVxZfiJQwEByCWOAGp4gXU\nafITRP4KojAJWDgQTqGzkX+7oRUrTSPIScd7SXRoJBJPu5Ya8SCoUw6yh4ovn/LN\n4DrHEYBugpfHI3BXsbUZHqc5mOdJ1VhYriFnKmY103AKYcw93exoUop4dCwY/2Gr\nV2mTtvEpVnd1",
  "3BXsbUZHqc5mOdJ1VhYriFnKmY103AKYcw93exoUop4dCwY/2Gr\nV2mTtvEpVnd1dxEkLWfdx2d07yoYdepI8iCSRh3rTqCLAmXjCFLGS5LQ/ghBPfRN\nyqUFgVZGJu52nQbTszETw3xmsl63XpH0nzOUEROA1We+BVMh7+pr6cz2p8k5r3\n1T4GN/BIPVPYTlcXNa0bgrNrYhJp1rpBJswWhPL3omqY3DpVnonuCJoAXZkLFb\n2l3alLMGVNuH8",
  "TlcXNa0bgrNrYhJp1rpBJswWhPL3omqY3DpVnonuCJoAXZkLFb\n2l3alLMGVNuH8HTjUvJT/+fulHPj6pls2yMf9INqGiosxcFZnw/6hoCDcpPL8gc\nvlWjwIFAPXirh+o6GjuV4YptIPXZQEIpJoS/R8hex6h5TR3Bn0wT1FQKmXvhmQqF\nBjqKubAJGhm+43TomUIhOMmzOMZRpUeacXPzQfIZIrZvLYi7Mzap7QZVG6F43uJwd\nBW4OZzKw4P",
  "43TomUIhOMmzOMZRpUeacXPzQfIZIrZvLYi7Mzap7QZVG6F43uJwd\nBW4OZzKw4PUEaDJp9BWqohy1Eyx2ZIx6/6hYl5lr9ZA3RacV87ONtj3oF4xO\nGYb8bLCBxyMmFnUkqgv2N86JLEc7UFds+n6ds+qjVfkqkdO1y3KUm9bS/dtsO9\nogf8bNPR203iEYs6EtXV9pB6xHK0B3W587jpOguH6zYlqXeaR6ftcGcmv7R/gh2q\nGablMqh2falst",
  "s6EtXV9pB6xHK0B3W587jpOguH6zYlqXeaR6ftcGcmv7R/gh2q\nGablMqh2falst+EsKipqJ1ivbHtik0Ii0nZteA3VvYE3Dy6VhPC4nYhupoJYGnIJ\nT6FJoTFZgl3zTaG1U2HulWmcxGyGxCWHzMEnzWTQiLMRVjp3jKsgyJTYjkcYTzOK\nJ5zLCUuSQ8IpljRMiUck2ofJR2JRPA0hi1NnY0Bj2QqUINtkEsF3TmFc6Zp9AsVn\nQW91wN965o",
  "pljRMiUck2ofJR2JRPA0hi1NnY0Bj2QqUINtkEsF3TmFc6Zp9AsVn\nQW91wN965oWDNUoQlgaYusMb+/5VxkAU6xXz+T0uESyMpoArexs02d6e4viCqykw\nuiS0svKb2w9ILSQ0sPKc0tJU8EQbRrKXk6CaJzS8pPbD0gNLS0pLSnqU9SiNLI0o\nfWfqI0tDSkNI1S9co1ZaSHSncESzdp3Rk6YjSI0uPKH1h6QtKn1j6hNKXlr6k9I2\nlbyh9YO",
  "SkNI1S9co1ZaSHSncESzdp3Rk6YjSI0uPKH1h6QtKn1j6hNKXlr6k9I2\nlbyh9YOkDSpmljNJ1S9cp5ZaSVwdBtGrpKqWBpeTZD9apduUZpZmlD609CGlQ0vJ\nUzHczywl2xu4MVoqKX1q6VNKhaXk+S2Inlv6nNLE0oTSZ5Y+o/S1pa8pfWzpY0pj\nS8m7AdidWLpHqX0LVBWU7li6Q+mZpWfu9wJ8NoyBa2Ju2Qq2KE0tTSndsJQ8KcBWw\ntJT",
  "didWLpHqX0LVBWU7li6Q+mZpWfu9wJ8NoyBa2Ju2Qq2KE0tTSndsJQ8KcBWw\ntJTsp+MVHtVm75tIte1SM24g7UZnx5Nch6pGXew9uo0PZpcnyI14yPS9fWD2YsUS\nClc6Qfziyv4LSwtHPywtPLT0r2de4v3V9s3tNe9r7yvdveivezd974m17PS/0/\nvb+nfPm5hbEwu8Lfyz82ajX5tpjvA6n4W/gOPjvfG</latexit>@f3\n@h3\n=\n@\n@h3",
  "nfPm5hbEwu8Lfyz82ajX5tpjvA6n4W/gOPjvfG</latexit>@f3\n@h3\n=\n@\n@h3\n(\u03b23 + !3h3) = !3",
  "Comparing vector and matrix\nf0 = \u03b20 + !0x\nh1 = sin[f0]\nf1 = \u03b21 + !1h1\nh2 = exp[f1]\nf2 = \u03b22 + !2h2\nh3 = cos[f2]\nf3 = \u03b23 + !3h3\nh4 = log[f3]\ny = \u03b24 + !4h4\n<latexit sha1\n_base64=\"g1XiOW07MVqJywqE\nv+mr3kz7w=\">AYBHiclZhZb\n9w2EMfX7pW6V9IDfihaCDVSFG\n1qeO30eCmQ2HEuO7Udn4nXWVBa\nSsuYomQd9jrCvrZfpm9FX/s9+i\nX6GTqUtMtwhn7oAvZy5/fnDkc\nSpT8",
  "nXWVBa\nSsuYomQd9jrCvrZfpm9FX/s9+i\nX6GTqUtMtwhn7oAvZy5/fnDkc\nSpT8VIq8WFr6Z2b2jTfevuda\n+/Ovf+Bx9+dP3Gxwd5UmYB3w8\nSmWRHPsu5FIrvF6KQ/CjNOIt9y\nQ/90zXND895lotE7RWXKT+JWa\nREKAJWgKl/Y+Z3r+eHYb9aGntf\n/6LbPi9Y/fM7+LEV86j+Be1RX/\nRUosrY5nX681p8bBfdScd2fHE\n0wnRha/pmgBd",
  "6LbPi9Y/fM7+LEV86j+Be1RX/\nRUosrY5nX681p8bBfdScd2fHE\n0wnRha/pmgBdK0B3PHmCrGMQ\n3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2\nE9GlRwfT/zdutTopH9YWlxqf\n54tNFtGwud9rPdv/HZoDdIgjLm\nqgky/Pj7lJanFQsK0Qg+XiuV+\nY8ZcEpi/gxNBWLeX5S1SUz9m6C\nZeCFSQZ/qvBq6+s9Khbn+",
  "anFQsK0Qg+XiuV+\nY8ZcEpi/gxNBWLeX5S1SUz9m6C\nZeCFSQZ/qvBq6+s9Khbn+WXsg\nzJmxTDHTBtd7Lgswp9PKqHSsuA\nqaAKFpfSKxNP15w1ExoNCXkKDB\nZmAsXrBkGUsKBK53qKXwRJHD\nM1qHqr6zvjqufzSKiKn5V1xY7H\ntma91nBoXqVYfbQ39SIKHotXnD\nipJdrJFQIejauKL0aLGAgOQCx\nyAhLFc/Cp8+OHXhdR2KEScNUA\n1S",
  "SIKHotXnD\nipJdrJFQIejauKL0aLGAgOQCx\nyAhLFc/Cp8+OHXhdR2KEScNUA\n1SC93RMXKuCR5ATS/acyKCRSj6\nyVGtEBUsZW5JdkHjeTU8DXmSwC\njBU+OJoDXZTpsaTfgUfFVlc5d\nqGI2RMRbwOAVMOmNQzshWqlBK6\nBpbqV6x6ytRpm7gkrYeaQtS7W\nW2pshoXtTA1tQWpIijGxVbUE\nqCdfTAYsZLlt92HCsactbqlQW\nCpIYW5niW/",
  "7W\nW2pshoXtTA1tQWpIijGxVbUE\nqCdfTAYsZLlt92HCsactbqlQW\nCpIYW5niW/HTrUF1+Yohf1i69Y\nrkv5zhjKiDbD79LdgKuC2fC2Zq\nr1Jcs5rvW7wkTeExbK7sCxqpj\nUJArNqbWOqrHOFlDRbYMqSC1up\nR+OQ8lTYE9QGvOnKTKjwNdmtug\nUlq829WzDVrJT8+PvFH/jopFr\nS20b/I9kER3mZuhxp8/9wNIA7O\nK4vsODFSyRaPDUi",
  "29WzDVrJT8+PvFH/jopFr\nS20b/I9kER3mZuhxp8/9wNIA7O\nK4vsODFSyRaPDUi5dIuL6jpWM\nZLmxtqdcOGkIxKYpLtP1FpOw+\ntQUPNonRWMGg/cI3EwotchjaYm\n3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF\n83uJz2gjbcHM75Fd19lFG/yae\nflGrAMpTMkV7S0YteXsAWc+3+e\nsmbplMV8bONh6MC1anDAJ+",
  "M75Fd19lFG/yae\nflGrAMpTMkV7S0YteXsAWc+3+e\nsmbplMV8bONh6MC1anDAJ+1t/\nA6xERFdVI5AsOf05fkqgc8cDX\ntFxfH1m18eJbUtqRQ+tWSuK3Ha\nVb7dBeMQJ+tukY7SbRERXVSOSr\nHSHVEZUjHvhy53HTNQuH1q2UxO\n8kj061QztVovIP94ZwnNXHpEQ\nO9LEvkb3GhIUFRZOYaKPxLawM\nWFhXNoq+I0luwJuHraqMWHhdi5\nsm",
  "wnNXHpEQ\nO9LEvkb3GhIUFRZOYaKPxLawM\nWFhXNoq+I0luwJuHraqMWHhdi5\nsmTZg0YBLPIXGhIXNFraVrQ1L\nNx3STbeUyXSIlI0JCx+wGM+6MW\nFhRIWRU3jK0hQJGxPJ4xDncUjz\nmGJR6hLhFUkdK0JKylVQ2TCxR\ndqARSMUbeQIBiOQiUIBWyMW57T\nycmflKVTFilbxvivw/hWBC4Yca\ngMWbZE95vW2nJvMxymGY5Yryal\nAqpQmcB",
  "7T\nycmflKVTFilbxvivw/hWBC4Yca\ngMWbZE95vW2nJvMxymGY5Yryal\nAqpQmcBtrtqlmcvrzw4qc5Pzw\n0tBLSi8MvaD0NBDSjNDyROBHz\n41lDyd+OG5oeUHh6QGlpaEnp\nvqH7lIaGhpTeN/Q+pYGhAaVrh\nq5RWhKTqRwRzB0j9KhoUNKjw\n9ovSZoc8ofWjoQ0qfG/qc0leGv\nqL0rqF3KWGMkrXDV2nlBtKXh3\n4aqhq5T6hpJnP9hrhm",
  "8ofWjoQ0qfG/qc0leGv\nqL0rqF3KWGMkrXDV2nlBtKXh3\n4aqhq5T6hpJnP9hrhm5Tmhqa\nUnrP0HuUDgwlT8VwPzOUHG/gxm\niopPSRoY8oFYaS5zc/fGLoE0pj\nQ2NKHxv6mNKXhr6k9IGhDyiND\nCXvBuB0YugupeYtUJVTumPoDqV\nnhp653wvw6TL6rsLcMg62KE0MT\nSjdMJQ8KcBRwtBTcp4MVXtVm7x\ntIte1UE25g7UZn/QmOQ/VlDt",
  "sLcMg62KE0MT\nSjdMJQ8KcBRwtBTcp4MVXtVm7x\ntIte1UE25g7UZn/QmOQ/VlDtY\ne3Wa9CbXp1BN+ZAMf1g+iIFUg\npX+v71hS5+C0sbB8uL3R8Xb+/c\nXriz2r6hvdb5vPNV5tOt/NT5\n07nYWe7s98JZv6d/XT2i9kv53+\nb/2P+z/m/GunsTNvnk471mf/7P\n0kNWgs=</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2",
  "mf/7P\n0kNWgs=</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\nScalar derivatives:\nMatrix derivatives:\n<latexit sha1_base64=\"V3CYFDq4r+\nPwPuPiQT3UunDjVrg=\">AXDHiclZhbc9w0FIA3XEu5pTDkBR48ZMoUKDsJLZ\ncXZtqk6S0puW6SNpvuyF7Zq0aWHVtONvXsX2D4MbwxvPIf+CM8c2R7V/E",
  "KDsJLZ\ncXZtqk6S0puW6SNpvuyF7Zq0aWHVtONvXsX2D4MbwxvPIf+CM8c2R7V/E5ygM7\nk6z2fJ8l+UiyZfupFLleWvpn7o03r7nXevXf9/Q8+/Oj+Ruf7OdJkQW8Fy\nQyQ59lnMpFO9poSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEBvO/e\n/0wY0HZT1mBZNe3w/DwZ1JKzAalHcmE+8X5LqkvuShvgU/fa6ZiXjfwo/NmEf\nVr",
  "/0wY0HZT1mBZNe3w/DwZ1JKzAalHcmE+8X5LqkvuShvgU/fa6ZiXjfwo/NmEf\nVr6nXz0Q0l+bKi/Rl+XeZDC/uNRdqj4eLSw3hcVO89ka3Phs2B8mQRFzpQPJ8\nvxoeSnVx6XpWSD5Hq/yHnKghMW8SMoKhbz/LisUjfxbkJk6IVJBn9Ke1X08hE\nli/P8IvbBjJke5ZiZoIsdFTr8+bgUKi0V0HdUFhITyeGQdvKDIeaHkBRZkAv\nrqBSMG6dU",
  "vbBjJke5ZiZoIsdFTr8+bgUKi0V0HdUFhITyeGQdvKDIeaHkBRZkAv\nrqBSMG6dUwWtf7ip8HSRwzNSz7K2vbkGqfR0KV/LSoRm4yaTtrlcOheJWx8mRv\nVovQPBavOamkUkwlVwg8mpQl70ZdDAQHILqcgETxHOo0+fFDbxlRmKkSMHA/GU\nPnQm9nQqpWmkeQk5b2gmhQSCUft6xVYsFQxi1lFxTPu+kZwHUGowBdhS+OxmA3\nZWoyPU7zsc7i",
  "keQk5b2gmhQSCUft6xVYsFQxi1lFxTPu+kZwHUGowBdhS+OxmA3\nZWoyPU7zsc7iMjcx3ELGVMSrJuCUA1gaO9hQhZRwaNCyfsXWDlMnTeKStOpqZiL\nI2svajs5oXtSw7VQRZMEkjNpWFUGWhOvKkMUMstyUB3DCsWciblUorAoyMbeyx\nG+3nZoInpvjFNZL21srSfrPGMqICcDqM9+CqYC39dVkZnvT5JxVvinwsTeCwWo\nfwrKoPq1pI3BWT",
  "ZL21srSfrPGMqICcDqM9+CqYC39dVkZnvT5JxVvinwsTeCwWo\nfwrKoPq1pI3BWTWxCzSpXyKTZglCWnLdN0xuHylPRPkETwIuyIQKL2m3qxJMW\nRPu34ZTzQrJj7r/sDHx+WSWTbmH8kmVJQXqasiE/4fFQ3hTobnF0Tw4CUSDR4E\nqsFLJFzf0dCxDE9sE6nGDgpCMSn0BVr+IlLtY6oI7mwSo75CwNQL30woNMh2J\nZNwMjwDfdkxwQK0EkG9",
  "6nGDgpCMSn0BVr+IlLtY6oI7mwSo75CwNQL30woNMh2J\nZNwMjwDfdkxwQK0EkG9TkGMsmLjJOLH5rPEKl0c1nMhLlZtS+o0gjt6waXs6Og\nDeHM37F4T7KqF/n08KNWQZSubYDOn4ZT/XsMRcq78a8rotCJ+ut60B/2C0S\nmCgJ8O1vF4RMSijkR1wSbIWZcklqM9qGs2XS/3rFx/+Q2Z2pHDdZuS1Nv0m073\nCt6wE83HL3dIB6xqCNRXU0PqU",
  "cklqM9qGs2XS/3rFx/+Q2Z2pHDdZuS1Nv0m073\nCt6wE83HL3dIB6xqCNRXU0PqUcsR3tQlzuPG6zcLhuU5J6p3l02g53ZqLpH+6\nNYNtqtkmJHJptXyL7dQiLmoraKSZmc9sW6xAW46JtwW+s7Aq4ebStOoTFrVy0N\nRPA0pBLfAp1CIv1Em6bTQyrGw51w60ymY6QWYew+IjF+KzrEBYjKkZO8YSlKRL\nrEMnjCOdxRPOYil1SXhEUseIkCnlm",
  "60ymY6QWYew+IjF+KzrEBYjKkZO8YSlKRL\nrEMnjCOdxRPOYil1SXhEUseIkCnlmlDZKGlLJoClMWpt7GgMeiAThRpsgljO6c\nzLnTNPoVms6CzuRruXdGwZqhCE8DSJljXn/Tuch8nGLYZrmSnApkpTSBW9jZ\nos509+eHJdnJ+eGFpReUnlt6TumBpQeUZpaSJwI/3LGUPJ34ZmlZ5TuW7pPaW\nFpQWnP0h6loaUhpQ8tfUhpYGlA6aqlq5RqS",
  "paSJwI/3LGUPJ34ZmlZ5TuW7pPaW\nFpQWnP0h6loaUhpQ8tfUhpYGlA6aqlq5RqS8mOFO4Ilu5ROrJ0ROmhpYeUPrf0\nOaWPLX1M6QtLX1D62tLXlN639D6lzFJG6Zqla5RyS8mrAz9csXSFUt9S8uwHa8\n3SLUpTS1NKH1j6gNKhpeSpGO5nlpLtDdwYLZWUPrH0CaXCUvL85ofPLH1GaWxpT\nOlTS59S+srSV5Q+svQRpZGl5N0A7E4s3aXUvg",
  "UPrH0CaXCUvL85ofPLH1GaWxpT\nOlTS59S+srSV5Q+svQRpZGl5N0A7E4s3aXUvgUqc0q3Ld2m9NTSU/d7AT4bRt8\n1MTdtBZuUJpYmlK5bSp4UYCth6QnZT4aquapN3zaR61qoZtzBmoxPjyY5D9WMO\n1hzdZoeTa5PoZrxEen62v7sRQqkFK70g/nFZfwWlhb2v+8u/9i9u3138d5K84b\n2WufzpedW53lzk+de53Hna1OrxN0/p37Yu7m3FcL",
  "lhb2v+8u/9i9u3138d5K84b\n2WufzpedW53lzk+de53Hna1OrxN0/p37Yu7m3FcLvy38sfDnwl+1+sZc8ynd\nZn4e/ANm3Bws=</latexit> @f3\n@h3\n=\n@\n@h3\n(\u03b23 + \u23263h3) = \u2326T\n3\n<latexit sha1_base64=\"UL2kuIrH0fw\ncv78EJSVz1qpcjbU=\">AW53iclZjLbtw2FEDl9JWmL6dFvelGqBEgbVPDbtLHp\nkBix3nZqZ9jO/E4A0pDaRhT",
  "3iclZjLbtw2FEDl9JWmL6dFvelGqBEgbVPDbtLHp\nkBix3nZqZ9jO/E4A0pDaRhTlCxR9jCfEN3Rbf9pC7Jd32UtIMo3vpRQewh3PEU\nldkhKlIJOi0MvL/8xde+fd97/4PqHNz76+JNP5u/+flBkZ5yHthKtP8KGAFl0\nLxnhZa8qMs5ywJD8MTtcMPzneSFSta8vM36SsFiJSIRMQ2gwf9qPchZW/YzlWj\nDpR4O7E/trNKjuTib+rz6yiHK7H",
  "FSta8vM36SsFiJSIRMQ2gwf9qPchZW/YzlWj\nDpR4O7E/trNKjuTib+rz6yiHK7H3DNBne/6cJj6HQxL8xhzYR+DWYX1xeWq4/Pi2\nstIVFr/1sD25+OewP07BMuNKhZEVxvLKc6ZPKNB1KPrnRLwuesfCUxfwYiolvDi\np6qxM/FsQGfpRmsOf0n4dfuIiVFcZkEYCZMjwrMTNDFjksd/XJSCZWVmquwaSgq\npa9T36TYH4qch1peQoGFuYC+uG",
  "FcZkEYCZMjwrMTNDFjksd/XJSCZWVmquwaSgq\npa9T36TYH4qch1peQoGFuYC+uGIQI1DMSNvuIXYZokTA2r/ur6DuQy4LFQFT8r\n60GZTLrOeu1wKF5lrD7dn9UiNE/EG04qRVTyRUCjydVxZfiJQwEByCWOAGp4gXU\nafITRP4KojAJWDgQTqGzkX+7oRUrTSPIScd7SXRoJBJPu5Ya8SCoUw6yh4ovn/LN\n4DrHEYBugpfHI3BXsbUZHqc5mO",
  "rTSPIScd7SXRoJBJPu5Ya8SCoUw6yh4ovn/LN\n4DrHEYBugpfHI3BXsbUZHqc5mOdJ1VhYriFnKmY103AKYcw93exoUop4dCwY/2Gr\nV2mTtvEpVnd1dxEkLWfdx2d07yoYdepI8iCSRh3rTqCLAmXjCFLGS5LQ/ghBPfRN\nyqUFgVZGJu52nQbTszETw3xmsl63XpH0nzOUEROA1We+BVMh7+pr6cz2p8k5r3\n1T4GN/BIPVPYTlcXNa0bgrNrY",
  "sl63XpH0nzOUEROA1We+BVMh7+pr6cz2p8k5r3\n1T4GN/BIPVPYTlcXNa0bgrNrYhJp1rpBJswWhPL3omqY3DpVnonuCJoAXZkLFb\n2l3alLMGVNuH8HTjUvJT/+fulHPj6pls2yMf9INqGiosxcFZnw/6hoCDcpPL8gc\nvlWjwIFAPXirh+o6GjuV4YptIPXZQEIpJoS/R8hex6h5TR3Bn0wT1FQKmXvhmQqF\nBjqKubAJGhm+43TomUIhOMmzO",
  "XZQEIpJoS/R8hex6h5TR3Bn0wT1FQKmXvhmQqF\nBjqKubAJGhm+43TomUIhOMmzOMZRpUeacXPzQfIZIrZvLYi7Mzap7QZVG6F43uJwd\nBW4OZzKw4PUEaDJp9BWqohy1Eyx2ZIx6/6hYl5lr9ZA3RacV87ONtj3oF4xO\nGYb8bLCBxyMmFnUkqgv2N86JLEc7UFds+n6ds+qjVfkqkdO1y3KUm9bS/dtsO9\nogf8bNPR203iEYs6EtXV9pB6xHK",
  "c7UFds+n6ds+qjVfkqkdO1y3KUm9bS/dtsO9\nogf8bNPR203iEYs6EtXV9pB6xHK0B3W587jpOguH6zYlqXeaR6ftcGcmv7R/gh2q\nGablMqh2falst+EsKipqJ1ivbHtik0Ii0nZteA3VvYE3Dy6VhPC4nYhupoJYGnIJ\nT6FJoTFZgl3zTaG1U2HulWmcxGyGxCWHzMEnzWTQiLMRVjp3jKsgyJTYjkcYTzOK\nJ5zLCUuSQ8IpljRMiUck2ofJ",
  "GyGxCWHzMEnzWTQiLMRVjp3jKsgyJTYjkcYTzOK\nJ5zLCUuSQ8IpljRMiUck2ofJR2JRPA0hi1NnY0Bj2QqUINtkEsF3TmFc6Zp9AsVn\nQW91wN965oWDNUoQlgaYusMb+/5VxkAU6xXz+T0uESyMpoArexs02d6e4viCqykw\nuiS0svKb2w9ILSQ0sPKc0tJU8EQbRrKXk6CaJzS8pPbD0gNLS0pLSnqU9SiNLI0o\nfWfqI0tDSkNI1S9co1ZaS",
  "JU8EQbRrKXk6CaJzS8pPbD0gNLS0pLSnqU9SiNLI0o\nfWfqI0tDSkNI1S9co1ZaSHSncESzdp3Rk6YjSI0uPKH1h6QtKn1j6hNKXlr6k9I2\nlbyh9YOkDSpmljNJ1S9cp5ZaSVwdBtGrpKqWBpeTZD9apduUZpZmlD609CGlQ0vJ\nUzHczywl2xu4MVoqKX1q6VNKhaXk+S2Inlv6nNLE0oTSZ5Y+o/S1pa8pfWzpY0pj\nS8m7AdidWLpHqX0LVB",
  "q6VNKhaXk+S2Inlv6nNLE0oTSZ5Y+o/S1pa8pfWzpY0pj\nS8m7AdidWLpHqX0LVBWU7li6Q+mZpWfu9wJ8NoyBa2Ju2Qq2KE0tTSndsJQ8KcBWw\ntJTsp+MVHtVm75tIte1SM24g7UZnx5Nch6pGXew9uo0PZpcnyI14yPS9fWD2YsUS\nClc6Qfziyv4LSwtHPywtPLT0r2de4v3V9s3tNe9r7yvdveivezd974m17PS/0/\nvb+nfPm5hbEwu8Lf",
  "HPywtPLT0r2de4v3V9s3tNe9r7yvdveivezd974m17PS/0/\nvb+nfPm5hbEwu8Lfyz82ajX5tpjvA6n4W/gOPjvfG</latexit>@f3\n@h3\n=\n@\n@h3\n(\u03b23 + !3h3) = !3",
  "Comparing vector and matrix\nf0 = \u03b20 + !0x\nh1 = sin[f0]\nf1 = \u03b21 + !1h1\nh2 = exp[f1]\nf2 = \u03b22 + !2h2\nh3 = cos[f2]\nf3 = \u03b23 + !3h3\nh4 = log[f3]\ny = \u03b24 + !4h4\n<latexit sha1\n_base64=\"g1XiOW07MVqJywqE\nv+mr3kz7w=\">AYBHiclZhZb\n9w2EMfX7pW6V9IDfihaCDVSFG\n1qeO30eCmQ2HEuO7Udn4nXWVBa\nSsuYomQd9jrCvrZfpm9FX/s9+i\nX6GTqUtMtwhn7oAvZy5/fnDkc\nSpT8",
  "nXWVBa\nSsuYomQd9jrCvrZfpm9FX/s9+i\nX6GTqUtMtwhn7oAvZy5/fnDkc\nSpT8VIq8WFr6Z2b2jTfevuda\n+/Ovf+Bx9+dP3Gxwd5UmYB3w8\nSmWRHPsu5FIrvF6KQ/CjNOIt9y\nQ/90zXND895lotE7RWXKT+JWa\nREKAJWgKl/Y+Z3r+eHYb9aGntf\n/6LbPi9Y/fM7+LEV86j+Be1RX/\nRUosrY5nX681p8bBfdScd2fHE\n0wnRha/pmgBd",
  "6LbPi9Y/fM7+LEV86j+Be1RX/\nRUosrY5nX681p8bBfdScd2fHE\n0wnRha/pmgBdK0B3PHmCrGMQ\n3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2\nE9GlRwfT/zdutTopH9YWlxqf\n54tNFtGwud9rPdv/HZoDdIgjLm\nqgky/Pj7lJanFQsK0Qg+XiuV+\nY8ZcEpi/gxNBWLeX5S1SUz9m6C\nZeCFSQZ/qvBq6+s9Khbn+",
  "anFQsK0Qg+XiuV+\nY8ZcEpi/gxNBWLeX5S1SUz9m6C\nZeCFSQZ/qvBq6+s9Khbn+WXsg\nzJmxTDHTBtd7Lgswp9PKqHSsuA\nqaAKFpfSKxNP15w1ExoNCXkKDB\nZmAsXrBkGUsKBK53qKXwRJHD\nM1qHqr6zvjqufzSKiKn5V1xY7H\ntma91nBoXqVYfbQ39SIKHotXnD\nipJdrJFQIejauKL0aLGAgOQCx\nyAhLFc/Cp8+OHXhdR2KEScNUA\n1S",
  "SIKHotXnD\nipJdrJFQIejauKL0aLGAgOQCx\nyAhLFc/Cp8+OHXhdR2KEScNUA\n1SC93RMXKuCR5ATS/acyKCRSj6\nyVGtEBUsZW5JdkHjeTU8DXmSwC\njBU+OJoDXZTpsaTfgUfFVlc5d\nqGI2RMRbwOAVMOmNQzshWqlBK6\nBpbqV6x6ytRpm7gkrYeaQtS7W\nW2pshoXtTA1tQWpIijGxVbUE\nqCdfTAYsZLlt92HCsactbqlQW\nCpIYW5niW/",
  "7W\nW2pshoXtTA1tQWpIijGxVbUE\nqCdfTAYsZLlt92HCsactbqlQW\nCpIYW5niW/HTrUF1+Yohf1i69Y\nrkv5zhjKiDbD79LdgKuC2fC2Zq\nr1Jcs5rvW7wkTeExbK7sCxqpj\nUJArNqbWOqrHOFlDRbYMqSC1up\nR+OQ8lTYE9QGvOnKTKjwNdmtug\nUlq829WzDVrJT8+PvFH/jopFr\nS20b/I9kER3mZuhxp8/9wNIA7O\nK4vsODFSyRaPDUi",
  "29WzDVrJT8+PvFH/jopFr\nS20b/I9kER3mZuhxp8/9wNIA7O\nK4vsODFSyRaPDUi5dIuL6jpWM\nZLmxtqdcOGkIxKYpLtP1FpOw+\ntQUPNonRWMGg/cI3EwotchjaYm\n3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF\n83uJz2gjbcHM75Fd19lFG/yae\nflGrAMpTMkV7S0YteXsAWc+3+e\nsmbplMV8bONh6MC1anDAJ+",
  "M75Fd19lFG/yae\nflGrAMpTMkV7S0YteXsAWc+3+e\nsmbplMV8bONh6MC1anDAJ+1t/\nA6xERFdVI5AsOf05fkqgc8cDX\ntFxfH1m18eJbUtqRQ+tWSuK3Ha\nVb7dBeMQJ+tukY7SbRERXVSOSr\nHSHVEZUjHvhy53HTNQuH1q2UxO\n8kj061QztVovIP94ZwnNXHpEQ\nO9LEvkb3GhIUFRZOYaKPxLawM\nWFhXNoq+I0luwJuHraqMWHhdi5\nsm",
  "wnNXHpEQ\nO9LEvkb3GhIUFRZOYaKPxLawM\nWFhXNoq+I0luwJuHraqMWHhdi5\nsmTZg0YBLPIXGhIXNFraVrQ1L\nNx3STbeUyXSIlI0JCx+wGM+6MW\nFhRIWRU3jK0hQJGxPJ4xDncUjz\nmGJR6hLhFUkdK0JKylVQ2TCxR\ndqARSMUbeQIBiOQiUIBWyMW57T\nycmflKVTFilbxvivw/hWBC4Yca\ngMWbZE95vW2nJvMxymGY5Yryal\nAqpQmcB",
  "7T\nycmflKVTFilbxvivw/hWBC4Yca\ngMWbZE95vW2nJvMxymGY5Yryal\nAqpQmcBtrtqlmcvrzw4qc5Pzw\n0tBLSi8MvaD0NBDSjNDyROBHz\n41lDyd+OG5oeUHh6QGlpaEnp\nvqH7lIaGhpTeN/Q+pYGhAaVrh\nq5RWhKTqRwRzB0j9KhoUNKjw\n9ovSZoc8ofWjoQ0qfG/qc0leGv\nqL0rqF3KWGMkrXDV2nlBtKXh3\n4aqhq5T6hpJnP9hrhm",
  "8ofWjoQ0qfG/qc0leGv\nqL0rqF3KWGMkrXDV2nlBtKXh3\n4aqhq5T6hpJnP9hrhm5Tmhqa\nUnrP0HuUDgwlT8VwPzOUHG/gxm\niopPSRoY8oFYaS5zc/fGLoE0pj\nQ2NKHxv6mNKXhr6k9IGhDyiND\nCXvBuB0YugupeYtUJVTumPoDqV\nnhp653wvw6TL6rsLcMg62KE0MT\nSjdMJQ8KcBRwtBTcp4MVXtVm7x\ntIte1UE25g7UZn/QmOQ/VlDt",
  "sLcMg62KE0MT\nSjdMJQ8KcBRwtBTcp4MVXtVm7x\ntIte1UE25g7UZn/QmOQ/VlDtY\ne3Wa9CbXp1BN+ZAMf1g+iIFUg\npX+v71hS5+C0sbB8uL3R8Xb+/c\nXriz2r6hvdb5vPNV5tOt/NT5\n07nYWe7s98JZv6d/XT2i9kv53+\nb/2P+z/m/GunsTNvnk471mf/7P\n0kNWgs=</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2",
  "mf/7P\n0kNWgs=</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\nScalar derivatives:\nMatrix derivatives:\n<latexit sha1_base64=\"/x7p9\nRpc72xRw18nLvEnH/fdg=\">AW5HiclZhJb9w2FICVdEvTzWlR\nX3oRagQouh2ky6XAIkdZ7NTr2M78TgDSkNpGFOULFH2OML8g96K\nXvuTeu/6LVFHyXN0HqPnQ",
  "h2ky6XAIkdZ7NTr2M78TgDSkNpGFOULFH2OML8g96K\nXvuTeu/6LVFHyXN0HqPnQAezjv+0RSj6REKcikKPTS0l/Xr/19\njvnfj/ZsfPjRx5/M3fp0v0jLPOS9MJVpfhiwgkuheE8Lflhln\nOWBJIfBCerh+c8bwQqdrTFxk/TlisRCRCpiE0mBv1o5yFVT9juR\nZM+tGgujOZ2N/9gGtWx+75XfWylCY8ZoM7k8a+80bgMNG5p9/z18\nezC0sLS",
  "juR\nZM+tGgujOZ2N/9gGtWx+75XfWylCY8ZoM7k8a+80bgMNG5p9/z18\nezC0sLS7VH58WltvCgtd+tga3Ph/2h2lYJlzpULKiOFpeyvRxZdoM\nJZ/c7JcFz1h4wmJ+BEXFEl4cV3VKJv5tiAz9KM3hT2m/jl4+omJ\nUVwkAZgJ06MCMxN0saNSRz8fV0JlpeYqbBqKSunr1Df59Yci56GWF\n1BgYS6gr34YpA4DaNws6/4eZgmCVPDqr+ytg1JD",
  "eYqbBqKSunr1Df59Yci56GWF\n1BgYS6gr34YpA4DaNws6/4eZgmCVPDqr+ytg1JDHgsVMVPy3pEJp\nOus1Y7HIpXGStP92a1CM0T8YaTSmrFVHKFwONJVfHFeBEDwQGIRU5\nAqngBdZr8BJG/jCjMQAkYeJCOoXORvzMhVSvNY8hJR3tJNChko8\n71iqxYCiTjrILiu/f9g3gOodRgK7CF0djsJsxNZkep/lY50lVmBhu\nIWcq5nUTcMohTPodbK",
  "TjrILiu/f9g3gOodRgK7CF0djsJsxNZkep/lY50lVmBhu\nIWcq5nUTcMohTPodbKhSjg07Fi/YGuHqZM2cWlWdzU3EWTt5V1H5\nzQvath16giyYBLGXauOIEvC9WLIEgZbsDOHENxG3KhRWBZmYW\n3kadNvOTATPzXEG6XrVUk/WcMZcQEYPWZb8FUyLv6ajqz/Wlyzm\nrfFPjYH8FgdQ9hedyc1rQROKs2NqFmnStk0mxBKE/Pu6bpjUPlmei\ne",
  "qz/Wlyzm\nrfFPjYH8FgdQ9hedyc1rQROKs2NqFmnStk0mxBKE/Pu6bpjUPlmei\neoAngRVfmQkWXtG/rEkxZE+5/C6eal5Ifbf4Ax8fV0tm2Zh/JtQ\nUVFmropM+H9UNIQ7FJ5fEMGDl0o0eBCoBy+VcH1HQ8dyPLFNpB47\nKAjFpNAXaPmLWHWPqSO4s2mC+goBUy98M6HQIEdRVzYBI8M3GsdE\nyhEJxk25xjKtChzTi5+aD5DpNbNZTEX5mbV",
  "+goBUy98M6HQIEdRVzYBI8M3GsdE\nyhEJxk25xjKtChzTi5+aD5DpNbNZTEX5mbVvaBKI3SvG1zOjoIy3B\nzO+BWHByijQZPIC3VkOUomWMzpONX/ULDEnOt/nrIm6LTivnpet\nse9AtGpwxDfjpYx+MRE4s6EtUFmxtnXZJYjvagrtl0vdyzav3V12R\nqxw7XbUpSb9tLt+1wr+gBP91w9HaDeMSijkR1tT2kHrEc7UFd7jxu\nuM7C4bpNSeqd5",
  "7XbUpSb9tLt+1wr+gBP91w9HaDeMSijkR1tT2kHrEc7UFd7jxu\nuM7C4bpNSeqd5tFpO9yZiaZ/tDeCjanZJqVyaLZ9qew3ISxqKmqn\nWO9wu2ITwmJSdi34jZVdATePrtWEsLhViK5mAlgacolPoQlhsVnCX\nbONYXDoW64VSazETKbEBYfswSfdRPCYkzF2CmesCxDYhMieRzhPI\n5oHjMsZS4Jj0jmGBEypVwTKh+lXckEsDRGrY0djUEPZKp",
  "2CmesCxDYhMieRzhPI\n5oHjMsZS4Jj0jmGBEypVwTKh+lXckEsDRGrY0djUEPZKpQg20QywW\ndeYVz5ik0ixWdxT1Xw70rGtYMVWgCWNoka8zvbzoXWYBT3Dy40uE\nSyMpoArews0Wd6e4viCqykwuiC0svKD239JzSA0sPKM0tJU8EQbRj\nKXk6CaIzS8o3bd0n9LS0pLSnqU9SiNLI0ofWfqI0tDSkNJVS1cp1\nZaSHSncESzdo3Rk6YjSQ0sP",
  "bd0n9LS0pLSnqU9SiNLI0ofWfqI0tDSkNJVS1cp1\nZaSHSncESzdo3Rk6YjSQ0sPKX1h6QtKn1j6hNKXlr6k9I2lbyh9Y\nOkDSpmljNI1S9co5ZaSVwdBtGLpCqWBpeTZD9apVuUZpZmlD609C\nGlQ0vJUzHczywl2xu4MVoqKX1q6VNKhaXk+S2Inlv6nNLE0oTSZ5Y\n+o/S1pa8pfWzpY0pjS8m7AdidWLpLqX0LVBWUblu6Tempafu9wJ8\nNo",
  "E0oTSZ5Y\n+o/S1pa8pfWzpY0pjS8m7AdidWLpLqX0LVBWUblu6Tempafu9wJ8\nNoyBa2Ju2go2KU0tTSldt5Q8KcBWwtITsp+MVHtVm75tIte1SM24\ng7UZnx5Nch6pGXew9uo0PZpcnyI14yPS9bX92YsUSClc6QdzC8v4L\nSwt7H+/uPzj4t3tuwv3V9o3tDe8L7wva+8Ze8n73xNvyel7o/e\nn97f3j/Tsfzf86/9v87416/Vp7zGde5zP/x3",
  "tDe8L7wva+8Ze8n73xNvyel7o/e\nn97f3j/Tsfzf86/9v87416/Vp7zGde5zP/x3/pD/m8</latexit>@f3\n@\u03b23\n=\n@\n@!3\n\u03b23 + !3h3 = 1\n<latexit sha1_base64=\"1XwlPuZS2V\nTaACUSuS8Tb9Ldi8U=\">AW9HiclZhbc9w0FIA3UKAUWlIY8sKLh0xnCpRMQs\nvlpTNt0rRNk5LNZO02TQje2WvGl2fNls6tl/whvDK/+HF34LR7Z3VZ2jPLAz",
  "MQs\nvlpTNt0rRNk5LNZO02TQje2WvGl2fNls6tl/whvDK/+HF34LR7Z3VZ2jPLAzq\ndXzfZbkI8mW7adS5MXy8j9zH3x47aOP7n+6Y3Pr9564v5218e5EmZBbwXJDL\nJjnyWcykU7xWikPwozTiLfckP/bM1zQ9HPMtFovaLy5SfxCxSIhQBKyB0Oj/uhx\nkLqn7KskIw6fX9MDyt7k8mVsjnBaujDz3btzQt3Z/cnfr3f4DSdsyj+lQoD/Xx\nO+",
  "Lqn7KskIw6fX9MDyt7k8mVsjnBaujDz3btzQt3Z/cnfr3f4DSdsyj+lQoD/Xx\nO+hrnHjdH5xeWm5/nm0sNIWFjvtr3t6+tBf5AEZcxVEUiW58cry2lxUunmA8\nknN/plzlMWnLGIH0NRsZjnJ1Wdol3ByIDL0wy+FOFV0fP6NicZ5fxj6YMSuGO\nWY6GLHZRH+dlIJlZYFV0HTUFhKr0g8nW9vIDIeFPISCizIBPTVC4YMcljAqNz\noK34RJHM1",
  "RH+dlIJlZYFV0HTUFhKr0g8nW9vIDIeFPISCizIBPTVC4YMcljAqNz\noK34RJHM1KDqr67vQD59HglV8fOyHqHJxHbWa4dD8SpjdWN/VosoeCzecVJre\nhKrhB4NKkqvhQtYSA4ALHECUgUz6FOnR8/9FYQhRkpAQP3kzF0LvR2J6RqVfAI\ncmJpr4kGhVTysWtEQuGMraUPVA8746nAS8yGAXoKhw4GoO9lKnJ9LyCj4srnI\ndwy1kTEW8bgIuO",
  "sWtEQuGMraUPVA8746nAS8yGAXoKhw4GoO9lKnJ9LyCj4srnI\ndwy1kTEW8bgIuOYD5v4sNVUoJpwaW9Tu2dpk6axOXpHVXMx1B1n5mO0VG86IGt\nlNHkAWTMLKtOoIsCfePAYsZLktn8IFx56OuFWhsCrIxOxmiW+3neoInpvjFNaL\n7a1XJP0jhjKiA7D69FEwFXBbX0tmtjdNzqj2dYGPvSEMln0Ky6LmsqaNwFW1sQ\nk161whk2YLQlyYZu6",
  "9FEwFXBbX0tmtjdNzqj2dYGPvSEMln0Ky6LmsqaNwFW1sQ\nk161whk2YLQlyYZu6Nw6Vp8K+QB3Ai67MhArf0+7VJZiyOty/B5ealZIf/7j0M\nx+fVMt62eh/SDahorxMXRXp8P+oaABPLDy/IHL5Fo8CBQD14i4f6Oho5leGL\nrSD12UBCKSVFcouUvImWfU0dwZ5MY9RUCul4MqHQIehLeuAluEIz17HBArQRQ\nbNQYycuMk5sfms8QqXV9W8y",
  "U0dwZ5MY9RUCul4MqHQIehLeuAluEIz17HBArQRQ\nbNQYycuMk5sfms8QqXV9W8yEfljZN1SpBfu+weXsLCjDw2HErzjdRxn1m3z6\nSakGLEPJHOshHb/p5wUsMdfqr4e8KTqtiJ9vtu1Bv2B0yiDg56ebeDwiYlFHor\npgs+OsSxL0R7UNZu7/es2nzPZnakcN1m5LU2/bSbTvcK3rAz7cvd0iHrGoI\n1FdbQ+pRyxHe1CXO49brqtwuG5Tknqn",
  "kcN1m5LU2/bSbTvcK3rAz7cvd0iHrGoI\n1FdbQ+pRyxHe1CXO49brqtwuG5TknqneXTaDndmoukf7g9h6q3SYkc6G1fIvt\nNCIsFQunmOiNri02ISzGpW3B/7GyJ+DhYVtNCIvdXNiaDmBpwCW+hCaExWYJ2\nYbw+qWQ91yq0ymQ2Q2ISw+YzG+6iaExYiKkVM8Y2mKxCZE8jEeRzSPKZYSl0S\nHpHUMSJkSrkmVDZMbEkHsDRGrY0djUEPZKJQg",
  "VM8Y2mKxCZE8jEeRzSPKZYSl0S\nHpHUMSJkSrkmVDZMbEkHsDRGrY0djUEPZKJQg20Qyzmdeblz5ik0ixWdxT1Xw70\nrGi4YqlAHsLRN1pjX3YuMh+nuHk9pcMlkJXSBHax06XOdPfnhxXZyfnhpaGXl\nF4YekHpoaGHlGaGkjcCP9w1lLyd+OHI0BGlB4YeUFoaWlLaM7RHaWhoSOlTQ59S\nGhgaULpm6BqlhaFkRwpPBEP3KR0aOqT0yNAjSl8",
  "UFoaWlLaM7RHaWhoSOlTQ59S\nGhgaULpm6BqlhaFkRwpPBEP3KR0aOqT0yNAjSl8Z+orS54Y+p/S1oa8pfWfoO0\nofG/qYUmYo3Td0HVKuaHk04Efrhq6SqlvKHn3g7VmaJfS1NCU0ieGPqF0YCh5K\n4bnmaFkewMPRkMlpRuGblAqDCXvb3740tCXlMaGxpS+MPQFpW8NfUvpM0OfURo\nZSr4NwO7E0D1KzVegKqd0x9AdSs8NPXd/F+CzYfRdE",
  "S+MPQFpW8NfUvpM0OfURo\nZSr4NwO7E0D1KzVegKqd0x9AdSs8NPXd/F+CzYfRdE3PbVLBNaWJoQumoeRNAb\nYShp6R/WSo2rva9GsTua+FasYdrM349GyS81DNuIO1d6fp2eT+FKoZH5Kurx/M\nPqRASuFOfzq/uIK/wtLCwU9LK78sPdh5sPhotf1Ce73zTefbzt3OSufXzqPO80\n630+sEnX/nrs3dnLu1MFr4Y+HPhb8a9YO59pyvOtZv4e/",
  "Tefbzt3OSufXzqPO80\n630+sEnX/nrs3dnLu1MFr4Y+HPhb8a9YO59pyvOtZv4e/APkL/P8=</latexit\n> @f3\n@\u03b23\n=\n@\n@\u03b23\n(\u03b23 + \u23263h3) = I",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "The forward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n1. Write this as a series of \nintermediate calculations",
  "The forward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKD",
  "BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY",
  "LJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzou",
  "fwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL",
  "JyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19N",
  "ab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2Fg",
  "Mw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16",
  "g37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0i",
  "aCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGd",
  "+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0",
  "ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P",
  "YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY",
  "Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY9niczZhb9s2FIDdZJcuzTdMCDAXoQFHbqhC+K2u7wMaJOmt6TL1UnaODUomZLZ\nUJQiUYlTwX9lb8Ne93f2tp+yQ8k2o3OYQP2MAOp2fN9vB2Skiw/lSLXy8t/XJuZfefd96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM5",
  "d96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEejd\nnLrphxoKym7JMCya9LpeyV4rR6FLID8Ne2R6NvrJw7YfDnrlXadN1ZCqde25ruShvu1s/B5t/K678X\nukahLRnzF/O6NupmIBvpr9ud+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/v",
  "d+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/vdfhIUMVc6kCzPj9rLqT4uTeuB5DCtIucpC05YxI+gqFjM8+OyOiQj7xZE+l6YZPCntFdFL\n9coWZznF7EPZsz0IMfMBF3sqNDhj8elUGmhuQrqjsJCejrxzInz+iLjgZYXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+",
  "YXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+JVxsqzvWkrQvNYvOWkUoxjVwh8GhUlnwpWsJA\ncABiROQKJ5DmyY/fui1EYVrkgQM3E+GMLjQ2xmRpXmEeSkob0iGhRSyYcNa5VYsJRxQ9kFxfNueQZ\nwncEqwFDhi6M12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmg",
  "12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmgqy\n9rOnojOZF9ZtOFUEWbMKoaVURZEm4g/RZzCDL43IPJhx7JuJWhcKqIBtzK0v8Zt+pieC9OUzhvDS9tZ\nKk/4yhjJgAnD7zLZgKeFNfTa2N0nOWeWbAh96A1isZhWRfW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0D",
  "fW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0DU80KyY+XfqOD4/LZXNszD8km9BQXqSuhkz4XzTUh2cWvL8g\nhcvkWjxIFAtXiLh+o6WjmV4Y5tItXZQEIpJoS/Q8ReRatapIniwSYzGCgHTLnwzodAih2FTNgEjwzc8f\nTk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZ",
  "Tk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZdSv8+knheqzDCVz\naJZ0+LqbazhirtNfLXldFoRP10f9wfjgtUpgoCf9tbxekTEo5EbcHjrMtSxHf9DWdLteHlm5/vo\nbsrUjh+s2JWl3PEq37XCvGAE/3XCMdoN4xKORG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3U",
  "RG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3UIi5qK2ikmMY+QWIewGBdNC/6PlV0BN4+mVYewuJWLpmYCWOpziadQh7BYH+\nGmOY5hdcOhbrhVJtMBMusQFp+wGM+6DmExomLkFE9YmiKxDpE8DnAeBzSPKZSl4RXJHWsCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61g",
  "sCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61gw1aAJY2iRnzOtuOg+Zj1Mj1muJ\nKcCWSlN4BZ2tqgzefrzw5I8yfnhaUXlJ5bek7pgaUHlGaWkl8EfrhjKfl14odnlp5Rum/pPqWFpQWlH\nUs7lIaWhpQ+tvQxpYGlAaWrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPK",
  "Wrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPKWMkrXLF2jlFtKXh34YqlK5T6lpLfnDWLN2iNLU0pfSRpY8o7VtKfhXD/cxS8ngDN0ZLJaXP\nLH1GqbCU/H7zwxeWvqA0tjSm9Lmlzyl9Y+kbSp9Y+oTSyFLybgCeTizdpdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlT",
  "pdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlTjq9rkbRO5roVqyh1snPFJbZLzUE25g42vTpPa5P\noUqikfkKGv7U9fpEBK4Urfm19s47ewtLB/d6n9/dL97fuLD1bGb2ivt75ofdm63Wq3fmg9aD1tbU6rW\nDmz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n=",
  "mz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n= @h2\n@f1\n@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n@f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9C",
  "64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87",
  "wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UO",
  "uCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgw",
  "AYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4",
  "KYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkR",
  "KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J",
  "XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRF",
  "WLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLS",
  "UmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfU",
  "NLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfd",
  "n8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKD",
  "BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY",
  "LJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzou",
  "fwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL",
  "JyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19N",
  "ab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2Fg",
  "Mw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16",
  "g37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0i",
  "aCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGd",
  "+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0",
  "ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P",
  "YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY",
  "Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY9niczZhb9s2FIDdZJcuzTdMCDAXoQFHbqhC+K2u7wMaJOmt6TL1UnaODUomZLZ\nUJQiUYlTwX9lb8Ne93f2tp+yQ8k2o3OYQP2MAOp2fN9vB2Skiw/lSLXy8t/XJuZfefd96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM5",
  "d96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEejd\nnLrphxoKym7JMCya9LpeyV4rR6FLID8Ne2R6NvrJw7YfDnrlXadN1ZCqde25ruShvu1s/B5t/K678X\nukahLRnzF/O6NupmIBvpr9ud+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/v",
  "d+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/vdfhIUMVc6kCzPj9rLqT4uTeuB5DCtIucpC05YxI+gqFjM8+OyOiQj7xZE+l6YZPCntFdFL\n9coWZznF7EPZsz0IMfMBF3sqNDhj8elUGmhuQrqjsJCejrxzInz+iLjgZYXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+",
  "YXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+JVxsqzvWkrQvNYvOWkUoxjVwh8GhUlnwpWsJA\ncABiROQKJ5DmyY/fui1EYVrkgQM3E+GMLjQ2xmRpXmEeSkob0iGhRSyYcNa5VYsJRxQ9kFxfNueQZ\nwncEqwFDhi6M12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmg",
  "12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmgqy\n9rOnojOZF9ZtOFUEWbMKoaVURZEm4g/RZzCDL43IPJhx7JuJWhcKqIBtzK0v8Zt+pieC9OUzhvDS9tZ\nKk/4yhjJgAnD7zLZgKeFNfTa2N0nOWeWbAh96A1isZhWRfW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0D",
  "fW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0DU80KyY+XfqOD4/LZXNszD8km9BQXqSuhkz4XzTUh2cWvL8g\nhcvkWjxIFAtXiLh+o6WjmV4Y5tItXZQEIpJoS/Q8ReRatapIniwSYzGCgHTLnwzodAih2FTNgEjwzc8f\nTk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZ",
  "Tk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZdSv8+knheqzDCVz\naJZ0+LqbazhirtNfLXldFoRP10f9wfjgtUpgoCf9tbxekTEo5EbcHjrMtSxHf9DWdLteHlm5/vo\nbsrUjh+s2JWl3PEq37XCvGAE/3XCMdoN4xKORG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3U",
  "RG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3UIi5qK2ikmMY+QWIewGBdNC/6PlV0BN4+mVYewuJWLpmYCWOpziadQh7BYH+\nGmOY5hdcOhbrhVJtMBMusQFp+wGM+6DmExomLkFE9YmiKxDpE8DnAeBzSPKZSl4RXJHWsCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61g",
  "sCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61gw1aAJY2iRnzOtuOg+Zj1Mj1muJ\nKcCWSlN4BZ2tqgzefrzw5I8yfnhaUXlJ5bek7pgaUHlGaWkl8EfrhjKfl14odnlp5Rum/pPqWFpQWlH\nUs7lIaWhpQ+tvQxpYGlAaWrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPK",
  "Wrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPKWMkrXLF2jlFtKXh34YqlK5T6lpLfnDWLN2iNLU0pfSRpY8o7VtKfhXD/cxS8ngDN0ZLJaXP\nLH1GqbCU/H7zwxeWvqA0tjSm9Lmlzyl9Y+kbSp9Y+oTSyFLybgCeTizdpdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlT",
  "pdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlTjq9rkbRO5roVqyh1snPFJbZLzUE25g42vTpPa5P\noUqikfkKGv7U9fpEBK4Urfm19s47ewtLB/d6n9/dL97fuLD1bGb2ivt75ofdm63Wq3fmg9aD1tbU6rW\nDmz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n=",
  "mz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n= @h2\n@f1\n@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n@f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9C",
  "64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87",
  "wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UO",
  "uCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgw",
  "AYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4",
  "KYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkR",
  "KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J",
  "XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRF",
  "WLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLS",
  "UmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfU",
  "NLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfd",
  "n8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3",
  "Yikes!\n\u2022 But:\n\u2022 Quite similar to:\n<latexit sha1_base64=\n\"V3CYFDq4r+PwPuPiQT3UunDjVrg=\">AXDHiclZh\nbc9w0FIA3XEu5pTDkBR48ZMoUKDsJLZcXZtqk6S0pu\nW6SNpvuyF7Zq0aWHVtONvXsX2D4MbwxvPIf+CM8c2R\n7V/E5ygM7k6z2fJ8l+UiyZfupFLleWvpn7o03r7n\nXevXf9/Q8+/Oj+Ruf7OdJkQW8FyQyQ59lnMpFO9\npoSU/TDPOYl/yA/9k1fC",
  "r7n\nXevXf9/Q8+/Oj+Ruf7OdJkQW8FyQyQ59lnMpFO9\npoSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhF\nwDSEBvO/e/0wY0HZT1mBZNe3w/DwZ1JKzAalHcmE+\n8X5LqkvuShvgU/fa6ZiXjfwo/NmEfVr6nXz0Q0l+b\nKi/Rl+XeZDC/uNRdqj4eLSw3hcVO89ka3Phs2B8mQR\nFzpQPJ8vxoeSnVx6XpWSD5Hq/yHnKghMW8SMoKhbz\n/",
  "w3hcVO89ka3Phs2B8mQR\nFzpQPJ8vxoeSnVx6XpWSD5Hq/yHnKghMW8SMoKhbz\n/LisUjfxbkJk6IVJBn9Ke1X08hEli/P8IvbBjJke5\nZiZoIsdFTr8+bgUKi0V0HdUFhITyeGQdvKDIeaHk\nBRZkAvrqBSMG6dUwWtf7ip8HSRwzNSz7K2vbkGqfR\n0KV/LSoRm4yaTtrlcOheJWx8mRvVovQPBavOamkUkw\nlVwg8mpQl70ZdDAQHILqcgETx",
  "LSoRm4yaTtrlcOheJWx8mRvVovQPBavOamkUkw\nlVwg8mpQl70ZdDAQHILqcgETxHOo0+fFDbxlRmKkSM\nHA/GUPnQm9nQqpWmkeQk5b2gmhQSCUft6xVYsFQxi1\nlFxTPu+kZwHUGowBdhS+OxmA3ZWoyPU7zsc7iMjcx\n3ELGVMSrJuCUA1gaO9hQhZRwaNCyfsXWDlMnTeKStO\npqZiLI2svajs5oXtSw7VQRZMEkjNpWFUGWhOvKkMUM\nsty",
  "NCyfsXWDlMnTeKStO\npqZiLI2svajs5oXtSw7VQRZMEkjNpWFUGWhOvKkMUM\nstyUB3DCsWciblUorAoyMbeyxG+3nZoInpvjFNZL21\nsrSfrPGMqICcDqM9+CqYC39dVkZnvT5JxVvinwsTeC\nwWofwrKoPq1pI3BWTWxCzSpXyKTZglCWnLdN0xuHyl\nPRPkETwIuyIQKL2m3qxJMWRPu34ZTzQrJj7r/sD\nHx+WSWTbmH8kmVJQXqasiE/4fF",
  "PkETwIuyIQKL2m3qxJMWRPu34ZTzQrJj7r/sD\nHx+WSWTbmH8kmVJQXqasiE/4fFQ3hTobnF0Tw4CUSD\nR4EqsFLJFzf0dCxDE9sE6nGDgpCMSn0BVr+IlLtY6o\nI7mwSo75CwNQL30woNMh2JZNwMjwDfdkxwQK0EkG9\nTkGMsmLjJOLH5rPEKl0c1nMhLlZtS+o0gjt6waXs6O\ngDeHM37F4T7KqF/n08KNWQZSubYDOn4ZT/XsMRc\nq78a8ro",
  "ZtS+o0gjt6waXs6O\ngDeHM37F4T7KqF/n08KNWQZSubYDOn4ZT/XsMRc\nq78a8rotCJ+ut60B/2C0SmCgJ8O1vF4RMSijkR1wS\nbIWZcklqM9qGs2XS/3rFx/+Q2Z2pHDdZuS1Nv0m07\n3Ct6wE83HL3dIB6xqCNRXU0PqUcsR3tQlzuPG6zcL\nhuU5J6p3l02g53ZqLpH+6NYNtqtkmJHJptXyL7dQiL\nmoraKSZmc9sW6xAW46JtwW+s7Aq4eb",
  "02g53ZqLpH+6NYNtqtkmJHJptXyL7dQiL\nmoraKSZmc9sW6xAW46JtwW+s7Aq4ebStOoTFrVy0NR\nPA0pBLfAp1CIv1Em6bTQyrGw51w60ymY6QWYew+Ij\nF+KzrEBYjKkZO8YSlKRLrEMnjCOdxRPOYil1SXhEU\nseIkCnlmlDZKGlLJoClMWpt7GgMeiAThRpsgljO6cz\nLnTNPoVms6CzuRruXdGwZqhCE8DSJljXn/Tuch8n\nGLYZrmSnApk",
  "hRpsgljO6cz\nLnTNPoVms6CzuRruXdGwZqhCE8DSJljXn/Tuch8n\nGLYZrmSnApkpTSBW9jZos509+eHJdnJ+eGFpReUnlt\n6TumBpQeUZpaSJwI/3LGUPJ34ZmlZ5TuW7pPaWFpQ\nWnP0h6loaUhpQ8tfUhpYGlA6aqlq5RqS8mOFO4Ilu\n5ROrJ0ROmhpYeUPrf0OaWPLX1M6QtLX1D62tLXlN63\n9D6lzFJG6Zqla5RyS8mrAz9csXSFUt9S8",
  "eUPrf0OaWPLX1M6QtLX1D62tLXlN63\n9D6lzFJG6Zqla5RyS8mrAz9csXSFUt9S8uwHa83SLU\npTS1NKH1j6gNKhpeSpGO5nlpLtDdwYLZWUPrH0CaXC\nUvL85ofPLH1GaWxpTOlTS59S+srSV5Q+svQRpZGl5N\n0A7E4s3aXUvgUqc0q3Ld2m9NTSU/d7AT4bRt81MTd\ntBZuUJpYmlK5bSp4UYCth6QnZT4aquapN3zaR61qoZ\ntzBmoxPjyY5",
  "4bRt81MTd\ntBZuUJpYmlK5bSp4UYCth6QnZT4aquapN3zaR61qoZ\ntzBmoxPjyY5D9WMO1hzdZoeTa5PoZrxEen62v7sRQq\nkFK70g/nFZfwWlhb2v+8u/9i9u3138d5K84b2Wufz\npedW53lzk+de53Hna1OrxN0/p37Yu7m3FcLvy38sfD\nnwl+1+sZc8yndZn4e/ANm3Bws=</latexit> @f3\n@h3\n=\n@\n@h3\n(\u03b23 + \u23263h3) = \u2326T\n3\n<latexit sh",
  "ndZn4e/ANm3Bws=</latexit> @f3\n@h3\n=\n@\n@h3\n(\u03b23 + \u23263h3) = \u2326T\n3\n<latexit sha1_base6\n4=\"yZBKTnT7qspEL36MxtIKQ+8n3t8=\">AW9\nHiclZhb9s2FICdrtu6bu3SDcvLXoQFBbqtM5K\n1u7wMaJOmt6SLc3GSJk4NSqZkNhSlSJTjVPA/2\nduw1/2fvey37FCWzeoc5mEGEtPn+0RSh6REyU+\nlyPXKyj8L1z64/uFH9/45Oan926/fn",
  "2fvey37FCWzeoc5mEGEtPn+0RSh6REyU+\nlyPXKyj8L1z64/uFH9/45Oan926/fninS8O\n8qTIAt4NEplkRz7LuRSKd7XQkh+lGWexL/mhf7\nZu+OGIZ7lI1L6+TPlpzCIlQhEwDaH+4tjrhRkL\nyl7KMi2Y9ML+g4n9NeyXDyYT7zdkEaMneajv9X\nyuWf+B930viXnEDKlwLxPRUH9rqpmD/uLySnu\nl+ni0sFoXlv1p9O/89WgN0iCIuZKB5L",
  "B930viXnEDKlwLxPRUH9rqpmD/uLySnu\nl+ni0sFoXlv1p9O/89WgN0iCIuZKB5Ll+cnqS\nqpPS9OPQPLJzV6R85QFZyziJ1BULOb5aVmlaOL\ndhcjAC5M/pT2quj7R5QszvPL2AczZnqY2aCL\nnZS6PDX01KotNBcBdOGwkJ6OvFMvr2ByHig5SU\nUWJAJ6KsXDBkU8Oo3OwpfhEkczUoOytbexA\nYn0eCVXy86Iaocmk6WxUDofiVcbai/15LU",
  "KsXDBkU8Oo3OwpfhEkczUoOytbexA\nYn0eCVXy86Iaocmk6WxUDofiVcbai/15LULzWL\nzjpJKMZVcIfBoUpa8HbUxEByAaHMCEsVzqNPk\nxw+9VURhRkrAwP1kDJ0Lvd0JqVpHkFOGtox0a\nCQSj5uWOvEgqGMG8oeKJ531zOA6wxGAboKXxyN\nwV7K1GR2nOZjncVlbmK4hYypiFdNwCkHsB2s\naEKeHQoGH9jq1dps7qxCVp1dXMRJC1nzUdn",
  "nOZjncVlbmK4hYypiFdNwCkHsB2s\naEKeHQoGH9jq1dps7qxCVp1dXMRJC1nzUdndG\n8qEHTqSLIgkYNa0qgiwJ148BixlkuS734YRjz\n0TcqlBYFWRidrLEb7admgiem+MU1kvT2yhJ+kc\nMZcQEYPWZb8FUwJv6ejK3vVlyRpVvCnzsDWGwm\noewLJqe1qwROKs6NqFmlStk0mxBKEsumqbpjU\nPlqWieoAngRVdkQoXvaferEkxZE+7dh1PNC",
  "wROKs6NqFmlStk0mxBKEsumqbpjU\nPlqWieoAngRVdkQoXvaferEkxZE+7dh1PNCslP\nfmj/xMen5YpZNuYfySZUlBepqyIT/h8VDeCOhe\ncXRPDgJRINHgSqwUskXN/R0LEMT2wTqcYOCkIx\nKfQlWv4iUs1jqgjubBKjvkLA1AvfTCg0yGHYl\nE3AyPAN917HBArQSQbTcwxkhcZJxc/NJ8hUun\nmspgJc7NqXlClEZrXDS7nR0EZbg4jfsXhP",
  "17HBArQSQbTcwxkhcZJxc/NJ8hUun\nmspgJc7NqXlClEZrXDS7nR0EZbg4jfsXhPsqoP\n82nxRqwDKUzLEZ0vGbXq5hiblWfzXk06LTivj\n5Zt0e9AtGpwgCft7fxOMREYs6EtUFmx1nXZJYj\nvagrvl0fb9n5eab78jUjhyu25Sk3rqXbtvhXt\nEDfr7l6O0W8YhFHYnqntIPWI52oO63Hncp2F\nw3WbktQ7y6PTdrhzE03/cH8IO1WzTUrkwGz",
  "0W8YhFHYnqntIPWI52oO63Hncp2F\nw3WbktQ7y6PTdrhzE03/cH8IO1WzTUrkwGz7Et\nmbhrCoqaidYrWxbYrTEBbjomnBb6zsCbh5NK1p\nCIudXDQ1E8DSgEt8CtMQFqdLuGnWMaxuOdQt8\npkOkTmNITFZyzGZz0NYTGiYuQUz1iaInEaInk\nc4jwOaR5TLKUuCY9I6hgRMqVcEyobJk3JBLA0R\nq2NHY1BD2SiUIN1EMs5nXm5c+YpNIsVncV",
  "LKUuCY9I6hgRMqVcEyobJk3JBLA0R\nq2NHY1BD2SiUIN1EMs5nXm5c+YpNIsVncVdV8P\ndKxrWDFVoAljaJmvM6207F5mPU+xVz6Z0uASyU\nprADnY61Jnt/vywJDs5P7y09JLSC0svKD209J\nDSzFLyROCHu5aSpxM/HFk6ovTA0gNKC0sLSruW\ndikNLQ0pfWrpU0oDSwNK1y1dp1RbSnakcEewdJ\n/SoaVDSo8sPaL0taWvKX1u6XNKjy09pv",
  "pfWrpU0oDSwNK1y1dp1RbSnakcEewdJ\n/SoaVDSo8sPaL0taWvKX1u6XNKjy09pvSdpe8o\nfWzpY0qZpYzSDUs3KOWklcHfrhm6RqlvqXk2Q\n/WmqUdSlNLU0qfWPqE0oGl5KkY7meWku0N3Bg\ntlZS+sPQFpcJS8vzmh68sfUVpbGlM6UtLX1L61\ntK3lD6z9BmlkaXk3QDsTizdo9S+BSpzSncs3aH\n03NJz93sBPh9G3zUxt20F25QmliaUbl",
  "z9BmlkaXk3QDsTizdo9S+BSpzSncs3aH\n03NJz93sBPh9G3zUxt20F25QmliaUblpKnhRgK\n2HpGdlPhq+qs3eNpHrWqjm3MHqjM+OJjkP1Zw\n7WH1mh1Nrk+hmvMh6frGwfxFCqQUrvT9xeV\n/BaWFg5+bK/+3H6483D50Vr9hvZG6+vWN617rd\nXWL61HretTqvbClr/LlxfuLVwe2m09MfSn0t/\nTdVrC/UxX7Yan6W/wNvHPzC</latexit>",
  "etTqvbClr/LlxfuLVwe2m09MfSn0t/\nTdVrC/UxX7Yan6W/wNvHPzC</latexit>@f3\n@h3\n=\n@\n@h3\n(\u03b23 + !3h3) = !3",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKD",
  "BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY",
  "LJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzou",
  "fwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL",
  "JyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19N",
  "ab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2Fg",
  "Mw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16",
  "g37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0i",
  "aCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGd",
  "+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0",
  "ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P",
  "YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY",
  "Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY9niczZhb9s2FIDdZJcuzTdMCDAXoQFHbqhC+K2u7wMaJOmt6TL1UnaODUomZLZ\nUJQiUYlTwX9lb8Ne93f2tp+yQ8k2o3OYQP2MAOp2fN9vB2Skiw/lSLXy8t/XJuZfefd96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM5",
  "d96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEejd\nnLrphxoKym7JMCya9LpeyV4rR6FLID8Ne2R6NvrJw7YfDnrlXadN1ZCqde25ruShvu1s/B5t/K678X\nukahLRnzF/O6NupmIBvpr9ud+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/v",
  "d+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/vdfhIUMVc6kCzPj9rLqT4uTeuB5DCtIucpC05YxI+gqFjM8+OyOiQj7xZE+l6YZPCntFdFL\n9coWZznF7EPZsz0IMfMBF3sqNDhj8elUGmhuQrqjsJCejrxzInz+iLjgZYXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+",
  "YXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+JVxsqzvWkrQvNYvOWkUoxjVwh8GhUlnwpWsJA\ncABiROQKJ5DmyY/fui1EYVrkgQM3E+GMLjQ2xmRpXmEeSkob0iGhRSyYcNa5VYsJRxQ9kFxfNueQZ\nwncEqwFDhi6M12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmg",
  "12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmgqy\n9rOnojOZF9ZtOFUEWbMKoaVURZEm4g/RZzCDL43IPJhx7JuJWhcKqIBtzK0v8Zt+pieC9OUzhvDS9tZ\nKk/4yhjJgAnD7zLZgKeFNfTa2N0nOWeWbAh96A1isZhWRfW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0D",
  "fW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0DU80KyY+XfqOD4/LZXNszD8km9BQXqSuhkz4XzTUh2cWvL8g\nhcvkWjxIFAtXiLh+o6WjmV4Y5tItXZQEIpJoS/Q8ReRatapIniwSYzGCgHTLnwzodAih2FTNgEjwzc8f\nTk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZ",
  "Tk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZdSv8+knheqzDCVz\naJZ0+LqbazhirtNfLXldFoRP10f9wfjgtUpgoCf9tbxekTEo5EbcHjrMtSxHf9DWdLteHlm5/vo\nbsrUjh+s2JWl3PEq37XCvGAE/3XCMdoN4xKORG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3U",
  "RG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3UIi5qK2ikmMY+QWIewGBdNC/6PlV0BN4+mVYewuJWLpmYCWOpziadQh7BYH+\nGmOY5hdcOhbrhVJtMBMusQFp+wGM+6DmExomLkFE9YmiKxDpE8DnAeBzSPKZSl4RXJHWsCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61g",
  "sCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61gw1aAJY2iRnzOtuOg+Zj1Mj1muJ\nKcCWSlN4BZ2tqgzefrzw5I8yfnhaUXlJ5bek7pgaUHlGaWkl8EfrhjKfl14odnlp5Rum/pPqWFpQWlH\nUs7lIaWhpQ+tvQxpYGlAaWrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPK",
  "Wrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPKWMkrXLF2jlFtKXh34YqlK5T6lpLfnDWLN2iNLU0pfSRpY8o7VtKfhXD/cxS8ngDN0ZLJaXP\nLH1GqbCU/H7zwxeWvqA0tjSm9Lmlzyl9Y+kbSp9Y+oTSyFLybgCeTizdpdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlT",
  "pdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlTjq9rkbRO5roVqyh1snPFJbZLzUE25g42vTpPa5P\noUqikfkKGv7U9fpEBK4Urfm19s47ewtLB/d6n9/dL97fuLD1bGb2ivt75ofdm63Wq3fmg9aD1tbU6rW\nDmz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n=",
  "mz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n= @h2\n@f1\n@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n@f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9C",
  "64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87",
  "wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UO",
  "uCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgw",
  "AYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4",
  "KYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkR",
  "KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J",
  "XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRF",
  "WLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLS",
  "UmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfU",
  "NLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfd",
  "n8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n<latexit sha1_base64=\"V3CYFDq4r+PwPuPiQT3UunDjVrg=\">AXDHiclZhbc9w0FIA3XEu5pTDkBR48\nZMoUKDsJ",
  "Dq4r+PwPuPiQT3UunDjVrg=\">AXDHiclZhbc9w0FIA3XEu5pTDkBR48\nZMoUKDsJLZcXZtqk6S0puW6SNpvuyF7Zq0aWHVtONvXsX2D4MbwxvPIf+CM8c2R7V/E5ygM7k6z2fJ8l+UiyZfupFLleWvpn7o03r7nXevXf9/Q8+/Oj+Ruf7OdJkQW8FyQyQ59lnMpFO9poSU/TDPOYl/yA/9k1fC\nDM57lIlF7+iLlxzGLlAhFwDSEBvO/e/0wY0HZ",
  "nMpFO9poSU/TDPOYl/yA/9k1fC\nDM57lIlF7+iLlxzGLlAhFwDSEBvO/e/0wY0HZT1mBZNe3w/DwZ1JKzAalHcmE+8X5LqkvuShvgU/fa6ZiXjfwo/NmEfVr6nXz0Q0l+bKi/Rl+XeZDC/uNRdqj4eLSw3hcVO89ka3Phs2B8mQRFzpQPJ8vxoeSnVx6XpWSD\n5Hq/yHnKghMW8SMoKhbz/LisUjfxbkJk6IVJBn9Ke1X08hEli/P8IvbBjJke5Z",
  "D\n5Hq/yHnKghMW8SMoKhbz/LisUjfxbkJk6IVJBn9Ke1X08hEli/P8IvbBjJke5ZiZoIsdFTr8+bgUKi0V0HdUFhITyeGQdvKDIeaHkBRZkAvrqBSMG6dUwWtf7ip8HSRwzNSz7K2vbkGqfR0KV/LSoRm4yaTtrlcOheJ\nWx8mRvVovQPBavOamkUkwlVwg8mpQl70ZdDAQHILqcgETxHOo0+fFDbxlRmKkSMHA/GUPnQm9nQqpWmkeQk5b2gmhQ",
  "mpQl70ZdDAQHILqcgETxHOo0+fFDbxlRmKkSMHA/GUPnQm9nQqpWmkeQk5b2gmhQSCUft6xVYsFQxi1lFxTPu+kZwHUGowBdhS+OxmA3ZWoyPU7zsc7iMjcx3ELGVMSrJuCUA1gaO9hQh\nZRwaNCyfsXWDlMnTeKStOpqZiLI2svajs5oXtSw7VQRZMEkjNpWFUGWhOvKkMUMstyUB3DCsWciblUorAoyMbeyxG+3nZoInpvjFNZL21srSfrPGMq",
  "WFUGWhOvKkMUMstyUB3DCsWciblUorAoyMbeyxG+3nZoInpvjFNZL21srSfrPGMqICcDqM9+CqYC39dVkZnvT5JxVvinwsTeCwWofwrKoPq1pI3BWTWxCz\nSpXyKTZglCWnLdN0xuHylPRPkETwIuyIQKL2m3qxJMWRPu34ZTzQrJj7r/sDHx+WSWTbmH8kmVJQXqasiE/4fFQ3hTobnF0Tw4CUSDR4EqsFLJFzf0dCxDE9sE6nGDgpCMSn0BVr+",
  "VJQXqasiE/4fFQ3hTobnF0Tw4CUSDR4EqsFLJFzf0dCxDE9sE6nGDgpCMSn0BVr+IlLtY6oI7mwSo75CwNQL30woNMh\n2JZNwMjwDfdkxwQK0EkG9TkGMsmLjJOLH5rPEKl0c1nMhLlZtS+o0gjt6waXs6OgDeHM37F4T7KqF/n08KNWQZSubYDOn4ZT/XsMRcq78a8rotCJ+ut60B/2C0SmCgJ8O1vF4RMSijkR1wSbIWZcklqM9qGs2XS/3rFx",
  "Rcq78a8rotCJ+ut60B/2C0SmCgJ8O1vF4RMSijkR1wSbIWZcklqM9qGs2XS/3rFx\n/+Q2Z2pHDdZuS1Nv0m073Ct6wE83HL3dIB6xqCNRXU0PqUcsR3tQlzuPG6zcLhuU5J6p3l02g53ZqLpH+6NYNtqtkmJHJptXyL7dQiLmoraKSZmc9sW6xAW46JtwW+s7Aq4ebStOoTFrVy0NRPA0pBLfAp1CIv1Em6bTQy\nrGw51w60ymY6QWYew+IjF+Kz",
  "7Aq4ebStOoTFrVy0NRPA0pBLfAp1CIv1Em6bTQy\nrGw51w60ymY6QWYew+IjF+KzrEBYjKkZO8YSlKRLrEMnjCOdxRPOYil1SXhEUseIkCnlmlDZKGlLJoClMWpt7GgMeiAThRpsgljO6czLnTNPoVms6CzuRruXdGwZqhCE8DSJljXn/Tuch8nGLYZrmSnApkpTSBW9jZos\n509+eHJdnJ+eGFpReUnlt6TumBpQeUZpaSJwI/3LGUPJ34ZmlZ5",
  "pkpTSBW9jZos\n509+eHJdnJ+eGFpReUnlt6TumBpQeUZpaSJwI/3LGUPJ34ZmlZ5TuW7pPaWFpQWnP0h6loaUhpQ8tfUhpYGlA6aqlq5RqS8mOFO4Ilu5ROrJ0ROmhpYeUPrf0OaWPLX1M6QtLX1D62tLXlN639D6lzFJG6Zqla5RyS8mrA\nz9csXSFUt9S8uwHa83SLUpTS1NKH1j6gNKhpeSpGO5nlpLtDdwYLZWUPrH0CaXCUvL85ofPLH1Ga",
  "uwHa83SLUpTS1NKH1j6gNKhpeSpGO5nlpLtDdwYLZWUPrH0CaXCUvL85ofPLH1GaWxpTOlTS59S+srSV5Q+svQRpZGl5N0A7E4s3aXUvgUqc0q3Ld2m9NTSU/d7AT4bRt81MTdtBZuUJpYmlK5bSp4UYCth\n6QnZT4aquapN3zaR61qoZtzBmoxPjyY5D9WMO1hzdZoeTa5PoZrxEen62v7sRQqkFK70g/nFZfwWlhb2v+8u/9i9u3138d5K84b2",
  "O1hzdZoeTa5PoZrxEen62v7sRQqkFK70g/nFZfwWlhb2v+8u/9i9u3138d5K84b2WufzpedW53lzk+de53Hna1OrxN0/p37Yu7m3FcLvy38sfDnwl+1+sZc8yndZn4e/\nANm3Bws=</latexit> @f3\n@h3\n=\n@\n@h3\n(\u03b23 + \u23263h3) = \u2326T\n3",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKD",
  "BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY",
  "LJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzou",
  "fwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL",
  "JyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19N",
  "ab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2Fg",
  "Mw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16",
  "g37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0i",
  "aCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGd",
  "+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0",
  "ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P",
  "YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY",
  "Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY9niczZhb9s2FIDdZJcuzTdMCDAXoQFHbqhC+K2u7wMaJOmt6TL1UnaODUomZLZ\nUJQiUYlTwX9lb8Ne93f2tp+yQ8k2o3OYQP2MAOp2fN9vB2Skiw/lSLXy8t/XJuZfefd96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM5",
  "d96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEejd\nnLrphxoKym7JMCya9LpeyV4rR6FLID8Ne2R6NvrJw7YfDnrlXadN1ZCqde25ruShvu1s/B5t/K678X\nukahLRnzF/O6NupmIBvpr9ud+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/v",
  "d+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/vdfhIUMVc6kCzPj9rLqT4uTeuB5DCtIucpC05YxI+gqFjM8+OyOiQj7xZE+l6YZPCntFdFL\n9coWZznF7EPZsz0IMfMBF3sqNDhj8elUGmhuQrqjsJCejrxzInz+iLjgZYXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+",
  "YXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+JVxsqzvWkrQvNYvOWkUoxjVwh8GhUlnwpWsJA\ncABiROQKJ5DmyY/fui1EYVrkgQM3E+GMLjQ2xmRpXmEeSkob0iGhRSyYcNa5VYsJRxQ9kFxfNueQZ\nwncEqwFDhi6M12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmg",
  "12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmgqy\n9rOnojOZF9ZtOFUEWbMKoaVURZEm4g/RZzCDL43IPJhx7JuJWhcKqIBtzK0v8Zt+pieC9OUzhvDS9tZ\nKk/4yhjJgAnD7zLZgKeFNfTa2N0nOWeWbAh96A1isZhWRfW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0D",
  "fW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0DU80KyY+XfqOD4/LZXNszD8km9BQXqSuhkz4XzTUh2cWvL8g\nhcvkWjxIFAtXiLh+o6WjmV4Y5tItXZQEIpJoS/Q8ReRatapIniwSYzGCgHTLnwzodAih2FTNgEjwzc8f\nTk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZ",
  "Tk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZdSv8+knheqzDCVz\naJZ0+LqbazhirtNfLXldFoRP10f9wfjgtUpgoCf9tbxekTEo5EbcHjrMtSxHf9DWdLteHlm5/vo\nbsrUjh+s2JWl3PEq37XCvGAE/3XCMdoN4xKORG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3U",
  "RG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3UIi5qK2ikmMY+QWIewGBdNC/6PlV0BN4+mVYewuJWLpmYCWOpziadQh7BYH+\nGmOY5hdcOhbrhVJtMBMusQFp+wGM+6DmExomLkFE9YmiKxDpE8DnAeBzSPKZSl4RXJHWsCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61g",
  "sCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61gw1aAJY2iRnzOtuOg+Zj1Mj1muJ\nKcCWSlN4BZ2tqgzefrzw5I8yfnhaUXlJ5bek7pgaUHlGaWkl8EfrhjKfl14odnlp5Rum/pPqWFpQWlH\nUs7lIaWhpQ+tvQxpYGlAaWrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPK",
  "Wrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPKWMkrXLF2jlFtKXh34YqlK5T6lpLfnDWLN2iNLU0pfSRpY8o7VtKfhXD/cxS8ngDN0ZLJaXP\nLH1GqbCU/H7zwxeWvqA0tjSm9Lmlzyl9Y+kbSp9Y+oTSyFLybgCeTizdpdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlT",
  "pdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlTjq9rkbRO5roVqyh1snPFJbZLzUE25g42vTpPa5P\noUqikfkKGv7U9fpEBK4Urfm19s47ewtLB/d6n9/dL97fuLD1bGb2ivt75ofdm63Wq3fmg9aD1tbU6rW\nDmz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n=",
  "mz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n= @h2\n@f1\n@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n@f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9C",
  "64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87",
  "wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UO",
  "uCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgw",
  "AYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4",
  "KYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkR",
  "KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J",
  "XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRF",
  "WLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLS",
  "UmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfU",
  "NLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfd",
  "n8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3",
  "Derivative of ReLU",
  "Derivative of ReLU\n<latexit sha1_base64=\"Hg/RfzPx0d42l4dwgRiyNcSI3cQ=\"\n>AWi3iclZhb9s2FIDVXbtuXdMNy8tehAUFhqEznK27YBiGNqnbpkXp4mTtHEaUDIls6EoRaISJ4J/yV63H7V/s0NZNqtzmIcZS\nM2e7xMvh6REK8ikKHS3+N97/4MOPr75ya1P7v9+Z2lu1/sF2mZh3wQpjLNDwNWcCkUH2ihJT/Mcs6SQPKD4HTd8INznhciVXv\n6MuPHC",
  "2lu1/sF2mZh3wQpjLNDwNWcCkUH2ihJT/Mcs6SQPKD4HTd8INznhciVXv\n6MuPHCYuViETINIROlu4ME6bHQVBtTI+u/ugenytdDvd+uPTwmpTWPGaT/k7lej4SgNy4QrHUpWFEer3UwfVyzXIpR8emtYFjxj\n4SmL+REUFUt4cVzVPZ/69yAy8qM0hz+l/Tr67hUVS4riMgnANP0sMDNBFzsqdfTrcSVUVmquwlDUSl9nfomDf5I5DzU8hIKLMw",
  "r67hUVS4riMgnANP0sMDNBFzsqdfTrcSVUVmquwlDUSl9nfomDf5I5DzU8hIKLMwF9N\nUPxyxnoYZk3RoqfhGmScLUqBqu9Xam1TDgsVAVPyvrxE2nbadXOxyK1xlrG3uLWoTmibjipJaMZVcI/B4WlW8E3cwEByA6HACUsUL\nqLOe48hfRQWigQMPEgn0LnIfzklVSvNY8hJS3tNChk9a1jqxYCqTlrILiu/f8w3gOodZgK7CF0dzsJsxNZ1fp/",
  "IfzklVSvNY8hJS3tNChk9a1jqxYCqTlrILiu/f8w3gOodZgK7CF0dzsJsxNZ1fp/lE50lVmBhuIW\ncq5nUTMOSQSTOitqFKeHSsGX9ia2XTJ02iUuzuqu5iSBrL287Oqd5UaO2U0eQBYswblt1BFkStvWIJQy3JRPYMCJbyJuVSisCrI\nw+3katNvOTASvzUkG+6Xt9SqS/nOGMmICsPvMt2Aq5G19PV3Y/jw57VvCnzij2Gy2pewPJ4Na94IjKq",
  "kG+6Xt9SqS/nOGMmICsPvMt2Aq5G19PV3Y/jw57VvCnzij2Gy2pewPJ4Na94IjKqJTalZ5wqZNFsQytOLtml64\n1B5JtoDNAG86cpcqOgd7X5dgiVrwsP7MNS8lPzo+85PfHJcdc2Mf+QbEJFRZm5KjLh/1HRCB4keH1BE9eKtHkQaCevFTC/R1NHcv\nxwjaReu6gIBSTQl+i7S9i1b6mjuDOpgnqKwRMvfDNhEKTHEVt2QSMDN/wSHQsoBANMpy",
  "Reu6gIBSTQl+i7S9i1b6mjuDOpgnqKwRMvfDNhEKTHEVt2QSMDN/wSHQsoBANMpyNMZRpUeac3PzQeoZIrZvbYi7Mw6p9Q5VG\naN83uFxcBWV4OJzay4PUEaDWT6DtFQjlqNkTsyUTt4MCw1bzLX76ymfFZ1WzM82m/agXzA7ZRjys5NPB8xsagjUV1wBnHWJYnlaA\n/qWizXd3tWb75jizt2OG6TUnqbXrpth3uNT3gZ1uO3m4Rj1jUkaiupofU",
  "JYnlaA\n/qWizXd3tWb75jizt2OG6TUnqbXrpth3uNT3gZ1uO3m4Rj1jUkaiupofUI5ajPajLnct1ygcrtuUpN5Hp2w12YaPlHe2OumTkm\npXJkjn2pHM5CWNRU1E4xTXiMxFkIi0nZtuD/WNkV8PBoW7MQFvuFaGsmgKURl3gIsxAWZ1u4bTYxrG451C23ymQ2RuYshMWnLMGjno\nWwGFMxdoqnLMuQOAuRPI5xHsc0jxmWMpeEZyRzAhZUq4FlY",
  "Q2RuYshMWnLMGjno\nWwGFMxdoqnLMuQOAuRPI5xHsc0jxmWMpeEZyRzAhZUq4FlY/TtmQCWJqg1iaOxqAHMlWowSaI5YKuvMK58hRaxYqu4oGr4cE1DWu\nGKjQBLG2TPeYPt52bLMAphmOWK8mZQFZGE9jHTp8689NfEFXkJBdEl5ZeUnph6QWlB5YeUJpbSn4RBNFLS8mvkyA6t/Sc0n1L9yktL\nS0pHVg6oDSyNKL0iaVPKA0tDSldt3SdUm0p",
  "n4RBNFLS8mvkyA6t/Sc0n1L9yktL\nS0pHVg6oDSyNKL0iaVPKA0tDSldt3SdUm0pOZHCE8HSPUrHlo4pPbT0kNJXlr6i9Jmlzyh9belrSq8svaL0kaWPKGWMkp7lvYo5Za\nSVwdBtGbpGqWBpeS3H+w1S/uUZpZmlD629DGlI0vJr2J4nlKjfwYLRUrph6QalwlLy+y2IXlj6gtLE0oTS5Y+p/StpW8pfWrp\nU0pjS8m7ATidWLpLqX0LVBWU7li",
  "lwlLy+y2IXlj6gtLE0oTS5Y+p/StpW8pfWrp\nU0pjS8m7ATidWLpLqX0LVBWU7li6Q+mZpWfu9wJ8MY2Ba2Fu2wq2KU0tTSndtJT8UoCjhKWn5DwZqeauNn/bRO5rkVpwB2syPr+a5D\nxSC+5gzd1pfjW5P0Vqwcek6739xYsUSCnc6U+WVlbxW1ha2P+hs/pz58HOg5WHa80b2pve1943rfeqveL9B75vW9gRd6pfeX97f3\nz/Lt5R+Xf1v+fa+",
  "pz58HOg5WHa80b2pve1943rfeqveL9B75vW9gRd6pfeX97f3\nz/Lt5R+Xf1v+fa+d6O5kuv9Vnu/Qcl5NSP</latexit>I[z > 0]\n\u201cIndicator function\u201d",
  "Derivative of RELU\n<lat\nexit s\nha1_ba\nse64=\"A\nKqpdOE\nF3auNV\nSX+yCY\ng5tU0q\nI=\">A\nWmHicl\nZhb9s\n2FIDV7t\nZ1t3TD\n8rDtQV\nhQYBhaI\nxm6y8u\nANmap\nkmXq5O0\ncRpQMi\nWzoShF\nohKngl/\n2a/a6/\nZv9mx1K\nshmdwz\nzMQGvm\nfJ94OSQ\nlWkEmR\naEXF/+\n9dfu9z\n/48KM7\nH9/95N\nPv9i7\nt6XB0V\na5iHvh\n6lM86OA\nFVwKxf\nt",
  "aEXF/+\n9dfu9z\n/48KM7\nH9/95N\nPv9i7\nt6XB0V\na5iHvh\n6lM86OA\nFVwKxf\ntaMmPs\npyzJD\n8MDhbM\nfzwgueF\nSNW+vs\nr4ScJi\nJSIRMg2\nh07nvB\nkHE/N/\n9QRKk4w\nr+8Hf5\nZn9yDK\nXg5HRuY\nbG3WH9\n8WlhqCw\nte+9k+\nvf1cD\nBMwzLhS\noeSFcX\nx0mKmT\nyqWaxFK\nPrk7KA\nuesfCM\nxfwYio\nlvDip6\nmFM/Ps\nQGfpRm",
  "oeSFcX\nx0mKmT\nyqWaxFK\nPrk7KA\nuesfCM\nxfwYio\nlvDip6\nmFM/Ps\nQGfpRms\nM/pf06\nev2KiV\nFcZUEY\nCZMjwr\nMTNDFjk\nsd/XZS\nCZWVmq\nuwaSgqp\na9T3+T\nEH4qch\n1peQYGF\nuYC+u\nGI5SzU\nkLm7A8U\nvwzRJm\nBpWg+XV\nnQmkic\ndCVfy8\nrLM4mXS\nd1drhU\nLzJWF7\nfn9UiNE\n/EO04q\nqRVTyQ\n0CjydVx\nXtxDwP\nB",
  "fy8\nrLM4mXS\nd1drhU\nLzJWF7\nfn9UiNE\n/EO04q\nqRVTyQ\n0CjydVx\nXtxDwP\nBAYgeJ\nyBVvIA6\nTX5gnp\ncQhVUjA\nVfXVsK\nEVK0j\nyEnHe01\n0aCQST\n7uWCvE\ngqlMOso\neKL5/3\nzeA6x\nmAboKXx\nzNwV7G\n1GR6ne\nZjnSdVY\nWK4hZy\npmNdNw\nJBDJs2I\nuoYqpY\nRLw471B\n7Z2mTp\nrE5dmd\nVdzE0HW\nft51dE\n7zoZd\np4gCx",
  "JBDJs2I\nuoYqpY\nRLw471B\n7Z2mTp\nrE5dmd\nVdzE0HW\nft51dE\n7zoZd\np4gCxZ\nh3LXqC\nLIk7PE\nhSxhkuS\n2fwoAT\n30Tcql\nBYFWRhb\nudp0G0\n7MxG8Ns\ncZ7Jeu\nt1qR9F\n8wlBETg\nN1nvgV\nTIe/qK\n+nM9qfJ\nuah9U+\nBjfwST\n1b2E5XE\nzrGkjM\nKo2NqF\nmnStk0m\nxBKE8v\nu6bpjUP\nlmegO0\nATwpit\nzoaJr2o\nO6BEvW",
  "rGkjM\nKo2NqF\nmnStk0m\nxBKE8v\nu6bpjUP\nlmegO0\nATwpit\nzoaJr2o\nO6BEvW\nhAcPYK\nh5Kfnxw\n97PfHx\nSLZptY\n/4j2YSK\nijJzVW\nTC/6Oi\nITxV8Pq\nCJ68V\nKLJg0A9\neamE+z\nuaOpbj\nhW0i9dx\nBQSgmh\nb5C21/\nEqntNHc\nGdTRPU\nVwiYeu\nGbCYUmO\nYq6sgk\nYGb7h+\nehYQCEa\nZNiMZ\nRpUeac3\nPzQeoZ\nIrZvbY\ni7",
  "Yeu\nGbCYUmO\nYq6sgk\nYGb7h+\nehYQCEa\nZNiMZ\nRpUeac3\nPzQeoZ\nIrZvbY\ni7Mw6p7\nQ5VG6N\n43uJxd\nBWV4OFz\nwGy4PU\nEaDJp9\nBWqohy1\nEyx2ZK\nx28GhY\nYt5tr9\nZQ3Rac\nV8/ONtj\n3oF8xO\nGYb8/H\nQDz0dML\nOpIVBc\ncSJx1S\nWI52oO6\nZsv1es\n+qjTc/\nkqUdO1y\n3KUm9b\nS/dtsO\n9oQf8fN\nPR203i\nEYs6EtX\nV9p",
  "6\nZsv1es\n+qjTc/\nkqUdO1y\n3KUm9b\nS/dtsO\n9oQf8fN\nPR203i\nEYs6EtX\nV9pB6x\nHK0B3W\n587jpGo\nXDdZuS\n1DvNo9\nN2uDMTL\nf9of8Q\n1M8ekV\nA7NsS+V\ngyaERU\n1F7RT\nhMdIbEJ\nYTMquB\nX9jZU/A\nw6NrNS\nEsbhei\nq5kAloZ\nc4iE0I\nSw2W7h\nrtjGsbj\nrUTbfK\nZDZCZh\nPC4hpL8\nKibEBZ\njKsZO8\nYxlGRKb\nEMnjC",
  "Sw2W7h\nrtjGsbj\nrUTbfK\nZDZCZh\nPC4hpL8\nKibEBZ\njKsZO8\nYxlGRKb\nEMnjCO\ndxRPOY\nSlzSXh\nGMseMk\nCXlWlD5\nKO1KJo\nClMWpt\n7GgMeiB\nThRpsg\n1gu6Mo\nrnCtPoV\nWs6Cru\nuxru39\nCwZqhCE\n8DSFtl\nj/mDLu\nckCnGI4\nZrmSnA\nlkZTSB2\n9jZps7\n09BdEF\nTnJBdGV\npVeUXl\np6Semh\npYeU5pa\nSXwRBt\nGsp+XU",
  "ZTSB2\n9jZps7\n09BdEF\nTnJBdGV\npVeUXl\np6Semh\npYeU5pa\nSXwRBt\nGsp+XU\nSRBeWXl\nB6YOkB\npaWlJa\nV9S/uUR\npZGlD6\nz9Bmloa\nUhpSuW\nrlCqLS\nUnUngiW\nLpP6cj\nSEaVHl\nh5R+srS\nV5Q+t/\nQ5pa8t\nfU3pO0v\nfUfrE0\nieUMks\nZpauWrl\nLKLSWv\nDoJo2dJ\nlSgNLy\nW8/2Gu\nWblOaWZ\npR+tTS\np5QOLS\nW/iuF5Z",
  "uWrl\nLKLSWv\nDoJo2dJ\nlSgNLy\nW8/2Gu\nWblOaWZ\npR+tTS\np5QOLS\nW/iuF5Z\nik53sC\nD0VJ6\nbql65QK\nS8nvty\nB6aelL\nShNLE0p\nfWPqC0\nreWvqV0\nzdI1Sm\nNLybsB\nOJ1Yuke\npfQtUF\nZTuWLp\nD6bml5+\n73Anw2\njYFrYW\n7ZCrYoT\nS1NKd2\nwlPxSg\nKOEpWfk\nPBmp9q\n42fdtE7\nmuRmnE\nHazM+v\nZrkPFIz\n7mDt3W\nl6",
  "d2\nwlPxSg\nKOEpWfk\nPBmp9q\n42fdtE7\nmuRmnE\nHazM+v\nZrkPFIz\n7mDt3W\nl6Nbk/\nRWrGR6T\nrqwezF\nymQUrj\nTn84tLO\nG3sLRw\n8FNv6Z\nfeo51HC\n4+X2ze\n0d7xve\n+9H7wl\n71fvsf\nfc2/b6X\nuj96f3\nl/e39M\n/N/OP5\ntfn1Rr\n19q73m\nK6/zmd/\n9D+WM2\nUw=</l\natexit>a = ReLU[b]\n<latexit sha1_base64=\"XBb10PlJk\nwj",
  "K6/zmd/\n9D+WM2\nUw=</l\natexit>a = ReLU[b]\n<latexit sha1_base64=\"XBb10PlJk\nwj2OHDfpKlkaDFPFc=\">AWr3iclZhb9s2FIDVXbvulm5YXvYiLCgwDJ0Rt\n93lZUCbNL0lXZImTtLGqUfJlMyGohSJSpwK/iH7NXvdfsL+zQ4l2azOYR5mIDZ\nzvk+8HJK6BZkUhV5d/fae+9/8OFH1/5Man3+xZdLN786KNIyD/kgTGWaH\nwWs4FIoPtBCS36",
  "6BZkUhV5d/fae+9/8OFH1/5Man3+xZdLN786KNIyD/kgTGWaH\nwWs4FIoPtBCS36U5ZwlgeSHwem64YfnPC9Eqvb1ZcZPEhYrEYmQaQiNlu4Og4j\n5v/nDgMdCVUHCdC6mM5+N+sMhfN+pv+8OuRov4GhpZbW3Wn98Wui3hRWv/eyM\nbn4zHo7TsEy40qFkRXHcX830ScVyLULJZzeGZcEzFp6ymB9DUbGEFydVPbqZfw\nsiYz9Kc/hT2q+j7x5R",
  "XHcX830ScVyLULJZzeGZcEzFp6ymB9DUbGEFydVPbqZfw\nsiYz9Kc/hT2q+j7x5RsaQoLpMATOjgpMDMBF3suNTRryeVUFmpuQqbhqJS+jr1\nTar8sch5qOUlFiYC+irH05YzkINCb0xVPwiTJOEQWaGaxu7s6rNIT8r6+TOZl\n1no3ZMIq8y1p7uL2oRmifiLSeV1Iqp5AqBx7Oq4r24h4HgAESPE5AqXkCdJj9B\n5PcRhcUkAQMP0il0LvJfzE",
  "iLSeV1Iqp5AqBx7Oq4r24h4HgAESPE5AqXkCdJj9B\n5PcRhcUkAQMP0il0LvJfzEjVSvMYctLRXhENCpnk0461TiyYyqSj7IHi+7d8A\nzgswNCv12HI0RzsZUzN5sdpPtV5UhUmhlvImYp53QMOWTSjKhrqFJKODTsWL9\nj6wVTp23i0qzuam4iyNrPu47OaV7UuOvUEWTBIoy7Vh1BloStP2YJgy35REMO\nPFNxK0KhVBFuZOngbdtjMTwWtzms",
  "V7UuOvUEWTBIoy7Vh1BloStP2YJgy35REMO\nPFNxK0KhVBFuZOngbdtjMTwWtzmsF+6XobFUn/OUMZMQHYfeZXMBXyr6eLmx\n/npz2jcFPvUnMFndQ1geN8OaNwKjamMzata5QibNFoTy9KJrmt4VJ6J7gBNA\nG+6Mhcqeke7XZdgyZrw8DYMNS8lP/6x9xOfnlSrZtuYL5JNqKgoM1dFJvw/Kh\nrDxQavL4jgyUslmjwI1JOXSji/o6ljOV7YJl",
  "fnlSrZtuYL5JNqKgoM1dFJvw/Kh\nrDxQavL4jgyUslmjwI1JOXSji/o6ljOV7YJlLPHRSEYlLoS7T9Ray6x9QR3Nk0\nQX2FgKkXfplQaJKjqCubgJHhFy6bjgUokGzRhDmRZlzsnJD61niNS6OS3mwl\nysuidUaYTueYPLxVFQhovDOb/i8ABlNGjyGaSlGrMcJXNqpnT6elho2GKu3V9P\neVN0WjE/2zbg37B7JRhyM9Gm3g+YmJR6K64D7FWZc",
  "McJXNqpnT6elho2GKu3V9P\neVN0WjE/2zbg37B7JRhyM9Gm3g+YmJR6K64D7FWZcklqM9qGuxXN/tWbX5+g\neytGOH6zYlqbftpdt2uFf0gJ9tOXq7RTxiUeiutoeUo9YjvagLncet1yjcLh\nuU5J653l02g53YaLlH+1PuGbmNimVY3Pbl8phE8KipqJ2imnCYyQ2ISwmZdeC/\n7GyJ+Di0bWaEBZ3CtHVTABLYy7xEJoQFpst3DXbGFa3HOqW2",
  "nCYyQ2ISwmZdeC/\n7GyJ+Di0bWaEBZ3CtHVTABLYy7xEJoQFpst3DXbGFa3HOqW2UymyCzCWHxMUv\nwqJsQFmMqxk7xlGUZEpsQyeME53FC85hKXNJeEYyx4yQJeVaUPk7UomgKUpa\nm3qaAx6IFOFGmyDWC7oyiucK0+hVazoKh64Gh5c0bBmqEITwNI2WP+cNu5yQK\ncYvNs6khyJpCV0QTuYGeHOvO7vyCqyJ1cEF1aeknphaUXlB5aekhpbi",
  "+cNu5yQK\ncYvNs6khyJpCV0QTuYGeHOvO7vyCqyJ1cEF1aeknphaUXlB5aekhpbil5Igi\nF5aSp5MgOrf0nNIDSw8oLS0tKR1YOqA0sjSi9JGljygNLQ0pXbd0nVJtKbkjhS\nuCpfuUTiydUHpk6RGlLy19SekTS59Q+srSV5S+tfQtpQ8sfUAps5RumHpBqXc\nUvLqIjWLF2jNLCUPvBXrN0h9LM0ozSh5Y+pHRsKXkqhuZpeT2Bi6MlkpKn1\nr",
  "Xc\nUvLqIjWLF2jNLCUPvBXrN0h9LM0ozSh5Y+pHRsKXkqhuZpeT2Bi6MlkpKn1\nr6lFJhKXl+C6Lnlj6nNLE0ofSZpc8ofWPpG0ofW/qY0thS8m4A7k4s3aPUvgWq\nCkp3Ld2l9MzSM/d7Ab6YxsC1MLdtBduUpamlG5aSp4U4FbC0lNyPxmp9qw2f\n9tEzmuRWnAHazM+P5rkPFIL7mDt2Wl+NDk/RWrBJ6TrGweLFymQUjTj5ZW+vg\ntLC0c3O",
  "uRWnAHazM+P5rkPFIL7mDt2Wl+NDk/RWrBJ6TrGweLFymQUjTj5ZW+vg\ntLC0c3On1f+7d2723cn+tfUN73fvW+873ut7v3j3vSfejfwQu9P7y/vb+f5\nf7y4fLr5T8a9b1r7TFfe53PsvgP3+fixg=</latexit>\na =\n2\n4\na1\na2\na3\n3\n5\n<latexit sha1_base64=\"yYwenEiqtEQKi9jT3S2VI9uBiSc=\">AW\nr3iclZhb9s2FIDVXbvulm5YXv",
  "se64=\"yYwenEiqtEQKi9jT3S2VI9uBiSc=\">AW\nr3iclZhb9s2FIDVXbvulm5YXvYiLCgwDJ0Rt93lZUCbNL0lXZImTtLGqUfJlMyGohSJSpwK/iH7NXvdfsL+zQ4l2azOYR5mIDZzvk+8HJK6BZkU\nhV5d/fae+9/8OFH1/5Man3+xZdLN786KNIyD/kgTGWaHwWs4FIoPtBCS36U5ZwlgeSHwem64YfnPC9Eqvb1ZcZPEhYrEYmQaQiNlu4Ogy",
  "WaHwWs4FIoPtBCS36U5ZwlgeSHwem64YfnPC9Eqvb1ZcZPEhYrEYmQaQiNlu4Ogyj\nwf/OHAY+FqoKE6VxMZ34w6g+H8H2n/r475Gq8gKOldXeav3xaHfFla89rMzuvnNeDhOwzLhSoeSFcVxfzXTJxXLtQgln90YlgXPWHjKYn4MRc\nUSXpxU9ehm/i2IjP0ozeFPab+OvntExZKiuEwCMKGDkwIzE3Sx41JHv5UQmWl5ipsGopK6evUN6nyxyL",
  "0ozeFPab+OvntExZKiuEwCMKGDkwIzE3Sx41JHv5UQmWl5ipsGopK6evUN6nyxyLnoZaXUGBhLqCvfjhOQs1JPTGUPGLME\n0SBpkZrm3szqo2h/ysrJM7m3WdjdoxibzKWHu6v6hFaJ6It5xUiumkisEHs+qivfiHgaCAxA9TkCqeAF1mvwEkd9HFBaTBAw8SKfQuch/MSNVK\n81jyElHe0U0KGSTzvWOrFgKpOsgeK79/yDeCwAEO/XochR3Owl",
  "SKfQuch/MSNVK\n81jyElHe0U0KGSTzvWOrFgKpOsgeK79/yDeCwAEO/XochR3OwlzE1mx+n+VTnSVWYG4hZyrmdRMw5JBJM6KuoUop4dCwY/2OrRdMnbaJS7O6q\n7mJIGs/7zo6p3lR465TR5AFizDuWnUEWRK2/pglDLclkcw4MQ3EbcqFYFWZg7eRp0285MBK/NaQb7petVCT95wxlxARg95lfwVTIu/p6urD9\neXLOa98U+NSfwGR1D2F53Ax",
  "MBK/NaQb7petVCT95wxlxARg95lfwVTIu/p6urD9\neXLOa98U+NSfwGR1D2F53Axr3giMqo3NqFnCpk0WxDK04uaXrjUHkmugM0Abzpylyo6B3tdl2CJWvCw9sw1LyU/PjH3k98elKtm1jvkg2oaK\nizFwVmfD/qGgMFxu8viCJy+VaPIgUE9eKuH8jqaO5Xhm0g9d1AQikmhL9H2F7HqHlNHcGfTBPUVAqZe+GVCoUmOoq5sAkaGX7hsOhZQiAY",
  "hm0g9d1AQikmhL9H2F7HqHlNHcGfTBPUVAqZe+GVCoUmOoq5sAkaGX7hsOhZQiAYZNmM\nMZVqUOScnP7SeIVLr5rSYC3Ox6p5QpRG65w0uF0dBGS4O5/yKwOU0aDJZ5CWasxylMypmdLp62GhYu5dn895U3RacX8bLNtD/oFs1OGIT8be\nL5iIlFHYnqgvsUZ12SWI72oK7Fcn23Z9Xm6x/I0o4drtuUpN62l27b4V7RA3625ejtFvGIR2J6mp7S",
  "Z12SWI72oK7Fcn23Z9Xm6x/I0o4drtuUpN62l27b4V7RA3625ejtFvGIR2J6mp7SD1iOdqDutx53HKNwuG6TUnqnefRaTvchY\nmWf7Q/4ZqZ26RUjs1tXyqHTQiLmoraKaYJj5HYhLCYlF0L/sfKnoCLR9dqQljcKURXMwEsjbnEQ2hCWGy2cNdsY1jdcqhbpXJbILMJoTFxyzBo\n25CWIypGDvFU5ZlSGxCJI8TnMcJzWOGpcwl4RnJHDNClpRr",
  "pXJbILMJoTFxyzBo\n25CWIypGDvFU5ZlSGxCJI8TnMcJzWOGpcwl4RnJHDNClpRrQeWTtCuZAJamqLWpozHogUwVarANYrmgK69wrjyFVrGiq3jganhwRcOaoQpNAEvb\nZI/5w23nJgtwis2zqSPJmUBWRhO4g50d6szv/oKoIndyQXRp6SWlF5ZeUHpo6SGluaXkiSCIXlhKnk6C6NzSc0oPLD2gtLS0pHRg6YDSyNKI0keW\nPqI0tDSkdN3SdU",
  "luaXkiSCIXlhKnk6C6NzSc0oPLD2gtLS0pHRg6YDSyNKI0keW\nPqI0tDSkdN3SdUq1peSOFK4Ilu5TOrF0QumRpUeUvrT0JaVPLH1C6StLX1H61tK3lD6w9AGlzFJG6YalG5RyS8mrgyBas3SN0sBS8uwHe83SHUo\nzSzNKH1r6kNKxpeSpGK5nlpLbG7gwWiopfWrpU0qFpeT5LYieW/qc0sTShNJnlj6j9I2lbyh9bOljSmNLybsBuDuxdI9S+",
  "iopfWrpU0qFpeT5LYieW/qc0sTShNJnlj6j9I2lbyh9bOljSmNLybsBuDuxdI9S+xaoKijdtXSX0jNLz9\nzvBfhiGgPXwty2FWxTmlqaUrpKXlSgFsJS0/J/WSk2rPa/G0TOa9FasEdrM34/GiS80gtuIO1Z6f50eT8FKkFn5CubxwsXqRASuFMP1pa6eO3sL\nRwcKfX/7l3b/feyv219g3tde9b7zve6/v/eLd954O97AC70/vb+8v71/lvLh8u",
  "L\nRwcKfX/7l3b/feyv219g3tde9b7zve6/v/eLd954O97AC70/vb+8v71/lvLh8uvl/9o1Peutcd87XU+y+I/OMDiyg=</latexit>\nb =\n2\n4\nb1\nb2\nb3\n3\n5\n<lat\nexit sha\n1_base64\n=\"Q32+u3\nZS3rPpGB\nqEuoGKSo\nj9upU=\">\nAXBXic\nlZjZbtw2\nFEDH6Zam\nm9MifinQ\nCjUCFEU6\n8CTp8lI\ngseNsdup\n1bMfWZEB\npKA1jipK\n12",
  "FEDH6Zam\nm9MifinQ\nCjUCFEU6\n8CTp8lI\ngseNsdup\n1bMfWZEB\npKA1jipK\n12OMI81z\n0Y/pW9LX\nf0a/oL/R\nS0gyje+m\nHGkiHved\nwuyS1eYk\nUWb6y8s/\nCtXfefe/\n9D65/eO\nOjz/59L\nPFm58fZH\nGR+rzvxz\nJOjzyWcS\nkU7+cil/\nwoSTmLPM\nkPvdM1zQ\n/PeZqJWO\n3nlwkfRC\nxUIhA+y\nE0XPzd9X\ngoVOlFL\nE/FZOqwY",
  "PM\nkPvdM1zQ\n/PeZqJWO\n3nlwkfRC\nxUIhA+y\nE0XPzd9X\ngoVOlFL\nE/FZOqwY\ndmbOq6rC\n3d1AX7vT\nV2uRsb5x\ncG13MiLJ\n+Uu3+xPT\n7xhbwD12\n6G7NHRv0\nGp1uLi80\nl2p/hxa6\nDWF5U7zt\nz28eWvk\njmK/iLjK\nfcmy7KS3\nkuSDkqW5\n8CWf3nCL\njCfMP2Uh\nP4GiYhHP\nBmWVs6lz\nGyIjJ4hT\n+Kdyp4q+\nXaNkUZd\nRh6YM",
  "f3nCL\njCfMP2Uh\nP4GiYhHP\nBmWVs6lz\nGyIjJ4hT\n+Kdyp4q+\nXaNkUZd\nRh6YMBx\nhpkO2thJ\nkQc/D0q\nhkiLnyq8\n7Cgrp5LG\njF8AZiZT\n7ubyEAvN\nTAWN1/DF\nLmZ/DMt1\nwFb/w4yh\nikBl3dX1\nnWjap5md\nFtWTadt\nZrxydyKu\nM1Wf781\nZEziPxhp\nNGKkU3co\nXAw2lZ8m\n7YxUBwAK\nLCYgVz6\nBNnR8vcH\nqIwhaVgM\nt6R7",
  "ziPxhp\nNGKkU3co\nXAw2lZ8m\n7YxUBwAK\nLCYgVz6\nBNnR8vcH\nqIwhaVgM\nt6R7hg7E\n5J0yrnIe\nSkpR0TDQ\nqJ5JOWtU\nYsWMqop\neyB4ji3H\nQ04bEDfq\nfahz9Ea7\nCVMTWf1c\nj7J06jMd\nAz3kDIV8\nqoLmLPp\nJ5R21CFl\nFDVb1m/Y\nmuXqdMmc\nXFSDTXVE\nWTtp20n\nT2le1Kjt\nVBFkwSYM\n21YVQZaE\nC8qIRQy\n3JSHMOHI\n0R",
  "XFSDTXVE\nWTtp20n\nT2le1Kjt\nVBFkwSYM\n21YVQZaE\nC8qIRQy\n3JSHMOHI\n0RG7KhRW\nBdmY2ns\ntftOdATv\nzUkC56Xt\nrZck/ecM\nZUQH4PTp\nX8GUz9v\n6Wjy3nVl\nyzitfF/j\nEGcNitau\nwNKynNes\nEZtXEptS\nscoVMmi0\nIpfF29S\njsag8Ee0\nJ6gA+dEU\nqVPCWdqc\nqwZbVYfc\nOTDUtJD\n/5vsDnw\nzKFX1s9H\n9INqGhrE\nh",
  "0\nJ6gA+dEU\nqVPCWdqc\nqwZbVYfc\nOTDUtJD\n/5vsDnw\nzKFX1s9H\n9INqGhrE\nhsDenw/2\nhoBLcwvL\n8ghcvlm\njxIFAtXi\nzh+o6Wjq\nV4Y+tItX\nZQEIpJkV\n+i4y9C1a\n5TRfBg4\nwiNFQK6X\nfhlQqFD\noK2rANah\nl+4GVs2k\nI8m6dz9\nGWcFSknF\nz+0nyFS6\nfqymAp9s\n2pfUKUW2\ntcNLue1o\nAw3h3N+R\nXUPZdSr\n8+nFhRqx",
  "F\nz+0nyFS6\nfqymAp9s\n2pfUKUW2\ntcNLue1o\nAw3h3N+R\nXUPZdSr\n8+nFhRqx\nFCVzopd0\n8srNcjhi\ntNfLXld\ntFohP9to\n+oNxweoU\nvs/Pht4\nPUJiUei\ntuDpx9qW\nJalP2hr\nvl3fHlm5\n8eo7srV\nDi2s3JWm\n3GaXdtrh\nXjICfbVp\nGu0k8YlF\nHoraEVK\nPWJb+oC1\n7Hjdts7C\n4dlOSdmd\n5tNoWd26\ni7R/sj3n\nO9GNSLEf",
  "F\nHoraEVK\nPWJb+oC1\n7Hjdts7C\n4dlOSdmd\n5tNoWd26\ni7R/sj3n\nO9GNSLEf\n6sS+Wbh\n3CYk7F3C\nrGEQ+RWI\newGBVtC/\n4fK3sCbh\n5tqw5hcT\nsTbU0HsD\nTiEk+hDm\nGxPsJts4\nlhdOibt\npVJpMxMu\nsQFp+wCM\n+6DmExp\nGJoFU9Zk\niCxDpE8j\nnEexzSPC\nZYSm4RXJ\nLGsCNlSt\ng2VjuO2p\nANYmqDeJ\npbOYAQyV\nqjDJoj",
  "E8j\nnEexzSPC\nZYSm4RXJ\nLGsCNlSt\ng2VjuO2p\nANYmqDeJ\npbOYAQyV\nqjDJojlj\nO68zLrzF\nNrFiu7iv\nq3j/hUd\n5w1qANY\n2iJnzHG3\nrIfMwynW\nr7CWJCcC\nWQlN4DZ2\ntqkze/rz\ngpI8yXnB\npaGXlF4Y\nekHpoaGH\nlKaGkjcC\nL9g1lLyd\neMG5oeU\nHh6QGl\nhaEFp39A\n+pYGhAaW\nPDX1MqW+\noT+maoWu\nU5oaSJ1K\n4Ixi6",
  "MG5oeU\nHh6QGl\nhaEFp39A\n+pYGhAaW\nPDX1MqW+\noT+maoWu\nU5oaSJ1K\n4Ixi6T+n\nY0DGlR4Y\neUfrS0Je\nUPjX0KaX\nHh5T+sb\nQN5Q+NPQ\nhpcxQRu\nm6oeuUck\nPJpwMvWD\nV0lVLPUP\nLuB2fN0G\n1KE0MTSh\n8Z+ojSka\nHkrRjuZ4\naSxu4MR\noqKX1m6D\nNKhaHk/c\n0LXhj6gt\nLI0IjS5\n4Y+p/S1o\na8pfWLoE\n0pDQ8m3A\nXg6",
  "qKX1m6D\nNKhaHk/c\n0LXhj6gt\nLI0IjS5\n4Y+p/S1o\na8pfWLoE\n0pDQ8m3A\nXg6MXSPU\nvMVqMwo3\nTF0h9IzQ\n8/s3wX4f\nBk928bcM\ng1sURobG\nlO6YSh5U\n4BHCUNPy\nfNkoJqr\n2uxrE7mu\nBWrOLazJ\n+Kw2yXmg\n5tzCmqvT\nrDa5PgVq\nzsdk6OsH\n8w8pkFK4\n0g8Xl3v4\nKywtHNzt\n9n7s3t+5\nv/xgtflC\ne73zZe\nbzredXue",
  "sH\n8w8pkFK4\n0g8Xl3v4\nKywtHNzt\n9n7s3t+5\nv/xgtflC\ne73zZe\nbzredXue\nnzoPO085\n2p9/xO/8\nu3Fr4auH\nrpd+W/lj\n6c+mvWr2\n20NT5otP\n6W/r7P5e\nCBIA=</l\natexit>2\n4\na1\na2\na3\n3\n5 =\n2\n4\nReLU[b1]\nReLU[b2]\nReLU[b3]\n3\n5\n1. Consider:\nwhere:\n2. We could equivalently write:\n3. Taking the derivative\n<latexit sha1_base64=\"/yDqcr4akJisw/46DNZC\niqBFY0M=\">AYnHi",
  "Taking the derivative\n<latexit sha1_base64=\"/yDqcr4akJisw/46DNZC\niqBFY0M=\">AYnHiclZhbT9xGFIA39JbSG2lVCalSaxUlqoULUl6eUmVQEhCIOW6QIJGnvH3gnjsfEFl\nj72l/T1/a/9N/0jO3dwefMSnQVsrPn+zwzPnPxUukyPJu98bM+9/8GH938ePaTz/7/Iu5W1/uZ3GR+r\nznxzJODz2WcSkU7+Uil/wSTmLPMkPvNMVzQ/OeZqJWO3lwk/jlioRC",
  "1/uZ3GR+r\nznxzJODz2WcSkU7+Uil/wSTmLPMkPvNMVzQ/OeZqJWO3lwk/jlioRCB8lkPo5NbMd26QMr90E5bmgknH9Q\nI2av30Rs5D+OahUKUXsTwVw5GDjmIn5dLoymFe9du5cw3v3jW9+C57iwV712zYexNaxh7Uxu+f82GsTetYe\nxVDXPVn2R9iEaBhe+B5Xro2OqvZ/7x47d7rwz3Wdrm7D8KOqYRDuAGkw/LVruF/V0GrzZG6hu9itPg4",
  "e+B5Xro2OqvZ/7x47d7rwz3Wdrm7D8KOqYRDuAGkw/LVruF/V0GrzZG6hu9itPg4tLD\nWFhU7z2Tq59Xf7cd+EXGV+5Jl2dFSN8mPS31evuSjWbfIeML8UxbyIygqFvHsuKym8si5DZG+E8Qp/Kncqa\nJXjyhZlGWXEczJ27rnGWY6aGNHR78dlwKlRQ5V37dUFBIJ48dvS6cvki5n8tLKDA/FdBXx8wGKIcVs+sq/\niFH0cRg8y4y6vbMFD1SPC",
  "dUFBIJ48dvS6cvki5n8tLKDA/FdBXx8wGKIcVs+sq/\niFH0cRg8y4y6vbMFD1SPCzolpJo1HbWa0cnchpxvLa3qQWkfNIvOkrRlUwReDgqS74YLmIgOACxyAmIFc\n+gzmrUA2cJUdg5JGDgXjyEzgXOzohUrXIeQk5a2muiQSGRfNiyVogFQxm1lF1QHOe2owGHCeg71Tz0ORqD3Y\nSp0fi4nA/zNCozHcMtpEyFvGoCTtmHhbWDVICYf6LesPbO",
  "wGHCeg71Tz0ORqD3Y\nSp0fi4nA/zNCozHcMtpEyFvGoCTtmHhbWDVICYf6LesPbO0wdokLk6qrqY6gqy9tO3kKc2L6redKoIsmI\nRh26oiyJKwz/dZxCDLTfkETjhydMSuCoVQSbmVhp7bYTHcFzc5jAeml7qyVJ/zlDGdEBWH36WzDl87a+E\nk9sZ5yc8rXBT50BjBY7UNYGtanNW4EzqJjahZ5QqZNFsQSuOLtql7Y1F5ItonqAN40RWpUMEV7",
  "BT50BjBY7UNYGtanNW4EzqJjahZ5QqZNFsQSuOLtql7Y1F5ItonqAN40RWpUMEV7W5Vgimrw\n+5dONW0kPzop8Wf+fC47Oplo/8j2YSKsiKxVaTD/6OiPtxZ4PkFETx4sUSDB4Fq8GIJ+zsaOpbia0j1dhBQ\nSgmRX6Jlr8IVfuYKoI7G0eorxDQ9cI3EwoNchC0ZR3QMnzDPZJlAvnoJP36H0Z0XKyeaH5jNEKl1vi6nQF\n6v2hiq10N43uJwcB",
  "chC0ZR3QMnzDPZJlAvnoJP36H0Z0XKyeaH5jNEKl1vi6nQF\n6v2hiq10N43uJwcBW4OJzKYd7KNenU8vLlSfpSiZQz2kwzdulsMSs63+asjrotUK+dl60x70C0an8H1+d\nrKOxyMkFnUkqgtuSq1SWJZ2oO6JtP1as/K9Tc/kqkdWly7KUm9TS/tsWd0gN+tmHp7QbxiEUdiepqekg9Y\nlnag7rsedywnYXFtZuS1DvOo9W2uBMTf9gb8Bz/Tjix",
  "mHp7QbxiEUdiepqekg9Y\nlnag7rsedywnYXFtZuS1DvOo9W2uBMTf9gb8Bz/TjixbKvb/ti6dYhLOZUzK1iHPEQiXUIi1HRtuA3VnYFX\nDzaVh3C4lYm2poOYKnPJT6FOoTFegm3zSaG1Q2LumFXmUwGyKxDWHzGInzWdQiLIRVDq3jKkgSJdYjkcYDzO\nKB5TLCU2CQ8IolRMiUsk2odBC3JR3A0hC1NrQ0Bj2QsUINkEsZ3TmZdaZp9AsVnQW9",
  "5TLCU2CQ8IolRMiUsk2odBC3JR3A0hC1NrQ0Bj2QsUINkEsZ3TmZdaZp9AsVnQW92wN96Y0nDNUoQ5ga\nZOsMcfdtC4yD6dYv2iwJDkRyEpoArews0Wd8d2fF5TkTs4Lg29pPTC0AtKDw9oDQ1lDwReMGOoeTpxAvOD\nT2ndN/QfUoLQwtKe4b2KA0MDSh9auhTSn1DfUpXDF2hNDeU3JHCFcHQPUoHhg4oPT0kNJXhr6i9Lmhzyl9\nbehrSt8Z+",
  "uhTSn1DfUpXDF2hNDeU3JHCFcHQPUoHhg4oPT0kNJXhr6i9Lmhzyl9\nbehrSt8Z+o7Sx4Y+pQZyihdNXSVUm4oeXgBcuGLlPqGUqe/WCtGbpFaWJoQukTQ59Q2jeUPBXD9cxQcnsD\nF0ZDJaVrhq5RKgwlz29e8NLQl5RGhkaUvjD0BaVvDX1L6TNDn1EaGkreDcDdiaG7lJq3QGVG6bah25SeGXpm\nfy/AJ8Po2Sbmpqlgk9LY0JjSdUPJkwLcS",
  "reDcDdiaG7lJq3QGVG6bah25SeGXpm\nfy/AJ8Po2Sbmpqlgk9LY0JjSdUPJkwLcSh6Su4nA9XsauO3TWRfC9SEW1iT8fHRJOeBmnALa3an8dFkfwrU\nhA9I1f3Jy9SIKWw05/MLSzht7C0sH9vcemXxQfbDxYeLTdvaG92vul83/mhs9T5tfOo87yz1el1/Jk/Z/6a\n+Xvmn/lv5/Mr8+/rNWZG80xX3Van/n9/wBKoJfw</latexit>\n@a\n@b",
  "/Jk/Z/6a\n+Xvmn/lv5/Mr8+/rNWZG80xX3Van/n9/wBKoJfw</latexit>\n@a\n@b =\n2\n64\n@a1\n@b1\n@a1\n@b2\n@a1\n@b3\n@a2\n@b1\n@a2\n@b2\n@a2\n@b3\n@a3\n@b1\n@a3\n@b2\n@a3\n@b3\n3\n75 =\n2\n4\nI[b1 > 0]\n0\n0\n0\nI[[b2 > 0]\n0\n0\n0\nI[b3 > 0]\n3\n5\n4. We can equivalently pointwise multiply by diagonal\n<latexit sha1_base64=\"dyXUwN+YjCvLH3D9DuTkJnFtg0=\"\n>AWk3iclZhb9s2FIDV7",
  "sha1_base64=\"dyXUwN+YjCvLH3D9DuTkJnFtg0=\"\n>AWk3iclZhb9s2FIDV7tZ1t3RDs4e9CAsKDMNmOEN3AQYMbdK0TZMuThMnaeM0oGRKZkNRikQlTgX/mr1uP2j/ZoeSbFbnMA8zk\nJo93ydeDkmJVpBJUeh+/98bN97/4MP7r18e1Pv3s8y+W7nx5UKRlHvJhmMo0PwpYwaVQfKiFlvwoyzlLAskPg7N1w8veF6IVO3\nrq4yfJCxWIhIh0",
  "KRlHvJhmMo0PwpYwaVQfKiFlvwoyzlLAskPg7N1w8veF6IVO3\nrq4yfJCxWIhIh0xA6Xfp6lDA9CYJqc3Y8CqLgj/7JKB2n+nRpd/r1x+fFlbworXfgand+6OR+M0LBOudChZURyv9jN9UrFci1Dy\n2e1RWfCMhWcs5sdQVCzhxUlVj2Dm34PI2I/SHP6U9uvou1dULCmKqyQA0/S3wMwEXey41NFvJ5VQWam5CpuGolL6OvVNOvyxyHmo5R\nUWJ",
  "vou1dULCmKqyQA0/S3wMwEXey41NFvJ5VQWam5CpuGolL6OvVNOvyxyHmo5R\nUWJgL6KsfTljOQg1Juz1S/DJMk4SpcTVa29idVaOAx0JV/LysEzibdZ2N2uFQvM5Y29xf1CI0T8RbTiqpFVPJNQKPZ1XFe3EPA8EB\niB4nIFW8gDruY78VURhwUjAwIN0Cp2L/BczUrXSPIacdLRXRINCJvm0Y60TC6Yy6Sh7oPj+Pd8ArnOYBegqfHE0B3sZU7",
  "2L/BczUrXSPIacdLRXRINCJvm0Y60TC6Yy6Sh7oPj+Pd8ArnOYBegqfHE0B3sZU7P5dZpPdZ\n5UhYnhFnKmYl43AUMOmTQj6hqlBIuDTvWn9h6wdRZm7g0q7uamwiy9vOuo3OaFzXuOnUEWbAI465VR5AlYXuPWcIgy235FAac+Cb\niVoXCqiALc5CnQbftzETw2pxmsF+63kZF0n/BUEZMAHaf+RZMhbyr6cL258n56L2TYFP/QlMVvcSlsfNsO",
  "ETw2pxmsF+63kZF0n/BUEZMAHaf+RZMhbyr6cL258n56L2TYFP/QlMVvcSlsfNsOaNwKja2Iyada6QSbMFo\nTy97JqmNw6VZ6I7QBPAm67MhYre0X6oS7BkTXj0Aw1LyU/rH3M5+eVH2zbcw/JtQUVFmropM+H9UNIYHCl5fEMGTl0o0eRCoJy+\nVcH9HU8dyvLBNpJ47KAjFpNBXaPuLWHWvqSO4s2mC+goBUy98M6HQJEdRVzYBI8M3PBodCyh",
  "yvLBNpJ47KAjFpNBXaPuLWHWvqSO4s2mC+goBUy98M6HQJEdRVzYBI8M3PBodCyhEgwybMYyLcqck5sfWs8QqXVzW8yF\neVh1b6jSCN37BpeLq6AMD4cLfs3lAcpo0OQzSEs1ZjlK5tRM6fT1qNCwxVy7v57ypui0Yn6+1bYH/YLZKcOQn59u4fmIiUdieqCs4\nizLksR3tQ12K5vtuzauv192Rpxw7XbUpSb9tLt+1wr+kBP929HabeMSijkR1t",
  "s4\nizLksR3tQ12K5vtuzauv192Rpxw7XbUpSb9tLt+1wr+kBP929HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN2mJPXO8+i0He7CRMs/\n2p9wzcwxKZVjc+xL5agJYVFTUTvFNOExEpsQFpOya8H/sbIn4OHRtZoQFgeF6GomgKUxl3gITQiLzRbum0Mq9sOdutMplNkNmEsP\niEJXjUTQiLMRVjp3jGsgyJTYjkcYLzOKF5zLCUuSQ8I5ljRsiSci",
  "tMplNkNmEsP\niEJXjUTQiLMRVjp3jGsgyJTYjkcYLzOKF5zLCUuSQ8I5ljRsiSci2ofJ2JRPA0hS1NnU0Bj2QqUINtkEsF3TlFc6Vp9AqVnQVD10\nND69pWDNUoQlgaYfsMX+049xkAU4xHLNcSc4EsjKawAF2BtSZn/6CqCInuSC6svSK0ktLyk9tPSQ0txS8osgiF5YSn6dBNGFpReUH\nlh6QGlpaUnp0NIhpZGlEaWPLX1MaWhpSOm6peuUak",
  "S8osgiF5YSn6dBNGFpReUH\nlh6QGlpaUnp0NIhpZGlEaWPLX1MaWhpSOm6peuUakvJiRSeCJbuUzqxdELpkaVHlL609CWlTy19SukrS19R+tbSt5Q+tPQhpcxSRum\nGpRuUckvJq4MgWrN0jdLAUvLbD/apQNKM0szSh9Z+ojSsaXkVzE8zywlxt4MFoqKd20dJNSYSn5/RZEzy19TmliaULpM0ufUfrG\n0jeUPrH0CaWxpeTdAJxOLN2j1L4Fqg",
  "0dJNSYSn5/RZEzy19TmliaULpM0ufUfrG\n0jeUPrH0CaWxpeTdAJxOLN2j1L4FqgpKdy3dpfTc0nP3ewG+mMbAtTB3bAU7lKaWpRuWUp+KcBRwtIzcp6MVHtXm79tIve1SC24g7\nUZn19Nch6pBXew9u40v5rcnyK14BPS9Y2DxYsUSCnc6U+XVlbxW1haOPipt/pL7/7u/ZUHa+0b2lveN963nfeqver98B76g28oRd6\nM+8v72/vn+W7y78vry",
  "ipt/pL7/7u/ZUHa+0b2lveN963nfeqver98B76g28oRd6\nM+8v72/vn+W7y78vry0/atSbN9prvI6n+Xn/wELotf9</latexit>I[b > 0]\u2299\n<latexit sha1_base64=\"ZgI+QBtlS/dTX3t6x21c9\n/heZOc=\">AYAHiclZhb9s2FICd7NZlt3TDEGx7ERZsGIbOiNPu8tgmTW9Jl6uTtHFqUDIls6EoRZfEqeC\nX7c/sbdjr/sn+wn7FDiXZjM6h",
  "IbOiNPu8tgmTW9Jl6uTtHFqUDIls6EoRZfEqeC\nX7c/sbdjr/sn+wn7FDiXZjM6hgSxAa/p8n0jq8FCS5cZSpNnKyj9z82+9/c67916f+GDz/6+JPF258eplG\neLzrRTJKjl2WcikU72Yik/w4TjgLXcmP3LN1zY8ueJKSB1kVzE/DVmghC8lkGof3vu957LA6EKN2RZIkb\njhZ6fMK/oxSzJBJMO6xed8dh8d8vzrcO9VZv6N2lXq93g3FXbzju",
  "A6EKN2RZIkb\njhZ6fMK/oxSzJBJMO6xed8dh8d8vzrcO9VZv6N2lXq93g3FXbzjuLA+Pu3rDce/ecNxZHh4Xvi/0uBpMc95f\nXF5pr5R/Dm106sZyq/7b6d/+fNAbRF4ecpV5kqXpSWclzk4LPYnOXSfpzxm3hkL+Ak0FQt5elqU5TJ2voHI\nwPGjBP6pzCmj148oWJimV6ELJkxwmGKmgzZ2kmf+L6eFUHGeceVA/m5dLI0bXnDETCvUxeQYN5i",
  "j148oWJimV6ELJkxwmGKmgzZ2kmf+L6eFUHGeceVA/m5dLI0bXnDETCvUxeQYN5iYC5Ot6Q\nQboyqNCFnuKXhSGDLTW9vYhaRVpcnP87JadquOxuloxM5y1h7ejDtRWQ8FG846aRUdCczB6Mi4K3gzYG\ngMQbU5ApHgKfer8uL7TQR2pwQM3I1GMDnf2RuTrlXGA8hJQ3tJNGjEko8a1jqxYCnDhrIPiuN842jAoQA9\np6xDj6M12I+ZGk+Oy/go",
  "lXGA8hJQ3tJNGjEko8a1jqxYCnDhrIPiuN842jAoQA9\np6xDj6M12I+ZGk+Oy/goS8Ii1TE8QsJUwMsh4JQ9KPI9bKhcSjUa1i/YmuPqbM6cVFcTjXREWQdJE0nS2he1\nKDplBFkQREGTauMIEvCtXTAQgZrt9OHQ0RG7KhRWBSnMnSRym2PHOoJrcxTDfml6GwVJ/wVDGdEB2H36U\nzDl8a+Hk1tZ5Kci9LXDT5yhrBYzUNYElSnNRkEzqOjalZ",
  "6GwVJ/wVDGdEB2H36U\nzDl8a+Hk1tZ5Kci9LXDT5yhrBYzUNYElSnNRkEzqOjalZ5gqZNFsQSqLpqlnY1F5LJonqAN40+WJUP417\nU7ZgpLV4d4dONUkl/zkh/aPfHRarOhto/8j2YSO0jy2daTD/6OjAdy9cX1BC9eJNHiQaBcvEjC9R0tHUtwY\netIuXbQEIpJkV2h7S8C1TymjODJRiGaKwR0v/DJhEKL7PtNWQe0DJ/wHGIpIA+dpFedoyej",
  "EIpJkV2h7S8C1TymjODJRiGaKwR0v/DJhEKL7PtNWQe0DJ/wHGIpIA+dpFedoyejNE84ufiheoZIqe\nvLYiL0zap5QZVaF43uJweBW24OVzwGYe7KNulU83ytWAJSiZI72ko1e9NIMtZtv95ZJXTasV8PNejyYF6\nxO7n8vL+J1yMgFnUk6gse/Kx9SWJZxoO+puV6fWbF5qvSWkHFtduStJvPUu7bXFnzICfb1lmu0U8YlFHor\n7qGVKPWJbxoC",
  "O+puV6fWbF5qvSWkHFtduStJvPUu7bXFnzICfb1lmu0U8YlFHor\n7qGVKPWJbxoC97HrdsZ2Fx7aYk/U7yaLUt7tRE5e8fDHnG9GNSJAf6sS+SvSqExYyKmVWMQh4gsQphMcybFn\nzHyr6Am0fTqkJY3ElFU9MBLA24xKdQhbBYbeGmWcewumVRt+wqk/EQmVUIi49ZiM+6CmExoGJgFc9YHCOxCp\nE8DnEehzSPMZim4RXJLasCkpW0Elw6gp6Q",
  "VUIi49ZiM+6CmExoGJgFc9YHCOxCp\nE8DnEehzSPMZim4RXJLasCkpW0Elw6gp6QCWRmi0kWUwmIGMFBqwDmI5pZWXWitPoSpWtIq7toG7MwbOGOp\nQB7C0TfaY09u2bjIXpxges2xJjgWyYprAHezsUGfy9Of6BXmSc/0rQ68ovT0ktIjQ48oTQwlvwhcf89Q8uv\nE9S8MvaD0NBDSnNDc0q7hnYp9Q31KX1k6CNKPUM9StcNXac0M5Q8kcIdwd",
  "9Q8uv\nE9S8MvaD0NBDSnNDc0q7hnYp9Q31KX1k6CNKPUM9StcNXac0M5Q8kcIdwdADSoeGDik9NvSY0heGvqD0iaF\nPKH1p6EtK3xj6htIHhj6glBnKN0wdINSbih5deD6a4auUeoaSn7wV4zdIfS2NCY0oeGPqR0YCj5VQz3M0P\nJ4w3cGA2VlD419CmlwlDy+831nxv6nNLQ0JDSZ4Y+o/S1oa8pfWzoY0oDQ8m7AXg6MXSfUvMWqEgp3TV0l9",
  "Dy+831nxv6nNLQ0JDSZ4Y+o/S1oa8pfWzoY0oDQ8m7AXg6MXSfUvMWqEgp3TV0l9Jz\nQ8/t7wX4dBldW2Fumw62KY0MjSjdNJT8UoBHCUPyPOkr+qr2uRtE7mu+WrKLazO+ORoknNfTbmF1VenydHk\n+uSrKR+SqW8cTl+kQErhSt9fXO7gt7C0cbja7vzUvrd7b/n+Wv2G9lbrq9bXre9andbPrfutJ62dVrflzf07\nvzj/xfyXS78t/bH059Jf",
  "rd7b/n+Wv2G9lbrq9bXre9andbPrfutJ62dVrflzf07\nvzj/xfyXS78t/bH059JflTo/Vx/zWavxt/T3f7ZMY2c=</latexit>2\n64\n@a1\n@b1\n@a2\n@b1\n@a3\n@b1\n@a1\n@b2\n@a2\n@b2\n@a3\n@b2\n@a1\n@b3\n@a2\n@b3\n@a3\n@b3\n3\n75\n<latexit sha1_base64=\"/yDqcr4akJisw/46DNZC\niqBFY0M=\">AYnHiclZhbT9xGFIA39JbSG",
  "1_base64=\"/yDqcr4akJisw/46DNZC\niqBFY0M=\">AYnHiclZhbT9xGFIA39JbSG2lVCalSaxUlqoULUl6eUmVQEhCIOW6QIJGnvH3gnjsfEFl\nj72l/T1/a/9N/0jO3dwefMSnQVsrPn+zwzPnPxUukyPJu98bM+9/8GH938ePaTz/7/Iu5W1/uZ3GR+r\nznxzJODz2WcSkU7+Uil/wSTmLPMkPvNMVzQ/OeZqJWO3lwk/jlioRCB8lkPo5NbMd",
  "nxzJODz2WcSkU7+Uil/wSTmLPMkPvNMVzQ/OeZqJWO3lwk/jlioRCB8lkPo5NbMd26QMr90E5bmgknH9Q\nI2av30Rs5D+OahUKUXsTwVw5GDjmIn5dLoymFe9du5cw3v3jW9+C57iwV712zYexNaxh7Uxu+f82GsTetYe\nxVDXPVn2R9iEaBhe+B5Xro2OqvZ/7x47d7rwz3Wdrm7D8KOqYRDuAGkw/LVruF/V0GrzZG6hu9itPg4tLD\nWFhU7z2",
  "Z/7x47d7rwz3Wdrm7D8KOqYRDuAGkw/LVruF/V0GrzZG6hu9itPg4tLD\nWFhU7z2Tq59Xf7cd+EXGV+5Jl2dFSN8mPS31evuSjWbfIeML8UxbyIygqFvHsuKym8si5DZG+E8Qp/Kncqa\nJXjyhZlGWXEczJ27rnGWY6aGNHR78dlwKlRQ5V37dUFBIJ48dvS6cvki5n8tLKDA/FdBXx8wGKIcVs+sq/\niFH0cRg8y4y6vbMFD1SPCzolpJo1HbWa",
  "6cvki5n8tLKDA/FdBXx8wGKIcVs+sq/\niFH0cRg8y4y6vbMFD1SPCzolpJo1HbWa0cnchpxvLa3qQWkfNIvOkrRlUwReDgqS74YLmIgOACxyAmIFc\n+gzmrUA2cJUdg5JGDgXjyEzgXOzohUrXIeQk5a2muiQSGRfNiyVogFQxm1lF1QHOe2owGHCeg71Tz0ORqD3Y\nSp0fi4nA/zNCozHcMtpEyFvGoCTtmHhbWDVICYf6LesPbO0wdokLk6qrq",
  "ORqD3Y\nSp0fi4nA/zNCozHcMtpEyFvGoCTtmHhbWDVICYf6LesPbO0wdokLk6qrqY6gqy9tO3kKc2L6redKoIsmI\nRh26oiyJKwz/dZxCDLTfkETjhydMSuCoVQSbmVhp7bYTHcFzc5jAeml7qyVJ/zlDGdEBWH36WzDl87a+E\nk9sZ5yc8rXBT50BjBY7UNYGtanNW4EzqJjahZ5QqZNFsQSuOLtql7Y1F5ItonqAN40RWpUMEV7W5Vgimrw\n+5",
  "YGtanNW4EzqJjahZ5QqZNFsQSuOLtql7Y1F5ItonqAN40RWpUMEV7W5Vgimrw\n+5dONW0kPzop8Wf+fC47Oplo/8j2YSKsiKxVaTD/6OiPtxZ4PkFETx4sUSDB4Fq8GIJ+zsaOpbia0j1dhBQ\nSgmRX6Jlr8IVfuYKoI7G0eorxDQ9cI3EwoNchC0ZR3QMnzDPZJlAvnoJP36H0Z0XKyeaH5jNEKl1vi6nQF\n6v2hiq10N43uJwcBW4OJzKYd7KN",
  "DPZJlAvnoJP36H0Z0XKyeaH5jNEKl1vi6nQF\n6v2hiq10N43uJwcBW4OJzKYd7KNenU8vLlSfpSiZQz2kwzdulsMSs63+asjrotUK+dl60x70C0an8H1+d\nrKOxyMkFnUkqgtuSq1SWJZ2oO6JtP1as/K9Tc/kqkdWly7KUm9TS/tsWd0gN+tmHp7QbxiEUdiepqekg9Y\nlnag7rsedywnYXFtZuS1DvOo9W2uBMTf9gb8Bz/TjixbKvb/ti6dYh",
  "iepqekg9Y\nlnag7rsedywnYXFtZuS1DvOo9W2uBMTf9gb8Bz/TjixbKvb/ti6dYhLOZUzK1iHPEQiXUIi1HRtuA3VnYFX\nDzaVh3C4lYm2poOYKnPJT6FOoTFegm3zSaG1Q2LumFXmUwGyKxDWHzGInzWdQiLIRVDq3jKkgSJdYjkcYDzO\nKB5TLCU2CQ8IolRMiUsk2odBC3JR3A0hC1NrQ0Bj2QsUINkEsZ3TmZdaZp9AsVnQW92wN96Y0nDNU",
  "lRMiUsk2odBC3JR3A0hC1NrQ0Bj2QsUINkEsZ3TmZdaZp9AsVnQW92wN96Y0nDNUoQ5ga\nZOsMcfdtC4yD6dYv2iwJDkRyEpoArews0Wd8d2fF5TkTs4Lg29pPTC0AtKDw9oDQ1lDwReMGOoeTpxAvOD\nT2ndN/QfUoLQwtKe4b2KA0MDSh9auhTSn1DfUpXDF2hNDeU3JHCFcHQPUoHhg4oPT0kNJXhr6i9Lmhzyl9\nbehrSt8Z+o7Sx4Y+pQZy",
  "DF2hNDeU3JHCFcHQPUoHhg4oPT0kNJXhr6i9Lmhzyl9\nbehrSt8Z+o7Sx4Y+pQZyihdNXSVUm4oeXgBcuGLlPqGUqe/WCtGbpFaWJoQukTQ59Q2jeUPBXD9cxQcnsD\nF0ZDJaVrhq5RKgwlz29e8NLQl5RGhkaUvjD0BaVvDX1L6TNDn1EaGkreDcDdiaG7lJq3QGVG6bah25SeGXpm\nfy/AJ8Po2Sbmpqlgk9LY0JjSdUPJkwLcSh6Su4nA9Xsa",
  "Jq3QGVG6bah25SeGXpm\nfy/AJ8Po2Sbmpqlgk9LY0JjSdUPJkwLcSh6Su4nA9XsauO3TWRfC9SEW1iT8fHRJOeBmnALa3an8dFkfwrU\nhA9I1f3Jy9SIKWw05/MLSzht7C0sH9vcemXxQfbDxYeLTdvaG92vul83/mhs9T5tfOo87yz1el1/Jk/Z/6a\n+Xvmn/lv5/Mr8+/rNWZG80xX3Van/n9/wBKoJfw</latexit>\n@a\n@b",
  "/Jk/Z/6a\n+Xvmn/lv5/Mr8+/rNWZG80xX3Van/n9/wBKoJfw</latexit>\n@a\n@b =\n2\n64\n@a1\n@b1\n@a1\n@b2\n@a1\n@b3\n@a2\n@b1\n@a2\n@b2\n@a2\n@b3\n@a3\n@b1\n@a3\n@b2\n@a3\n@b3\n3\n75 =\n2\n4\nI[b1 > 0]\n0\n0\n0\nI[[b2 > 0]\n0\n0\n0\nI[b3 > 0]\n3\n5",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKD",
  "BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY",
  "LJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzou",
  "fwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL",
  "JyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19N",
  "ab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2Fg",
  "Mw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16",
  "g37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0i",
  "aCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGd",
  "+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0",
  "ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P",
  "YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY",
  "Take derivatives of \noutput with respect to \nintermediate quantities\n<latexit sha1_base64=\"4CBrqWxMv9p5w4tTB6\nHKgiX37+Q=\">AY9niczZhb9s2FIDdZJcuzTdMCDAXoQFHbqhC+K2u7wMaJOmt6TL1UnaODUomZLZ\nUJQiUYlTwX9lb8Ne93f2tp+yQ8k2o3OYQP2MAOp2fN9vB2Skiw/lSLXy8t/XJuZfefd96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM5",
  "d96/sHchx9\n9/MmN+Zuf7udJkQW8EyQyQ59lnMpFO9oSU/TDPOYl/yA/9k1fCDM57lIlF7+iLlxzGLlAhFwDSEejd\nnLrphxoKym7JMCya9LpeyV4rR6FLID8Ne2R6NvrJw7YfDnrlXadN1ZCqde25ruShvu1s/B5t/K678X\nukahLRnzF/O6NupmIBvpr9ud+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/v",
  "d+6c5WTY5cY67ZSdKcFqXftvMvIfpPv/kOje/OLy0nL18WihPS4st\nsafrd7Nz/vdfhIUMVc6kCzPj9rLqT4uTeuB5DCtIucpC05YxI+gqFjM8+OyOiQj7xZE+l6YZPCntFdFL\n9coWZznF7EPZsz0IMfMBF3sqNDhj8elUGmhuQrqjsJCejrxzInz+iLjgZYXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+",
  "YXUGBJmCsXjBgkCsN53Ku\nq/h5kMQxU/2yu7K2DenyeSRUyU+L6oya5brsrFUOh+JVxsqzvWkrQvNYvOWkUoxjVwh8GhUlnwpWsJA\ncABiROQKJ5DmyY/fui1EYVrkgQM3E+GMLjQ2xmRpXmEeSkob0iGhRSyYcNa5VYsJRxQ9kFxfNueQZ\nwncEqwFDhi6M12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmg",
  "12E2ZGk3qaT7UWVzmJoZ7yJiKeNUFTDmA7b2DVICVWDhvUztnaYOhknLkmroWYmgqy\n9rOnojOZF9ZtOFUEWbMKoaVURZEm4g/RZzCDL43IPJhx7JuJWhcKqIBtzK0v8Zt+pieC9OUzhvDS9tZ\nKk/4yhjJgAnD7zLZgKeFNfTa2N0nOWeWbAh96A1isZhWRfW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0D",
  "fW0Jp3ArMaxETWrXCGTZgtCWXLeNM1oHC\npPRXOCJoAPXZEJFV7S7lQl2LIm3L0DU80KyY+XfqOD4/LZXNszD8km9BQXqSuhkz4XzTUh2cWvL8g\nhcvkWjxIFAtXiLh+o6WjmV4Y5tItXZQEIpJoS/Q8ReRatapIniwSYzGCgHTLnwzodAih2FTNgEjwzc8f\nTk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZ",
  "Tk2UIAmGdRzDGSFxknFz+0nyFS6eaymAlzs2peUKURmtcNLqe1oAw3hzN+RXUfZdSv8+knheqzDCVz\naJZ0+LqbazhirtNfLXldFoRP10f9wfjgtUpgoCf9tbxekTEo5EbcHjrMtSxHf9DWdLteHlm5/vo\nbsrUjh+s2JWl3PEq37XCvGAE/3XCMdoN4xKORG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3U",
  "RG2NR0g9Yjn6g7bcedxwzcLhuk1J2p3k0Wk73KmJtn+\n4N+CamcekRPbNY18iu3UIi5qK2ikmMY+QWIewGBdNC/6PlV0BN4+mVYewuJWLpmYCWOpziadQh7BYH+\nGmOY5hdcOhbrhVJtMBMusQFp+wGM+6DmExomLkFE9YmiKxDpE8DnAeBzSPKZSl4RXJHWsCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61g",
  "sCNlSrg2VDZ\nKmZAJYGqLeho7OYAQyUajDcRDLOd15uXPnKbSLFd3FHVfHnSs61gw1aAJY2iRnzOtuOg+Zj1Mj1muJ\nKcCWSlN4BZ2tqgzefrzw5I8yfnhaUXlJ5bek7pgaUHlGaWkl8EfrhjKfl14odnlp5Rum/pPqWFpQWlH\nUs7lIaWhpQ+tvQxpYGlAaWrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPK",
  "Wrlq5Sqi0lT6RwR7B0j9KBpQNKDy09pPSlpS8pfWrpU0pfWfqK0reWvqX0\noaUPKWMkrXLF2jlFtKXh34YqlK5T6lpLfnDWLN2iNLU0pfSRpY8o7VtKfhXD/cxS8ngDN0ZLJaXP\nLH1GqbCU/H7zwxeWvqA0tjSm9Lmlzyl9Y+kbSp9Y+oTSyFLybgCeTizdpdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlT",
  "pdS+BSpzSrct3ab01NJT93s\nBPl1G37UxN20Dm5QmliaUrltKfinAo4SlJ+R5MlTjq9rkbRO5roVqyh1snPFJbZLzUE25g42vTpPa5P\noUqikfkKGv7U9fpEBK4Urfm19s47ewtLB/d6n9/dL97fuLD1bGb2ivt75ofdm63Wq3fmg9aD1tbU6rW\nDmz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n=",
  "mz9n3Zm/Mzi8MF35Z+HXht1qduTau81mr8Vn4/S+J08WC</latexit>@`i\n@f1\n= @h2\n@f1\n@f2\n@h2\n\u2713@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n@`i\n@f0\n= @h1\n@f0\n@f1\n@h1\n\u2713@h2\n@f1\n@f2\n@h2\n@h3\n@f2\n@f3\n@h3\n@`i\n@f3\n\u25c6\n<latexit sha1_base64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9C",
  "64=\"BZypoYz/Da/RHUFJTym\nnEPT+04=\">AXJniclZhb9s2FICd7tZ1l6Yblpe9CAsKDENnJG3X7WVYmzS9JV2uTtLGqUHJlMyGohSJ\nSpwK/j/DfszehmFv+yk7lGQzOocBNgOp2fN94uWQlGj5qRS5Xlr6e+7ae+9/8OFH1z+8cmn31+c/7WF/t\n5UmQB7wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87",
  "wWJTLJDn+VcCsV7WmjJD9OMs9iX/MA/WTX84IxnuUjUnr5I+XHMIiVCETANocH87/0wY0HZT1mB\nZNen0s5KMVkcinkh+GgvDuZ/IxdPxwNyntOl6rh4N7Eca3HztwbzKYX1zqLlUfjxaWm8Jip/lsDW59NewP\nk6CIudKBZHl+tLyU6uPSVBtIPrnRL3KesuCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UO",
  "uCERfwIiorFPD8uq6ROvNsQGXphksGf0l4VvXxFyeI8v4h9MG\nOmRzlmJuhiR4UOfzouhUoLzVQNxQW0tOJZ2bIG4qMB1peQIEFmYC+esGIQZI0zONvuLnQRLHTA3L/sraN\nuTJ5FQJT8tqjmdTNrOWuVwKF5lrDzfm9UiNI/FO04qRTyRUCjyZlybtRFwPBAYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgw",
  "AYguJyBRPIc6TX780Ft\nGFNawBAzcT8bQudDbmZCqleYR5KSlvSYaFLJxy1rlVgwlXFL2QXF8257BnCdwSxAV+GLoznYTZmaTK/TfK\nyzuMxNDLeQMRXxqgkYcgDregcbqpASLg1a1q/Y2mHqpElcklZdzUwEWXtZ29EZzYsatp0qgixYhFHbqiLI\nknDHGbKYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4",
  "KYQZab8gAGHsm4laFwqogC3MrS/x26mJ4LU5TmG/tL21kqT/jKGMmADsPvMtmAp4W19NZrY3Tc5\nZ5ZsCH3sjmKz2JSyL6mFNG4FRNbEJNatcIZNmC0JZct42TW8cKk9Fe4AmgDdkQkVXtLuVCVYsibcvwNDz\nQrJj7v/sDHx+WS2TbmH5JNqCgvUldFJvw/KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkR",
  "KhrCMw6vL4jgyUskmjwIVJOXSLi/o6ljGV7YJlLNHRSEYlLo\nC7T9RaTa1QR3NkRn2FgKkXvplQaJLDsC2bgJHhG57WjgUoEG9RgDmeRFxsnND61niFS6uS1mwjys2j\ndUaYT2fYPL2VQhofDGb/ich9l1K/z6SeFGrIMJXNspnT8p9r2GKu3V9NeV10WhE/XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J",
  "XW/ag37B7BRBwE8H6\n3g+ImJR6K64HjkrEsSy9Ee1DVbrpd7Vq6/+Y4s7cjhuk1J6m16bYd7hU94Kcbjt5uEI9Y1JGorqaH1CO\nWoz2oy53HDdcoHK7blKTeaR6dtsOdmWj5h3sjrpk5JiVyaI59iezXISxqKmqnmMQ8QmIdwmJctC34P1Z2BT\nw82lYdwuJWLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRF",
  "WLtqaCWBpyCUeQh3CYr2F2YTw+qGQ91wq0ymI2TWISw+ZTEedR3CYkTFyCmesDRFYh0ieRzh\nPI5oHlMspS4Jz0jqmBGypFwLKhslbckEsDRGrY0djUEPZKJQg0Qyzldeblz5Sm0ihVdxT1Xw70rGtYMVWg\nCWNoke8zrbzo3mY9TDMcsV5JTgayUJnALO1vUmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLS",
  "UmZ7+/LAkJzk/vLD0gtJzS8pPbD0gNLMUvKLwA93LCW/T\nvzwzNIzSvct3ae0sLSgtGdpj9LQ0pDSJ5Y+oTSwNKB01dJVSrWl5EQKTwRL9ygdWTqi9NDSQ0pfWfqK0meW\nPqP0taWvKX1n6TtKH1n6iFJmKaN0zdI1Srml5NWBH65YukKpbyn57Qd7zdItSlNLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfU",
  "NLU0ofW/qY0qGl5FcxPM\n8sJcbeDBaKil9bulzSoWl5PebH7609CWlsaUxpS8sfUHpW0vfUvrU0qeURpaSdwNwOrF0l1L7FqjMKd2d\nJvSU0tP3e8F+GwafdfC3LQVbFKaWJpQum4p+aUARwlLT8h5MlTNXW36tonc10I14w7WZHx6Ncl5qGbcwZq\n70/Rqcn8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfd",
  "n8K1YyPSNfX9mcvUiClcKcfzC8u47ewtLB/t7v8oHt/+/7iw5XmDe31ztedbzrfdpY7P3Yedp51tj\nq9TjA3P/dg7pe5hwu/Lfyx8OfCX7V6ba65stO67Pwz7/X2hOG</latexit>@`i\n@f2\n= @h3\n@f2\n@f3\n@h3\n@`i\n@f3\n<latexit sha1_base64=\"Ngj2wtcr8WoZYc/V2IjUXbzLzd4=\"\n>AWkniclZhb9s2FIDV7tZ1t3SXvOxFWFBgGDrDK",
  "tcr8WoZYc/V2IjUXbzLzd4=\"\n>AWkniclZhb9s2FIDV7tZ1t3SXvOxFWFBgGDrDKboL9jC0SdM2Tbo4TZykiVODkimZDUpEpU41fxn9r9of2bHcqyWZ3DPMxAa\nvZ8n3g5JCVaQSZFobvdf2/cfO/9Dz786NbHtz/59LPv1i68+VBkZ5yPthKtP8KGAFl0LxvhZa8qMs5ywJD8MztYNP7zgeSFSta+\nvMn6asFiJSIRMQ2i4tDxImB4HQbU5PRkE",
  "vhZa8qMs5ywJD8MztYNP7zgeSFSta+\nvMn6asFiJSIRMQ2i4tDxImB4HQbU5PRkEUTSs7k9/754Ol1a6nW798WlhtSmseM2nN7zWgwSsMy4UqHkhXFyWo306cVy7UIJZ/e\nHpQFz1h4xmJ+AkXFEl6cVvUApv5diIz8KM3hT2m/jr57RcWSorhKAjBNdwvMTNDFTkod/XpaCZWVmqtw1lBUSl+nvsmGPxI5D7W8g\nILcwF9cMxy1moIWe3B4pfhm",
  "NDFTkod/XpaCZWVmqtw1lBUSl+nvsmGPxI5D7W8g\nILcwF9cMxy1moIWe3B4pfhmSMDWqBmsbu9NqEPBYqIqfl3X+ptO2s1E7HIrXGWub+4tahOaJeMtJbViKrlG4PG0qngn7mAgOADR\n4QSkihdQZz3Vkb+KwXCRh4kE6gc5H/ckqVprHkJOWdkw0KGST1rWOrFgKpOWsgeK79/1DeA6h1mArsIXR3OwlzE1nV+n+UTnSV\nWYG4hZyrmdRMw5JB",
  "WOrFgKpOWsgeK79/1DeA6h1mArsIXR3OwlzE1nV+n+UTnSV\nWYG4hZyrmdRMw5JBJM6K2oUop4dKwZf2BrZdMnTWJS7O6q7mJIGs/bzs6p3lRo7ZTR5AFizBuW3UEWRJ294glDLclIcw4MQ3Ebc\nqFYFWZi9PA3abWcmgtfmJIP90vY2KpL+C4YyYgKw+8y3YCrkbX09Xdj+PDkXtW8KfOKPYbLal7A8ng1r3giMqolNqVnCpk0WxDK0\n8u2aXrj",
  "YCrkbX09Xdj+PDkXtW8KfOKPYbLal7A8ng1r3giMqolNqVnCpk0WxDK0\n8u2aXrjUHkm2gM0Abzpylyo6B3tXl2CJWvCg3sw1LyU/OTHzk98clp1zbYx/5BsQkVFmbkqMuH/UdEInid4fUET14q0eRBoJ68VML\n9HU0dy/HCNpF67qAgFJNCX6HtL2LVvqaO4M6mCeorBEy98M2EQpMcRW3ZBIwM3/BkdCygEA0ynI0xlGlR5pzc/NB6hkitm9ti",
  "4M6mCeorBEy98M2EQpMcRW3ZBIwM3/BkdCygEA0ynI0xlGlR5pzc/NB6hkitm9tiLszD\nqn1DlUZo3ze4XFwFZXg4XPBrLg9QRoNZPoO0VCOWo2ROzJROXg8KDVvMtfvrKZ8VnVbMz7ea9qBfMDtlGPLz4Raej5hY1JGoLjiKO\nuSxHK0B3Utlu7Pau2Xv9AlnbscN2mJPU2vXTbDveaHvDzbUdvt4lHLOpIVFfTQ+oRy9Ee1OXO47ZrFA7XbUpS",
  "lnbscN2mJPU2vXTbDveaHvDzbUdvt4lHLOpIVFfTQ+oRy9Ee1OXO47ZrFA7XbUpS7zyPTtvhLky0/KP9\nMdfMHJNSOTLHvlQOZiEsaipqp5gmPEbiLITFpGxb8H+s7Al4eLStWQiLvUK0NRPA0ohLPIRZCIuzLdw2mxhWtx3qtltlMhsjcxbC4l\nOW4FHPQliMqRg7xTOWZUichUgexziPY5rHDEuZS8IzkjlmhCwp14LKx2lbMgEsTVBrE0dj0A",
  "iMqRg7xTOWZUichUgexziPY5rHDEuZS8IzkjlmhCwp14LKx2lbMgEsTVBrE0dj0AOZKtRgE8RyQVde4Vx5Cq1iRVdx39V\nw/5qGNUMVmgCWdsge8wc7zk0W4BTDMcuV5EwgK6MJ7GnR5356S+IKnKSC6IrS68ovbT0ktJDSw8pzS0lvwiC6KWl5NdJEF1YekHpg\naUHlJaWlpT2Le1TGlkaUfrE0ieUhpaGlK5buk6ptpScSOGJYOk+pWNLx5QeW",
  "Hpg\naUHlJaWlpT2Le1TGlkaUfrE0ieUhpaGlK5buk6ptpScSOGJYOk+pWNLx5QeWXpE6StLX1H6zNJnlB5bekzpW0vfUvrI0keUMksZpRu\nWblDKLSWvDoJozdI1SgNLyW8/2GuW9ijNLM0ofWzpY0pHlpJfxfA8s5Qcb+DBaKmkdNPSTUqFpeT3WxC9sPQFpYmlCaXPLX1O6RtL\n31D61NKnlMaWkncDcDqxdI9S+xaoKijdtXSX0nNLz93vBfh",
  "pYmlCaXPLX1O6RtL\n31D61NKnlMaWkncDcDqxdI9S+xaoKijdtXSX0nNLz93vBfhiGgPXwtyxFexQmlqaUrplKfmlAEcJS8/IeTJSzV1t/raJ3NciteAO1m\nR8fjXJeaQW3MGau9P8anJ/itSCj0nXNw4WL1IgpXCnHy6trOK3sLRwcL+z+nPnwe6DlYdrzRvaW963nfe96q94v30Hvm9by+F3p/\nen95f3v/LH+9/Nvyo+X1mXrzRnPNV17rs7z",
  "vaW963nfe96q94v30Hvm9by+F3p/\nen95f3v/LH+9/Nvyo+X1mXrzRnPNV17rs7z9H1Bg124=</latexit>I[f2 > 0]",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n4. Take derivatives w.r.t.\nparameters\n<latexit sha1_base64=\"hz6tenqe5IWv9",
  "Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n4. Take derivatives w.r.t.\nparameters\n<latexit sha1_base64=\"hz6tenqe5IWv9gUCVmB+AIbrj2w=\">AX0HicnZhb9xEFIA3XEu4tSCUB14s\noqICJUpQubwgtUnTW1Jyv7Rxuhp7x95pxmPHiebWivEKz+Jf8Ibr/ArOGN7d+JzJgiI1O7s+b65+MyMPesgk6LQi4u/z7z2+htvX2tXdm3v/Q8+vH7jo/0iLfOQ74WpTPDgBVcCsX3tNCSH2Y",
  "sgk6LQi4u/z7z2+htvX2tXdm3v/Q8+vH7jo/0iLfOQ74WpTPDgBVcCsX3tNCSH2Y5Z0kg+UFwsmL4wRnP\nC5GqX2R8eOExUpEImQaQv0bMzt+lLOw8jOWa8Gk53Mp+5UYjy+FgijgmvVPxt7nP3rYD6KoX51c4f+rxpv6nq9SVSYBz31/lvTjbN6XPNK3Jt+hEe8r+LKR8Lj+BuVh/ZmLeKi/+H+D8Xzfo8P5x8q3+9fnFxcW6z+PFpba\nwnyv/d",
  "hEe8r+LKR8Lj+BuVh/ZmLeKi/+H+D8Xzfo8P5x8q3+9fnFxcW6z+PFpba\nwnyv/dvs3/hk4A/SsEy40qFkRXG0tJjp48q0HEo+nvXLgmcsPGExP4KiYgkvjqt6+sfeTYgMvCjN4Z/SXh29XKNiSVFcJAGYCdPDAjMTdLGjUkc/HFdCZaXmKmw6ikrp6dQza8kbiJyHWl5AgYW5gLF64ZBnjSsuFlf8fMw\nTRKmBpW/vLoFqQp4LFTFT8t69Y3HXW",
  "kbiJyHWl5AgYW5gLF64ZBnjSsuFlf8fMw\nTRKmBpW/vLoFqQp4LFTFT8t69Y3HXWe1djgUrzKWH+9OWxGaJ+IVJ43UimnkCoH46riC/ECBoIDEAucgFTxAto0+QkibwlR2G0SMPAgHcHgIm97TJpWmseQk472nGhQyCQfdawVYsFUJh1lBxTPu+kZwHUOswBDhQ+O5mAn\nY2o8qaf5SOdJVZgY7iFnKuZ1F3DJISztbWyoUkqoGnasn7C1zdRJm7",
  "DhQ+O5mAn\nY2o8qaf5SOdJVZgY7iFnKuZ1F3DJISztbWyoUkqoGnasn7C1zdRJm7g0q4eamwiydvOuo3OaFzXoOnUEWbAI465VR5Al4d4YAmDLflPlxw4pmIWxUKq4IszM08Dbp9ZyaC1+Yog/3S9VYrkv4zhjJiArD7zKdgKuRdfSW\nd2t4kOWe1bwp85A1hsrpVWB43lzXpBK6qjY2pWecKmTRbEMrT865pRuNQeSa6F2gCeNOVuVDRJe12XYI",
  "hsrpVWB43lzXpBK6qjY2pWecKmTRbEMrT865pRuNQeSa6F2gCeNOVuVDRJe12XYIla8L+bjUvJT86OuFb/nouFo028b8R7IJDRVl5mrIhP9DQwN4GuP1BRE8ealEkweBevJSCfd3NHUsxwvbROq5g4J\nQTAp9gba/iFW3Th3Bg0TNFYImHbhkwmFJjmKurIJGBk+4VzhWEAhusiwucZQpkWZc3LzQ+sZIrVubou5MA+r7g1VGqF73+ByWgvK8HA4",
  "GBk+4VzhWEAhusiwucZQpkWZc3LzQ+sZIrVubou5MA+r7g1VGqF73+ByWgvK8HA41dUD1BGgyafQVqActRMkdmSkcv/ELDFnPt/nrKm6LTivnpWtsfjAtmpwxDftp\nfw/MRE4s6ErUFBzlnW5JYjv6grelyvTyau3Fl2Rpxw7XbUrSbjtKt+1wrxgBP13jHadeMSijkRtSOkHrEc/UFb7jyu67C4bpNSdqd5NFpO9ypiZ/tDuE07A5JqVyYI59qf",
  "deMSijkRtSOkHrEc/UFb7jyu67C4bpNSdqd5NFpO9ypiZ/tDuE07A5JqVyYI59qfSbEBY1FbVTM2Juis2ISwmZdeC71jZEfD\nw6FpNCIubhehqJoClAZf4EpoQFpst3DXbGFbXHeq6W2UyGyKzCWHxIUvwVTchLMZUjJ3iCcsyJDYhkschzuOQ5jHDUuaS8IxkjhkhS8q1oPJh2pVMAEsj1NvI0RmMQKYKdgGsVzQlVc4V5Cq1jRVbzn6njvio41",
  "xkjhkhS8q1oPJh2pVMAEsj1NvI0RmMQKYKdgGsVzQlVc4V5Cq1jRVbzn6njvio41Qw2aAJY\n2yB7z/A3nJgtwiuGY5UpyJpCV0QRuYmeTOpPTXxBV5CQXRBeWXlB6buk5pQeWHlCaW0p+EQTRtqXk10kQnVl6Rum+pfuUlpaWlO5ZukdpZGlE6QNLH1AaWhpSumLpCqXaUnIihSeCpbuUDi0dUnpo6SGlzyx9RukjSx9R+tz\nS5S+svQVpfcsvUc",
  "SumLpCqXaUnIihSeCpbuUDi0dUnpo6SGlzyx9RukjSx9R+tz\nS5S+svQVpfcsvUcps5RumrpKqXcUvLqIiWLV2mNLCU/PaDvWbpJqWZpRml9y29T+nAUvKrGJ5nlpLjDTwYLZWUPrb0MaXCUvL7LYieWvqU0sTShNInlj6h9KWlLyl9aOlDSmNLybsBOJ1YukOpfQtUFZRuWbpF6amlp+\n73Anw6jYFrYW7YBjYoTS1NKV2zlPxSgKOEpSfkPBmp",
  "kOpfQtUFZRuWbpF6amlp+\n73Anw6jYFrYW7YBjYoTS1NKV2zlPxSgKOEpSfkPBmp9q42edtE7muRmnIHazM+qU1yHqkpd7D27jSpTe5PkZryIRn6v70RQqkFO70/evzS/gtLC3sf7Ow9N3Cna0783eX2ze013qf9j7r3eot9b7v3e096m329nrhzG8zf8\nz8OfPX3PbcaO7nuV8a9bWZts7Hvc7f3K9/A0ECVHY=</latexit> @`i\n@\u03b2k\n=",
  "f8\nz8OfPX3PbcaO7nuV8a9bWZts7Hvc7f3K9/A0ECVHY=</latexit> @`i\n@\u03b2k\n= @fk\n@\u03b2k\n@`i\n@fk\n=\n@\n@\u03b2k\n(\u03b2k + \u2326khk) @`i\n@fk\n= @`i\n@fk\n,",
  "The backward pass\n<latexit sha1_base64=\"g1\nXiOW07MVqJywqEv+mr3kz7w=\">AYBHiclZhZb9w2EMf\nX7pW6V9IDfihaCDVSFG1qeO30eCmQ2HEuO7Udn4nXWVBaSs\nuYomQd9jrCvrZfpm9FX/s9+iX6GTqUtMtwhn7oAvZy5/fnD\nDkcSpT8VIq8WFr6Z2b2jTfevuda+/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo",
  "/Ovf+Bx9+dP3Gxwd\n5UmYB3w8SmWRHPsu5FIrvF6KQ/CjNOIt9yQ/90zXND895lo\ntE7RWXKT+JWaREKAJWgKl/Y+Z3r+eHYb9aGntf/6LbPi9Y/\nfM7+LEV86j+Be1RX/RUosrY5nX681p8bBfdScd2fHE0wnR\nha/pmgBdK0B3PHmCrGMQ3TdIZbtEMtWiOXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxq",
  "OXxJkrxAoOsew\nOsWKHWLFCrIwnzlBXLmW/Ek3X2E9GlRwfT/zdutTopH9YW\nlxqf54tNFtGwud9rPdv/HZoDdIgjLmqgky/Pj7lJanFQsK\n0Qg+XiuV+Y8ZcEpi/gxNBWLeX5S1SUz9m6CZeCFSQZ/qvB\nq6+s9Khbn+WXsgzJmxTDHTBtd7Lgswp9PKqHSsuAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zv",
  "suAqaAKFpf\nSKxNP15w1ExoNCXkKDBZmAsXrBkGUsKBK53qKXwRJHDM1q\nHqr6zvjqufzSKiKn5V1xY7Htma91nBoXqVYfbQ39SIKHotX\nnDipJdrJFQIejauKL0aLGAgOQCxyAhLFc/Cp8+OHXhdR2KE\nScNUA1SC93RMXKuCR5ATS/acyKCRSj6yVGtEBUsZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQz",
  "sZW5JdkH\njeTU8DXmSwCjBU+OJoDXZTpsaTfgUfFVlc5dqGI2RMRbwOA\nVMOmNQzshWqlBK6BpbqV6x6ytRpm7gkrYeaQtS7W2psho\nXtTA1tQWpIijGxVbUEqCdfTAYsZLlt92HCsactbqlQWC\npIYW5niW/HTrUF1+Yohf1i69Yrkv5zhjKiDbD79LdgKuC2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ",
  "C2f\nC2Zqr1Jcs5rvW7wkTeExbK7sCxqpjUJArNqbWOqrHOFlDRb\nYMqSC1upR+OQ8lTYE9QGvOnKTKjwNdmtugUlq829WzDVrJT\n8+PvFH/jopFrS20b/I9kER3mZuhxp8/9wNIA7OK4vsODFSy\nRaPDUi5dIuL6jpWMZLmxtqdcOGkIxKYpLtP1FpOw+tQUPN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxU",
  "PN\nonRWMGg/cI3EwotchjaYm3QYviGs4ijgAI0yaCZYyCTvMw4\nufihegZLdeXxUzom5V9QZVaYF83uJz2gjbcHM75Fd19lF\nG/yaeflGrAMpTMkV7S0YteXsAWc+3+esmbplMV8bONh6MC\n1anDAJ+1t/A6xERFdVI5AsOf05fkqgc8cDXtFxfH1m18eJb\nUtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2",
  "UtqRQ+tWSuK3HaVb7dBeMQJ+tukY7SbRERXVSOSrHSHVEZU\njHvhy53HTNQuH1q2UxO8kj061QztVovIP94ZwnNXHpEQO9L\nEvkb3GhIUFRZOYaKPxLawMWFhXNoq+I0luwJuHraqMWHhd\ni5smTZg0YBLPIXGhIXNFraVrQ1LNx3STbeUyXSIlI0JCx+w\nGM+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMU",
  "M+6MWFhRIWRU3jK0hQJGxPJ4xDncUjzmGJR6hLhFUkdK0J\nKylVQ2TCxRdqARSMUbeQIBiOQiUIBWyMW57TycmflKVTFi\nlbxvivw/hWBC4YcagMWbZE95vW2nJvMxymGY5YryalAqpQm\ncBtrtqlmcvrzw4qc5Pzw0tBLSi8MvaD0NBDSjNDyROBHz4\n1lDyd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9Kh",
  "yd+OG5oeUHh6QGlpaEnpvqH7lIaGhpTeN/Q+pYGhAa\nVrhq5RWhKTqRwRzB0j9KhoUNKjw9ovSZoc8ofWjoQ0qfG\n/qc0leGvqL0rqF3KWGMkrXDV2nlBtKXh34aqhq5T6hpJn\nP9hrhm5TmhqaUnrP0HuUDgwlT8VwPzOUHG/gxmiopPSRoY8\noFYaS5zc/fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp6",
  "fGLoE0pjQ2NKHxv6mNKXhr6k9IGhDyiNDCXvB\nuB0YugupeYtUJVTumPoDqVnhp653wvw6TL6rsLcMg62KE0M\nTSjdMJQ8KcBRwtBTcp4MVXtVm7xtIte1UE25g7UZn/QmOQ/\nVlDtYe3Wa9CbXp1BN+ZAMf1g+iIFUgpX+v71hS5+C0sbB8\nuL3R8Xb+/cXriz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsT",
  "riz2r6hvdb5vPNV5tOt/NT507nYWe7s98JZ\nv6d/XT2i9kv53+b/2P+z/m/GunsTNvnk471mf/7P0kNWgs=\n</latexit> f0 = \u03b20 + \u23260xi\nh1 = a[f0]\nf1 = \u03b21 + \u23261h1\nh2 = a[f1]\nf2 = \u03b22 + \u23262h2\nh3 = a[f2]\nf3 = \u03b23 + \u23263h3\n`i = l[f3, yi]\n1. Write this as a series of \nintermediate calculations\n2. Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n4. Take derivatives w.r.t.\nparameters\n<latexit sha1_base64=\"Efnr2HehvClZo",
  "Compute these \nintermediate quantities\n3. Take derivatives of \noutput with respect to \nintermediate quantities\n4. Take derivatives w.r.t.\nparameters\n<latexit sha1_base64=\"Efnr2HehvClZo1HJZRLxjYK4pOI=\">AX3HicnZhbc9w0FMc35VbCrYVh8sCLh0\nyZAiWTMOXywkybNL0lJfdLGyc7slf2qivLji0nm3r2jTeGVz4S34HvwCs8c2TvruJzFAbYmdbK+f1JB0dybKCTIpCLy7+PnPtdfePOt62/PvPue+9/cOPmh/tFWuYh3wtTmeaHASu4FIrv",
  "B0dybKCTIpCLy7+PnPtdfePOt62/PvPue+9/cOPmh/tFWuYh3wtTmeaHASu4FIrvaElP8xyzpJA8oNgsGL4wRnP\nC5GqX2R8eOExUpEImQaTN2bM10/ylY+RnLtWDS87mU3UqMRpdMQbSR8Jh1ByPvsx8XCGIom41uKrCv3LfOPB8laoyCXju+7OkIbd/X/JI3wZDwDUzXrwvpxT+gnK/fuYi7uvP/19vPN/3aH/+sXLd7uCk2h1b8wvLizWP4\n8",
  "I3wZDwDUzXrwvpxT+gnK/fuYi7uvP/19vPN/3aH/+sXLd7uCk2h1b8wvLizWP4\n8WlsaF+c74t9m9+XHP76VhmXClQ8mK4mhpMdPHlWkilHw065cFz1g4YDE/gqJiCS+OqzoVRt4tsPS8KM3hn9Jeb1co2JUVwkASgTpvsFZsboYkeljr4/roTKSs1V2DQUldLTqWfyuJnIdaXkCBhbmAvnphn0HANGTfrK/4e\nZgmCVO9yl9e3YKYBTwWquKnZ2",
  "dLTqWfyuJnIdaXkCBhbmAvnphn0HANGTfrK/4e\nZgmCVO9yl9e3YKYBTwWquKnZ2Jo1Fbs1prOBSvUiw/2Z16EZon4hUnTmqJcXKFgMejquIL8QIGgMQC5yAVPECfJr4BJG3hCisPAkYeJAOoXORtz0irpXmMcSkJXtBZFDIJB+2VCtEBVOZtCQ7IPG8W54BXOcwC9BVeHA0Bzs\nZU6NJPc2HOk+qwthwCzlTMa+bgCGHkOPbWKFKaFq2FL9iFXb",
  "XOcwC9BVeHA0Bzs\nZU6NJPc2HOk+qwthwCzlTMa+bgCGHkOPbWKFKaFq2FL9iFXbTA3GgUuzuqu5sSDVbt7W6JzGRfXamtqCVJCEcVtVW5BKwj7ZYwmDKI/LXRhw4hmLWyoUlgqSmJt5GrTbzowF5+Ywg/XS1q1WJPxnDEXEGD1madgKuRt+Uo6V\nXuT4JzVelPgQ68Pk9WuwvK4GdakERjV2DaiyjpWSEmjBaY8PW8rTW8cUp6J9gCNAS+6Mhcq",
  "elPgQ68Pk9WuwvK4GdakERjV2DaiyjpWSEmjBaY8PW8rTW8cUp6J9gCNAS+6MhcquiS7U5cgZY3ZvwNDzUvJj75a+IYPj6tFs2zMfySa4KgoM5cjY/4PjnrwZsb5BRY8ealEkweGevJSCfs7mjqW48Q2lnruoCAUk0J\nfoOUvYtWuU1twZ9ME9RUMxi8mVBokqOoLTYGI4YnDEcCRSiQYbNGEOZFmXOyeaH8hkstdxsi7kwL6v2hiqNoL1vcDmtB",
  "kqOoLTYGI4YnDEcCRSiQYbNGEOZFmXOyeaH8hkstdxsi7kwL6v2hiqNoL1vcDmtBWV4OZzxK6oHKJBE8gLVWP5SiYQzOlwxO/0LDEXKu/nvKm6FTF/HRt3B70C2anDEN+2l3D8xETFd\nVI5AsOdU5fkqgc7YGvabpe7lm1dvIFSe3YoXUrJfE7qVb7dBe0QN+u7o7TrRERXVSORr3EOqIypHe+DLHcd1ygcWrdSEr+TODrVDu1UidI/2u3Dsdgck",
  "+u7o7TrRERXVSORr3EOqIypHe+DLHcd1ygcWrdSEr+TODrVDu1UidI/2u3Dsdgck1LZM8e+VPqNCQs1FWqnMDVH67awMWFhUrZV8DeW7Ah4ebRVjQkL\nNwvRlhkDFvW4xENoTFjYLOG2cmzD0nWHdN0tZTLrI2VjwsJHLMGjbkxYGFNh7BQOWJYhYWMicezjOPZpHDMsylwiPCOZY0ZISrkSKu+nbZExYNEQtTZ0NAY9kKlCDY6NWFzQzCucmadQ",
  "OPZpHDMsylwiPCOZY0ZISrkSKu+nbZExYNEQtTZ0NAY9kKlCDY6NWFzQzCucmadQFiuaxXuhveuaFgz5NAYsGiDrDHP3\nAusgCHGI5ZriBnAqkyGsBNrNmkmsnpL4gqcpILogtLyg9t/Sc0gNLDyjNLSVfBEG0bSn5OgmiM0vPKN23dJ/S0tKS0j1L9yiNLI0ofWjpQ0pDS0NKVyxdoVRbSk6k8EawdJfSvqV9Sg8tPaT0uaXPKX1s6WNKX1j6gt",
  "0ofWjpQ0pDS0NKVyxdoVRbSk6k8EawdJfSvqV9Sg8tPaT0uaXPKX1s6WNKX1j6gtJXlr6i9\nL6l9yljJKVy1dpZRbSq4OgmjZ0mVKA0vJtx+sNUs3Kc0szSh9YOkDSnuWkq9ieJ9ZSo438GK0VFL6xNInlApLyfdbED2z9BmliaUJpU8tfUrpS0tfUvrI0keUxpaSuwE4nVi6Q6m9BaoKSrcs3aL01NJT970An05j4ErMDet\ng9LU0pTSNUvJlwI",
  "UxpaSuwE4nVi6Q6m9BaoKSrcs3aL01NJT970An05j4ErMDet\ng9LU0pTSNUvJlwIcJSwdkPNkpMa72uS2iexrkZpyBxtHfFKbxDxSU+5g491pUpvsT5Ga8j7p+ur+9CIFQgo7fG/BK+haWF/a8Xlr5duLt1d/7e8viG9nrnk86ndudpc53nXudx53Nzl4nPlt5o+ZP2f+mjuZ+2nu57lfG\num1mXGdjzqt39yvfwOMbVmp</latexit>",
  "3Nzl4nPlt5o+ZP2f+mjuZ+2nu57lfG\num1mXGdjzqt39yvfwOMbVmp</latexit> @`i\n@\u2326k\n= @fk\n@\u2326k\n@`i\n@fk\n=\n@\n@\u2326k\n(\u03b2k + \u2326khk) @`i\n@fk\n= @`i\n@fk\nhT\nk",
  "Gradients\n\u2022 Backpropagation intuition\n\u2022 Toy model\n\u2022 Jupyter notebook example of backprop and autograd\n\u2022 Matrix calculus\n\u2022 Backpropagation matrix forward pass\n\u2022 Backpropagation matrix backward pass\n\u2022 Matrix backprop summary",
  "Pros and cons\n\u2022 Extremely efficient\n\u2022 Only need matrix multiplication and thresholding for ReLU functions\n\u2022 Memory hungry \u2013 must store all the intermediate quantities\n\u2022 Sequential\n\u2022 can process multiple batches in parallel\n\u2022 but things get harder if the whole model doesn\u2019t fit on one machine.",
  "Feedback?",
  "Lecture 07b\nInitialization\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Where we are\n=== Foundational Concepts ===\n\u00fc 02 -- Supervised learning refresher\n\u00fc 03 -- Shallow networks and their representation capacity\n\u00fc 04 -- Deep networks and depth efficiency\n\u00fc 05 -- Loss function in terms of maximizing likelihoods\n\u00fc 06 \u2013 Fitting models with different optimizers\n\u00fc 07a \u2013 Gradients on deep models and backpropagation\n\u2022 07b \u2013 Initialization to avoid vanishing and exploding weights & \ngradients\n\u2022 08 \u2013 Measuring performance, test sets, overfitting and double \ndescent\n\u2022 09 \u2013 Regularization to improve fitting on test sets and unseen data\n=== Network Architectures and Applications ===\n\u2022 10 \u2013 Convolutional Networks\n\u2022 11 \u2013 Residual Networks\n\u2022 12 \u2013 Transformers\n\u2022 Large Language and other Foundational Models\n\u2022 Generative Models\n\u2022 Graph Neural Networks\n\u2022 \u2026",
  "Agenda\n\u2022 Finish Adam optimizer from lecture 06 \u2013 Fitting Models\n\u2022 Quick tips on how to read a research paper\n\u2022 Model Initialization",
  "Model Initialization\n\u2022 The need for weights initialization\n\u2022 Expectations Refresher\n\u2022 The He (Kaiming) Initialization",
  "Initialization\n\u2022 Consider standard building block of NN in terms of pre-activations:\n\u2022 How do we initialize the biases and weights?\n\u2022 Equivalent to choosing starting point in our gradient descent searches\n<latexit sha1_base64=\"8s5v1dPi\n/IDnmI/FaXZVPyw4VcQ=\">AW5XiclZhb9xEFIDdcivhloLICy8WURGC\nNsoiCrwgtUnTW1KSNc2m67G3rF3mvHYscfJptb+BN4Qr/wknvkhvAJnbO9\nOfc6kEislO3u+z3M5M7bHDjIpCr28/NeVq2+9/c6717f+6Dz/6+JP565\n/uF2mZh3wvTGWaHwas4",
  "+z3M5M7bHDjIpCr28/NeVq2+9/c6717f+6Dz/6+JP565\n/uF2mZh3wvTGWaHwas4FIovqeFlvwyzlLAskPgpNVw/OeF6IVO3qi4wf\nJyxWIhIh0xAazIt+EWD6mTif/WzD+WAa1b/BZ+bCY8rn9BeVR/q1SVScD\nzfn/ujX4SpOKTY7a2m/1JseD+cXlpeX649NCry0seu1na3D982F/mIZlw\npUOJSuKo95ypo8rlmsRSj6Z65cFz1h4wmJ+BEX",
  "49NCry0seu1na3D982F/mIZlw\npUOJSuKo95ypo8rlmsRSj6Z65cFz1h4wmJ+BEXFEl4cV3VOJv4NiAz9KM3h\nT2m/jr5+RMWSorhIAjATpkcFZiboYkeljn46roTKSs1V2DQUldLXqW8S7A9\nFzkMtL6DAwlxAX/1wxHIWapiGub7i52GaJEwNq/7K2vak6gc8Fqrip2U9J\nZNJ1mrHQ7Fy4yVR7uzWoTmiXjFSW1Yiq5RODxpKr4UryEgeAxBI",
  "gc8Fqrip2U9J\nZNJ1mrHQ7Fy4yVR7uzWoTmiXjFSW1Yiq5RODxpKr4UryEgeAxBInIFW8\ngDpNfoLI7yEKS1ACrpqFAKvAfzohVSvNY8hJR3tONChko871iqxYCqTjr\nIDiu/f8A3gOodZgK7CF0dzsJMxNZkep/lY50lVmBhuIWcq5nUTMOSQSTOir\nqFKeHQsGP9gq2nTJ20iUuzuqu5iSBrN+86Oqd5UcOuU0eQBYsw7lp1BFkS\nLhDljDIcl",
  "HQsGP9gq2nTJ20iUuzuqu5iSBrN+86Oqd5UcOuU0eQBYsw7lp1BFkS\nLhDljDIclsewIAT30TcqlBYFWRhbuVp0G07MxG8NscZnC9db60i6T9jKC\nMmAGef+RZMhbyr6Yz258m56z2TYGP/RFMVvcQlsfNsKaNwKja2ISada6QS\nbMFoTw975qmNw6VZ6I7QBPAJ12ZCxW9pt2sS7BkTbh/E4al5If3Vq6zcf\nH1bI5bcw/k2oqCgzV0Um/D8qGs",
  "BPAJ12ZCxW9pt2sS7BkTbh/E4al5If3Vq6zcf\nH1bI5bcw/k2oqCgzV0Um/D8qGsItCq8viODJSyWaPAjUk5dKuL6jqWM5Xt\ngmUs8dFIRiUugLdPqLWHWPqSO4s2mC+goBUy98M6HQJEdRVzYBI8M3GwdC\nyhEgwybMYyLcqck4sfWs8QqXVzWcyFuVl1L6jSCN3rBpezo6AMN4czfsn\nhAcpo0OQzSEs1ZDlK5thM6fhFv9BwirnO/nrKm6LTi",
  "jSCN3rBpezo6AMN4czfsn\nhAcpo0OQzSEs1ZDlK5thM6fhFv9BwirnO/nrKm6LTivnpetse9AtmpwxDfj\npYx/MRE4s6EtUFuxtnXZJYjvagrtlyfb1n1fqLb8jSjh2u25Sk3raXbtvhX\ntIDfrh6O0G8YhFHYnqantIPWI52oO63HncI3C4bpNSeqd5tFpO9yZiZ\n/tDuCjarZJqVyaLZ9qew3ISxqKmqnmJrNbldsQlhMyq4Fv7GyI+Dm0bWaE",
  "9yZiZ\n/tDuCjarZJqVyaLZ9qew3ISxqKmqnmJrNbldsQlhMyq4Fv7GyI+Dm0bWaEB\na3CtHVTABLQy7xEJoQFptTuGu2MaxuONQNt8pkNkJmE8LiA5bgUTchLMZU\njJ3iCcsyJDYhkscRzuOI5jHDUuaS8IxkjhkhS8q1oPJR2pVMAEtj1NrY0Rj\n0QKYKNdgGsVzQlVc4V5Cq1jRVbznanjvkoY1QxWaAJY2yTnm9zedJ1mAUw\nzbLFeSM4GsjC",
  "gGsVzQlVc4V5Cq1jRVbznanjvkoY1QxWaAJY2yTnm9zedJ1mAUw\nzbLFeSM4GsjCZwCztb1Jnu/oKoIju5ILqw9ILSc0vPKT2w9IDS3FLyRBE\nTy0lTydBdGbpGaX7lu5TWlpaUrpn6R6lkaURpfctvU9paGlI6aqlq5RqS8m\nOFO4Ilu5SOrJ0ROmhpYeUPrP0GaUPLX1I6XNLn1P6ytJXlN619C6lzFJG6\nZqla5RyS8mrgyBasXSF0sBS8uwH",
  "rP0GaUPLX1I6XNLn1P6ytJXlN619C6lzFJG6\nZqla5RyS8mrgyBasXSF0sBS8uwH5qlW5RmlmaU3rP0HqVDS8lTMdzPLCXb\nG7gxWiopfWTpI0qFpeT5LYieWPqE0sTShNLHlj6m9KWlLyl9YOkDSmNLybs\nB2J1YukOpfQtUFZRuW7pN6amlp+73Anw2jYFrYW7aCjYpTS1NKV23lDwpw\nFbC0hOyn4xUe1Wbvm0i17VIzbiDtRmfHk1yHqkZd7",
  "rYW7aCjYpTS1NKV23lDwpw\nFbC0hOyn4xUe1Wbvm0i17VIzbiDtRmfHk1yHqkZd7D26jQ9mlyfIjXjI9L1\ntf3ZixRIKVzpB/OLPfwWlhb2v1vq/bB0e/v7xTsr7Rva94X3pfe17P+9\nG74z30trw9L/T+9P72/vH+XYgXfl34beH3Rr16pT3mM6/zWfjPwN+bI=<\n/latexit>fk = \u03b2k + \u2326khk\n= \u03b2k + \u2326ka[fk\u22121]",
  "Forward Pass\n\u2022 Consider standard building block of NN in terms of pre-activations:\n\u2022 Set all the biases to 0\n\u2022 Set weights to be normally distributed \n\u2022 mean 0 \n\u2022 variance \ud835\udf0e!\n\"\n\u2022 What will happen as we move through the network if \ud835\udf0e!\n\" is very small?\n\u2022 What will happen as we move through the network if \ud835\udf0e!\n\" is very large?\n<latexit sha1_base64=\"7Ouk\nKVR0hBtygihROvA9RIuMq1k=\">AWjniclZhb9s2FIDV7tZ1t\n7TD8rIXYUGBYegMZ+i6vRrk6ZNm3RxmjhJE6cGJVMyG4pSJCpx\nKviv7HX7S/s3O5RlszqHeZiB1sz5PvFySEq0gkyKQne7/",
  "hJE6cGJVMyG4pSJCpx\nKviv7HX7S/s3O5RlszqHeZiB1sz5PvFySEq0gkyKQne7/964+d\nHn3z62a3Pb3/x5Vdf7N05+5BkZ5yPthKtP8KGAFl0LxvhZa8\nqMs5ywJD8MztYNP7zgeSFSta+vMn6asFiJSIRMQ2i4dHcQRAHX\nbFidTf1HPvzVHS6tdDvd+uPTwmpTWPGaT29457vRYJSGZcKVDiU\nripPVbqZPK5ZrEUo+vT0oC56x8IzF/ASKi",
  "PTwmpTWPGaT29457vRYJSGZcKVDiU\nripPVbqZPK5ZrEUo+vT0oC56x8IzF/ASKiW8OK3qzk/9exAZ+V\nGawz+l/Tr64RUVS4riKgnATJgeF5iZoIudlDr6/bQSKis1V+Gso\naiUvk59kwl/JHIeankFBRbmAvrqh2OWs1BDvm4PFL8M0yRhalQN\n1jZ2p9Ug4LFQFT8v69xNp21no3Y4FK8z1l7sL2oRmifiPSeV1I\nqp5BqBx9Oq4p24g4HgAE",
  "g4LFQFT8v69xNp21no3Y4FK8z1l7sL2oRmifiPSeV1I\nqp5BqBx9Oq4p24g4HgAESHE5AqXkCdJj9B5K8iCmtFAgYepBPoX\nOS/npKqleYx5KSlHRMNCpnk5a1TiyYyqSl7IHi+/d8A7jOYRag\nq/DF0RzsZUxN59dpPtF5UhUmhlvImYp53QMOWTSjKhtqFJKuDR\nsWX9i6zVTZ03i0qzuam4iyNrP247OaV7UqO3UEWTBIozbVh1Blo\nSdPWIJ",
  "JKuDR\nsWX9i6zVTZ03i0qzuam4iyNrP247OaV7UqO3UEWTBIozbVh1Blo\nSdPWIJgyw35SEMOPFNxK0KhVBFmYvT4N25mJ4LU5yWC/tL2Ni\nqT/gqGMmADsPvMtmAp5W19PF7Y/T85F7ZsCn/hjmKz2JSyPZ8Oa\nNwKjamJTata5QibNFoTy9LJtmt4VJ6J9gBNAG+6Mhcq+kC7X5\ndgyZrw4D4MNS8lP/m58yufnFZds23MfySbUFRZq6KTPh/V",
  "gBNAG+6Mhcq+kC7X5\ndgyZrw4D4MNS8lP/m58yufnFZds23MfySbUFRZq6KTPh/VDSCZ\nwleXxDBk5dKNHkQqCcvlXB/R1PHcrywTaSeOygIxaTQV2j7i1i1\nr6kjuLNpgvoKAVMvfDOh0CRHUVs2ASPDNzwVHQsoRIMZ2MZVq\nUOSc3P7SeIVLr5raYC/Owat9QpRHa9w0uF1dBGR4OF/yaywOU0W\nCWzyAt1YjlKJkTM6WTt4NCwxZz7f56ymd",
  "at9QpRHa9w0uF1dBGR4OF/yaywOU0W\nCWzyAt1YjlKJkTM6WTt4NCwxZz7f56ymdFpxXz862mPegXzE4Zh\nvx8uIXnIyYWdSqC4hzroksRztQV2L5fphz6qtz+RpR07XLcp\nSb1NL92w72mB/x829HbeIRizoS1dX0kHrEcrQHdbnzuO0ahc\nN1m5LUO8+j03a4CxMt/2h/DKdRc0xK5cgc+1I5mIWwqKmonWKa8\nBiJsxAWk7Jtwd9Y2RPw8Ghb",
  "3a4CxMt/2h/DKdRc0xK5cgc+1I5mIWwqKmonWKa8\nBiJsxAWk7Jtwd9Y2RPw8GhbsxAWe4VoayaApRGXeAizEBZnW7ht\nNjGsbjvUbfKZDZG5iyExecswaOehbAYUzF2imcsy5A4C5E8jnE\nexzSPGZYyl4RnJHPMCFlSrgWVj9O2ZAJYmqDWJo7GoAcyVajBJo\njlgq68wrnyFrFiq7ivqvh/jUNa4YqNAEs7ZA95g92nJswCmGY\n5YryZlAVkY",
  "BJo\njlgq68wrnyFrFiq7ivqvh/jUNa4YqNAEs7ZA95g92nJswCmGY\n5YryZlAVkYT2MNOjzrz018QVeQkF0RXl5RemnpJaWHlh5SmltK\nfhE0WtLya+TILqw9ILSA0sPKC0tLSntW9qnNLI0ovSZpc8oDS\n0NKV23dJ1SbSk5kcITwdJ9SseWjik9svSI0jeWvqF09JNSo8tP\nab0vaXvKX1i6RNKmaWM0g1LNyjlpJXB0G0ZukapYGl5Lcf7DVL",
  "WvqF09JNSo8tP\nab0vaXvKX1i6RNKmaWM0g1LNyjlpJXB0G0ZukapYGl5Lcf7DVL\ne5RmlmaUPrX0KaUjS8mvYnieWUqON/BgtFRS+sLSF5QKS8nvtyB\n6ZekrShNLE0pfWvqS0neWvqP0uaXPKY0tJe8G4HRi6R6l9i1QV\nC6a+kupeWnrvfC/DFNAauhbljK9ihNLU0pXTLUvJLAY4Slp6R8\n2Skmrva/G0Tua9FasEdrMn4/GqS80gtuIM1d6",
  "jK9ihNLU0pXTLUvJLAY4Slp6R8\n2Skmrva/G0Tua9FasEdrMn4/GqS80gtuIM1d6f51eT+FKkFH5O\nubxwsXqRASuFOP1xaWcVvYWnh4JfO6sPOg90HK4/Xmje0t7zvR\n+8H71V7zfvsbfp9by+F3oT7y/vb+f5aXlh8uPlv+YqTdvNd86\n7U+y5v/AWMN1VU=</latexit>\u03b2k = 0\n<latexit sha1_base64=\"8s5v1dPi\n/IDnmI/FaXZVPyw4VcQ=",
  "U=</latexit>\u03b2k = 0\n<latexit sha1_base64=\"8s5v1dPi\n/IDnmI/FaXZVPyw4VcQ=\">AW5XiclZhb9xEFIDdcivhloLICy8WURGC\nNsoiCrwgtUnTW1KSNc2m67G3rF3mvHYscfJptb+BN4Qr/wknvkhvAJnbO9\nOfc6kEislO3u+z3M5M7bHDjIpCr28/NeVq2+9/c6717f+6Dz/6+JP565\n/uF2mZh3wvTGWaHwas4FIovqeFlvwyzlLAskPgpN",
  "q2+9/c6717f+6Dz/6+JP565\n/uF2mZh3wvTGWaHwas4FIovqeFlvwyzlLAskPgpNVw/OeF6IVO3qi4wf\nJyxWIhIh0xAazIt+EWD6mTif/WzD+WAa1b/BZ+bCY8rn9BeVR/q1SVScD\nzfn/ujX4SpOKTY7a2m/1JseD+cXlpeX649NCry0seu1na3D982F/mIZlw\npUOJSuKo95ypo8rlmsRSj6Z65cFz1h4wmJ+BEXFEl4cV3VOJv4NiAz9KM3h",
  "IZlw\npUOJSuKo95ypo8rlmsRSj6Z65cFz1h4wmJ+BEXFEl4cV3VOJv4NiAz9KM3h\nT2m/jr5+RMWSorhIAjATpkcFZiboYkeljn46roTKSs1V2DQUldLXqW8S7A9\nFzkMtL6DAwlxAX/1wxHIWapiGub7i52GaJEwNq/7K2vak6gc8Fqrip2U9J\nZNJ1mrHQ7Fy4yVR7uzWoTmiXjFSW1Yiq5RODxpKr4UryEgeAxBInIFW8\ngDpNfoLI7yEKS1A",
  "7Fy4yVR7uzWoTmiXjFSW1Yiq5RODxpKr4UryEgeAxBInIFW8\ngDpNfoLI7yEKS1ACrpqFAKvAfzohVSvNY8hJR3tONChko871iqxYCqTjr\nIDiu/f8A3gOodZgK7CF0dzsJMxNZkep/lY50lVmBhuIWcq5nUTMOSQSTOir\nqFKeHQsGP9gq2nTJ20iUuzuqu5iSBrN+86Oqd5UcOuU0eQBYsw7lp1BFkS\nLhDljDIclsewIAT30TcqlBYFWRhbuV",
  "5iSBrN+86Oqd5UcOuU0eQBYsw7lp1BFkS\nLhDljDIclsewIAT30TcqlBYFWRhbuVp0G07MxG8NscZnC9db60i6T9jKC\nMmAGef+RZMhbyr6Yz258m56z2TYGP/RFMVvcQlsfNsKaNwKja2ISada6QS\nbMFoTw975qmNw6VZ6I7QBPAJ12ZCxW9pt2sS7BkTbh/E4al5If3Vq6zcf\nH1bI5bcw/k2oqCgzV0Um/D8qGsItCq8viODJSyWaPAjUk5d",
  "h/E4al5If3Vq6zcf\nH1bI5bcw/k2oqCgzV0Um/D8qGsItCq8viODJSyWaPAjUk5dKuL6jqWM5Xt\ngmUs8dFIRiUugLdPqLWHWPqSO4s2mC+goBUy98M6HQJEdRVzYBI8M3GwdC\nyhEgwybMYyLcqck4sfWs8QqXVzWcyFuVl1L6jSCN3rBpezo6AMN4czfsn\nhAcpo0OQzSEs1ZDlK5thM6fhFv9BwirnO/nrKm6LTivnpetse9AtmpwxDfj\npYx",
  "hAcpo0OQzSEs1ZDlK5thM6fhFv9BwirnO/nrKm6LTivnpetse9AtmpwxDfj\npYx/MRE4s6EtUFuxtnXZJYjvagrtlyfb1n1fqLb8jSjh2u25Sk3raXbtvhX\ntIDfrh6O0G8YhFHYnqantIPWI52oO63HncI3C4bpNSeqd5tFpO9yZiZ\n/tDuCjarZJqVyaLZ9qew3ISxqKmqnmJrNbldsQlhMyq4Fv7GyI+Dm0bWaEB\na3CtHVTABLQy7xEJoQF",
  "Z9qew3ISxqKmqnmJrNbldsQlhMyq4Fv7GyI+Dm0bWaEB\na3CtHVTABLQy7xEJoQFptTuGu2MaxuONQNt8pkNkJmE8LiA5bgUTchLMZU\njJ3iCcsyJDYhkscRzuOI5jHDUuaS8IxkjhkhS8q1oPJR2pVMAEtj1NrY0Rj\n0QKYKNdgGsVzQlVc4V5Cq1jRVbznanjvkoY1QxWaAJY2yTnm9zedJ1mAUw\nzbLFeSM4GsjCZwCztb1Jnu/oKoIju5ILq",
  "anjvkoY1QxWaAJY2yTnm9zedJ1mAUw\nzbLFeSM4GsjCZwCztb1Jnu/oKoIju5ILqw9ILSc0vPKT2w9IDS3FLyRBE\nTy0lTydBdGbpGaX7lu5TWlpaUrpn6R6lkaURpfctvU9paGlI6aqlq5RqS8m\nOFO4Ilu5SOrJ0ROmhpYeUPrP0GaUPLX1I6XNLn1P6ytJXlN619C6lzFJG6\nZqla5RyS8mrgyBasXSF0sBS8uwH5qlW5RmlmaU3rP0HqVDS8",
  "JXlN619C6lzFJG6\nZqla5RyS8mrgyBasXSF0sBS8uwH5qlW5RmlmaU3rP0HqVDS8lTMdzPLCXb\nG7gxWiopfWTpI0qFpeT5LYieWPqE0sTShNLHlj6m9KWlLyl9YOkDSmNLybs\nB2J1YukOpfQtUFZRuW7pN6amlp+73Anw2jYFrYW7aCjYpTS1NKV23lDwpw\nFbC0hOyn4xUe1Wbvm0i17VIzbiDtRmfHk1yHqkZd7D26jQ9mlyfIjXjI9L1\ntf",
  "w\nFbC0hOyn4xUe1Wbvm0i17VIzbiDtRmfHk1yHqkZd7D26jQ9mlyfIjXjI9L1\ntf3ZixRIKVzpB/OLPfwWlhb2v1vq/bB0e/v7xTsr7Rva94X3pfe17P+9\nG74z30trw9L/T+9P72/vH+XYgXfl34beH3Rr16pT3mM6/zWfjPwN+bI=<\n/latexit>fk = \u03b2k + \u2326khk\n= \u03b2k + \u2326ka[fk\u22121]",
  "Backward Pass\n\u2022 What will happen as we propagate backwards \nthrough the network if \ud835\udf0e!\n\" is very small?\n\u2022 What will happen as we propagate backwards \nthrough the network if \ud835\udf0e!\n\" is very large?",
  "Initialize weights to different variances\nExploding gradients\nVanishing gradients\n100D Input\n~ \ud835\udc41(0,1)",
  "How do we initialize weights to keep variance \nstable across layers?",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9",
  "sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01Yr",
  "W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGY",
  "TiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l",
  "qx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SO",
  "HGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgm",
  "/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7U",
  "ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RC",
  "n\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nB",
  "1XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVb",
  "Hw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn",
  "m3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\" \u2212 \ud835\udd3c \ud835\udc53$\n\"\n#\nDefinition of variance:",
  "Agenda\n\u2022 The need for weights initialization\n\u2022 Expectations Refresher\n\u2022 The He (Kaiming) Initialization",
  "Expectations\n<latexit sha1_base64=\"z9FcnqQPc\nhuVvH1ugbLNdn4d8IM=\">AXHniclZhb9s2FICd7tZlt3TD8rIXYUGHduiMZ\nOjWYkCBNmnapkXp4mTtJFrUDIls6YoRZfEqeD/MuzH7G3Y6/ZvdijZnQOA2w\nBOnPn+3g7pKiLl0iR5aur/yxce+/9Dz786PrHi598+tnXyzd+PIwi4vU510/l\nnF67LGMS6F4Nxe5MdJylnkSX7kjTY0PzrjaSZidZBfJ",
  "+tnXyzd+PIwi4vU510/l\nnF67LGMS6F4Nxe5MdJylnkSX7kjTY0PzrjaSZidZBfJLwXsVCJQPgsh1B/6Xd\n3c5xwP+cDd12E8sSNvHhchpOTcU8H0p7z3QPHzYqoX4mzpyOek4nvTV+MLrtu\nO7if2pDqNy5RKr6t53B2JXM47Lkp7+wJOl3tuq2quE95sHkzmJ/aW1vVr9ObS\nwNi2stKZ/nf6NrwfuIPaLiKvclyzLTtZWk7xXsjQXvuSTRbfI",
  "zmJ/aW1vVr9ObS\nwNi2stKZ/nf6NrwfuIPaLiKvclyzLTtZWk7xXsjQXvuSTRbfIeML8EQv5CRQV\ni3jWK6tkTpybEBk4QZzCPxhyFb1co2Rl1EHpgRy4cZjpoYydFHtzvlUIlRc\n6VX3cUFNLJY0evjDMQKUxcXkCB+amAsTr+kKUMEptCS4qf+3EUMTUo3fXNvUnp\nejwUChJXVMmaTJrOZuVwKF5lrG8dzFsROY/EO04aqRTdyBUCDydlydth",
  "fXNvUnp\nejwUChJXVMmaTJrOZuVwKF5lrG8dzFsROY/EO04aqRTdyBUCDydlydthGwPBAY\ng2JyBWPIM2dX68wFlDFPauBFzWO8UF4+WENK1yHkJOGtprokEhkXzcsDaIBUsZ\nNZR9UBznpqMBz1NYBRgq/HC0BvsJU5NZvZyP8zQqMx3DPaRMhbzqAqbsM6ln1D\nRUISVU9RvWr9h6ydRomrg4qYa6giyDtKmk6c0L2rQdKoIsmAThk2riBLwk",
  "ln1D\nRUISVU9RvWr9h6ydRomrg4qYa6giyDtKmk6c0L2rQdKoIsmAThk2riBLwk\nzYBGDLE/LfZhw5OiIXRUKq4JszE4ae82+Ex3Be7M6KJreZknSf8ZQRnQArj79K\n5jyeVPfiOe2M0vOWeXrAh87Q1isZhWhvW0Zp3ArKaxCTWrXCGTZgtCaXzeNPV\noLCpPRHOCOoAvuiIVKrik3alKsGV12L0DU0LyU9+aP/Ex71yV82+j8km9BQV\niS2hnT",
  "pPRHOCOoAvuiIVKrik3alKsGV12L0DU0LyU9+aP/Ex71yV82+j8km9BQV\niS2hnT4fzQ0gHsb3l8QwYsXS7R4EKgWL5ZwvqOlYyne2DpSrR0UhGJS5Bfo8he\nhatapIniwcYTGCgHdLvwyodAiB0FT1gEtwy/cpS0byEeT9Os5+jLOipSTw/tZ\n4hUuj4WU6FvVs0DVWqheW5wOa8FZbg5nPErqnso16dTy8u1IClKJljvaTjN26\nWwyVmu/qrJa",
  "FvVs0DVWqheW5wOa8FZbg5nPErqnso16dTy8u1IClKJljvaTjN26\nWwyVmu/qrJa+LVivkp9vT/mBcsDqF7/PT/jZej5BY1JGoLXgsrYliWXpD9qa\nb9fLIyu3xPtnZoce2mJO1OR2m3Le4VI+CnO5bR7hCPWNSRqK3pCKlHLEt/0J\nY9jzu2WVhcuylJu7M8Wm2LOzfR9g8Ohjxn+jEplgP92BdLtw5hMadibhXjiIdI\nrENYjIqmBf+PlX0BN4",
  "8Wm2LOzfR9g8Ohjxn+jEplgP92BdLtw5hMadibhXjiIdI\nrENYjIqmBf+PlX0BN4+mVYew2MlEU9MBLA24xFOoQ1isL+GmOY1hdcei7thVJp\nMhMusQFp+yCM+6DmExpGJoFUfw7oHEOkTyOMR5HNI8JlhKbBJekcSyImRL2TZU\nOoybkg5gaYx6G1s6gxHIWKEOp0EsZ3TnZdadp9AuVnQXd20d6/oOGeoQR3A0i\n65xhx313qReTjF8JhlS3Ii",
  "WKEOp0EsZ3TnZdadp9AuVnQXd20d6/oOGeoQR3A0i\n65xhx313qReTjF8JhlS3IikJXQBHaw06HO7OnPC0ryJOcF4ZeUHpu6DmlR4Ye\nUZoaSt4IvOCloeTtxAvOD2j9NDQ0oLQwtKu4Z2KQ0MDSh9YugTSn1DfUo3D\nN2gNDeUPJHCHcHQA0qHhg4pPTb0mNJXhr6i9Jmhzyh9behrSt8Z+o7SR4Y+opQ\nZyijdNHSTUm4o+XTgBeuGrlPqGUre",
  "NJXhr6i9Jmhzyh9behrSt8Z+o7SR4Y+opQ\nZyijdNHSTUm4o+XTgBeuGrlPqGUre/eBaM7RDaWJoQuljQx9TOjCUvBXD/cxQ8\nngDN0ZDJaVbhm5RKgwl729e8MLQF5RGhkaUPjf0OaVvDX1L6VNDn1IaGkq+DcD\nTiaH7lJqvQGVG6Z6he5SeGnpq/y7A58vo2Tbmrmlgl9LY0JjSbUPJmwI8Shg6I\ns+TgZqearOvTeRcC9ScW9g047PaJOeBm",
  "vo2Tbmrmlgl9LY0JjSbUPJmwI8Shg6I\ns+TgZqearOvTeRcC9ScW9g047PaJOeBmnMLm5Os9rkfArUnA/J0DcP5x9SIKV\nX7i+0F64t3B/+bflP5b/XP6rVq8tTOt81Wr8Lf/9L528DA4=</latexit>w0veXVtbwV1haOPyxvfZz+7e3ZWH69MvtNdb37S+bd1qrbXutR62nrU6rW7L\nE\nh\ng[x]\ni\n=\nX\nk\ng[k]Pr(x =",
  "e3ZWH69MvtNdb37S+bd1qrbXutR62nrU6rW7L\nE\nh\ng[x]\ni\n=\nX\nk\ng[k]Pr(x = k)\nE\nh\ng[x]\ni\n=\nZ\ng[x]Pr(x)dx,\nInterpretation: what is the average value of g[x] when taking into account the probability of x? \nConsider discrete case and assume uniform probability so calculating g[x] reduces to taking average:  \n<latexit sha1_base\n64=\"oh92z4RfHSLrKO6oyU3i3EjKbJc=\">A\nAWz3iclZhJb9tGFICZdEvTzWlRX3ohagQogl\nSwiqbtpUBiR9nsxHa8JqYsDKkhNfFwSHOx5R\nAqe",
  "GFICZdEvTzWlRX3ohagQogl\nSwiqbtpUBiR9nsxHa8JqYsDKkhNfFwSHOx5R\nAqeu1P6k/pqdf2X/QNSWnC98aHCrA1et/HWd\n7McPNTKfJidfWva9fe/+Dz+68fHNTz797P\nMvlm59eZAnZRbw/SCRSXbks5xLofh+IQrJj9\nKMs9iX/NA/Xdf8JxnuUjUXnGZ8mHMIiVCEb\nACQqOlXS9mxcT3q8HMWxORPZiP5lW0ex4Ot\nSBbOh6LE2zZOp6",
  "nGZ8mHMIiVCEb\nACQqOlXS9mxcT3q8HMWxORPZiP5lW0ex4Ot\nSBbOh6LE2zZOp6YcaCqj+rXsy8vIxHlfq1Pz\nuBX645ZqRO7gxHSyurvdX649JCvy2sO1ne3\nTr67E3ToIy5qoIJMvz4/5qWgwrlhUikHx20y\ntznrLglEX8GIqKxTwfVvXoZ+5tiIzdMngTx\nVuHX3iIrFeX4Z+2DqseaY6aCNHZdF+MuwEi\notC6CpqGwlG6RuDqV7lhkPCj",
  "gTx\nVuHX3iIrFeX4Z+2DqseaY6aCNHZdF+MuwEi\notC6CpqGwlG6RuDqV7lhkPCjkJRYkAnoqx\ntMGCSpgITf9BS/CJI4ZmpceWuDnVnl+TwSqu\nJnZ382azrDGqHQ/EqY+3p3qIWUfBYvOWkl\nrRlVwh8GhWVbwX9TAQHIDocQISxXOos14nod\ntHFBabBFw1a8AD4+WMVK0KHkFOtprokEhlX\nzasdaJBVMZd5RdUFz3tqsBLzKYBegqfHE0B",
  "w1a8AD4+WMVK0KHkFOtprokEhlX\nzasdaJBVMZd5RdUFz3tqsBLzKYBegqfHE0B\n7spU7P5cQWfFlc5TqGW8iYinjdBAw5YFKPq\nGuoUko4NOhYL7D1kqnTNnFJWnc10xFk7WVdp\n8hoXtS469QRZMEijLpWHUGWhFPDmMUMstyWR\nzDg2NURuyoUVgVZmNtZ4nfbTnUEr81pCvul6\nw0qkv5zhjKiA7D79LdgKuBdfT1Z2O48Oe1r\nwt86k5gsr",
  "Z4nfbTnUEr81pCvul6\nw0qkv5zhjKiA7D79LdgKuBdfT1Z2O48Oe1r\nwt86k5gsrqHsCxqhjVvBEbVxmbUrHOFTJotC\nGXJRdfUvbGoPBXdAeoA3nRlJlT4jna3LsGS1\nWHvLgw1KyU/r53j0+H1areNvofySZUlJepr\nSId/h8VjeFihNcXRPDkJRJNHgTqyUsknN/R1\nLEML2wdqecOCkIxKYpLtP1FpLrH1BHc2SRGf\nYWArhe+mVBoksOwK+",
  "yUsknN/R1\nLEML2wdqecOCkIxKYpLtP1FpLrH1BHc2SRGf\nYWArhe+mVBoksOwK+uAluEbLquWBRSgQbNG\nAOZ5GXGyckPrWeI1Lo+LWZCX6y6J1Sphe5g\n8vFUVCGi8M5v+JwH2XUb/LpJ6Uaswlc6qnd\nHri5QVsMdvur6e8KVqtiJ9tO1Bv2B2yiDgZ\n6MNPB8RsagjUV1wH2OtSxL0h7UtViu7/as2\nji5Q5Z2ZHtpiT1tr202xb3ih7ws01",
  "MNPB8RsagjUV1wH2OtSxL0h7UtViu7/as2\nji5Q5Z2ZHtpiT1tr202xb3ih7ws01LbzeJR\nyzqSFRX20PqEcvSHtRlz+OmbRQW125KUu8j\n1b4i5MtPzDvQkvmL5NSuRY3/Yl0mtCWCyoW\nFjFJOYREpsQFuOya8FvrOwKuHh0rSaExe1c\ndDUdwNKYSzyEJoTFZgt3zTaG1U2LumlXmUwn\nyGxCWHzMYjzqJoTFiIqRVTyFhxIkNiGSxwnO\n4Tm",
  "oTFZgt3zTaG1U2LumlXmUwn\nyGxCWHzMYjzqJoTFiIqRVTyFhxIkNiGSxwnO\n4TmMcVSapPwjKSWGSFLyragsknSlXQAS1PU\n2tTSGPRAJgo12AaxnNOVl1tXnkKrWNFVvG9r\neP+KhguGKtQBLG2RPeZ6W9ZN5uMUw2WLcmp\nQFZKE7iNnW3qzO/+/LAid3J+eGnoJaUXhl5Q\nemjoIaWZoeSJwA9fGkqeTvzw3NBzSg8MPaC0\nNLSkdN/QfUpD",
  "J+eGnoJaUXhl5Q\nemjoIaWZoeSJwA9fGkqeTvzw3NBzSg8MPaC0\nNLSkdN/QfUpDQ0NKHxn6iNLA0IDSdUPXKS0M\nJXekcEUwdI/SiaETSo8MPaL0laGvKH1i6BNK\nXxv6mtK3hr6l9IGhDyhlhjJKB4YOKOWGklcH\nfrhm6BqlvqHk2Q/2mqHblKaGpQ+NPQhpWND\nyVMxXM8MJbc3cGE0VFL61NCnlApDyfObHz43\n9DmlsaExpc8MfUbpG0Pf",
  "QhpWND\nyVMxXM8MJbc3cGE0VFL61NCnlApDyfObHz43\n9DmlsaExpc8MfUbpG0PfUPrY0MeURoaSdwNw\nd2LoLqXmLVCVU7pj6A6lZ4ae2d8L8MU0+raF\nuWUq2KI0MTShdMNQ8qQAtxKGnpL7yVC1Z7X5\n2yZyXgvVgltYm/H50STnoVpwC2vPTvOjyfkp\nVAs+IV0fHCxepEBK4Uw/Wlrp47ewtHDwQ6/\nU+/ezo8r9faN7Q3nG+cb53vnL7zs",
  "VAs+IV0fHCxepEBK4Uw/Wlrp47ewtHDwQ6/\nU+/ezo8r9faN7Q3nG+cb53vnL7zs3PfeJs\nO/tO4Pzp/O384/y7vLN8sfzb8u+Nev1ae8x\nXTuez/Md/uyjx4g=</latexit>\nE\nh\ng[x]\ni\n\u21e1 1\nN\nN\nX\nn=1\ng[x\u21e4\nn]\n<latexit sha1_base64=\"K4H1PDYo+Gp6DOir+j9XHOgb2OI=\">AWpniclZhb\nU9w2FICd9JamN9JOemLp0xm0ky7A5",
  "Yo+Gp6DOir+j9XHOgb2OI=\">AWpniclZhb\nU9w2FICd9JamN9JOemLp0xm0ky7A52m7WMCITdIWAILJCzZyl7ZqyDLxpZhiWf/Q39NX9u/0X/TI9u7is8RD92ZsMr5PutyJNlaB5kUhV5d/fa9Q8+/OjT258ev\nOz7/48qulW18fFGmZh3wQpjLNjwJWcCkUH2ihJT/Kcs6SQPLD4HTD8MNznhciVfv6MuMnCYuViETINIRGS3eHSZBOq4sJz/ls",
  "H2ihJT/Kcs6SQPLD4HTD8MNznhciVfv6MuMnCYuViETINIRGS3eHSZBOq4sJz/lseFaycf3Hn4qNXtT3Z0NC5H4/fzO\n9IfR0spqb7X+LSw1hZWvPbTH936djwcp2GZcKVDyYrieG010ycVy7UIJZ/dHJYFz1h4ymJ+DEXFEl6cVPWgZv5tiIz9KM3hn9J+HX3/iolRXGZBGAmTE8KzEzQxY\n5LHf1+UgmVlZqrsGkoKqWvU9kyB+LnIdaXkKBhb",
  "X3/iolRXGZBGAmTE8KzEzQxY\n5LHf1+UgmVlZqrsGkoKqWvU9kyB+LnIdaXkKBhbmAvrhOUs1JDHm0PFL8I0SZgaV8P1zd1ZNQx4LFTFIW8mp7NZ19msHQ7Fq4z1p/uLWoTmiXjHSW1Yiq5QuDxr\nKp4L+5hIDgA0eMEpIoXUKfJTxD5a4jCGpKAq2Z5DMF4OSNVK81jyElHe0KGSTzvWBrFgKpOsgeK79/2DeA6h1mArsIXR3OwlzE1m1+n+VTn",
  "OSNVK81jyElHe0KGSTzvWBrFgKpOsgeK79/2DeA6h1mArsIXR3OwlzE1m1+n+VTnSVWYG4hZyrmdRM\nw5JBJM6KuoUop4dKwY73A1kumTtvEpVnd1dxEkLWfdx2d07yocdepI8iCRh3rTqCLAk7fswSBluyMYcOKbiFsVCquCLMx+ngbdtjMTwWtzmsF+6XqbFUn/OUMZM\nQHYfeZbMBXyr6RLmx/npz2jcFPvUnMFndS1geN8OaNwKjamMzata5",
  "bFUn/OUMZM\nQHYfeZbMBXyr6RLmx/npz2jcFPvUnMFndS1geN8OaNwKjamMzata5QibNFoTy9KJrmt4VJ6J7gBNAG+6Mhcqek/7sS7BkjXh4Y8w1LyU/Pin3j0+PalWzbYxf0g2\noaKizFwVmfD/qGgMzxi8viCJy+VaPIgUE9eKuH+jqaO5Xhm0g9d1AQikmhL9H2F7HqXlNHcGfTBPUVAqZe+GZCoUmOoq5sAkaGb3haOhZQiAYZNmMZVqUOSc",
  "mhL9H2F7HqXlNHcGfTBPUVAqZe+GZCoUmOoq5sAkaGb3haOhZQiAYZNmMZVqUOSc3P7\nSeIVLr5raYC/Ow6t5QpRG69w0uF1dBGR4O5/yKywOU0aDJZ5CWasxylMypmdLpm2GhYu5dn895U3RacX8bKtD/oFs1OGIT8beH5iIlFHYnqguOJsy5JLEd7UNdi\nub7fs2rzV2ytGOH6zYlqbftpdt2uFf0gJ9tO3q7TxiUeiutoeUo9YjvagLncet12",
  "ub7fs2rzV2ytGOH6zYlqbftpdt2uFf0gJ9tO3q7TxiUeiutoeUo9YjvagLncet12jcLhuU5J653l02g53YaLlH+1PuGbmJTKsTn2pXLYhLCoqaidYprwGIlNCI\ntJ2bXg/1jZE/Dw6FpNCIv9QnQ1E8DSmEs8hCaExWYLd802htVth7rtVpnMJshsQlh8zBI86iaExZiKsVM8ZVmGxCZE8jBeZzQPGZYylwSnpHMSNkSbkWVD5Ju5IJY\nGmKWps6",
  "86iaExZiKsVM8ZVmGxCZE8jBeZzQPGZYylwSnpHMSNkSbkWVD5Ju5IJY\nGmKWps6GoMeyFShBtsglgu68grnylNoFSu6igeuhgdXNKwZqtAEsLRD9pg/3HFusgCnGI5ZriRnAlkZTWAfO3qzE9/QVSRk1wQXVp6SemFpReUHlp6SGluKflFEQ\nvLSW/ToLo3NJzSg8sPaC0tLSkdGDpgNLI0ojSR5Y+ojS0NKR0w9INSrWl5EQKTwRL9ymdWDqh9",
  "NJzSg8sPaC0tLSkdGDpgNLI0ojSR5Y+ojS0NKR0w9INSrWl5EQKTwRL9ymdWDqh9MjSI0pfWfqK0ieWPqH0taWvKX1n6TtKH1j6gFJmKaN09JNSrml5\nNVBEK1buk5pYCn57Qd7zdI+pZmlGaUPLX1I6dhS8qsYnmeWkuMNPBgtlZQ+tfQpcJS8vstiJ5b+pzSxNKE0meWPqP0raVvKX1s6WNKY0vJuwE4nVi6R6l9C1QVlO5a\nukvpmaVn7vcCf",
  "zSxNKE0meWPqP0raVvKX1s6WNKY0vJuwE4nVi6R6l9C1QVlO5a\nukvpmaVn7vcCfDGNgWth7tgKdihNLU0p3bKU/FKAo4Slp+Q8Gan2rjZ/20Tua5FacAdrMz6/muQ8UgvuYO3daX41uT9FasEnpOubB4sXKZBSuNOPlbW8FtYWj4ub\nf2a+/e7i8r9fbN7Q3vO+87073pr3m3fe+L1vYEXen96f3l/e/8s31l+sTxYPmzU69fa7xOp/lP/4DCIb",
  "3vO+87073pr3m3fe+L1vYEXen96f3l/e/8s31l+sTxYPmzU69fa7xOp/lP/4DCIbgUw=</latexit>where\nx\u21e4\nn \u21e0 Pr(x)",
  "Common Expectation Functions",
  "Rules for manipulating expectation\n<latexit sha1_bas\ne64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo\n=\">AYCHicpZjZbtw2FEDH7pamW9Kic\nIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlG\nJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6B\nb2UNEPrXvohqIHMPcbmkKI68VIq8W\nF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T\n/fzpMx8vucnMskOPZzKRTfK0Qh+WGacR\nZ7kh94wzXND854",
  "K+9f/eDjz7+5Nr1T\n/fzpMx8vucnMskOPZzKRTfK0Qh+WGacR\nZ7kh94wzXND854lotE7RbjlB/HLFIiFD\n4rIHRyfe53tz9KuV/wF0VkTwa6q/s2P\nn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnR\nZ2L6EurbZxQu3cstZ9ud84t/5fo9HR+A\n0au6ytWR3uacmC+sNpUyXCSRN1RrfHba\nklKuAphw9VTE6uLS4vLd/Di2stIXFXv\nu3dXL98ANE",
  "cmC+sNpUyXCSRN1RrfHba\nklKuAphw9VTE6uLS4vLd/Di2stIXFXv\nu3dXL98ANEr+M4Wpfsjw/WlOi+OKZY\nXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5\nNiAROmGTwTxVOHb14RcXiPB/HpgxKwY\n5ZjpoY0dlEf58DCNLy4Irv2koLKVTJI5\neg04gMsifHEOB+ZmAvjr+gGUMcpBTYq\nf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaT\nr9GsHcn",
  "fHEOB+ZmAvjr+gGUMcpBTYq\nf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaT\nr9GsHcnipsfp4d1aLKHgsXnNSa3oSi4\nReDSpKr4ULWEgOACxAlIFM+hTp0fL3R\nWEIW7VAKumXgvFsQqpWBY8gJx3tBdGg\nkEo+6lhrxIKpjDvKDiOc9PRgBcZzAJ0\nFb4moOdlKnJ9LqCj4osrnIdwy1kTEW8\nbgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4\nJK27mu",
  "lKnJ9LqCj4osrnIdwy1kTEW8\nbgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4\nJK27mukIsnazrlNkNC8q6Dp1BFmwCKOu\nVUeQJWFPDVjMIMt+QGHDs6YleFwqog\nC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEd\ngLtPfwumfN7V15KZ7UyTc1b7usBHzgAm\nq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd\n03dG4vKU9EdoA7gm67MhAovaLfrEixZH\nXZv",
  "6JmWNGYFRtbELNOlfIpNmCUJacd\n03dG4vKU9EdoA7gm67MhAovaLfrEixZH\nXZvw1CzUvKj75Z+4KPjalnfNvqDZBMqy\nsvUVpEOv0FATzF8fqCJ68RKLJg0A9e\nYmE/R1NHcvwtaReu6gIBSTohij219Eq\nntNHcGdTWLUVwjoeuGbCYUmOQy7sg5oG\nb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1\nvW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5H",
  "hPGJZQD4apN+M0ZdJXmacbH5oPUOk1\nvW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5H\nIPZdRr8uklpQpYhpI50lM6eunmBdxitr\nu/nvKmaLUifretgf9gtkpfZ+fnqzj+Y\niIR2J6oIDoLUuSxLe1DXbLle7Fm1/v\nJbsrQji2s3Jam37aXdtriX9ICfblh6u0\nE8YlFHoraHlKPWJb2oC57Hjdso7C4dl\nOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+",
  "0\nE8YlFHoraHlKPWJb2oC57Hjdso7C4dl\nOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9i\nXSbUJYLKhYWMUk5hESmxAW47Jrwf+xsiP\ng4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw\n12xhWNyzqhl1lMh0gswlh8SGL8aibEBY\njKkZWcjSFIlNiORxgPM4oHlMsZTaJDw\njqWVGyJKyLahskHQlHcDSCLU2sjQGPZC\nJQg2QSzndOXl1pWn0CpWdBXv",
  "ZTaJDw\njqWVGyJKyLahskHQlHcDSCLU2sjQGPZC\nJQg2QSzndOXl1pWn0CpWdBXv2Rreu6T\nhgqEKdQBLm+Qec9xN603m4RTDMcuW5FQ\ngK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf\n0nNIDQw8ozQwlvwi8Jmh5NeJF54Zekbp\nvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvq\nU7pm6BqlhaHkRApPBEN3KR0YOqD0NBD\nSp8b+pzSR4Y+ovSFoS8o",
  "OkDQx9Q6hvq\nU7pm6BqlhaHkRApPBEN3KR0YOqD0NBD\nSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqP\nUmYo7RvaJ9Sbih5deCFq4auUuoZSn7\nwb1m6BalqaEpfcNvU9pYCj5VQzPM0PJ\n8QYejIZKSh8b+phSYSj5/eaFTw19Smls\naEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxOD\nN2h1LwFqnJKtw3dpvTU0FP7ewE+m0bPt\njA3TQWblCaGJpSuG",
  "0IaWRoeTdAJxOD\nN2h1LwFqnJKtw3dpvTU0FP7ewE+m0bPt\njA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gq\nt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7\nk7Tq8n+FKoZH5Cu9/dnL1IgpbDTn1xbX\nMFvYWlh/ulR+X7mzfWby72r6hvdL7s\nnej901vpfdT727vUW+rt9fz5/6d/2L+q\n/kbC78t/LHw58JfjTo/17zWa/zt/D3f\n5T7ZSo=</late",
  "vUW+rt9fz5/6d/2L+q\n/kbC78t/LHw58JfjTo/17zWa/zt/D3f\n5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent",
  "Agenda\n\u2022 The need for weights initialization\n\u2022 Expectations Refresher\n\u2022 The He (Kaiming) Initialization",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9",
  "sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01Yr",
  "W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGY",
  "TiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l",
  "qx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SO",
  "HGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgm",
  "/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7U",
  "ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RC",
  "n\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nB",
  "1XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVb",
  "Hw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn",
  "m3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n\ud835\udf0e!!\n\"# = \ud835\udd3c \ud835\udc53$\n\" \u2212 \ud835\udd3c \ud835\udc53$\n\"\n#\nDefinition of variance:",
  "Now let\u2019s prove:\n<latexit sha1_base64=\"ZepB\n7KjO5+Qtd/0U7z5fxFnMgk=\">AWu3iclZhbU9w2FICdXtP0R\ntopL3xlMlM2kl2gEkvL5kmkM0NUiCwQMIujOyVvQqybGwZlnj2\n5/TX9LV96L/pkde7wueIh+5MsuJ8n3U5kmWtg0yKQi8v/3vjgw\n8/+viT29+duvzL786uF29/sF2mZh7wXpjLNDwNWcCkU72mhJ\nT/Mcs6SQPKD4HTd8INznhciVXv6Mu",
  "786uF29/sF2mZh7wXpjLNDwNWcCkU72mhJ\nT/Mcs6SQPKD4HTd8INznhciVXv6MuODhMVKRCJkGkInC7/3u+OM\nh5oP+5JH+uju+H4/KX8Xu3nIh7pgf/QnxtH4+PVwf0rfw6OV08\nWlpY7y/XHp4WVprDkNZ/tk9vfDfvDNCwTrnQoWVEcrSxnelCxXI\ntQ8smtflnwjIWnLOZHUFQs4cWgqkc68e9AZOhHaQ7/lPbr6NUrK\npYUxWUSgJkwPSo",
  "8smtflnwjIWnLOZHUFQs4cWgqkc68e9AZOhHaQ7/lPbr6NUrK\npYUxWUSgJkwPSowM0EXOyp19NugEiorNVfhtKGolL5OfZM2fyhy\nGLS8hAILcwF9cMRyxkIoeaFL8I0yRhalj17o7k6of8Fioip\n+VdaInk7bTrR0OxeuMtRd781qE5ol4z0kltWIquUbg8aSqeCfuY\nCA4ANHhBKSKF1CnyU8Q+SuIwsKSgIEH6Rg6F/mvJ6RqpXkMOWlp\nb4",
  "8aSqeCfuY\nCA4ANHhBKSKF1CnyU8Q+SuIwsKSgIEH6Rg6F/mvJ6RqpXkMOWlp\nb4kGhUzyctaJxZMZdJSdkHx/Tu+AVznMAvQVfjiaA52M6Yms+s\n0H+s8qQoTwy3kTMW8bgKGHDJpRtQ2VCklXBq2rD+w9Zqp0yZxaV\nZ3NTcRZO3lbUfnNC9q2HbqCLJgEcZtq4gS8I2MGQJgyw35RMYc\nOKbiFsVCquCLMztPA3abWcmgtdmvUm0vW5F0n/OU",
  "cZtq4gS8I2MGQJgyw35RMYc\nOKbiFsVCquCLMztPA3abWcmgtdmvUm0vW5F0n/OUEZMAO4+8y2Y\nCnlbX0/ntj9LzntmwIf+yOYrPYlLI+nw5o1AqNqYhNq1rlCJs\n0WhPL0om2a3jhUnon2AE0A3RlLlR0RbtXl2DJmnD/Hgw1LyU/u\nt/5mY8H1bK5bcx/JtQUVFmropM+H9UNIQHD15fEMGTl0o0eRCo\nJy+VsL+jqWM5XtgmUs8dFIRiUuhL",
  "JtQUVFmropM+H9UNIQHD15fEMGTl0o0eRCo\nJy+VsL+jqWM5XtgmUs8dFIRiUuhLdPuLWLWvqSO4s2mC+goBUy9\n8M6HQJEdRWzYBI8M3PEIdCyhEgwynYwxlWpQ5J5sfWs8QqXWzLe\nbCPKzaG6o0Qnvf4HJ+FZTh4XDOr7k8QBkNpvkM0lINWY6SOTZTO\nj7uFxpuMdfdX0/5tOi0Yn620bQH/YLZKcOQn51s4PmIiUdieqC\nM4uzLksR3tQ13",
  "j7uFxpuMdfdX0/5tOi0Yn620bQH/YLZKcOQn51s4PmIiUdieqC\nM4uzLksR3tQ13y5Xu1ZtXH8E1nascN1m5LU2/TSbTvca3rAz\nYdvd0kHrGoI1FdTQ+pRyxHe1CXO4+brlE4XLcpSb2zPDpthzs30\nfKP9kZcM3NMSuXQHPtS2Z+GsKipqJ1imvAYidMQFpOybcHfWNkV\n8PBoW9MQFrcL0dZMAEtDLvEQpiEsTm/htnEsLrpUDfdKpPZCJn\nT",
  "pOybcHfWNkV\n8PBoW9MQFrcL0dZMAEtDLvEQpiEsTm/htnEsLrpUDfdKpPZCJn\nTEBafsQSPehrCYkzF2CmesixD4jRE8jCeRzRPGZYylwSnpHMS\nNkSbkWVD5K25IJYGmMWhs7GoMeyFShBpsglgu68grnylNoFSu6i\nnuhnvXNKwZqtAEsLRF7jG/v+W8yQKcYjhmuZKcCWRlNIHb2Nm\nzuz0F0QVOckF0aWl5ReWHpB6YGlB5TmlpJfBEH02l",
  "QKcYjhmuZKcCWRlNIHb2Nm\nzuz0F0QVOckF0aWl5ReWHpB6YGlB5TmlpJfBEH02lLy6ySIzi\n09p3Tf0n1KS0tLSnuW9iNLI0ofWrpU0pDS0NK1y1dp1RbSk6k8\nESwdI/SkaUjSg8tPaT0jaVvKH1u6XNK31r6ltL3lr6n9LGljyl\nljJKu5Z2KeWklcHQbRm6RqlgaXktx/ca5ZuU5pZmlH6xNInlA4\ntJb+K4XlmKTnewIPRUknpC0tfUCosJ",
  "Rm6RqlgaXktx/ca5ZuU5pZmlH6xNInlA4\ntJb+K4XlmKTnewIPRUknpC0tfUCosJb/fguiVpa8oTSxNKH1p6U\ntK31n6jtJnlj6jNLaUvBuA04mlu5Tat0BVQemOpTuUnl65n4vw\nOfTGLgW5patYIvS1NKU0g1LyS8FOEpYekrOk5FqdrXZ2yayr0V\nqzh2syfjsapLzSM25gzW70+xqsj9Fas5HpOvd/fmLFEgp7PQnC0\nsr+C0sLeyvdlZ+6T",
  "syfjsapLzSM25gzW70+xqsj9Fas5HpOvd/fmLFEgp7PQnC0\nsr+C0sLeyvdlZ+6TzYebD0aK15Q3vT+97wbvrXi/eo+859621\ntexit>/NC70/vL+9v75/Fh4vh4rtFOVU/uNFc863X+iyW/wGDr+eV</la\nE\n\u21e5\n(x \u2212 \u00b5)2\u21e4\n= E[x2] \u2212 E[x]2\nKeeping in mind:\n<latexit sha1_base64=\"LCHt\naEnOu97cFGH+XlOZpvFfOUw=\">AWjniclZhbT9xGFICd9Ja",
  "ha1_base64=\"LCHt\naEnOu97cFGH+XlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN\n5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOvRPGY2OPYm1\nf6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH3\n38yaef3fr89hdfvX1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP\n8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRM",
  "XaZmHfBCmMs0PA1ZwKRQfaKElP\n8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7g7k4yH\nmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzq\nUrCiOV5YzfVKxXItQ8untYVnwjIVnLObHUFQs4cVJVXd+6t+DyM\niP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1FyFT\nUNRKX2d+iYT/kjkMHB5BQU",
  "6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1FyFT\nUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1Kga\nrvZ3ptUw4LFQFT8v69xNp12nXzscitcZqy/25rUIzRPxnpNKas\nVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQ\nVfjiaA52",
  "l0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQ\nVfjiaA52M6ams+s0n+g8qQoTwy3kTMW8bgKGHDJpRtQ1VCklXBp\n2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8\nLOHrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6\nb9gKCMmALvPfAumQt7V19K57c+Sc1H7psAn/hgmq3sJy+NmW",
  "jMTwWuzPi6Xr8i6\nb9gKCMmALvPfAumQt7V19K57c+Sc1H7psAn/hgmq3sJy+NmWLNG\nYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEi\nxZEx7eh6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBeg\ntcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HMHBaGYFPoKbX8Rq+41\ndQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu",
  "9o6liOF7aJ1HMHBaGYFPoKbX8Rq+41\ndQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZl\nzcvih9QyRWjfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyW\neQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/HyjbQ/6BbNThiE/P\n93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9\nbS/dtsO9pgf8fNPR203",
  "TizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9\nbS/dtsO9pgf8fNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZ\nuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjUVtVNMEx4js\nQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41h\ndOhbrpVJrMxMpsQFp+zBI+6CWExpmLsFM9YliGxCZE8jnEexzS\nPGZYy",
  "cJds41h\ndOhbrpVJrMxMpsQFp+zBI+6CWExpmLsFM9YliGxCZE8jnEexzS\nPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVajBNojlgq\n68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXk\njOBrIwmcBs729SZPf0FUWe5ILoytIrSi8tvaT0wNIDSnNLyS+C\nIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZp",
  "i8tvaT0wNIDSnNLyS+C\nIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaGlIa\nVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48of\nW/pe0qfWPqEUmYpo7RvaZ9Sbil5dRBEq5auUhpYSn7wV6zdJvS\nzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXln\n6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8",
  "3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXln\n6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3b\nF0h9JzS8/d7wX4fBoD18LcshVsUZpamlK6YSn5pQCPEpaekefJS\nLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7\n+/EUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9\nFb8X7zHnvr3rY38EJv",
  "pBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9\nFb8X7zHnvr3rY38EJv4v3l/e39s7iw+HDx0eIfjXrzRnvNt17ns\n7j+H6oN1Xs=</latexit>E[x] = \u00b5",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "<latexit sha1\n_base64=\"9xk9LeHDNCaEfnqE\n3fWnwTqmjE=\">AXaXiclZhb\nb9s2FICd7tZ1t3TDlmF7ERZ06\nNbGSIzu8jKgTereki5XJ2ljJ6B\nkSmZDUYouiV3BL9uv3G8Y9h92\nKMtmdA49YAZaM+f7eDukKFluLE\nWara7+tXDjnXfe/+Dmx/e+uj\njTz79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5",
  "z79bPH254dplCce73iRjJjl\n6VcCsU7mcgkP4TzkJX8iP3fE\nPzo0uepCJSB9ko5r2QBUr4wmM\nZhM4W/+m2hzH3Mt4/uTtc6Yb5a\neuHnvP9b46JD09bK60hoHsl7n\nW7tzDvrZg/S7Xn3Ltm/Ee9FrBr\nobKetufKp61784X5pA1tn7bOF\npdXm6vlx6GFtaqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxU",
  "taqw3Kg+O2e3v+x\n3+5GXh1xlnmRperK2Gme9giWZ\n8CQf3+rmKY+Zd84CfgJFxUKe9\nopyXcbOHYj0HT9K4J/KnDJ6vUb\nBwjQdhS6YIcsGKWY6aGMneb/\n2iuEivOMK2/SkZ9LJ4scvchOXy\nQwdzmCAvMSAWN1vAFLGOQjgZY\nUv/KiMGSqX3TX27vjouvyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHA",
  "vyQKiCX\n+TlthiP6067dDgU5xnrzw9mrY\niMh+ItJ42Uim5kjsCDcVHwZtD\nEQHAoskJiBRPoU2dH9d31hCFy\n0ACBu5GQxic7+yNSdMq4wHkpK\na9JhoUYsmHNWuDWLCUYU3ZB8Vx\n7jga8CyBVYChwhdHa7AfMzWe1\nsv4MEvCItUx3EPCVMDLmDKHpN\n6RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg",
  "RnVD5VJCVa9m/Y6tPabOq8RF\ncTnUREeQdZDUnSyheVH9ulNGk\nAWbMKhbZQRZEg6tPgsZLkqn8G\nEQ0dH7KpQWBVkY+4kVvO9YR\nvDfLs6LutQuS/kuGMqIDcPXpb8\nGUx+v6RjSznWlyLktfF/jQGcB\ni1auwJhMa9oJzKqKjalZ5gqZN\nFsQSqKruqlHY1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9",
  "1F5LOoT1AF80e\nWJUP417X5Zgi2rw937MNUkl/x\nkpfkTH/aKVX3Z6P9INqGhNI9tD\nenw/2ioD7dJvL8ghcvkmjxIF\nAuXiThfEdLxK8sXWkXDsoCMWk\nyEbo8heBqtcpI3iwUYjGCgHdL\nnwzodAi+35d1gEtwzfc8C0byEO\nT9CZz9GSU5gknhx/azxApdX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0v",
  "dX0s\nJkLfrOoHqtRC/dzgclYLynBzu\nORzqrso+4kn26Uqz5LUDKHekm\nHp90g0vMdvWXSz4pWq2AX2xW\n/cG4YHVyz+MXZ5t4PQJiUeitu\nAJy9qWJalP2hrtl2vj6zYP2\nRbO3A4tpNSdqtRm3Le6cEfCL\nctot4hHLOpI1FY1QuoRy9IftG\nXP45ZtFhbXbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C",
  "XbkrS7jSPVtvizky\n0/f2DAc+YfkyKZF8/9kWyOwlhM\naNiZhWjkAdInISwGOZ1C/7Gyr\n6Am0fdmoSwuJOKuqYDWOpziacw\nCWFxcgnXzSqG1S2LumVXmYwHy\nJyEsPiUhXjWkxAWAyoGVvGcxTE\nSJyGSxwHO4DmMcZSbJPwisSW\nFSFbyrahkFUl3QAS0PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF",
  "PU29DSG\nYxARgp1WAWxnNKdl1p3nkK7WNF\nd3LF13JnTcZQgzqApW1yjTnd\nbetF5uIUw2OWLcmxQFZME7iDnR\n3qTJ/+XL8gT3KuPzJ0ROmVoVe\nUHhl6RGliKPlF4Pp7hpJfJ65/a\neglpYeGHlKaG5pT2jG0Q6lvqE\n/pE0OfUOoZ6lG6YegGpZmh5Ik\nU7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2p",
  "U7giGHlA6MHRA6bGhx5S+MvQVp\nc8MfUbpa0NfU/rW0LeUPjL0Ea\nXMUEZp29A2pdxQ8urA9dcNXafU\nNZT89oNrzdAdSmNDY0ofG/qY0\nr6h5Fcx3M8MJY83cGM0VFL63ND\nnlApDye83139p6EtKQ0NDSl8Y\n+oLSN4a+ofSpoU8pDQwl7wbg6\ncTQfUrNW6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWr",
  "W6AipXTX0F1KLwy9sL8\nX4LNldG0bc9s0sE1pZGhE6ah\n5JcCPEoYek6eJ31VnWrTt03kXP\nPVjFtYlfFpbZJzX824hVWn07Q\n2OZ98NeMDMvT24exFCqQUTvqz\nxeU1/BaWFg5bzbWfmw92Hyw/XK\n/e0N5sfNv4rnG3sdb4pfGw8ay\nx0+g0vIX9hdHCHwt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212",
  "wt/fvX30u2lr\n9Q</latexit>5e+mag3Fqo6XzRqn6XlfwGKWR\nE[(x \u2212 \u00b52)] = E[x2 \u2212 2x\u00b5 + \u00b52]\n= E[x2] \u2212 E[2x\u00b5] + E[\u00b52]\n= E[x2] \u2212 2\u00b5E[x] + \u00b52\n= E[x2] \u2212 2\u00b52 + \u00b52\n= E[x2] \u2212 \u00b52\n= E[x2] \u2212 E[x]2\n<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKF",
  "icpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YO",
  "54lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVO",
  "M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFs",
  "d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7z",
  "NC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gI",
  "ja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1",
  "0lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gs",
  "AW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8J",
  "3m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5",
  "Fq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzf",
  "j06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n=",
  "\u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\n<latexit sha1_base64=\"LCHtaEnOu97cFGH+X\nlOZpvFfOUw=\">AWjniclZhbT9xGFICd9JamN5KqvPTFKopUVekKqjTtS9QEsgkJpEBgcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8",
  "gcASNPaOv\nRPGY2OPYm1f6Wv7V/qv+kZ27sTnzM8dCWyk/N9nsuZi70OMikKvbz8742bH38yaef3fr89hdfvX\n1Nwt37u4XaZmHfBCmMs0PA1ZwKRQfaKElP8xyzpJA8oPgbM3wgwueFyJVe/oq4ycJi5WIRMg0hE4X7\ng7k4yHmo+OJyf+I3+YlKcLS8u95frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObH",
  "frj08JKW1jy2s/26Z3vRsNRGpYJVzqUrCiOV5YzfVKxXItQ8un\ntYVnwjIVnLObHUFQs4cVJVXd+6t+DyMiP0hz+lPbr6IdXVCwpiqskADNhelxgZoIudlzq6PeTSqis1\nFyFTUNRKX2d+iYT/kjkMHB5BQUW5gL6odjljNIRg41KX4ZpknC1KgarvZ3ptUw4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKL",
  "4LFQFT8v69xNp12\nnXzscitcZqy/25rUIzRPxnpNKasVUco3A42lV8V7cw0BwAKLHCUgVL6BOk58g8lcQhbUiAQMP0gl0L\nvJfT0nVSvMYctLRjogGhUzyScdaIxZMZdJRdkHx/Xu+AVznMAvQVfjiaA52M6ams+s0n+g8qQoTwy3\nkTMW8bgKGHDJpRtQ1VCklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQ",
  "CklXBp2rD+x9ZqpszZxaVZ3NTcRZO3lXUfnNC9q1HXqCLJgEcZdq4gS8LOH\nrGEQZb8ikMOPFNxK0KhVBFuZ2ngbdtjMTwWuzPi6Xr8i6b9gKCMmALvPfAumQt7V19K57c+Sc1H\n7psAn/hgmq3sJy+NmWLNGYFRtbErNOlfIpNmCUJ5edk3TG4fKM9EdoAngTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8Vje",
  "gTVfmQkUfaPfrEixZEx7eh\n6HmpeTHP/d+5ZOTatlsG/MPySZUVJSZqyIT/h8VjeBegtcXRPDkpRJNHgTqyUslnO9o6liOF7aJ1HM\nHBaGYFPoKbX8Rq+41dQR3Nk1QXyFg6oVvJhSa5CjqyiZgZPiGu6JjAYVokGEzxlCmRZlzcvih9QyRW\njfHYi7Mzap7oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/",
  "oEojdM8NLudXQRluDhf8msDlNGgyWeQlmrEcpTMiZnSydthoWGLuXZ/PeVN0WnF/Hy\njbQ/6BbNThiE/P93A8xETizoS1QWPIc6JLEc7UFd8+X6Yc+qjbc/kaUdO1y3KUm9bS/dtsO9pgf8f\nNPR203iEYs6EtXV9pB6xHK0B3W587jpGoXDdZuS1DvLo9N2uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwl",
  "uHMTLf9ob8w1M49JqRyZx75UDpsQFjU\nVtVNMEx4jsQlhMSm7FvwfK7sCbh5dqwlhcbsQXc0EsDTiEg+hCWGx2cJds41hdOhbrpVJrMxMpsQF\np+zBI+6CWExpmLsFM9YliGxCZE8jnEexzSPGZYyl4RnJHPMCFlSrgWVj9OuZAJYmqDWJo7GoAcyVaj\nBNojlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs7",
  "jlgq68wrnyFrFiq7igavhwTUNa4YqNAEsbZE95g+3nJswCmGxyxXkjOBrIwmcBs729SZPf0FU\nUWe5ILoytIrSi8tvaT0wNIDSnNLyS+CIHptKfl1EkQXl5Qum/pPqWlpSWlA0sHlEaWRpQ+s/QZpaG\nlIaVrlq5Rqi0lT6RwR7B0j9KxpWNKDy09pPSNpW8oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJv",
  "oXbd0ndIjS48ofW/pe0qfWPqEUmYpo7RvaZ9Sb\nil5dRBEq5auUhpYSn7wV6zdJvSzNKM0qeWPqV0ZCn5VQz3M0vJ4w3cGC2VlL6w9AWlwlLy+y2IXl\nn6itLE0oTSl5a+pPSdpe8ofW7pc0pjS8m7AXg6sXSXUvsWqCo3bF0h9JzS8/d7wX4fBoD18LcshVs\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdj",
  "s\nUZpamlK6YSn5pQCPEpaekefJSLWn2uxtEznXIjXnDtZmfHY1yXmk5tzB2tNpdjU5nyI152PS9f7+/E\nUKpBRO+tOFpRX8FpYW9n/prTzsPdh5sPR4tX1De8v73vB+9Fb8X7zHnvr3rY38EJv4v3l/e39s7iw\n+HDx0eIfjXrzRnvNt17ns7j+H6oN1Xs=</latexit>E[x] = \u00b5\nRule 1:\nRule 2:\nRule 3:\nDef\u2019n",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9",
  "sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01Yr",
  "W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGY",
  "TiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l",
  "qx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SO",
  "HGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgm",
  "/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7U",
  "ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RC",
  "n\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nB",
  "1XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVb",
  "Hw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn",
  "m3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"ZepB7KjO5+Qtd/0U7z5fxFnMgk=\">AW\nu3iclZhbU9w2FICdXtP0RtopL3xlMlM",
  "\"ZepB7KjO5+Qtd/0U7z5fxFnMgk=\">AW\nu3iclZhbU9w2FICdXtP0RtopL3xlMlM2kl2gEkvL5kmkM0NUiCwQMIujOyVvQqybGwZlnj25/TX9LV96L/pkde7wueIh+5MsuJ8n3U5kmWtg0y\nKQi8v/3vjgw8/+viT29+duvzL786uF29/sF2mZh7wXpjLNDwNWcCkU72mhJT/Mcs6SQPKD4HTd8INznhciVXv6MuODhMVKRCJkGkInC7/3u+O\nMh",
  "NWcCkU72mhJT/Mcs6SQPKD4HTd8INznhciVXv6MuODhMVKRCJkGkInC7/3u+O\nMh5oP+5JH+uju+H4/KX8Xu3nIh7pgf/QnxtH4+PVwf0rfw6OV08WlpY7y/XHp4WVprDkNZ/tk9vfDfvDNCwTrnQoWVEcrSxnelCxXItQ8smtfl\nnwjIWnLOZHUFQs4cWgqkc68e9AZOhHaQ7/lPbr6NUrKpYUxWUSgJkwPSowM0EXOyp19NugEiorNVfhtKGol",
  "kc68e9AZOhHaQ7/lPbr6NUrKpYUxWUSgJkwPSowM0EXOyp19NugEiorNVfhtKGolL5OfZM2fyhyGLS8hAILcwF9cMRy\nxkIoeaFL8I0yRhalj17o7k6of8Fioip+VdaInk7bTrR0OxeuMtRd781qE5ol4z0kltWIquUbg8aSqeCfuYCA4ANHhBKSKF1CnyU8Q+SuIwsKSg\nIEH6Rg6F/mvJ6RqpXkMOWlpb4kGhUzyctaJxZMZdJSdkHx/Tu+AVz",
  "Q+SuIwsKSg\nIEH6Rg6F/mvJ6RqpXkMOWlpb4kGhUzyctaJxZMZdJSdkHx/Tu+AVznMAvQVfjiaA52M6Yms+s0H+s8qQoTwy3kTMW8bgKGHDJpRtQ2VCklXBq2\nrD+w9Zqp0yZxaVZ3NTcRZO3lbUfnNC9q2HbqCLJgEcZtq4gS8I2MGQJgyw35RMYcOKbiFsVCquCLMztPA3abWcmgtdmvUm0vW5F0n/OUEZMAO4\n+8y2YCnlbX0/ntj9LzntmwI",
  "VCquCLMztPA3abWcmgtdmvUm0vW5F0n/OUEZMAO4\n+8y2YCnlbX0/ntj9LzntmwIf+yOYrPYlLI+nw5o1AqNqYhNq1rlCJs0WhPL0om2a3jhUnon2AE0A3RlLlR0RbtXl2DJmnD/Hgw1LyU/ut/5mY\n8H1bK5bcx/JtQUVFmropM+H9UNIQHD15fEMGTl0o0eRCoJy+VsL+jqWM5XtgmUs8dFIRiUuhLdPuLWLWvqSO4s2mC+goBUy98M6HQJEdR",
  "eRCoJy+VsL+jqWM5XtgmUs8dFIRiUuhLdPuLWLWvqSO4s2mC+goBUy98M6HQJEdRWzYBI8\nM3PEIdCyhEgwynYwxlWpQ5J5sfWs8QqXWzLebCPKzaG6o0Qnvf4HJ+FZTh4XDOr7k8QBkNpvkM0lINWY6SOTZTOj7uFxpuMdfdX0/5tOi0Yn620\nbQH/YLZKcOQn51s4PmIiUdieqCM4uzLksR3tQ13y5Xu1ZtXH8E1nascN1m5LU2/TSbTvca3rA",
  "OQn51s4PmIiUdieqCM4uzLksR3tQ13y5Xu1ZtXH8E1nascN1m5LU2/TSbTvca3rAzYdvd0kHrGoI1FdTQ+pRyxHe1CXO4+brlE4\nXLcpSb2zPDpthzs30fKP9kZcM3NMSuXQHPtS2Z+GsKipqJ1imvAYidMQFpOybcHfWNkV8PBoW9MQFrcL0dZMAEtDLvEQpiEsTm/htnEsLrpUDf\ndKpPZCJnTEBafsQSPehrCYkzF2CmesixD4jRE8jCeRzRP",
  "piEsTm/htnEsLrpUDf\ndKpPZCJnTEBafsQSPehrCYkzF2CmesixD4jRE8jCeRzRPGZYylwSnpHMSNkSbkWVD5K25IJYGmMWhs7GoMeyFShBpsglgu68grnylNoFSu6inu\nuhnvXNKwZqtAEsLRF7jG/v+W8yQKcYjhmuZKcCWRlNIHb2Nmzuz0F0QVOckF0aWl5ReWHpB6YGlB5TmlpJfBEH02lLy6ySIzi09p3Tf0n1KS0\ntLSnuW9iNLI0ofWr",
  "Wl5ReWHpB6YGlB5TmlpJfBEH02lLy6ySIzi09p3Tf0n1KS0\ntLSnuW9iNLI0ofWrpU0pDS0NK1y1dp1RbSk6k8ESwdI/SkaUjSg8tPaT0jaVvKH1u6XNK31r6ltL3lr6n9LGljyljJKu5Z2KeWklcHQbRm6\nRqlgaXktx/ca5ZuU5pZmlH6xNInlA4tJb+K4XlmKTnewIPRUknpC0tfUCosJb/fguiVpa8oTSxNKH1p6UtK31n6jtJnlj6jNLaUv",
  "XlmKTnewIPRUknpC0tfUCosJb/fguiVpa8oTSxNKH1p6UtK31n6jtJnlj6jNLaUvBuA04mlu5Ta\nt0BVQemOpTuUnl65n4vwOfTGLgW5patYIvS1NKU0g1LyS8FOEpYekrOk5FqdrXZ2yayr0Vqzh2syfjsapLzSM25gzW70+xqsj9Fas5HpOvd/fmL\natexit>FEgp7PQnC0sr+C0sLeyvdlZ+6TzYebD0aK15Q3vT+97wbvrXi/eo+859621/N",
  "it>FEgp7PQnC0sr+C0sLeyvdlZ+6TzYebD0aK15Q3vT+97wbvrXi/eo+859621/NC70/vL+9v75/Fh4vh4rtFOVU/uNFc863X+iyW/wGDr+eV</l\nE\n\u21e5\n(x \u2212 \u00b5)2\u21e4\n= E[x2] \u2212 E[x]2\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\"# \u2212 \ud835\udd3c \ud835\udc53$\n\" #\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\" \u2212 \ud835\udd3c \ud835\udc53$\n\"\n#",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9",
  "sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01Yr",
  "W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGY",
  "TiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l",
  "qx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SO",
  "HGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgm",
  "/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7U",
  "ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RC",
  "n\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nB",
  "1XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVb",
  "Hw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn",
  "m3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\"# \u2212 \ud835\udd3c \ud835\udc53$\n\" #\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\" \u2212 \ud835\udd3c \ud835\udc53$\n\"\n#",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"65sbw94EZd3W\nrTqnlFJwRWZRc8=\">AYBnicrZhb9xEFIA34Vb",
  "sha1_base64=\"65sbw94EZd3W\nrTqnlFJwRWZRc8=\">AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpe\nktK7pc2Tldj79g7yXjs2ONkU2vf4c/whnjlb/Av+Amc8Xp36nMmlZCIVHY43+e5nBm\nPxw4yKQrd7f49M/vGm2+9/c6Vd6+9/4H3507frHe0Va5iHfDVOZ5gcBK7gUiu9qoS\nU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwm",
  "07frHe0Va5iHfDVOZ5gcBK7gUiu9qoS\nU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwmLlYhEyDSEetdnfvP8lWHGQ837h9HXvUqMj\nryv7tigL3mkD/2Aa2agd9PzizLpVcd3Fkcvqvu9wchfT3hs4PFoAPGRn4t4oI98lao\nyCXju+f7V1XZ+N5NUjG6wN2ON23o/29nMpS28NrWu4MeV0/7Kf6dXAhb1r892Fbv\n3n0cJiU5jvNH8bveuf9v1+GpY",
  "3o/29nMpS28NrWu4MeV0/7Kf6dXAhb1r892Fbv\n3n0cJiU5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLe\nHFU1Ytm5N2ASN+L0hz+Ke3V0VevqFhSFBdJAGbC9KDAzARd7LDU0c9HlVBZqbkKxw1F\npfR06pkV6PVFDlmQF1BgYS6gr14YDmDzORQk+LnYZokTPUrf2lc1TBMZCVfy0rN\nfsaNR2VmqHQ/EyY+n",
  "1BgYS6gr14YDmDzORQk+LnYZokTPUrf2lc1TBMZCVfy0rN\nfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+gTpOfI\nPIWEYV7VAIGHqRD6FzkbY1I1UrzGHLS0p4TDQqZ5MOWtUwsmMqkpWyD4nk3PAO4zmEW\noKvw9EcbGdMjSbXaT7UeVIVJoZbyJmKed0EDlk0oyobahSrg0bFm/YGuLqZMmcW\nlWdzU3EWTt",
  "MjSbXaT7UeVIVJoZbyJmKed0EDlk0oyobahSrg0bFm/YGuLqZMmcW\nlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTMStCoVQRbmR\np4G7bYzE8Frs9412t5KRdJ/xlBGTADuPvMrmAp5W19Op7Y3Sc5Z7ZsCH3oDmKz2JSyP\nx8OaNAKjamIjata5QibNFoTy9Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8\nl",
  "NAKjamIjata5QibNFoTy9Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8\nlP/xu4TYfHlVdc9uY/5BsQkVFmbkqMuH/UFEfnuF4fUET14q0eRBoJ68VML+jqaO5\nXhm0g9d1AQikmhL9DtL2LVvqaO4M6mCeorBEy98MuEQpMcRW3ZBIwMv3AacSygEA0y\nHI8xlGlR5pxsfmg9Q6TWzbaYC/Owam+o0gjtfYPL6VQhofDGb/k8gBlNBjnM",
  "A0y\nHI8xlGlR5pxsfmg9Q6TWzbaYC/Owam+o0gjtfYPL6VQhofDGb/k8gBlNBjnM0hL1Wc\n5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZqcMQ37aW8XzEROLOhLVBc/Z12SW\nI72oK7pcn21Z9Xqi2/J0o4drtuUpN6ml27b4V7SA365ujtGvGIR2J6mp6SD1iOdqD\nutx5XHONwuG6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsa",
  "p6SD1iOdqD\nutx5XHONwuG6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsaipqp5iaY3RbHIe\nwmJRtC/4fK9sCHh5taxzC4kYh2poJYKnPJR7COITF8S3cNpsYVtc6pbZTIbIHMcwu\nJDluBRj0NYjKkYO8UTlmVIHIdIHgc4jwOaxwxLmUvCM5I5ZoQsKdeCygdpWzIBLA1R\na0NHY9ADmSrUYBPEckFXuFceQqtYkVX8a6r4d1LGtYMV",
  "oQsKdeCygdpWzIBLA1R\na0NHY9ADmSrUYBPEckFXuFceQqtYkVX8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NU\nvxnS6BLIymsAN7GxQZ3L6C6KnOSC6MLSC0rPLT2ndN/SfUpzS8kbQRBtWUreToLozN\nIzSvcs3aO0tLSkdNfSXUojSyNKH1j6gNLQ0pDSZUuXKdWkhMpPBEs3aF0YOmA0gNL\nDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZp",
  "ZUuXKdWkhMpPBEs3aF0YOmA0gNL\nDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q/\nuNUs3KM0szSi9b+l9SvuWkrdieJ5ZSo438GC0VFL62NLHlApLyftbED219CmliaUJpU\n8sfULpsaXHlD609CGlsaXk2wCcTizdptR+BaoKSjct3aT01NJT93cBPp3GwLUw120F6\n5SmlqaUrlpK3hTgKGHpCTlPRqrZ",
  "tR+BaoKSjct3aT01NJT93cBPp3GwLUw120F6\n5SmlqaUrlpK3hTgKGHpCTlPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hS\npKR+Qrq/sT+kQEphp+9dm1/EX2FpYe/7hcUfF25v/jB/d6n5Qnul83ny843ncXOT5\n27nUedjc5uJ5z5Z/az2S9mvblf536f+2Puz7E6O9Nc80mn9Tf317+0F2Qn</latexit\n>\nE[f 0\ni] = E\n2\n4\u03b2i",
  "vblf536f+2Puz7E6O9Nc80mn9Tf317+0F2Qn</latexit\n>\nE[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\nDh\nX\nConsider the mean of the pre-activations:",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\n<latexit sha1_base64=\"65sbw94EZd3WrTqnlF\nJwRWZRc8=\">AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpektK7pc2Tldj79g7yX",
  "AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpektK7pc2Tldj79g7yX\njs2ONkU2vf4c/whnjlb/Av+Amc8Xp36nMmlZCIVHY43+e5nBmPxw4yKQrd7f49M/vGm2+9/c6Vd6+\n9/4H3507frHe0Va5iHfDVOZ5gcBK7gUiu9qoSU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWHGQ837h9HXvUqMjryv7tigL3m",
  "2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWHGQ837h9HXvUqMjryv7tigL3mkD/2Aa2agd9PzizLpVcd3Fkcvqvu9wchfT3hs4PFo\nAPGRn4t4oI98laoyCXju+f7V1XZ+N5NUjG6wN2ON23o/29nMpS28NrWu4MeV0/7Kf6dXAhb1r89\n2Fbv3n0cJiU5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLeHFU",
  "U5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLeHFU1Ytm\n5N2ASN+L0hz+Ke3V0VevqFhSFBdJAGbC9KDAzARd7LDU0c9HlVBZqbkKxw1FpfR06pkV6PVFDlmQF1\nBgYS6gr14YDmDzORQk+LnYZokTPUrf2lc1TBMZCVfy0rNfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+g",
  "NfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+gTpOfIPIWEYV7VAIGHqRD6FzkbY1I1UrzGHLS0p4TDQqZ5M\nOWtUwsmMqkpWyD4nk3PAO4zmEWoKvw9EcbGdMjSbXaT7UeVIVJoZbyJmKed0EDlk0oyobahSrg0\nbFm/YGuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTM",
  "GuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTMStCoVQR\nbmRp4G7bYzE8Frs9412t5KRdJ/xlBGTADuPvMrmAp5W19Op7Y3Sc5Z7ZsCH3oDmKz2JSyPx8OaNAKj\namIjata5QibNFoTy9Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQkVFmbkqMuH/UFEfnuF4fUET1",
  "p2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQkVFmbkqMuH/UFEfnuF4fUET14q0eRBoJ68VML+jqaO5Xhm0g9d1AQikmhL9DtL2LVvqaO4M6m\nCeorBEy98MuEQpMcRW3ZBIwMv3AacSygEA0yHI8xlGlR5pxsfmg9Q6TWzbaYC/Owam+o0gjtfYPL6V\nVQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZ",
  "VQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZqcMQ37aW8XzEROL\nOhLVBc/Z12SWI72oK7pcn21Z9Xqi2/J0o4drtuUpN6ml27b4V7SA365ujtGvGIR2J6mp6SD1iOd\nqDutx5XHONwuG6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/4f\nK9sCHh5taxzC4kYh2poJY",
  "OyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/4f\nK9sCHh5taxzC4kYh2poJYKnPJR7COITF8S3cNpsYVtc6pbZTIbIHMcwuJDluBRj0NYjKkYO8UTlm\nVIHIdIHgc4jwOaxwxLmUvCM5I5ZoQsKdeCygdpWzIBLA1Ra0NHY9ADmSrUYBPEckFXuFceQqtYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NUvxnS6BLIymsAN7GxQZ3L6C6",
  "tYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NUvxnS6BLIymsAN7GxQZ3L6C6KnOSC6MLSC0rPLT2ndN\n/SfUpzS8kbQRBtWUreToLozNIzSvcs3aO0tLSkdNfSXUojSyNKH1j6gNLQ0pDSZUuXKdWkhMpPBEs\n3aF0YOmA0gNLDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q\n/uNUs3KM0szSi9b+l9",
  "+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q\n/uNUs3KM0szSi9b+l9SvuWkrdieJ5ZSo438GC0VFL62NLHlApLyftbED219CmliaUJpU8sfULpsaXHl\nD609CGlsaXk2wCcTizdptR+BaoKSjct3aT01NJT93cBPp3GwLUw120F65SmlqaUrlpK3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hSpKR+Qrq",
  "K3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hSpKR+Qrq/sT+kQEphp+9dm1/EX2FpYe/7h\ncUfF25v/jB/d6n5Qnul83ny843ncXOT527nUedjc5uJ5z5Z/az2S9mvblf536f+2Puz7E6O9Nc80m\nn9Tf317+0F2Qn</latexit>\nE[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\n=",
  "E[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ij] E [hj]\n= 0 +\nDh\nX\nj=1\n0 \u00b7 E [hj] = 0",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\n<latexit sha1_base64=\"65sbw94EZd3WrTqnlF\nJwRWZRc8=\">AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpektK7pc2Tldj79g7yX",
  "AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpektK7pc2Tldj79g7yX\njs2ONkU2vf4c/whnjlb/Av+Amc8Xp36nMmlZCIVHY43+e5nBmPxw4yKQrd7f49M/vGm2+9/c6Vd6+\n9/4H3507frHe0Va5iHfDVOZ5gcBK7gUiu9qoSU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWHGQ837h9HXvUqMjryv7tigL3m",
  "2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWHGQ837h9HXvUqMjryv7tigL3mkD/2Aa2agd9PzizLpVcd3Fkcvqvu9wchfT3hs4PFo\nAPGRn4t4oI98laoyCXju+f7V1XZ+N5NUjG6wN2ON23o/29nMpS28NrWu4MeV0/7Kf6dXAhb1r89\n2Fbv3n0cJiU5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLeHFU",
  "U5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLeHFU1Ytm\n5N2ASN+L0hz+Ke3V0VevqFhSFBdJAGbC9KDAzARd7LDU0c9HlVBZqbkKxw1FpfR06pkV6PVFDlmQF1\nBgYS6gr14YDmDzORQk+LnYZokTPUrf2lc1TBMZCVfy0rNfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+g",
  "NfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+gTpOfIPIWEYV7VAIGHqRD6FzkbY1I1UrzGHLS0p4TDQqZ5M\nOWtUwsmMqkpWyD4nk3PAO4zmEWoKvw9EcbGdMjSbXaT7UeVIVJoZbyJmKed0EDlk0oyobahSrg0\nbFm/YGuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTM",
  "GuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTMStCoVQR\nbmRp4G7bYzE8Frs9412t5KRdJ/xlBGTADuPvMrmAp5W19Op7Y3Sc5Z7ZsCH3oDmKz2JSyPx8OaNAKj\namIjata5QibNFoTy9Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQkVFmbkqMuH/UFEfnuF4fUET1",
  "p2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQkVFmbkqMuH/UFEfnuF4fUET14q0eRBoJ68VML+jqaO5Xhm0g9d1AQikmhL9DtL2LVvqaO4M6m\nCeorBEy98MuEQpMcRW3ZBIwMv3AacSygEA0yHI8xlGlR5pxsfmg9Q6TWzbaYC/Owam+o0gjtfYPL6V\nVQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZ",
  "VQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZqcMQ37aW8XzEROL\nOhLVBc/Z12SWI72oK7pcn21Z9Xqi2/J0o4drtuUpN6ml27b4V7SA365ujtGvGIR2J6mp6SD1iOd\nqDutx5XHONwuG6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/4f\nK9sCHh5taxzC4kYh2poJY",
  "OyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/4f\nK9sCHh5taxzC4kYh2poJYKnPJR7COITF8S3cNpsYVtc6pbZTIbIHMcwuJDluBRj0NYjKkYO8UTlm\nVIHIdIHgc4jwOaxwxLmUvCM5I5ZoQsKdeCygdpWzIBLA1Ra0NHY9ADmSrUYBPEckFXuFceQqtYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NUvxnS6BLIymsAN7GxQZ3L6C6",
  "tYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NUvxnS6BLIymsAN7GxQZ3L6C6KnOSC6MLSC0rPLT2ndN\n/SfUpzS8kbQRBtWUreToLozNIzSvcs3aO0tLSkdNfSXUojSyNKH1j6gNLQ0pDSZUuXKdWkhMpPBEs\n3aF0YOmA0gNLDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q\n/uNUs3KM0szSi9b+l9",
  "+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q\n/uNUs3KM0szSi9b+l9SvuWkrdieJ5ZSo438GC0VFL62NLHlApLyftbED219CmliaUJpU8sfULpsaXHl\nD609CGlsaXk2wCcTizdptR+BaoKSjct3aT01NJT93cBPp3GwLUw120F65SmlqaUrlpK3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hSpKR+Qrq",
  "K3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hSpKR+Qrq/sT+kQEphp+9dm1/EX2FpYe/7h\ncUfF25v/jB/d6n5Qnul83ny843ncXOT527nUedjc5uJ5z5Z/az2S9mvblf536f+2Puz7E6O9Nc80m\nn9Tf317+0F2Qn</latexit>\nE[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\n=",
  "E[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ij] E [hj]\n= 0 +\nDh\nX\nj=1\n0 \u00b7 E [hj] = 0",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\nSet all the biases to 0\nWeights normally distributed \nmean 0 \nvariance \ud835\udf0e!\n\"\n \n<latexit sha1_base64=\"65sbw94EZd3WrTqnlF\nJwRWZRc8=\">AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosB",
  "4EZd3WrTqnlF\nJwRWZRc8=\">AYBnicrZhb9xEFIA34VbKrQVBHnjAIiogWqINosBLpTZpektK7pc2Tldj79g7yX\njs2ONkU2vf4c/whnjlb/Av+Amc8Xp36nMmlZCIVHY43+e5nBmPxw4yKQrd7f49M/vGm2+9/c6Vd6+\n9/4H3507frHe0Va5iHfDVOZ5gcBK7gUiu9qoSU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWH",
  "BK7gUiu9qoSU/yHLOkDy/eBk2fD9M54XIlU7+iLjRwmLlYhEyD\nSEetdnfvP8lWHGQ837h9HXvUqMjryv7tigL3mkD/2Aa2agd9PzizLpVcd3Fkcvqvu9wchfT3hs4PFo\nAPGRn4t4oI98laoyCXju+f7V1XZ+N5NUjG6wN2ON23o/29nMpS28NrWu4MeV0/7Kf6dXAhb1r89\n2Fbv3n0cJiU5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo",
  "7Kf6dXAhb1r89\n2Fbv3n0cJiU5jvNH8bveuf9v1+GpYJVzqUrCgOF7uZPqpYrkUo+eiqXxY8Y+EJi/khFBVLeHFU1Ytm\n5N2ASN+L0hz+Ke3V0VevqFhSFBdJAGbC9KDAzARd7LDU0c9HlVBZqbkKxw1FpfR06pkV6PVFDlmQF1\nBgYS6gr14YDmDzORQk+LnYZokTPUrf2lc1TBMZCVfy0rNfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4R",
  "nYZokTPUrf2lc1TBMZCVfy0rNfsaNR2VmqHQ/EyY+nxzrQWoXkiXnJS\nSa2YSi4ReDyqKr4QL2AgOACxwAlIFS+gTpOfIPIWEYV7VAIGHqRD6FzkbY1I1UrzGHLS0p4TDQqZ5M\nOWtUwsmMqkpWyD4nk3PAO4zmEWoKvw9EcbGdMjSbXaT7UeVIVJoZbyJmKed0EDlk0oyobahSrg0\nbFm/YGuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGb",
  "ed0EDlk0oyobahSrg0\nbFm/YGuLqZMmcWlWdzU3EWTt5G1H5zQvqt926giyYBHGbauOIEvCjtpnCYMsN+UeDjxTMStCoVQR\nbmRp4G7bYzE8Frs9412t5KRdJ/xlBGTADuPvMrmAp5W19Op7Y3Sc5Z7ZsCH3oDmKz2JSyPx8OaNAKj\namIjata5QibNFoTy9Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQ",
  "Lxtmt4VJ6J9gBNAN90ZS5U9Ip2qy7BkjVh/xYMNS8lP/xu4TYfHlVdc9uY/5\nBsQkVFmbkqMuH/UFEfnuF4fUET14q0eRBoJ68VML+jqaO5Xhm0g9d1AQikmhL9DtL2LVvqaO4M6m\nCeorBEy98MuEQpMcRW3ZBIwMv3AacSygEA0yHI8xlGlR5pxsfmg9Q6TWzbaYC/Owam+o0gjtfYPL6V\nVQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ou",
  "TWzbaYC/Owam+o0gjtfYPL6V\nVQhofDGb/k8gBlNBjnM0hL1Wc5SubQTOnwhV9ouMVcd3895eOi04r56WrTHvQLZqcMQ37aW8XzEROL\nOhLVBc/Z12SWI72oK7pcn21Z9Xqi2/J0o4drtuUpN6ml27b4V7SA365ujtGvGIR2J6mp6SD1iOd\nqDutx5XHONwuG6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/",
  "G6TUnqneTRaTvcqYmWf7QzgGOyOSalsm+Ofan0xyEsaipqp5iaY3RbHIewmJRtC/4f\nK9sCHh5taxzC4kYh2poJYKnPJR7COITF8S3cNpsYVtc6pbZTIbIHMcwuJDluBRj0NYjKkYO8UTlm\nVIHIdIHgc4jwOaxwxLmUvCM5I5ZoQsKdeCygdpWzIBLA1Ra0NHY9ADmSrUYBPEckFXuFceQqtYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7N",
  "NHY9ADmSrUYBPEckFXuFceQqtYkVX\n8a6r4d1LGtYMVWgCWFon95g3fn0lZoBT7NUvxnS6BLIymsAN7GxQZ3L6C6KnOSC6MLSC0rPLT2ndN\n/SfUpzS8kbQRBtWUreToLozNIzSvcs3aO0tLSkdNfSXUojSyNKH1j6gNLQ0pDSZUuXKdWkhMpPBEs\n3aF0YOmA0gNLDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlg",
  "mA0gNLDyh9ZukzSh9Z+ojS5Y+p/SlpS8pvWfpPUqZpYzSFUtXKOWk8HQbRk6RKlgaXk3Q\n/uNUs3KM0szSi9b+l9SvuWkrdieJ5ZSo438GC0VFL62NLHlApLyftbED219CmliaUJpU8sfULpsaXHl\nD609CGlsaXk2wCcTizdptR+BaoKSjct3aT01NJT93cBPp3GwLUw120F65SmlqaUrlpK3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc",
  "BPp3GwLUw120F65SmlqaUrlpK3hTgKGHpCTl\nPRqrZ1SZfm8i+Fqkpd7Am45OrSc4jNeUO1uxOk6vJ/hSpKR+Qrq/sT+kQEphp+9dm1/EX2FpYe/7h\ncUfF25v/jB/d6n5Qnul83ny843ncXOT527nUedjc5uJ5z5Z/az2S9mvblf536f+2Puz7E6O9Nc80m\nn9Tf317+0F2Qn</latexit>\nE[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E",
  "n9Tf317+0F2Qn</latexit>\nE[f 0\ni] = E\n2\n4\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n3\n5\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ijhj]\n= E [\u03b2i] +\nDh\nX\nj=1\nE [\u2326ij] E [hj]\n= 0 +\nDh\nX\nj=1\n0 \u00b7 E [hj] = 0",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8",
  "BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0",
  "SoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk",
  "g7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5",
  "zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpb",
  "h859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHor",
  "9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxL",
  "IuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2",
  "FVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdn",
  "LGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni",
  "5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n<latexit sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9",
  "sha1_base64=\"cW4r\ntVaEv2VIWDQZIFCt2xiZPrE=\">AWxXiclZhbU9w2FICd9JbSG\n2mnvPTFUyZtp0l3oNO0felMAiGEQMoSWCBhCSN7Za+CLBtbXpZ4\ndvqT+mv60Jf2r/TI613F54iH7kxicb7PuhzJtuwgk6LQKyt/37\nj5zrvf/BrQ8XPvr4k08/W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01Yr",
  "W7z9+WGRlnIe2Eq0/w4YAWXQvGeF\nlry4yznLAkPwrO1w0/GvG8EKk60FcZP01YrEQkQqYhdLa45feD\naOh/85vfT4J0XMFfPpucwCE6vdXqSqTgOd+v79gxOjb2gyigGv\nm34XCbsJjVldxtri80lmpfz4trDaFZa/5dc9ufznoD9KwTLjSoW\nRFcbK6kunTiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGY",
  "TiuVahJPFvplwTMWnrOYn0BRsYQXp1U95ol/ByIDP\n0pz+Ke0X0fPqNiSVFcJQGYCdPDAjMTdLGTUke/nlZCZaXmKpw2\nFJXS16lvEugPRM5DLa+gwMJcQF/9cMhyFmpI80Jf8cswTRKmBl\nV/bWNvAhnlsVAVvyjrlE8mbWejdjgUrzPWtg7mtQjNE/Gk0pqx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l",
  "qx\nVRyjcDjSVXxTtzBQHAosMJSBUvoE6TH1gSq4jCEpOAK7tonk9I\n1UrzGHLS0l4SDQqZ5OWtU4smMqkpeyD4vt3fAO4zmEWoKtw4Gg\nO9jOmJrPzNB/rPKkKE8Mt5EzFvG4ChwyaUbUNlQpJZwatqzfsf\nWcqfMmcWlWdzU3EWQd5G1H5zQvatB26giyYBHGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SO",
  "HGbauOIEvCDWHAE\ngZbspnMODENxG3KhRWBVmY3TwN2m1nJoLX5jiD6XtbVQk/SOG\nMmICcPWZo2Aq5G19PZ3b/iw5o9o3BT72hzBZ7VNYHk+HNWsERt\nXEJtSsc4VMmi0I5el2zS9cag8E+0BmgC+6MpcqOgt7V5dgiVrw\nv17MNS8lPzkh859Pj6tVsxlY/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgm",
  "/4j2YSKijJzVWTC/6OiATyC8PqC\nCJ68VKLJg0A9eamE+zuaOpbjhW0i9dxBQSgmhb5Cl7+IVfucOoI\n7myaorxAw9cKRCYUmOYrasgkYGY7wMHUsoBANMpyOMZRpUeac3P\nzQeoZIrZvbYi7Mw6p9Q5VGaN83uJyfBWV4OIz4NacHKPBNJ9BW\nqoBy1Eyx2ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7U",
  "ZKx6/6hYZLzHX1M+LTqtmF9sN+1Bv2B2yjDkF2fb\neD5iYlFHorpg9+KsSxL0R7UNV+ub/es2n71PVnascN1m5LU2/\nTSbTvca3rAL3Ycvd0hHrGoI1FdTQ+pRyxHe1CXO487rlE4XLcpS\nb2zPDpthzs30fKPDoawIzXbpFQOzLYvlf1pCIuaitopmZn2xan\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RC",
  "n\nISwmZduCv7GyL+Dh0bamISx2C9HWTABLAy7xEKYhLE4v4bZxLC\n641B3CqT2RCZ0xAWN1mCRz0NYTGmYuwUz1mWIXEaInkc4jwOaR\n4zLGUuCc9I5pgRsqRcCyofpm3JBLA0Rq2NHY1BD2SqUINEMsFX\nXmFc+UptIoVXcU9V8O9axrWDFVoAljaJdeY391XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nB",
  "1XmQBTjFs1xJ\nzgSyMprALna61Jnt/oKoIju5ILqy9IrS0svKT2y9IjS3FLyRh\nBEzy0lbydBNLJ0ROmhpYeUlpaWlPYs7VEaWRpR+tjSx5SGloaUr\nlu6Tqm2lOxI4Ylg6QGlQ0uHlB5bekzpC0tfUPrE0ieUvrT0JaVv\nLH1D6UNLH1LKLGWUbli6QSm3lHw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVb",
  "Hw6CKI1S9coDSwl735wrVnapTS\nzNKP0kaWPKB1YSt6K4XlmKdnewIPRUknplqVblApLyftbED2z9B\nmliaUJpU8tfUrpa0tfU7p6SalsaXk2wDsTizdp9R+BaoKSvcs3\naP0wtIL93cBPp/GwLUwd20Fu5SmlqaUbltK3hRgK2HpOdlPRq\n5q82+NpH7WqTm3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn",
  "m3MGajM/OJjmP1Jw7WHN3mp1N7k+RmvMh6frG4f\nxDCqQU7vRni8ur+CsLRz+2Fn9uXN/76flB2vNF9pb3lfe1953\nqr3i/fAe+J1vZ4Xen96f3n/eP8ubS4lS3pNFVv3mjO+cJr/Zb+\n+A/JU+lx</latexit>h = a[f],\nf 0 = \u03b2 + \u2326h\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\"# \u2212 \ud835\udd3c \ud835\udc53$\n\" # = \ud835\udd3c \ud835\udc53$\n\"#\n\ud835\udf0e!\"\n# = \ud835\udd3c \ud835\udc53$\n\" \u2212 \ud835\udd3c \ud835\udc53$\n\"\n#\n0",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\nSet all the biases to 0\nWeights normally distributed \nmean 0 \nvariance \ud835\udf0e!\n\"\n \n<latexit sha1_base64=\"dGgtaMvHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3",
  "vHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3bl0gJhNwgBcI1wctq7B17B8Zj4wsfzS/sr+h\nvZH9IxtdvA5Q6SsFHZyvs9nxmfG3rG9RIosn5/59rUe+9/8OFH1z+8cmn3+xc3pL3ezuEh9vuPHMk73PZxK\nRTfyUu+X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPl",
  "X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPlfPDfcdGSfcz/nwI\nBiUojos3SQVEXcWq/7dS+xHDfv6SMdVsSoij6eO697opHAlD/KD+u9t1+M50wf94mZFNCiP7i9A8keDUeWuRzU6\nKgaQbxyUxGO8p8OF5tG/+78pIe3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u",
  "Ie3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u\nfn649DGQtuY7bWfjcH010N3GPtFxFXuS5ZlBwvzSd4vWZoLX/LqhltkPGH+MQv5ATQVi3jWL+t1WTm3IDJ0gjiFf\nyp36ujlI0oWZdl5IEZsXyUYaDNnZQ5MEf/VKopMi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC",
  "Mi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC5DoUp+UtSXRV1nZXa4dC8ylh6tj3JInIeiTecJKkVneQKgYdVWfK5cA4DwQGIOU5ArHgGO\nXV9vMBZQBRuAxIwcC8ew+AC52VFUquch1CTjvaNBIJB93rGViwVRGHWULFMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41",
  "FMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41O9Yf2LrJVPHbeHipB5qiPI2k67Tp7Suqh16kjyIJFGHatOoIsCT\nftIYsYVLltD+CEI0dH7KpQWBVkYW6ksdftO9ERvDbre0bXWylJ+U8ZqogOwNWnvwVTPu/qy/HEdi6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3",
  "6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3RFKlRwSbtTt2DJ6rB7B041LSQ/uDt3j4/75b\ny+bPQfUk1IlBWJLZEOv0OiIWwT8PqCJ68WKLJg0A9ebGE+zuaOpbiha0j9dxBQygmRX6OLn8Rqu4xdQPNo7QWC\nGg8I3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe5",
  "3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe59g8vJUdCGH4dTfsXhHq\nqo19Tiws1ZCkq5lhP6fjQzXK4xGxXfz3lTdNqhfxkte0PxgWzU/g+Pxms4vkIiUdiXLBDtOaSxL0h/kmizXy\nMrVw9/Jks7tLh2U5K87SjtsW9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY",
  "W9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY\nLdv94mxXKot32xdJsQFnMq5lYx1rvgrtiEsBgVXQv+j5UtvbHuWk0IixuZ6Go6gKUhl/gUmhAWm0u4a7YxrK5Z1D\nW7ymQyQmYTwuITFuGzbkJYDKkYWsVjliRIbEKkjiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7Ou",
  "jiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7OuPIVWsaKreMfW8c4VHecMJdQBLK2Ta8xpnj6J6eESO/XzPp0ugayEFnADOxvUudj9eUFJdnJecG7\noOaVnhp5RumfoHqWpoeSJwAteGkqeTrzg1NBTSncN3aW0MLSgdMfQHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6F",
  "QHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6FNKXxv6mtI3hr6h9KGhDylhjJKVwxdoZQbSl4deMGSoUuUeoaSZz+41gzdoDQ\nxNKH0kaGPKB0aSp6K4fMULK9gR9GQyWlzwx9RqkwlDy/ecELQ19QGhkaUfrc0OeUHhl6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSm",
  "6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSmdNVQ8qQAWwlDj8l+MlDtXe3ibRO5rwVqwi2srfjF0aT\nnvY3eTs+f2po6n/pr6u9v/p2ZnpmZ+bZRp61x3zV63xmZv8HLmaHiw=</latexit>mgZpwC2vThdHk/tToCZ8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+",
  "8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+9B72\n\u03c32\nf 0 = E[f 02\ni ] \u2212 E[f 0\ni]2\n= E\n2\n64\n0\n@\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75 \u2212 0\n= E\n2\n64\n0\n@\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75\n=\nDh\nX\nj=1\nE\n\u21e5\n\u23262\nij\n\u21e4\nE\n\u21e5\nh2\nj\n\u21e4\n=\nDh\nX\nj=1\n\u03c32\n\u2326E\n\u21e5\nh2\nj\n\u21e4\n=",
  "\u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\nSet all the biases to 0\nWeights normally distributed \nmean 0 \nvariance \ud835\udf0e!\n\"\n \n<latexit sha1_base64=\"dGgtaMvHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3",
  "vHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3bl0gJhNwgBcI1wctq7B17B8Zj4wsfzS/sr+h\nvZH9IxtdvA5Q6SsFHZyvs9nxmfG3rG9RIosn5/59rUe+9/8OFH1z+8cmn3+xc3pL3ezuEh9vuPHMk73PZxK\nRTfyUu+X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPl",
  "X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPlfPDfcdGSfcz/nwI\nBiUojos3SQVEXcWq/7dS+xHDfv6SMdVsSoij6eO697opHAlD/KD+u9t1+M50wf94mZFNCiP7i9A8keDUeWuRzU6\nKgaQbxyUxGO8p8OF5tG/+78pIe3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u",
  "Ie3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u\nfn649DGQtuY7bWfjcH010N3GPtFxFXuS5ZlBwvzSd4vWZoLX/LqhltkPGH+MQv5ATQVi3jWL+t1WTm3IDJ0gjiFf\nyp36ujlI0oWZdl5IEZsXyUYaDNnZQ5MEf/VKopMi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC",
  "Mi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC5DoUp+UtSXRV1nZXa4dC8ylh6tj3JInIeiTecJKkVneQKgYdVWfK5cA4DwQGIOU5ArHgGO\nXV9vMBZQBRuAxIwcC8ew+AC52VFUquch1CTjvaNBIJB93rGViwVRGHWULFMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41",
  "FMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41O9Yf2LrJVPHbeHipB5qiPI2k67Tp7Suqh16kjyIJFGHatOoIsCT\nftIYsYVLltD+CEI0dH7KpQWBVkYW6ksdftO9ERvDbre0bXWylJ+U8ZqogOwNWnvwVTPu/qy/HEdi6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3",
  "6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3RFKlRwSbtTt2DJ6rB7B041LSQ/uDt3j4/75b\ny+bPQfUk1IlBWJLZEOv0OiIWwT8PqCJ68WKLJg0A9ebGE+zuaOpbiha0j9dxBQygmRX6OLn8Rqu4xdQPNo7QWC\nGg8I3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe5",
  "3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe59g8vJUdCGH4dTfsXhHq\nqo19Tiws1ZCkq5lhP6fjQzXK4xGxXfz3lTdNqhfxkte0PxgWzU/g+Pxms4vkIiUdiXLBDtOaSxL0h/kmizXy\nMrVw9/Jks7tLh2U5K87SjtsW9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY",
  "W9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY\nLdv94mxXKot32xdJsQFnMq5lYx1rvgrtiEsBgVXQv+j5UtvbHuWk0IixuZ6Go6gKUhl/gUmhAWm0u4a7YxrK5Z1D\nW7ymQyQmYTwuITFuGzbkJYDKkYWsVjliRIbEKkjiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7Ou",
  "jiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7OuPIVWsaKreMfW8c4VHecMJdQBLK2Ta8xpnj6J6eESO/XzPp0ugayEFnADOxvUudj9eUFJdnJecG7\noOaVnhp5RumfoHqWpoeSJwAteGkqeTrzg1NBTSncN3aW0MLSgdMfQHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6F",
  "QHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6FNKXxv6mtI3hr6h9KGhDylhjJKVwxdoZQbSl4deMGSoUuUeoaSZz+41gzdoDQ\nxNKH0kaGPKB0aSp6K4fMULK9gR9GQyWlzwx9RqkwlDy/ecELQ19QGhkaUfrc0OeUHhl6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSm",
  "6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSmdNVQ8qQAWwlDj8l+MlDtXe3ibRO5rwVqwi2srfjF0aT\nnvY3eTs+f2po6n/pr6u9v/p2ZnpmZ+bZRp61x3zV63xmZv8HLmaHiw=</latexit>mgZpwC2vThdHk/tToCZ8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+",
  "8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+9B72\n\u03c32\nf 0 = E[f 02\ni ] \u2212 E[f 0\ni]2\n= E\n2\n64\n0\n@\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75 \u2212 0\n= E\n2\n64\n0\n@\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75\n=\nDh\nX\nj=1\nE\n\u21e5\n\u23262\nij\n\u21e4\nE\n\u21e5\nh2\nj\n\u21e4\n=\nDh\nX\nj=1\n\u03c32\n\u2326E\n\u21e5\nh2\nj\n\u21e4\n=",
  "\u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\nSet all the biases to 0\nWeights normally distributed \nmean 0 \nvariance \ud835\udf0e!\n\"\n \n<latexit sha1_base64=\"dGgtaMvHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3",
  "vHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3bl0gJhNwgBcI1wctq7B17B8Zj4wsfzS/sr+h\nvZH9IxtdvA5Q6SsFHZyvs9nxmfG3rG9RIosn5/59rUe+9/8OFH1z+8cmn3+xc3pL3ezuEh9vuPHMk73PZxK\nRTfyUu+X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPl",
  "X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPlfPDfcdGSfcz/nwI\nBiUojos3SQVEXcWq/7dS+xHDfv6SMdVsSoij6eO697opHAlD/KD+u9t1+M50wf94mZFNCiP7i9A8keDUeWuRzU6\nKgaQbxyUxGO8p8OF5tG/+78pIe3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u",
  "Ie3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u\nfn649DGQtuY7bWfjcH010N3GPtFxFXuS5ZlBwvzSd4vWZoLX/LqhltkPGH+MQv5ATQVi3jWL+t1WTm3IDJ0gjiFf\nyp36ujlI0oWZdl5IEZsXyUYaDNnZQ5MEf/VKopMi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC",
  "Mi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC5DoUp+UtSXRV1nZXa4dC8ylh6tj3JInIeiTecJKkVneQKgYdVWfK5cA4DwQGIOU5ArHgGO\nXV9vMBZQBRuAxIwcC8ew+AC52VFUquch1CTjvaNBIJB93rGViwVRGHWULFMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41",
  "FMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41O9Yf2LrJVPHbeHipB5qiPI2k67Tp7Suqh16kjyIJFGHatOoIsCT\nftIYsYVLltD+CEI0dH7KpQWBVkYW6ksdftO9ERvDbre0bXWylJ+U8ZqogOwNWnvwVTPu/qy/HEdi6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3",
  "6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3RFKlRwSbtTt2DJ6rB7B041LSQ/uDt3j4/75b\ny+bPQfUk1IlBWJLZEOv0OiIWwT8PqCJ68WKLJg0A9ebGE+zuaOpbiha0j9dxBQygmRX6OLn8Rqu4xdQPNo7QWC\nGg8I3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe5",
  "3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe59g8vJUdCGH4dTfsXhHq\nqo19Tiws1ZCkq5lhP6fjQzXK4xGxXfz3lTdNqhfxkte0PxgWzU/g+Pxms4vkIiUdiXLBDtOaSxL0h/kmizXy\nMrVw9/Jks7tLh2U5K87SjtsW9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY",
  "W9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY\nLdv94mxXKot32xdJsQFnMq5lYx1rvgrtiEsBgVXQv+j5UtvbHuWk0IixuZ6Go6gKUhl/gUmhAWm0u4a7YxrK5Z1D\nW7ymQyQmYTwuITFuGzbkJYDKkYWsVjliRIbEKkjiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7Ou",
  "jiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7OuPIVWsaKreMfW8c4VHecMJdQBLK2Ta8xpnj6J6eESO/XzPp0ugayEFnADOxvUudj9eUFJdnJecG7\noOaVnhp5RumfoHqWpoeSJwAteGkqeTrzg1NBTSncN3aW0MLSgdMfQHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6F",
  "QHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6FNKXxv6mtI3hr6h9KGhDylhjJKVwxdoZQbSl4deMGSoUuUeoaSZz+41gzdoDQ\nxNKH0kaGPKB0aSp6K4fMULK9gR9GQyWlzwx9RqkwlDy/ecELQ19QGhkaUfrc0OeUHhl6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSm",
  "6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSmdNVQ8qQAWwlDj8l+MlDtXe3ibRO5rwVqwi2srfjF0aT\nnvY3eTs+f2po6n/pr6u9v/p2ZnpmZ+bZRp61x3zV63xmZv8HLmaHiw=</latexit>mgZpwC2vThdHk/tToCZ8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+",
  "8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+9B72\n\u03c32\nf 0 = E[f 02\ni ] \u2212 E[f 0\ni]2\n= E\n2\n64\n0\n@\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75 \u2212 0\n= E\n2\n64\n0\n@\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75\n=\nDh\nX\nj=1\nE\n\u21e5\n\u23262\nij\n\u21e4\nE\n\u21e5\nh2\nj\n\u21e4\n=\nDh\nX\nj=1\n\u03c32\n\u2326E\n\u21e5\nh2\nj\n\u21e4\n=",
  "\u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4\nFor all the cross terms, \n\ud835\udc38 \u03a9$% = 0 so only the \nsquared terms are left, then \nuse independence.",
  "<latexit sha1_base64=\"dBQAfIQvKlRTxv6XiOXt3wW6qVo=\n\">AYCHicpZjZbtw2FEDH7pamW9KicIG+qDFSFE1q2EW6vBRI7Ew2O7Udr4nlGJREaZihKFmLPRNhfqD9mb4Vfe1f9Df6Bb2U\nNEPrXvohqIHMPcbmkKI68VIq8WF7+Z27+rbfefe9K+9f/eDjz7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB",
  "7+5Nr1T/fzpMx8vucnMskOPZzKRTfK0Qh+WGacRZ7kh\n94wzXND854lotE7RbjlB/HLFIiFD4rIHRyfe53tz9KuV/wF0VkTwa6q/s2Pn6F2foulcJ9YOkcNzYS0ZVNDkaHV/QnRZ2L6Eur\nbZxQu3cstZ9ud84t/5fo9HR+A0au6ytWR3uacmC+sNpUyXCSRN1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlO",
  "N1RrfHbaklKuAphw9VTE6uLS4vLd/Di2stIXFXvu3dXL98A\nNEr+M4Wpfsjw/WlOi+OKZYXwJZ9cdcucp8wfsogfQVGxmOfHVb1sJs5NiAROmGTwTxVOHb14RcXiPB/HpgxKwY5ZjpoY0dlE\nf58DCNLy4Irv2koLKVTJI5eg04gMsifHEOB+ZmAvjr+gGUMcpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNS",
  "cpBTYqf+0kcMxVU7mp/e1K5Ho+EqjikTa/ayaTr9GsHcnipsfp\n4d1aLKHgsXnNSa3oSi4ReDSpKr4ULWEgOACxAlIFM+hTp0fL3RWEIW7VAKumXgvFsQqpWBY8gJx3tBdGgkEo+6lhrxIKpjD\nvKDiOc9PRgBcZzAJ0Fb4moOdlKnJ9LqCj4osrnIdwy1kTEW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmw",
  "EW8bgKG7DOpR9Q1VCklXOp3rF+x9YypYZu4JK27mukIsnazrlNk\nNC8q6Dp1BFmwCKOuVUeQJWFPDVjMIMt+QGHDs6YleFwqogC3MrS7xu26mO4LVZ7zdr1+R9J8xlBEdgLtPfwumfN7V15KZ7U\nyTc1b7usBHzgAmq3sJy6JmWNGYFRtbELNOlfIpNmCUJacd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZB",
  "cd03dG4vKU9EdoA7gm67MhAovaLfrEixZHXZvw1CzUvKj75Z+4KPja\nlnfNvqDZBMqysvUVpEOv0FATzF8fqCJ68RKLJg0A9eYmE/R1NHcvwtaReu6gIBSTohij219EqntNHcGdTWLUVwjoeuGbCYU\nmOQy7sg5oGb7hPGJZQD4apN+M0ZdJXmacbH5oPUOk1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdx",
  "1vW2mAn9sOpuqFIL3X2Dy9lVUIaHwxm/5HIPZdRr8uklpQpYhpI50lM6e\nunmBdxitru/nvKmaLUifretgf9gtkpfZ+fnqzj+YiIR2J6oIDoLUuSxLe1DXbLle7Fm1/vJbsrQji2s3Jam37aXdtriX9ICf\nblh6u0E8YlFHoraHlKPWJb2oC57Hjdso7C4dlOSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xs",
  "OSeqd5tNoWd2ai5R/uDnjB9DEpkYE+9iXSbUJYLKhYWMUk5hESmxAW47Jrwf\n+xsiPg4dG1mhAWt3LR1XQASwGXeAhNCIvNLdw12xhWNyzqhl1lMh0gswlh8SGL8aibEBYjKkZWcjSFIlNiORxgPM4oHlMsZTa\nJDwjqWVGyJKyLahskHQlHcDSCLU2sjQGPZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F",
  "ZCJQg2QSzndOXl1pWn0CpWdBXv2Rreu6ThgqEKdQBLm+Qec9xN603m4RTDMcuW5F\nQgK6UJ3MLOFnWmpz8vrMhJzgvHho4pPTf0nNIDQw8ozQwlvwi8Jmh5NeJF54ZekbpvqH7lJaGlpTuGbpHaWhoSOkDQx9Q6hvqU\n7pm6BqlhaHkRApPBEN3KR0YOqD0NBDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7",
  "BDSp8b+pzSR4Y+ovSFoS8ofW3oa0rvGXqPUmYo7RvaJ9Sbih5deCFq4auUuoZSn7wb1\nm6BalqaEpfcNvU9pYCj5VQzPM0PJ8QYejIZKSh8b+phSYSj5/eaFTw19SmlsaEzpE0OfUPrK0FeUPjT0IaWRoeTdAJxODN2h1\nLwFqnJKtw3dpvTU0FP7ewE+m0bPtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xb",
  "PtjA3TQWblCaGJpSuG0p+KcBRwtAhOU+Gqt3Vpm+byL4Wqhm3sDbj06tJzkM14xbW7k7Tq8n+\nFKoZH5Cu9/dnL1IgpbDTn1xbXMFvYWlh/ulR+X7mzfWby72r6hvdL7snej901vpfdT727vUW+rt9fz5/6d/2L+q/kbC78t/L\nHw58JfjTo/17zWa/zt/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] +",
  "/D3f5T7ZSo=</latexit>\nE\nh\nk\ni\n= k\nE\nh\nk \u00b7 g[x]\ni\n= k \u00b7 E\nh\ng[x]\ni\nE\nh\nf[x] + g[x]\ni\n= E\nh\nf[x]\ni\n+ E\nh\ng[x]\ni\nE\nh\nf[x]g[y]\ni\n= E\nh\nf[x]\ni\nE\nh\ng[y]\ni\nif\nx, y\nindependent\nRule 1:\nRule 2:\nRule 3:\nRule 4:\nSet all the biases to 0\nWeights normally distributed \nmean 0 \nvariance \ud835\udf0e!\n\"\n \n<latexit sha1_base64=\"dGgtaMvHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3",
  "vHKt1f4vTa9vZIE3\nXdU4=\">AYaXiclZhbT9xGFIA39Jamt9CqKmpfrK0aZsgQE3bl0gJhNwgBcI1wctq7B17B8Zj4wsfzS/sr+h\nvZH9IxtdvA5Q6SsFHZyvs9nxmfG3rG9RIosn5/59rUe+9/8OFH1z+8cmn3+xc3pL3ezuEh9vuPHMk73PZxK\nRTfyUu+X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPl",
  "X6SchZ5ku95x8ua753yNBOx2s7PE96PWKhEIHyWQ2gwfe0/x81EGLHDcrEalMGPlfPDfcdGSfcz/nwI\nBiUojos3SQVEXcWq/7dS+xHDfv6SMdVsSoij6eO697opHAlD/KD+u9t1+M50wf94mZFNCiP7i9A8keDUeWuRzU6\nKgaQbxyUxGO8p8OF5tG/+78pIe3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u",
  "Ie3dvCOaUlSfDzqxKQzKbpK3Q3N35YF52+LP2gSHy7is+pmc+5TnyR8W4LBzdn5u\nfn649DGQtuY7bWfjcH010N3GPtFxFXuS5ZlBwvzSd4vWZoLX/LqhltkPGH+MQv5ATQVi3jWL+t1WTm3IDJ0gjiFf\nyp36ujlI0oWZdl5IEZsXyUYaDNnZQ5MEf/VKopMi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC",
  "Mi58puOgkI6ezoRe4MRQo1kOfQYH4qYKyOP2Ipg7qkEnxM\nz+OIqaGpbu0slmVsC5DoUp+UtSXRV1nZXa4dC8ylh6tj3JInIeiTecJKkVneQKgYdVWfK5cA4DwQGIOU5ArHgGO\nXV9vMBZQBRuAxIwcC8ew+AC52VFUquch1CTjvaNBIJB93rGViwVRGHWULFMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41",
  "FMe5WjA8xRmAYKXxzNwVbCVHVxXM\n7HeRqVmY7hHlKmQl53AafsM6nPqGuoQko41O9Yf2LrJVPHbeHipB5qiPI2k67Tp7Suqh16kjyIJFGHatOoIsCT\nftIYsYVLltD+CEI0dH7KpQWBVkYW6ksdftO9ERvDbre0bXWylJ+U8ZqogOwNWnvwVTPu/qy/HEdi6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3",
  "6Kc1r7usHzg\ngmq3sIS8PmtC46gbNqYxU161ohk1YLQml81jX1aCwqT0T3BHUAX3RFKlRwSbtTt2DJ6rB7B041LSQ/uDt3j4/75b\ny+bPQfUk1IlBWJLZEOv0OiIWwT8PqCJ68WKLJg0A9ebGE+zuaOpbiha0j9dxBQygmRX6OLn8Rqu4xdQPNo7QWC\nGg8I3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe5",
  "3EwpNchB0ZR3QMnzDhseygHx0kn5zjr6MsyLl5OaH1jNEal3fFlOhf6y6N1Sphe59g8vJUdCGH4dTfsXhHq\nqo19Tiws1ZCkq5lhP6fjQzXK4xGxXfz3lTdNqhfxkte0PxgWzU/g+Pxms4vkIiUdiXLBDtOaSxL0h/kmizXy\nMrVw9/Jks7tLh2U5K87SjtsW9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY",
  "W9YgT8ZM0y2jXiEYs6EuVqR0g9Yln6g1z2Oq7ZzsLi2k1J8l7U0Wpb3ImJln+wPY\nLdv94mxXKot32xdJsQFnMq5lYx1rvgrtiEsBgVXQv+j5UtvbHuWk0IixuZ6Go6gKUhl/gUmhAWm0u4a7YxrK5Z1D\nW7ymQyQmYTwuITFuGzbkJYDKkYWsVjliRIbEKkjiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7Ou",
  "jiNcxGtY4KlxCbhGUksM0KWlG1BpaO4K+kAlsaot7GlMxiBjB\nXqsA1iOaMrL7OuPIVWsaKreMfW8c4VHecMJdQBLK2Ta8xpnj6J6eESO/XzPp0ugayEFnADOxvUudj9eUFJdnJecG7\noOaVnhp5RumfoHqWpoeSJwAteGkqeTrzg1NBTSncN3aW0MLSgdMfQHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6F",
  "QHUoDQwNKHxv6mFLfUJ/SZUOXKc0NJTtS+EU\nwdJvSkaEjSvcN3af0laGvKH1q6FNKXxv6mtI3hr6h9KGhDylhjJKVwxdoZQbSl4deMGSoUuUeoaSZz+41gzdoDQ\nxNKH0kaGPKB0aSp6K4fMULK9gR9GQyWlzwx9RqkwlDy/ecELQ19QGhkaUfrc0OeUHhl6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSm",
  "6ROkTQ59QGhpK3g3A7sT\nQLUrNW6Ayo3T0E1KTw9sb8X4JNp9GwLc90kWKc0NjSmdNVQ8qQAWwlDj8l+MlDtXe3ibRO5rwVqwi2srfjF0aT\nnvY3eTs+f2po6n/pr6u9v/p2ZnpmZ+bZRp61x3zV63xmZv8HLmaHiw=</latexit>mgZpwC2vThdHk/tToCZ8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+",
  "8RIa+sjt5kQIlhTv94ObsAn4LSxu7i3MLv83d2/x19sFS+4b2eu+73ve9272F3u+9B72\n\u03c32\nf 0 = E[f 02\ni ] \u2212 E[f 0\ni]2\n= E\n2\n64\n0\n@\u03b2i +\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75 \u2212 0\n= E\n2\n64\n0\n@\nDh\nX\nj=1\n\u2326ijhj\n1\nA\n23\n75\n=\nDh\nX\nj=1\nE\n\u21e5\n\u23262\nij\n\u21e4\nE\n\u21e5\nh2\nj\n\u21e4\n=\nDh\nX\nj=1\n\u03c32\n\u2326E\n\u21e5\nh2\nj\n\u21e4\n=",
  "\u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4\nBecause the \u03a9\u2019s are zero \nmean, this is the \nvariance.",
  "<latexit s\nha1_base64=\"kVj+lE9\nSoIyTm2A+dO6/hSln4\n=\">AYo3icpZjbts2G\nIDd7NRlp3bDgAC7ERa0\na4s2cIJ1202GNoe2adLF\nOThJGzkGJVOHhKIUiUq\ncCnqEPc1utwfZ2+ynJs\nVf+aiq4HWzP9IqmfB1\nNyEhZmotv98bMRx9/8u\nlnNz+f/eLr7+5tbtb\nw+yOE9d2ndjFqdHDskoC\nznti1AwepSklEQOo4fO\n2arkhxc0zcKY74",
  "r7+5tbtb\nw+yOE9d2ndjFqdHDskoC\nznti1AwepSklEQOo4fO\n2arkhxc0zcKY74urhA4i\n4vPQC10iIDS8PXPXsrP\nQj8hJsVQOC+n0rq73IS\nG9nZEfXKyBH/n0bA4XV\n4sT4q1YVBa9vo4oa6gI5\ntRTxwHAMuTJTsN/UAMb\nHv2rX8vnXYkROPi1261\nS+PVnf4L0rDLkYFo/g\nyxNXEGwKpoqtXnrPG57e\nt0ZV5MauGdHRASOU2w0\nDfze",
  "S+PVnf4L0rDLkYFo/g\nyxNXEGwKpoqtXnrPG57e\nt0ZV5MauGdHRASOU2w0\nDfzeHVTf9z+soa5qwqs\nT/D9r81LiFs0wL8EglyU\nMtrXcgLVhEZRaJUge3p\nrvLnSrj4ULi01hvtN8es\nPb34/sUezmEeXCZSTLj\nhe7iRgUJBWhy2g5a+cZT\nYh7Rnx6DEVOIpoNimpG\nl9YdiIwsL07hHxdWFX3\nioJEWXYVOWDK3Gc6k0E\nTO86F9ugCHmSC",
  "VOIpoNimpG\nl9YdiIwsL07hHxdWFX3\nioJEWXYVOWDK3Gc6k0E\nTO86F9ugCHmSC8rduiE\nvZ5aILbk8rFGYwpxkV1\nAgbhpCXy03IJAmAYto1u\nb0o2jiPBRYa+s75SF7\nVA/5AU9z6sFVZtZ71yK\nBSvM1Y29qe1hIJG4VuK\nKqkUWck1AvXLoqAL/oIO\nQgogXKAIxJxmUGc1bz1\nrUaOwgTDARb1ybDB2S1Q\n1F9SHnLS0N0iDQsLouG\nWtI",
  "QgogXKAIxJxmUGc1bz1\nrUaOwgTDARb1ybDB2S1Q\n1F9SHnLS0N0iDQsLouG\nWtIguGMmope6BY1h1LAi\npSGAXoKnxRbQz2EsLy\nXWCjkUaFZmM6S2khPu0a\ngJu2SVM3lHb4DljcKnb\nsv7QrV3Cz5rExUnV1VRG\nNGs/bTsixXnho7ZTRTQ\nLJqHftqIZjHY7kckIpD\nlpjyEG4sGTGrIdfVE\n3MXho7bYTGdHnZrWHt7\n31AqX/gmgZkQFYf",
  "jHY7kckIpD\nlpjyEG4sGTGrIdfVE\n3MXho7bYTGdHnZrWHt7\n31AqX/gmgZkQFYfI7J\nNylbX01ntrWJDkXlS8Ld\nGwFMFjtS0jq17c1aQTu\nqomV2KxypZk4WxBK48u2\nKXtjUGkStm9QBvRFl6e\nwm7+jPaxKMGVl2H4It5r\nmjB4/WnhMx4OiK5eN/A\n9lEyrK8sRUkQy/R0UjOG\nDo8wsi+uDFTBs8CFSDF\nzPY37WhI6k+sWkGjsoh\nJywU",
  "lEyrK8sRUkQy/R0UjOG\nDo8wsi+uDFTBs8CFSDF\nzPY37WhI6k+sWkGjsoh\nJywUFxpyz/0efuaKqJ3\nNo60vkJA1gvfJOTaIHte\nW5YBKcM3HJUME8jVbtK\nt79FlcZanFG1+2nyGSKX\nLbTEN5Y9Ve0NlUmjvG5\nRNr4Iy/Dhc0Gsud7SMOn\nU+nTjnI5JqyRzLIR2f2J\nmAJWZa/dWQ10Wj5dPz\naY96BeMTu69Hy4qY+Hj\nyzsMK0uOJsa62",
  "qyRzLIR2f2J\nmAJWZa/dWQ10Wj5dPz\naY96BeMTu69Hy4qY+Hj\nyzsMK0uOJsa62LIMrQH\ndU2n67s9KzZPHqCp7Rtc\ns8lQvU0vzbBvaYH9Hz\nL0Nst5CELO0yrq+kh9pB\nlaA/qMudxy3QXBtdsMl\nTvJI9G2+BOTW36e/sBFU\nQek2I2kse+mNl1SBcF\noVRjOU5ty3WIV2M8rYFf\n+vKnjwrt606pIu9LGxr\nMqBLI8r0W6hDulgv4bZ\nxHR",
  "oVRjOU5ty3WIV2M8rYFf\n+vKnjwrt606pIu9LGxr\nMqBLI8r0W6hDulgv4bZ\nxHR1y6BumVXCkAz65A\nuPieRftd1SBd9LPpG8Yw\nkiSbWIZTHQM9jgPOY6F\nJikvQRSQwjgqaUaUKlQd\nyWZECXxlprY0Nj0AMWc\n63BJqjLGZ5mXHmcW0Wc\nzyL+6aG+9c0LIhWoQzo\n0jZaY1b9fIlMR08xHLNM\nSU5CzUpwAnu608PO5PT\nneAU6yTnelaJX",
  "0LIhWoQzo\n0jZaY1b9fIlMR08xHLNM\nSU5CzUpwAnu608PO5PT\nneAU6yTnelaJXmF4qeon\npoaKHmKaKoicCx9tVFD\n2dON6FoheYHih6gGmuaI\n5pX9E+p6iHqbPFH2Gq\nauoi+mqoquYCkXRiR+E\nRTdxzRQNMD0SNEjTF8r\n+hrTF4q+wPSNom8wfavo\nW0yfKvoU6IowXRd0XV\nMqaLo1YHjrSi6gqmjKHr\n2g7WmaA/TRNE0zVF1z\nAdKY",
  "W0yfKvoU6IowXRd0XV\nMqaLo1YHjrSi6gqmjKHr\n2g7WmaA/TRNE0zVF1z\nAdKYqeiuH3TF0vIEfRk\nUZphuKbmAaKoqe3xzvl\naKvMI0UjTB9qehLTE8VP\ncX0uaLPMfUVRe8G4HSi\n6B6m6i1QkWG6o+gOpueK\nnpvfC9DpMDqmibmtKtj\nGNFY0xnRTUfSkAEcJRc/\nQedLjza42eduE9jWPT7\nmBNRmfXI1y7vEpN7Bmd5\npcjfYnj095gL",
  "fSkAEcJRc/\nQedLjza42eduE9jWPT7\nmBNRmfXI1y7vEpN7Bmd5\npcjfYnj095gLq+fjB9k\nQIphZ1+eGt+UX8LiwsHS\nwuLvyw83vl5/slK84b2\nZueHzo+de53Fzq+dJ50X\nnV6n3Fn/pz5a+bvmX/m\n7sxtzu3O7dfqzI3mu8\n6rc/c4D+bhZnt</latex\nit>\n\u03c32\nf 0 = \u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4\n=",
  "6rc/c4D+bhZnt</latex\nit>\n\u03c32\nf 0 = \u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nh2\nj\n\u21e4\n= \u03c32\n\u2326\nDh\nX\nj=1\nE\n\u21e5\nReLU[fj]2\u21e4\n= \u03c32\n\u2326\nDh\nX\nj=1\nZ 1\n\u22121\nReLU[fj]2Pr(fj)dfj\n= \u03c32\n\u2326\nDh\nX\nj=1\nZ 1\n\u22121\n(I[fj > 0]fj)2Pr(fj)dfj\n= \u03c32\n\u2326\nDh\nX\nj=1\nZ 1\n0\nf 2\nj Pr(fj)dfj\n= \u03c32\n\u2326\nDh\nX\nj=1\n\u03c32\nf\n2 =\nDh\u03c32\n\u2326\u03c32\nf\n2\nFrom the definition of expectation.",
  "1\n0\nf 2\nj Pr(fj)dfj\n= \u03c32\n\u2326\nDh\nX\nj=1\n\u03c32\nf\n2 =\nDh\u03c32\n\u2326\u03c32\nf\n2\nFrom the definition of expectation.\nOnly positive integral limits \nbecause of ReLU\n\u00bd of the variance for zero mean \ndistribution",
  "Aim: keep variance same between two layers\n<latexit sha1_base64=\"wqVi\nTfkXh3lhdP2nYAkf3Zl582o=\">AWuniclZhbU9w2FICdXtP0R\ntopL3xlMm0l3gGnaPrSdBEJukAKBRIWdmSv7FWQZWPLsMSz\n/6a/pq/tS/9Nj2zvKj5HPJSZdNXzfdblSJZlB5kUhV5e/vfGO+\n+9/4H9786NbHn3z62ecLt784KNIyD3k/TGWaHwWs4FIo3tdCS\n36U5ZwlgeSHwdm64YcXPC9Eqv",
  "Hn3z62ecLt784KNIyD3k/TGWaHwWs4FIo3tdCS\n36U5ZwlgeSHwdm64YcXPC9Eqvb1VcZPEhYrEYmQaQgNF373B4WI\nE3ZarU6HVfTt1P/NH0Q5C6uHw2o8beBwsJ3wmJ2utu4qmNMpXOE\nPF5aWe8v1n08LK21hyWv/doa3vxoNRmlYJlzpULKiOF5ZzvRJxX\nItQsmntwZlwTMWnrGYH0NRsYQXJ1U90Kl/ByIjP0pz+Ke0X0fv\nqJiSVFcJQG",
  "xX\nItQsmntwZlwTMWnrGYH0NRsYQXJ1U90Kl/ByIjP0pz+Ke0X0fv\nqJiSVFcJQGYCdPjAjMTdLHjUke/nFRCZaXmKmwaikrp69Q3WfNH\nIuehldQYGEuoK9+OGaQJg25vTVQ/DJMk4SpUTVY29idVoOAx0\nJV/Lys8zydp2N2uFQvM5Ye7o/r0Vonog3nFRSK6aSawQeT6uK9\n+IeBoIDED1OQKp4AXWa/ASRv4IorCsJGHiQTqBzkf9iSqpWms",
  "SK6aSawQeT6uK9\n+IeBoIDED1OQKp4AXWa/ASRv4IorCsJGHiQTqBzkf9iSqpWmseQ\nk472imhQyCSfdKx1YsFUJh1lDxTfv+MbwHUOswBdhR+O5mAvY2o\n6u07zic6TqjAx3ELOVMzrJmDIZNmRF1DlVLCpWH+gNbL5g6ax\nOXZnVXcxNB1n7edXRO86JGXaeOIAsWYdy16giyJOwCI5YwyHJbH\nsKAE9E3KpQWBVkYe7kadBtOzMRvDYnGdwvX",
  "aeOIAsWYdy16giyJOwCI5YwyHJbH\nsKAE9E3KpQWBVkYe7kadBtOzMRvDYnGdwvXW+jIum/YCgjJgB3\nn/kVTIW8q6+nc9ufJei9k2BT/wxTFb3EpbHzbBmjcCo2tiUmn\nWukEmzBaE8veyapjcOlWeiO0ATwDdmQsVvaXdrUuwZE14cBeGm\npeSH/Qu8cnJ9WyuW3Mf0g2oaKizFwVmfD/qGgEzx28viCJy+V\naPIgUE9eKmF/R1PHcrywTaSeO",
  "9WyuW3Mf0g2oaKizFwVmfD/qGgEzx28viCJy+V\naPIgUE9eKmF/R1PHcrywTaSeOygIxaTQV+j2F7HqXlNHcGfTBPU\nVAqZe+GVCoUmOoq5sAkaGX3iCOhZQiAYZNmMZVqUOSebH1rPEK\nl1sy3mwjysuhuqNEJ3+ByfhWU4eFwa+5PEAZDZp8BmpRixHy\nZyYKZ2cDgoNt5jr7q+nvCk6rZifb7btQb9gdsow5OfDTwfMbGo\nI1FdcGRx1iWJ5W",
  "yYKZ2cDgoNt5jr7q+nvCk6rZifb7btQb9gdsow5OfDTwfMbGo\nI1FdcGRx1iWJ5WgP6pov17d7Vm2efk+Wduxw3aYk9ba9dNsO95\noe8PMtR2+3iEcs6khUV9tD6hHL0R7U5c7jlmsUDtdtSlLvLI9O2\n+HOTbT8o/0x18wck1I5Mse+VA6aEBY1FbVTM05tys2ISwmZdeC\n/8fKnjkrd60mhMWdQnQ1E8DSiEs8hCaExeYW7ptDKtbDnXLrTK\nZ",
  "ys2ISwmZdeC\n/8fKnjkrd60mhMWdQnQ1E8DSiEs8hCaExeYW7ptDKtbDnXLrTK\nZjZHZhLD4mCV41E0IizEVY6d4xrIMiU2I5HGM8zimecywlLkPC\nOZY0bIknItqHycdiUTwNIEtTZxNAY9kKlCDbZBLBd05RXOlafQK\nlZ0FfdDfevaVgzVKEJYGmb3GN+835JzACnGI5ZriRnAlkZTeAO\ndnaoMzv9BVFTnJBdGXpFaWXl5SemjpIaW5peSNI",
  "835JzACnGI5ZriRnAlkZTeAO\ndnaoMzv9BVFTnJBdGXpFaWXl5SemjpIaW5peSNIheWEreTo\nLowtILSg8sPaC0tLSktG9pn9LI0ojSR5Y+ojS0NKR03dJ1SrWl5\nEQKTwRL9ykdWzqm9MjSI0pfWvqS0ieWPqH0laWvKH1j6RtKH1j6\ngFJmKaN0w9INSrml5NBEK1ZukZpYCl594N7zdIdSjNLM0ofWvq\nQ0pGl5K0YnmeWkuMNPBgtlZQ+tfQ",
  "l5NBEK1ZukZpYCl594N7zdIdSjNLM0ofWvq\nQ0pGl5K0YnmeWkuMNPBgtlZQ+tfQpcJS8v4WRM8tfU5pYmlC6T\nNLn1H62tLXlD629DGlsaXk2wCcTizdo9R+BaoKSnct3aX03NJz9\n3cBPp/GwLUwt20F25SmlqaUblpK3hTgKGHpGTlPRqrd1WZfm8i\n+Fqk5d7A247OrSc4jNecO1u5Os6vJ/hSpOR+Trm8czD+kQEphpx\n8uLK3gr7C0cLDa",
  "qk5d7A247OrSc4jNecO1u5Os6vJ/hSpOR+Trm8czD+kQEphpx\n8uLK3gr7C0cLDaW/mpd2/3x6X7a+0X2pve1943nfeivezd974\nu14fS/0/vT+8v72/ln8dTFYFItnjfrOjfaL73O36L+D3yI58k=\n</latexit>\n\u03c32\nf 0 =\nDh\u03c32\n\u2326\u03c32\nf\n2\n<latexit sha1_base64=\"XLmfGzSKSfZ\nYSCXPvpNpnhsJxmA=\">AWm3iclZhb9s2FIDV7tZ1t3T",
  "_base64=\"XLmfGzSKSfZ\nYSCXPvpNpnhsJxmA=\">AWm3iclZhb9s2FIDV7tZ1t3TD8jIMExYUGIbOSIru8\njKgTZreki5OEydp49SgZEpmQ1GKRCVOBT/u1+x1+zH7NzuUZLM6h3lYgM7c+T7xck\nhKlIJMikKvrv57fp73/w4Uc3Pr75yaef7F0q0vD4q0zEM+CFOZ5kcBK7gUig+\n0JIfZTlnSD5YXC6YfjhOc8Lkap9fZnxk4TFSkQiZBpCo6",
  "EM+CFOZ5kcBK7gUig+\n0JIfZTlnSD5YXC6YfjhOc8Lkap9fZnxk4TFSkQiZBpCo6XvhoWIEzYa7iQ8Zq/\nv+r/7wyhnYXV3Vj0cTWajpZXV3mr959PCWltY8dq/ujW1+PhOA3LhCsdSlYUx2ur\nmT6pWK5FKPns5rAseMbCUxbzYygqlvDipKpHMvNvQ2TsR2kO/5T26+i7V1QsKYrLJ\nAzYXpSYGaCLnZc6ui3k0qorNRchU1DUSl9nfomLf5Y5",
  "2kO/5T26+i7V1QsKYrLJ\nAzYXpSYGaCLnZc6ui3k0qorNRchU1DUSl9nfomLf5Y5DzU8hIKLMwF9NUPJwSo\nSF5N4eKX4RpkjA1robrm7uzahjwWKiKn5V1ImezrNZOxyKVxnrT/cXtQjNE/GWk0\npqxVRyhcDjWVXxXtzDQHAoscJSBUvoE6TnyDy1xCFhSMBAw/SKXQu8l/MSNVK8x\nhy0tFeEQ0KmeTjrVBLJjKpKPsgeL7t30DuM5hFqCr8M",
  "MBAw/SKXQu8l/MSNVK8x\nhy0tFeEQ0KmeTjrVBLJjKpKPsgeL7t30DuM5hFqCr8MPRHOxlTM3m12k+1XlSFSa\nGW8iZindBAw5ZNKMqGuoUkq4NOxYf2DrBVOnbeLSrO5qbiLI2s+7js5pXtS469QR\nZMEijLtWHUGWhG0+ZgmDLflEQw48U3ErQqFVUEWZj9Pg27bmYngtTnNYL90vc2K\npP+coYyYAOw+8yuYCnlX30gXtj9PzntmwKf+hOYrO",
  "9Pg27bmYngtTnNYL90vc2K\npP+coYyYAOw+8yuYCnlX30gXtj9PzntmwKf+hOYrO4lLI+bYc0bgVG1sRk161whk\n2YLQnl60TVNbxwqz0R3gCaAN12ZCxW9o92pS7BkTXh4B4al5If/9T7mU9PqlWzbc\nx/SDahoqLMXBWZ8P+oaAwPFry+InL5Vo8iBQT14q4f6Opo7leGbSD13UBCKSa\nEv0fYXsepeU0dwZ9ME9RUCpl74ZUKhSY6irmwCRoZ",
  "q4f6Opo7leGbSD13UBCKSa\nEv0fYXsepeU0dwZ9ME9RUCpl74ZUKhSY6irmwCRoZfeEQ6FlCIBhk2YwxlWpQ5Jzc\n/tJ4hUuvmtpgL87Dq3lClEbr3DS4XV0EZHg7n/IrLA5TRoMlnkJZqzHKUzKmZ0un\nrYaFhi7l2fz3lTdFpxfxsq20P+gWzU4YhPxt4fmIiUdieqCM4mzLksR3tQ12K5\nvtuzauv1j2Rpxw7XbUpSb9tLt+1wr+gBP9t29Ha",
  "mIiUdieqCM4mzLksR3tQ12K5\nvtuzauv1j2Rpxw7XbUpSb9tLt+1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu\n0bhcN2mJPXO8+i0He7CRMs/2p9wzcwxKZVjc+xL5bAJYVFTUTvF1Bxzu2ITwmJSd\ni34f6zsmZNz12pCWOwXoquZAJbGXOIhNCEsNlu4a7YxrG471G23ymQ2QWYTwuJjlu\nBRNyEsxlSMneIpyzIkNiGSxwnO4TmMcNS5p",
  "u4a7YxrG471G23ymQ2QWYTwuJjlu\nBRNyEsxlSMneIpyzIkNiGSxwnO4TmMcNS5pLwjGSOGSFLyrWg8knalUwAS1PU2tT\nRGPRApgo12AaxXNCVzhXnkKrWNFVPHA1PLiYc1QhSaApR2yx/zm9ZKYAU4xHLN\ncSc4EsjKawD52+tSZn/6CqCInuSC6tPS0gtLyg9tPSQ0txS8kYQRC8sJW8nQXRu\n6TmlB5YeUFpaWlI6sHRAaWRpROkjSx9RGlo",
  "Lyg9tPSQ0txS8kYQRC8sJW8nQXRu\n6TmlB5YeUFpaWlI6sHRAaWRpROkjSx9RGloaUrph6Qal2lJyIoUngqX7lE4snVB6Z\nOkRpS8tfUnpE0ufUPrK0leUvrX0LaUPLH1AKbOUbp6Sal3FLy6SCI1i1dpzSwl\nLz7wV6ztE9pZmlG6UNLH1I6tpS8FcPzFJyvIEHo6WS0qeWPqVUWEre34LouaXPKU\n0sTSh9ZukzSt9Y+obSx5Y+pjS2lHwbgNO",
  "vIEHo6WS0qeWPqVUWEre34LouaXPKU\n0sTSh9ZukzSt9Y+obSx5Y+pjS2lHwbgNOJpXuU2q9AVUHprqW7lJ5Zeub+LsAX0x\ni4FuaOrWCH0tTSlNItS8mbAhwlLD0l58lItXe1+dcmcl+L1I7WJvx+dUk5FacAd\nr707zq8n9KVILPiFd3zxYfEiBlMKdfrS0soa/wtLCwd3e2i+9e7v3Vu6vt19ob3jf\n2sc=</latexit>eN97P3hr3q/efe+J",
  "0soa/wtLCwd3e2i+9e7v3Vu6vt19ob3jf\n2sc=</latexit>eN97P3hr3q/efe+J1/cGXuj96f3l/e39s/zt8sbys+XtRr1+rb3mK6/ztz4D9kA\n\u03c32\n\u2326 =\n2\nDh\nShould choose:\nThis is called He initialization or Kaiming initialization. \n\ud835\udf0e!\"\n# = \ud835\udf0e!\n#\nSince:\nTo get:\nK. He, X. Zhang, S. Ren, and J. Sun, \u201cDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,\u201d Proc. IEEE International Conference on \nComputer Vision, 2015, pp. 1026\u20131034. Accessed: Feb. 11, 2024.",
  "He initialization (assumes ReLU)\n\u2022 Forward pass:  want the variance of hidden unit activations in layer \nk+1 to be the same as variance of activations in layer k:\n\u2022 Backward pass:  want the variance of gradients at layer k to be the \nsame as variance of gradient in layer k+1:\n<latexit sha1_base64=\"XLmfGzSKSfZ\nYSCXPvpNpnhsJxmA=\">AWm3iclZhb9s2FIDV7tZ1t3TD8jIMExYUGIbOSIru8j\nKgTZreki5OEydp49SgZEpmQ1GKRCVOBT/u1+x1+zH7NzuUZLM6h3lYgM7c+T7xckh\nKlIJMikKvrv57fp73/w4Uc3Pr75yaef",
  "u1+x1+zH7NzuUZLM6h3lYgM7c+T7xckh\nKlIJMikKvrv57fp73/w4Uc3Pr75yaef7F0q0vD4q0zEM+CFOZ5kcBK7gUig+0\n0JIfZTlnSD5YXC6YfjhOc8Lkap9fZnxk4TFSkQiZBpCo6XvhoWIEzYa7iQ8Zq/v+\nr/7wyhnYXV3Vj0cTWajpZXV3mr959PCWltY8dq/ujW1+PhOA3LhCsdSlYUx2urm\nT6pWK5FKPns5rAseMbCUxbzYygqlvDip",
  "ltY8dq/ujW1+PhOA3LhCsdSlYUx2urm\nT6pWK5FKPns5rAseMbCUxbzYygqlvDipKpHMvNvQ2TsR2kO/5T26+i7V1QsKYrLJA\nAzYXpSYGaCLnZc6ui3k0qorNRchU1DUSl9nfomLf5Y5DzU8hIKLMwF9NUPJwSoSF\n5N4eKX4RpkjA1robrm7uzahjwWKiKn5V1ImezrNZOxyKVxnrT/cXtQjNE/GWk0pq\nxVRyhcDjWVXxXtzDQHAoscJSBUvo",
  "n5V1ImezrNZOxyKVxnrT/cXtQjNE/GWk0pq\nxVRyhcDjWVXxXtzDQHAoscJSBUvoE6TnyDy1xCFhSMBAw/SKXQu8l/MSNVK8xhy0\ntFeEQ0KmeTjrVBLJjKpKPsgeL7t30DuM5hFqCr8MPRHOxlTM3m12k+1XlSFSaGW8\niZindBAw5ZNKMqGuoUkq4NOxYf2DrBVOnbeLSrO5qbiLI2s+7js5pXtS469QRZME\nijLtWHUGWhG0+ZgmDLflEQw48",
  "2DrBVOnbeLSrO5qbiLI2s+7js5pXtS469QRZME\nijLtWHUGWhG0+ZgmDLflEQw48U3ErQqFVUEWZj9Pg27bmYngtTnNYL90vc2KpP+c\noYyYAOw+8yuYCnlX30gXtj9PzntmwKf+hOYrO4lLI+bYc0bgVG1sRk161whk2YLQ\nnl60TVNbxwqz0R3gCaAN12ZCxW9o92pS7BkTXh4B4al5If/9T7mU9PqlWzbcx/SD\nahoqLMXBWZ8P+oaAwPFry+",
  "CxW9o92pS7BkTXh4B4al5If/9T7mU9PqlWzbcx/SD\nahoqLMXBWZ8P+oaAwPFry+InL5Vo8iBQT14q4f6Opo7leGbSD13UBCKSaEv0f\nYXsepeU0dwZ9ME9RUCpl74ZUKhSY6irmwCRoZfeEQ6FlCIBhk2YwxlWpQ5Jzc/tJ4\nhUuvmtpgL87Dq3lClEbr3DS4XV0EZHg7n/IrLA5TRoMlnkJZqzHKUzKmZ0unrYaFh\ni7l2fz3lTdFpxfxsq20P",
  "S4XV0EZHg7n/IrLA5TRoMlnkJZqzHKUzKmZ0unrYaFh\ni7l2fz3lTdFpxfxsq20P+gWzU4YhPxt4fmIiUdieqCM4mzLksR3tQ12K5vtuza\nuv1j2Rpxw7XbUpSb9tLt+1wr+gBP9t29HabeMSijkR1tT2kHrEc7UFd7jxu0bhcN\n2mJPXO8+i0He7CRMs/2p9wzcwxKZVjc+xL5bAJYVFTUTvF1Bxzu2ITwmJSdi34f6z\nsmZNz12pCWOwXoquZA",
  "9wzcwxKZVjc+xL5bAJYVFTUTvF1Bxzu2ITwmJSdi34f6z\nsmZNz12pCWOwXoquZAJbGXOIhNCEsNlu4a7YxrG471G23ymQ2QWYTwuJjluBRNyEs\nxlSMneIpyzIkNiGSxwnO4TmMcNS5pLwjGSOGSFLyrWg8knalUwAS1PU2tTRGPRAp\ngo12AaxXNCVzhXnkKrWNFVPHA1PLiYc1QhSaApR2yx/zm9ZKYAU4xHLNcSc4Esj\nKawD52+tSZn/6Cq",
  "kKrWNFVPHA1PLiYc1QhSaApR2yx/zm9ZKYAU4xHLNcSc4Esj\nKawD52+tSZn/6CqCInuSC6tPS0gtLyg9tPSQ0txS8kYQRC8sJW8nQXRu6TmlB5Y\neUFpaWlI6sHRAaWRpROkjSx9RGloaUrph6Qal2lJyIoUngqX7lE4snVB6ZOkRpS8\ntfUnpE0ufUPrK0leUvrX0LaUPLH1AKbOUbp6Sal3FLy6SCI1i1dpzSwlLz7wV6z\ntE9pZmlG6UNLH1",
  "leUvrX0LaUPLH1AKbOUbp6Sal3FLy6SCI1i1dpzSwlLz7wV6z\ntE9pZmlG6UNLH1I6tpS8FcPzFJyvIEHo6WS0qeWPqVUWEre34LouaXPKU0sTSh9Z\nukzSt9Y+obSx5Y+pjS2lHwbgNOJpXuU2q9AVUHprqW7lJ5Zeub+LsAX0xi4FuaOrW\nCH0tTSlNItS8mbAhwlLD0l58lItXe1+dcmcl+L1I7WJvx+dUk5FacAdr707zq8n\n9KVILPiFd3z",
  "8mbAhwlLD0l58lItXe1+dcmcl+L1I7WJvx+dUk5FacAdr707zq8n\n9KVILPiFd3zxYfEiBlMKdfrS0soa/wtLCwd3e2i+9e7v3Vu6vt19ob3jfeN97P3hr\nexit>3q/efe+J1/cGXuj96f3l/e39s/zt8sbys+XtRr1+rb3mK6/ztz4D9kA2sc=</lat\n\u03c32\n\u2326 =\n2\nDh\nNumber of units at layer k\n<latexit sha1_base64=\"yZbCaq0TL4B\nrpAtMIPVn01TXmo=\">A",
  "\u2326 =\n2\nDh\nNumber of units at layer k\n<latexit sha1_base64=\"yZbCaq0TL4B\nrpAtMIPVn01TXmo=\">AWniclZhb9s2FIDV7tZ1t3bD8rIXoUWxYeiMJOguLw\nPapOkt6eI0cZI2TgxKpmQ2FKVIVOJU8Pt+zV63v7J/s0NJNqtzmIcF6Myd7xMvh6R\nEKcikKPTy8r/Xrn/w4Ucf3Lj05uf7Fl1/duv31fpGWecgHYSrT/DBgBZdC8YEW\nWvLDLOcsCSQ/CE7XDT845",
  "f3Lj05uf7Fl1/duv31fpGWecgHYSrT/DBgBZdC8YEW\nWvLDLOcsCSQ/CE7XDT8453khUrWnLzN+nLBYiUiETENodOvOsBxwkbD7YTH7GTV/\n90fRjkLq9VZ9XhUTb6fzUa37i73lus/nxZW2sJdr/3rj25/Ox6O07BMuNKhZEVxt\nLKc6eOK5VqEks9uDsuCZyw8ZTE/gqJiCS+Oq3owM/8eRMZ+lObwT2m/jr5/RcWSor\nhMAjATpicFZiboYke",
  "Zyw8ZTE/gqJiCS+Oq3owM/8eRMZ+lObwT2m/jr5/RcWSor\nhMAjATpicFZiboYkeljn47roTKSs1V2DQUldLXqW8y49FzkMtL6HAwlxAX/1wiA\nXGvJ3c6j4RZgmCVPjari2sTOrhgGPhar4WVncjbrOhu1w6F4lbH2fG9Ri9A8Ee84\nqaRWTCVXCDyeVRXvxT0MBAcgepyAVPEC6jT5CSJ/BVFYOxIw8CdQuci/9WMVK0j\nyEnHe0N0aCQST7tW",
  "xT0MBAcgepyAVPEC6jT5CSJ/BVFYOxIw8CdQuci/9WMVK0j\nyEnHe0N0aCQST7tWOvEgqlMOsouKL5/zeA6xmAboKPxzNwW7G1Gx+neZTnSdVYW\nK4hZypmNdNwJBDJs2IuoYqpYRLw471B7ZeMXaJi7N6q7mJoKsvbzr6JzmRY27Th1\nBFizCuGvVEWRJ2OljljDIclsewYAT30TcqlBYFWRh9vM06LadmQhem9M9kvX26hI\n+s8ZyogJwO4zv4",
  "ljljDIclsewYAT30TcqlBYFWRh9vM06LadmQhem9M9kvX26hI\n+s8ZyogJwO4zv4KpkHf19XRh+/PknNe+KfCpP4HJ6l7C8rgZ1rwRGFUbm1GzhUya\nbYglKcXdP0xqHyTHQHaAJ405W5UNF72v26BEvWhIf3Yah5KfnRT72f+fS4Wjbxv\nyHZBMqKsrMVZEJ/4+KxvBswesLInjyUokmDwL15KUS7u9o6liOF7aJ1HMHBaGYFP\noSbX8Rq+41dQ",
  "J/4+KxvBswesLInjyUokmDwL15KUS7u9o6liOF7aJ1HMHBaGYFP\noSbX8Rq+41dQR3Nk1QXyFg6oVfJhSa5CjqyiZgZPiFp6RjAYVokGEzxlCmRZlzcvN\nD6xkitW5ui7kwD6vuDVUaoXvf4HJxFZTh4XDOr7g8QBkNmnwGanGLEfJnJopnZ4M\nCw1bzLX76ylvik4r5mebXvQL5idMgz52WgTz0dMLOpIVBcS5x1SWI52oO6Fsv1/\nZ5Vmyc/kq",
  "ylvik4r5mebXvQL5idMgz52WgTz0dMLOpIVBcS5x1SWI52oO6Fsv1/\nZ5Vmyc/kqUdO1y3KUm9bS/dtsO9ogf8bMvR2y3iEYs6EtXV9pB6xHK0B3W587jlGo\nXDdZuS1DvPo9N2uAsTLf9ob8I1M8ekVI7NsS+VwyaERU1F7RTc9Ltik0Ii0nZteD\n/sbJrDs9dqwlhsV+IrmYCWBpziYfQhLDYbOGu2cawuVQt9wqk9kEmU0Ii09Zgkfd\nhLAYU",
  "s9dqwlhsV+IrmYCWBpziYfQhLDYbOGu2cawuVQt9wqk9kEmU0Ii09Zgkfd\nhLAYUzF2iqcsy5DYhEgeJziPE5rHDEuZS8IzkjlmhCwp14LKJ2lXMgEsTVFrU0dj0\nAOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw4IqGNUMVmgCWtske85s3TGIGOMVwzHIlOR\nPIymgC+9jpU2d+guipzkgujS0ktKLy9oPTA0gNKc0vJG0EQvbKUvJ0E0bml5T\nuW7",
  "IymgC+9jpU2d+guipzkgujS0ktKLy9oPTA0gNKc0vJG0EQvbKUvJ0E0bml5T\nuW7pPaWlpSenA0gGlkaURpU8sfUJpaGlI6bql65RqS8mJFJ4Ilu5ROrF0QumhpYe\nUvrb0NaXPLH1G6RtL31D6ztJ3lD6y9BGlzFJG6YalG5RyS8mngyBas3SN0sBS8u4H\ne83SPqWZpRmljy19TOnYUvJWDM8zS8nxBh6MlkpKn1v6nFJhKXl/C6KXlr6kNLE0",
  "e83SPqWZpRmljy19TOnYUvJWDM8zS8nxBh6MlkpKn1v6nFJhKXl/C6KXlr6kNLE0o\nfSFpS8ofWvpW0qfWvqU0thS8m0ATieW7lJqvwJVBaU7lu5Qembpmfu7AF9MY+BamN\nu2gm1KU0tTSjctJW8KcJSw9JScJyPV3tXmX5vIfS1SC+5gbcbnV5OcR2rBHay9O82\nvJvenSC34hHR9Y3/xIQVSul9/l13BX2FpYX+1t/JL78HOg7sP19ovtDe87",
  "y9O82\nvJvenSC34hHR9Y3/xIQVSul9/l13BX2FpYX+1t/JL78HOg7sP19ovtDe87w73g/e\nexit>iver9B75vW9gRd6f3p/eX97/yz5S0+WXi5tN+r1a+013idv6XD/wBubNwE</lat\n\u03c32\n\u2326 =\n2\nDh0\nNumber of units at layer k+1",
  "Exploding gradients\nVanishing gradients\n<latexit sha1_base64=\"/2yxZlV4HCN4WS4AG1sltwjTg0=\">AWsXiclZhbU9w2FICd9JamN9JOemL\np0xmOp2UWjS9iUzCYTcIAUCyQs2cpe2asgy8aWYlnf0l/TV/bX9B/0yPbu4rPEQ/dGWLlfJ91OZJt2UEmRaF7vX+vXf/gw48+/uTGpzc/+/yL79auPX1QZGWecj7YSrT/ChgBZdC8b4WvKjLOcsCSQ/DE7XDT853k\nhUrWvLzN+k",
  "L79auPX1QZGWecj7YSrT/ChgBZdC8b4WvKjLOcsCSQ/DE7XDT853k\nhUrWvLzN+krBYiUiETENouHBvUIg4YcPBdsJj9mbVv+8PopyF1eq0ejSsxtOpf38eWOn1pvd7y73V4cISHOqfTwsrbWHJa387w1vfjgajNCwTrnQoWVEcr/QyfVKxXItQ8unNQVnwjIWnLObHUFQs4cVJVY9v6t+GyMiP0h\nz+lPbr6PtnVCwpiskADNhelxgZoIudlzq6",
  "IWnLObHUFQs4cVJVY9v6t+GyMiP0h\nz+lPbr6PtnVCwpiskADNhelxgZoIudlzq6LeTSqis1FyFTUNRKX2d+iZ/kjkPNTyEgoszAX01Q/HDLKhIaU3B4pfhGmSMDWqBmsbu9NqEPBYqIqflXV6p9Ous1E7HIpXGWvP9ue1CM0T8Y6TSmrFVHKFwONpVfHleBkDwQ\nGIZU5AqngBdZr8BJG/gigsJwkYeJBOoHOR/3JKqlax5CTjvaFDIJ90rHViw",
  "kDwQ\nGIZU5AqngBdZr8BJG/gigsJwkYeJBOoHOR/3JKqlax5CTjvaFDIJ90rHViwVQmHWUPFN+/7RvAdQ6zAF2FA0dzsJcxNZ2dp/lE50lVmBhuIWcq5nUTMOSQSTOirqFKeHUsGP9jq2XTJ2iUuzuqu5iSBrP+86Oqd5U\naOuU0eQBYsw7lp1BFkSLv4RSxhkuS0PYcCJbyJuVSisCrIwd/I06LadmQhem5Mrpeut1GR9J8zlBETgKvPHAVTIe/",
  "hkuS0PYcCJbyJuVSisCrIwd/I06LadmQhem5Mrpeut1GR9J8zlBETgKvPHAVTIe/q6+nc9mfJOa9U+ATfwyT1T2F5XEzrFkjMKo2NqVmnStk0mxBKE8vuqbpjUPlmegO0ATwRVfmQkXv\naXfqEixZEx7cgaHmpeTHPy3f45OTqmcuG/MPySZUVJSZqyIT/h8VjeBxg9cXRPDkpRJNHgTqyUsl3N/R1LEcL2wTqecOCkIxKfQluvxFrLrn1BHc2TR",
  "VjeBxg9cXRPDkpRJNHgTqyUsl3N/R1LEcL2wTqecOCkIxKfQluvxFrLrn1BHc2TRBfYWAqReOTCg0yVHUlU3AyHCEB6djAYVokGEzxlCmRZlzcvND6xk\nitW5ui7kwD6vuDVUaoXvf4HJ+FpTh4XDOrzg9QBkNmnwGalGLEfJnJgpnbwZFBouMdfVX095U3RaMT/bNuDfsHslGHIz4abeD5iYlFHorpgp+KsSxL0R7UNV+u7/es2nzI1nascN1m5",
  "aMT/bNuDfsHslGHIz4abeD5iYlFHorpgp+KsSxL0R7UNV+u7/es2nzI1nascN1m5LU2/bSbTvcK3rAz7Ycvd0iHr\nGoI1FdbQ+pRyxHe1CXO49brlE4XLcpSb2zPDpthzs30fKP9sdcM7NSuXIbPtSOWhCWNRU1E4xNZvfrtiEsJiUXQv+j5U9s5/uWk0IizuF6GomgKURl3gITQiLzSXcNdsYVrc6pZbZTIbI7MJYfEJS/ComxAWYyrGTvGUZ\nR",
  "uF6GomgKURl3gITQiLzSXcNdsYVrc6pZbZTIbI7MJYfEJS/ComxAWYyrGTvGUZ\nRkSmxDJ4xjncUzmGEpc0l4RjLHjJAl5VpQ+TjtSiaApQlqbeJoDHogU4UabINYLujK5wrT6FVrOgq7rsa7l/RsGaoQhPA0ja5xvzmpZOYAU4xbLNcSc4EsjKawB3s7FBntvsLors5ILo0tJLSi8svaD0NJDSnNLyRtBE\nL20lLydBNG5peUHlh6QGlpaUlp39",
  "tvsLors5ILo0tJLSi8svaD0NJDSnNLyRtBE\nL20lLydBNG5peUHlh6QGlpaUlp39I+pZGlEaWPLX1MaWhpSOm6peuUakvJjhSeCJbuUzq2dEzpkaVHlL6y9BWlTy19SulrS19T+s7Sd5Q+tPQhpcxSRumGpRuUckvJp4MgWrN0jdLAUvLuB9eapTuUZpZmlD6y9BGlI0vJ\nw9u40O5vcnyI152PS9Y2D+YcUSCnc6YcLSyv4KywtHKwur/yfHf37",
  "6y9BGlI0vJ\nw9u40O5vcnyI152PS9Y2D+YcUSCnc6YcLSyv4KywtHKwur/yfHf37tKDtfYL7Q3vO+97wdvxfvVe+A9Xa8vhd6f3p/eX97/yz+vPhq8Y/FoFGvX2vP+cbr/BZP/wOEc+Kg</latexit>WzE8zywl2xt4MFoqKX1m6TNKhaXk/S2IXlj6gtLE0oTS5Y+p/StpW8pfWLpE0pjS8m3AdidWLpHqf0KVBWU7lq6S+mZpWfu7wJ8Po2Ba2",
  "oTS5Y+p/StpW8pfWLpE0pjS8m3AdidWLpHqf0KVBWU7lq6S+mZpWfu7wJ8Po2Ba2Fu2wq2KU0tTSndtJS8KcBWwtJTsp+MVHtXm31tIve1SM25g7UZn51Nch6pOXe\n\u03c32\n\u2326 =\n2\nDh\n=\n2\n100 = 0.02",
  "Default Initialization in PyTorch\nhttps://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_uniform_",
  "Feedback?",
  "How to Read a Research Paper\n1\nAll images are property of their respective owners",
  "How to Read Research Papers\nReading Research Papers, Accelerating your Efficiency\n1. Compile list of papers (research papers, Medium posts, Blogs, etc.). You can use \ntools like Zotero or Mendeley to organize your bibliography.\n2. Skip around the list.\na. Glance through the list, maybe read only 10% of each paper.\nb. Weed out duds.\nc. Pick one of the papers to dive into.\n3. 15-20 papers gives you basic understanding maybe enough to implement. 50-100 \npapers give you very solid understanding, can start research.\n4. Shoot for reading 1-2 papers deeply per week. \u21d2 Reading Group!\n2\nAndrew Ng -- Reading Research Papers",
  "How to Read Research Papers - II\nHow to Read a Paper\nDo it in multiple passes.\n3\n1.\nTitle, abstract and figures. Especially the key figure(s).\n2.\nIntro, Conclusions, Figures then skim the rest\n3.\nMaybe skim or skip related work. Sometimes authors try to cite people that may be \nreviewers.\n4.\nRead the paper but skim/skip the math.\n5.\nRead whole thing but skip parts that don't make sense. Often even authors don't know \nwhat will make the biggest impact until later.\n6.\nGo Deeper: Re-derive the math from memory.\n7.\nGo Deeper: Re-implement the network from scratch.",
  "How to Read Research Papers - III\nSome questions to ask yourself.\n\u25cf\nWhat did authors try to accomplish?\n\u25cf\nWhat were the key elements of the approach?\n\u25cf\nWhat can you use yourself?\n\u25cf\nWhat other references do you want to follow?\n4",
  "Optional Exercise\nLook up \u201cDeep Residual Learning for Image Recognition\u201d on scholar.google.com.\nTake 7 minutes to scan and read the paper in multiple passes as previously \ndiscussed. Highlight key points and graphs.\nTake a few minutes and jot down the key takeaways.\n5",
  "Measuring Performance\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "Where we are\n=== Foundational Concepts ===\n\u00fc 02 -- Supervised learning refresher\n\u00fc 03 -- Shallow networks and their representation capacity\n\u00fc 04 -- Deep networks and depth efficiency\n\u00fc 05 -- Loss function in terms of maximizing likelihoods\n\u00fc 06 \u2013 Fitting models with different optimizers\n\u00fc 07a \u2013 Gradients on deep models and backpropagation\n\u00fc 07b \u2013 Initialization to avoid vanishing and exploding weights & \ngradients\n\u2022 08 \u2013 Measuring performance, test sets, overfitting and double \ndescent\n\u2022 09 \u2013 Regularization to improve fitting on test sets and unseen data\n=== Network Architectures and Applications ===\n\u2022 10 \u2013 Convolutional Networks\n\u2022 11 \u2013 Residual Networks\n\u2022 12 \u2013 Transformers\n\u2022 Large Language and other Foundational Models\n\u2022 Generative Models\n\u2022 Graph Neural Networks\n\u2022 \u2026\n2",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n3",
  "S. Greydanus, \u201cScaling down Deep Learning.\u201d arXiv, Dec. 04, 2020. doi: 10.48550/arXiv.2011.14439.\nMNIST1D\nhttps://github.com/greydanus/mnist1d \n\u201cA large number of deep learning innovations including dropout,  Adam, convolutional \nnetworks, generative adversarial networks, and variational autoencoders began life as \nMNIST experiments. Once these innovations proved themselves on small-scale \nexperiments, scientists found ways to scale them to larger and more impactful \napplications.\u201d\n4",
  "MNIST Dataset\n\u2022 28x28x1 grayscale images\n\u2022 60K Training, 10K Test\n\u2022 \u201cIs to Deep Learning what \nfruit flies are to genetics \nresearch\u201d\n28\n28\n2-3\ndays\nBut poorly differentiates model performance:\nModel Type\nAccuracy\nLogistic Regression\n94%\nMLP\n99+%\nCNN\n99+%\n5",
  "MNIST 1D Dataset\nFixed, 1-D,  length-12 \ntemplates for each of 10 digit \nclasses\nGenerate dataset by \nprogrammatically applying 6 \nparametric transformations.\nE.g. pad, shear, translate, correlated noise, i.i.d. noise, interpolation.\nSee https://github.com/greydanus/mnist1d/blob/master/building_mnist1d.ipynb \n6",
  "MNIST 1D\nDifferentiates performance of different \nmodel types much more than MNIST\n7",
  "Visualizing MNIST and MNIST-1D with tSNE\nhttps://twitter.com/hippopedoid \n8",
  "MNIST1D Train and Test Set\n\u2022 1D, Length 40 samples\n\u2022 4,000 training samples\n\u2022 1,000 test samples (80/20 split)\nDataset Samples\n9",
  "# choose cross entropy loss function\nloss_function = torch.nn.CrossEntropyLoss()\n# construct SGD optimizer and initialize learning rate and momentum\noptimizer = torch.optim.SGD(model.parameters(), lr = 0.1)\n# object that decreases learning rate by half every 10 epochs\nscheduler = StepLR(optimizer, step_size=10, gamma=0.5)\n# load the data into a class that creates the batches\ndata_loader = DataLoader(TensorDataset(x_train,y_train), batch_size=100, shuffle=True)\nNetwork\n\u2022 40 inputs\n\u2022 10 outputs \n\u2022 Two hidden layers\n\u2022 100 hidden units each\n\u2022 SGD with batch size 100, learning rate 0.1\n\u2022 6000 steps (?? Epochs)\nmodel = torch.nn.Sequential(\n    torch.nn.Linear(40, 100),\n    torch.nn.ReLU(),\n    torch.nn.Linear(100, 100),\n    torch.nn.ReLU(),\n    torch.nn.Linear(100, 10))\n============================================== \nLayer (type:depth-idx) Output Shape Param # \n==============================================\nSequential \n[1,",
  "nn.ReLU(),\n    torch.nn.Linear(100, 100),\n    torch.nn.ReLU(),\n    torch.nn.Linear(100, 10))\n============================================== \nLayer (type:depth-idx) Output Shape Param # \n==============================================\nSequential \n[1, 10] \n\u2013 \n\u251c\u2500Linear: 1-1 \n[1, 100] \n4,100 \n\u251c\u2500ReLU: 1-2 \n[1, 100] \n\u2013 \n\u251c\u2500Linear: 1-3 \n[1, 100] \n10,100 \n\u251c\u2500ReLU: 1-4 \n[1, 100] \n\u2013 \n\u251c\u2500Linear: 1-5 \n[1, 10] \n1,010 \n=============================================== \nTotal params: 15,210\nTrainable params: 15,210 \nNon-trainable params: 0 \nTotal mult-adds (Units.MEGABYTES): 0.02 \n================================================ \nInput size (MB): 0.00 \nForward/backward pass size (MB): 0.00 \nParams size (MB): 0.",
  "210 \nNon-trainable params: 0 \nTotal mult-adds (Units.MEGABYTES): 0.02 \n================================================ \nInput size (MB): 0.00 \nForward/backward pass size (MB): 0.00 \nParams size (MB): 0.06 \nEstimated Total Size (MB): 0.06 \n================================================\n# inference \u2013 just choose the max\npred_train = model(x_train)\npred_test = model(x_test)\n_, predicted_train_class = torch.max(pred_train.data, 1)\n_, predicted_test_class = torch.max(pred_test.data, 1)\n\u2026\n10",
  "Results\n0\n 0\n100\n6000\n0\n0.0\n3.0\n6000\n11",
  "Need to use separate test data\nThe model has not generalized well to the new data\n12",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n13",
  "Regression example with Toy Model\ny = \ud835\udc52!\"#(%&')\ny = \ud835\udc66 + \ud835\udca9(0, \ud835\udf0e))\n\ud835\udc65 = \ud835\udc65 + \ud835\udcb0(\u00b11/num_data)\nAdd small uniform noise to x:\nAdd small Gaussian noise to y:\n\ud835\udf0e!\n\u201dTrue\u201d function:\nDataset\n14",
  "Toy model\n\u2022 D hidden units\n\u2022 First layer fixed so \u201cjoints\u201d \ndivide interval evenly, e.g. \n0, 1/D, 2/D, \u2026, (D-1)/D\n\u2022 Second layer trained\n\u2022 But\u2026 now linear in h\n\u2022 so convex cost function\n\u2022 can find best soln in closed-\nform\n\u2022 A piecewise linear model \nwith D regions.\n15",
  "Three possible sources of error: \nnoise, bias and variance\n16",
  "Noise, bias, and variance\n\u2022 Genuine stochastic nature of the \nunderlying model\n\u2022 Noise in measurements, e.g. from sensors\n\u2022 Some variables not observed\n\u2022 Data mislabeled\nhttps://images.app.goo.gl/2PuBhaFpfdL9Pyjb8 \nhttps://images.app.goo.gl/CMDaXSCdX4pqN8Yx7 \n17",
  "Noise, bias, and variance\nBias occurs because the \nmodel lacks precision or \ncapacity to accurately match \nthe underlying function.\nE.g. optimal fit with 3 hidden \nunits and 3 line segments\n18",
  "Noise, bias, and variance\nNo way to distinguish change in the \ntrue underlying function from noise in \nthe data.\nVariability every time we capture \ntraining data and also from stochastic \nlearning algorithms.\n19",
  "Noise, bias, and variance\n20",
  "Least squares regression only\n<latexit sha1_base64=\"VA+P\nYFIy63/ZAhrl7ehFyrXxUE4=\">AWqniclZjZcts2FECZdEvTz\nWmnfukLp50o6jsTPp8tKZxI6z2anlRV4iKR6QAinEIEhzsaVw\n9Bf9mr62P9G/6QVJCeG98EM14wi5xDLBUBC9BIpsnxt7d8bNz\n/48KOP7n16e3Pv/iy6+W7nx9lMVF6vOeH8s4PfFYxqVQvJeLX\nPKTJOUs8iQ/9s43NT+5GkmYnWYTxM",
  "iy6+W7nx9lMVF6vOeH8s4PfFYxqVQvJeLX\nPKTJOUs8iQ/9s43NT+5GkmYnWYTxM+jFioRCB8lkPobKmz058M\n3R9+dweCOW9QeTFkzKY9SerAy9IxmJ4fwqChumPbx64Z0sra52\n16uPSwnpTWHGaT/fszrejwSj2i4ir3Jcsy/ra0k+LFmaC1/y2e\n1BkfGE+ecs5H0oKhbxbFhWA5u5dyEycoM4hT+Vu1X0/StKFmXZN\nPLAjFg+zjDTQRvrF",
  "GE+ecs5H0oKhbxbFhWA5u5dyEycoM4hT+Vu1X0/StKFmXZN\nPLAjFg+zjDTQRvrF3nw27AUKilyrvy6oaCQbh67OkvuSKTcz+U\nCsxPBfTV9csZX4Oubw9UPzKj6OIqVE52Njam5UDj4dClfyiqP\nI6m7WdrcrhULzO2HhxuKhF5DwS7zipFJ0JdcIPJyVJe+EHQwEB\nyA6nIBY8Qzq1PnxAncdUVhHEnBZLwpYD+7+jFStch5CTlra6JB\nIZF80",
  "+EHQwEB\nyA6nIBY8Qzq1PnxAncdUVhHEnBZLwpYD+7+jFStch5CTlra6JB\nIZF80rI2iQVTGbWUA1Bc96rAc9TmAXoKnxNAcHCVOz+XU5n+R\npVGY6hltImQp51QM2WdSj6htqEJKuNRvWX9ga5+p8yZxcVJ1Nd\nURZB2mbSdPaV7UqO1UEWTBIgzbVhVBloRdP2IRgyw35TMYcOTqi\nF0VCquCLMxuGnvthMdwWtzksB+aXtbJUn/JUMZ0QHYfp",
  "RdP2IRgyw35TMYcOTqi\nF0VCquCLMxuGnvthMdwWtzksB+aXtbJUn/JUMZ0QHYfpbMOXz\ntr4ZL2x3npzLytcFPnHMFntS1ga1sOaNwKjamIzala5QibNFo\nTS+Kpt6t5YVJ6I9gB1AG+6IhUqeE9brUqwZHV4sApDTQvJ+/c7P\n/PJsFzT20b/Q7IJFWVFYqtIh/9HRSN4zuD1BRE8ebFEkweBavJi\nCfd3NHUsxQtbR6q5g4JQTIp8ira/CFX7",
  "qtIh/9HRSN4zuD1BRE8ebFEkweBavJi\nCfd3NHUsxQtbR6q5g4JQTIp8ira/CFX7miqCOxtHqK8Q0PXCNxM\nKTXIQtGUd0DJ8wxPTsoB8NEi/HqMv46xIObn5ofUMkUrXt8VU6I\ndV+4YqtdC+b3C5uArK8HC45Ndc7qGMenU+vbhQI5aiZE70lE7eD\nLIctpht91dTXhetVsgvtpv2oF8wO4Xv84uzbTwfIbGoI1FdcESx\n1iWJZWkP6los1/d7",
  "pht91dTXhetVsgvtpv2oF8wO4Xv84uzbTwfIbGoI1FdcESx\n1iWJZWkP6los1/d7Vm6/+Yks7dDi2k1J6m16abct7jU94Bc7lt\n7uEI9Y1JGorqaH1COWpT2oy57HdsoLK7dlKTeR6tsVdmGj5B\n4djnjN9TIrlSB/7YjmoQ1jMqZhbxTjiIRLrEBajom3B/7FyIODh\n0bqEBa7mWhrOoClEZd4CHUIi/UWbptNDKs7FnXHrjKZjJFZh7D\n4jEV4",
  "7FyIODh\n0bqEBa7mWhrOoClEZd4CHUIi/UWbptNDKs7FnXHrjKZjJFZh7D\n4jEV41HUIiyEVQ6t4zpIEiXWI5HGM8zimeUywlNgkPCOJZUbIkr\nItqHQctyUdwNIEtTaxNAY9kLFCDTZBLGd05WXWlafQKlZ0FfdsD\nfeuaThnqEIdwNIu2WPuYNe6yTycYjhm2ZKcCGQlNIFd7HSpMz/9\neUFJTnJeMDV0SumVoVeUHht6TGlqKPlF4AX7hpJfJ",
  "hm2ZKcCGQlNIFd7HSpMz/9\neUFJTnJeMDV0SumVoVeUHht6TGlqKPlF4AX7hpJfJ15waeglpU\neGHlFaGFpQ2jO0R2lgaEDpU0OfUuob6lO6aegmpbmh5EQKTwRD\nykdGzqm9MTQE0pPDT2l9Lmhzyl9behrSt8Z+o7Sx4Y+pQZyijd\nMnSLUm4oeXgBRuGblDqGUp+8FeM7RLaWJoQukTQ59QOjKU/Cq\nG5mh5HgD0ZDJaUvDH1BqTCU/H7zgle",
  "lDqGUp+8FeM7RLaWJoQukTQ59QOjKU/Cq\nG5mh5HgD0ZDJaUvDH1BqTCU/H7zgleGvqI0MjSi9KWhLyl9a+\nhbSp8Z+ozS0FDybgBOJ4YeUGreApUZpXuG7lF6YeiF/b0AX0yjZ\n1uYu6aCXUpjQ2NKtw0lvxTgKGHoOTlPBq5q83fNpH7WqAW3MK\najM+vJjkP1IJbWHN3ml9N7k+BWvAx6frW0eJFCqQU7vRnSyvr+C\n0sLRw96Kz/0nm493Dl",
  "jkP1IJbWHN3ml9N7k+BWvAx6frW0eJFCqQU7vRnSyvr+C\n0sLRw96Kz/0nm493Dl0UbzhvaW853zvXPWXd+dR45z52u03N85\nt>0/nL+dv5/l1eX95dPlfq3evNFc843T+iyP/gOGd+A7</latexi\nL[x] =\n\ufffd\nf[x, \u03c6] \u2212 y[x]\n\ufffd2\n<latexit \nsha1_base64=\"bfJ\nQB7QvQIOHNFTBo3/\nX9sasPJ8=\">AXl\n3iclZhb9s2FIDt7\ntZ1t3bDEGx7URcU6",
  "B7QvQIOHNFTBo3/\nX9sasPJ8=\">AXl\n3iclZhb9s2FIDt7\ntZ1t3bDEGx7URcU6\nNbWSIru8jKsTZo2b\ndLFaeIkreUGlEzJb\nChKoaTEqaCH/Zq9b\nj9n/2aHkmxaPMzD\nDRmz/eJpA4vouUln\nKXZysq/3Svf/Bh\nx9d/fjaJ59+9vkX1\n298eZDGufTpwI95L\nI8klLOB1kLOP0K\nJGURB6nh97JuKHZ\n1SmLBb72UVCRxEJB\nQuYTzIHd/of",
  "I95L\nI8klLOB1kLOP0K\nJGURB6nh97JuKHZ\n1SmLBb72UVCRxEJB\nQuYTzIHd/ofutuT\nBPqZ3R8XLgRySY+4\ncXjsnTXWMiHC/CiH\nG4Pp6ORAnLk/OY4b\ni7GVHqS+LTQ4kIlT\nR0e/L3tRl48LYJyO\nL3rekEyYcMFcTS65\nwSq/Rz4SF0gf3hzv\n26pVHF17RmRjAifl\nuWdVstV9QtX37zn3\noTyQkXzKjxG0rJ07\nrS7nrIwIguWiFkKr",
  "HF17RmRjAifl\nuWdVstV9QtX37zn3\noTyQkXzKjxG0rJ07\nrS7nrIwIguWiFkKr\nfSOry+v9Faqj4MLq\n01hudN8+sc3vh674\n9jPIyoyn5M0Ha6uJ\nNmoIDJjPqflNTdPa\nUL8ExLSIRQFiWg6K\nqpRLJ1bEBk7QSzhn\n8icKrp4RUGiNL2IP\nDBV5lKTqaCNDfMs+\nHVUMJHkGRV+3VCQc\nyeLHTUlnDGTMHT8A\ngrElwz6vgTArnJY\nOJcwU9",
  "CNDfMs+\nHVUMJHkGRV+3VCQc\nyeLHTUlnDGTMHT8A\ngrElwz6vgTArnJY\nOJcwU9+MoImJcu\nGsbuyVknIZMFPQ0r\nyZRWbadjcqhULzMW\nHu2P6+FZTRi7yiqp\nFJUJZcINCyLgvbCn\ngkYBcB6FIFY0LSs5\n7gXOKsGhUXDATdTA\nKao87JEVYuMhpCTl\nvYaVBIOJ2rHVkw\nVBGLWUPFMe5ShAM\nwmjAF2FL2qMwV5CR\nDm7LqPTEZFqmJmC",
  "vYaVBIOJ2rHVkw\nVBGLWUPFMe5ShAM\nwmjAF2FL2qMwV5CR\nDm7LqPTEZFqmJmC\n5KIkFZN1AtN3VHbE\nDncKnfsv4wrZdEn\nDSJi5Oq1JFDGtft\np1M4ryIcdupIoYFk\nzBsW1XEsDhscWMSE\nchyU4bVKyNHRewqE\n6bK0MTsy9hrt52oi\nDk3q62u7W0UKP2wX\n7UdFYDV58w2sra+H\ns9tZ5acs8pXBTp1J\njBY7UuIDOvbmjUCd\n9XESmxW",
  "KP2wX\n7UdFYDV58w2sra+H\ns9tZ5acs8pXBTp1J\njBY7UuIDOvbmjUCd\n9XESmxWuTJMnC0Iy\nfi8bareWFSasPYNq\noC56HLJRLCg3a1KM\nGV2L0LtypzTof3e\nj/R6ahYUctG/UHZh\nIrSPLFVpML/o6IxP\nFTN+QURc/BibgweB\nKrBizns78bQEWlOb\nBWpxg4KTBDOsgtj+\nbNQtK+pImZn48joK\nwRUvfBNmDAGOQjas\ngoGb7heGCZQL5",
  "BWpxg4KTBDOsgtj+\nbNQtK+pImZn48joK\nwRUvfBNmDAGOQjas\ngoGb7heGCZQL5xk\n359jz6P01xStPkZ8\nxkila62RcnUw6q9o\nXIltPcNyudXQRkeD\nmf0ks9I6NenU8vh\nic3kUYyp2pIp2/cN\nIMlZlv91ZDXRasV0\ntOtpj3oF4xO7v09\nHjLHI8QWdjhRl1wH\nrPWxZFlaQ/qmk/Xx\nZ4VW29+RFM7tLh2k\n6N6m17abYt7SQ/o6\nbalt",
  "djhRl1wH\nrPWxZFlaQ/qmk/Xx\nZ4VW29+RFM7tLh2k\n6N6m17abYt7SQ/o6\nbalt9vIQxZ2uFX0\n0PsIcvSHtRlz+O27\nS4srt3kqN5ZHq2xZ\n2bxvQP9ic0I+qYFP\nOxOvbF3K1DphMb\nOKcURDQ6xDpgH6Z\nYF/zeVPXVmblt1yB\nT7KWtrKmBKY8rNW6\nhDplgv4bZxEx126\nJu21XCk4lh1iFTfE\noi867rkCmGWAyt4g\nlJEkOsQyiPEzOP",
  "hDplgv4bZxEx126\nJu21XCk4lh1iFTfE\noi867rkCmGWAyt4g\nlJEkOsQyiPEzOPE5\nzHxJQSm2SOSGIZET\nSlbBNKTuK2pAKmND\nVam1oagx7wWBgNk\nFTvHMS60zTxizWO\nBZPLA1PLik4YwYFa\nqAKe2gNea4O9ZF5p\nkphmOWLckJM6wEJ7\nBvOn3szE5/XlCgk5\nwXGh6gem5pueYHm\np6iKnUFP0i8IKXmq\nJfJ15wpukZpgeaHm\nCa5pj",
  "5/XlCgk5\nwXGh6gem5pueYHm\np6iKnUFP0i8IKXmq\nJfJ15wpukZpgeaHm\nCa5pjOtB0gGmgaY\nDpE02fYOpr6mO6ru\nk6pm6EQKTwRN9z\nGdaDrB9EjTI0xfaf\noK01NzF9relrTN\n9p+g7TR5o+wpRoSj\nDd0HQDU6openXgBW\nuarmHqaYp+8Fa07\nSPaJpguljTR9jOt\nYU/SqG5m6HgD0\nZNOabPNH2GKdMU/X\n7zgheavsA0jTC9L\nmzF9",
  "JpguljTR9jOt\nYU/SqG5m6HgD0\nZNOabPNH2GKdMU/X\n7zgheavsA0jTC9L\nmzF9q+lbTJ9q+h\nTUFP0bgBOJ5ruYa\nrfAhUprua7mJ6qu\nmp/b0AnQ+jZ5uYO7\nqCHUxjTWNMtzRFvx\nTgKHpCTpPBqLZ1W\nZvm9C+Fog5t7Am47\nOrUc4DMecW1uxOs6\nvR/hSIOZ+grm8czF\n+kQEphpz+vrxqvo\nXFhYP7vdWfew92Hy\nw/XGve0F7tfNf5v",
  "R/hSIOZ+grm8czF\n+kQEphpz+vrxqvo\nXFhYP7vdWfew92Hy\nw/XGve0F7tfNf5vn\nO7s9r5pfOws9npdw\nYdv/tn96/u391/lr\n5Z+n3pydJmrV7pNt\n4=</latexit>d81Wl9lnb/A2qBPL\nED\nh\nEy[L[x]]\ni\n= ED\nh\ufffd\nf[x, \u03c6[D]] \u2212 f\u00b5[x]\n\ufffd2i\n|\n{z\n}\nvariance\n+\n\ufffd\nf\u00b5[x]\u2212\u00b5[x]\n\ufffd2\n|\n{z\n}\nbias\n+ \u03c32\n|{z}\nnoise\n.",
  "i\n= ED\nh\ufffd\nf[x, \u03c6[D]] \u2212 f\u00b5[x]\n\ufffd2i\n|\n{z\n}\nvariance\n+\n\ufffd\nf\u00b5[x]\u2212\u00b5[x]\n\ufffd2\n|\n{z\n}\nbias\n+ \u03c32\n|{z}\nnoise\n.\nExpectation over noise \nin training data\nExpectation over\nnoise in test data\nBest possible model if \nwe had infinite data\nActual model\nTrue function\nFor derivation see Section 8.2.2 in UDL.\nMore complex interactions between noise, bias and variance in more \ncomplex models.\n\u2022 We can show that:\n\u2022 And then:\n21",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n22",
  "Variance\nWhen measuring (capturing) 6 \ndifferent data samples with a fixed \nmodel (e.g. 3 hidden units), we get \ndifferent optimal fits every time.\n23",
  "Variance\nCan reduce \nvariance by \nadding more \nsamples\n24",
  "Variance\nCan reduce \nvariance by \nadding more \nsamples\n25",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n26",
  "Reducing bias\n(example with the true function)\nWe can reduce bias by adding more model capacity.\nIn this case, adding more hidden units.\nBias\n27",
  "Reducing bias \u00e8 Increases variance!!\nBias\nVariance\n28",
  "Why does variance increase? Overfitting\nDescribes the training data better, but not the true underlying function (black curve)\nMany ways to fit a sample of 15 data points\n3 Regions\n10 Regions\n29",
  "Bias and variance trade-off for the simple \nlinear model\nNumber of hidden units\n<latexit sha1_base64=\"bfJQB7QvQIOHNFTBo3/X9sasPJ8=\">AXl3iclZhb9s2FIDt7tZ1t3bDEGx7URcU6NbWSIru8jKsTZo2bdLFaeIkreUGlEzJbChKoaTEqaCH/Z\nq9bj9n/2aHkmxaPMzDRmz/eJpA4vouUlnKXZysq/3Svf/Bhx9d/fjaJ59+9vkX1298eZDGufTpwI95LI8klLOB1kLOP0KJGURB6nh97JuKHZ1SmL",
  "9d/fjaJ59+9vkX1298eZDGufTpwI95LI8klLOB1kLOP0KJGURB6nh97JuKHZ1SmLBb72UVCRxEJBQuYTzIHd/ofutuTBPqZ3R8XLgRySY+4cXjsnTXWMiHC/CiHG4Pp6ORAnLk/OY4bi7GVHqS+LTQ4kIlTR0e/L3tRl48LYJyOL3rekEyYcMFcTS65wSq/Rz4SF0gf3hzv26pVHF17\nRmRjAifluWdVstV9QtX37zn3oTyQkXzKjxG0rJ07rS7n",
  "z4SF0gf3hzv26pVHF17\nRmRjAifluWdVstV9QtX37zn3oTyQkXzKjxG0rJ07rS7nrIwIguWiFkKrfSOry+v9Faqj4MLq01hudN8+sc3vh6749jPIyoyn5M0Ha6uJNmoIDJjPqflNTdPaUL8ExLSIRQFiWg6KqpRLJ1bEBk7QSzhn8icKrp4RUGiNL2IPDBV5lKTqaCNDfMs+HVUMJHkGRV+3VCQcyeLHTUlnDGTMHT8AgrElwz6vgTArnJYOJcwU",
  "lKTqaCNDfMs+HVUMJHkGRV+3VCQcyeLHTUlnDGTMHT8AgrElwz6vgTArnJYOJcwU9+MoImJcuGsbuyVknIZMFPQ0ryZR\nWbadjcqhULzMWHu2P6+FZTRi7yiqpFJUJZcINCyLgvbCngkYBcB6FIFY0LSs57gXOKsGhUXDATdTAKao87JEVYuMhpCTlvYaVBIOJ2rHVkwVBGLWUPFMe5ShAMwmjAF2FL2qMwV5CRDm7LqPTEZFqmJmC5KIkFZN1At",
  "OJ2rHVkwVBGLWUPFMe5ShAMwmjAF2FL2qMwV5CRDm7LqPTEZFqmJmC5KIkFZN1AtN3VHbEDncKnfsv4wrZdEnDSJi5Oq1JFDGtftp1M4ryIcdupIoYFkzBsW1XEsDhscWMSEchyU4bVKyNHRewqE6bK0MTsy9hrt52\noiDk3q62u7W0UKP2wX7UdFYDV58w2sra+Hs9tZ5acs8pXBTp1JjBY7UuIDOvbmjUCd9XESmxWuTJMnC0Iyfi8bareWFSa",
  "sra+Hs9tZ5acs8pXBTp1JjBY7UuIDOvbmjUCd9XESmxWuTJMnC0Iyfi8bareWFSasPYNqoC56HLJRLCg3a1KMGV2L0LtypzTof3ej/R6ahYUctG/UHZhIrSPLFVpML/o6IxPFTN+QURc/BibgweBKrBizns78bQEWlObBWpxg4KTBDOsgtj+bNQtK+pImZn48joKwRUvfBNmDAGOQjasgoGb7heGCZQL5xk359jz6P01\nxStPkZ8xkila62RcnUw",
  "8joKwRUvfBNmDAGOQjasgoGb7heGCZQL5xk359jz6P01\nxStPkZ8xkila62RcnUw6q9oXIltPcNyudXQRkeDmf0ks9I6NenU8vhic3kUYyp2pIp2/cNIMlZlv91ZDXRasV0tOtpj3oF4xO7v09HjLHI8QWdjhRl1wHrPWxZFlaQ/qmk/XxZ4VW29+RFM7tLh2k6N6m17abYt7SQ/o6balt9vIQxZ2uFX0PsIcvSHtRlz+O27S4srt3kqN5ZHq2xZ",
  "2k6N6m17abYt7SQ/o6balt9vIQxZ2uFX0PsIcvSHtRlz+O27S4srt3kqN5ZHq2xZ2bxvQP9ic0I+qYFPOxOvbF3K1DphMbOKcURDQ6xDpgH6ZYF/\nzeVPXVmblt1yBT7KWtrKmBKY8rNW6hDplgv4bZxEx126Ju21XCk4lh1iFTfEoi867rkCmGWAyt4glJEkOsQyiPEzOPE5zHxJQSm2SOSGIZETSlbBNKTuK2pAKmNDVam1oagx7wWBgNkFTvH",
  "kOsQyiPEzOPE5zHxJQSm2SOSGIZETSlbBNKTuK2pAKmNDVam1oagx7wWBgNkFTvHMS60zTxizWOBZPLA1PLik4YwYFaqAKe2gNea4O9ZF5pkphmOWLckJM6wEJ7BvOn3szE5/XlCgk5wXGh6gem5pueYHmp6iKnUFP0i8IKXmqJfJ15wpukZpgea\nMtzRFvxTgKHpCTpPBqLZ1WZvm9C+Fog5t7Am47OrUc4DMecW1uxOs6vR/hSIOZ+grm8czF+",
  "TgKHpCTpPBqLZ1WZvm9C+Fog5t7Am47OrUc4DMecW1uxOs6vR/hSIOZ+grm8czF+kQEphpz+vrxqvoXFhYP7vdWfew92Hyw/XGve0F7tfNf5vnO7s9r5pfOws9npdwYdv/tn96/u391/lr5Z+n3pydJmrV7pNtd81Wl9lnb/A2qBPL4=</latexit>HmCa5pjOtB0gGmgaYDpE02fYOpr6mO6ruk6pm6EQKTwRN9zGdaDrB9EjTI0xfafoK01NzF",
  "jOtB0gGmgaYDpE02fYOpr6mO6ruk6pm6EQKTwRN9zGdaDrB9EjTI0xfafoK01NzF9relrTN9p+g7TR5o+wpRoSjDd0HQDU6openXgBWuarmHqaYp+8Fa07SPaJpguljTR9jOtYU/SqG5m6HgD0ZNOabPNH2GKdMU/X7zgheavsA0jTC9LmzF9q+lbTJ9q+hTUFP0bgBOJ5ruYarfAhUprua7mJ6qump/b0AnQ+jZ5uYO7qCHUxjTWN\nED\nh\nEy[L",
  "UFP0bgBOJ5ruYarfAhUprua7mJ6qump/b0AnQ+jZ5uYO7qCHUxjTWN\nED\nh\nEy[L[x]]\ni\n= ED\nh\ufffd\nf[x, \u03c6[D]] \u2212 f\u00b5[x]\n\ufffd2i\n|\n{z\n}\nvariance\n+\n\ufffd\nf\u00b5[x]\u2212\u00b5[x]\n\ufffd2\n|\n{z\n}\nbias\n+ \u03c32\n|{z}\nnoise\n.\n30",
  "But does picking model capacity to \nminimize bias & variance hold for \nmore complex data and models?\n31",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n32",
  "Train and Test \nError versus # of \nHidden Layers\nTraining parameters = Training examples\n\u2022 10,000 training examples\n\u2022 5,000 test examples\n\u2022 Two hidden layers\n\u2022 Adam optimizer\n\u2022 Step size of 0.005\n\u2022 Full batch\n\u2022 4000 training steps\nModel has memorized the training set\nWhy do we say that?\nTest error keep \ndecreasing even as \nwe keep increasing \nmodel capacity!\n33",
  "Now randomize \n15% of the \ntraining labels \nNow we see what looks like bias-variance \ntrade-off as we increase capacity to the \npoint where the model fits training data.\nBut then???\nReminder: vertical dashed line is where: \n# training parameters = # training samples\n34",
  "Double \nDescent\nClassical or under-\nparameterized regime\nModern or over-\nparameterized \nregime\nCritical regime\nReminder: vertical dashed line is where: \n# training parameters = # training samples\n35",
  "Same \nphenomenon \nshows up on \nMNIST and \nCIFAR100\nReminder: vertical dashed line is where: \n# training parameters = # training samples\n36",
  "Double Descent\n\u2022 Note that training loss is very close \nto zero.\n\u2022 Whatever is happening isn\u2019t \nhappening at training data points\n\u2022 Model never sees test set during \ntraining\n\u2022 Must be happening between the \ndata points??\n37",
  "Potential explanation:  \n\u2022 can make smoother functions with more hidden units \n\u2022 being smooth between the datapoints is a reasonable thing to do\nBut why?\n38",
  "\u2022 All of these solutions are equivalent in terms of loss.  \n\u2022 Why should the model choose the smooth solution?\n\u2022 Tendency of model to choose one solution over another is inductive bias\n39",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n40",
  "Curse of dimensionality\n\u2022 40-dimensional data\n\u2022 10,000 data points\n\u2022 Consider quantizing each dimension into \n10 bins\n\u2022 10!\" bins\n\u2022 1 data point per 10#$ bins\n\u2022 The tendency of high-dimensional space \nto overwhelm the number of data points \nis called the curse of dimensionality \n2D: 10x10=100 bins\n3D: 10x10x10=1000 bins\n41",
  "Curse: Distances collapse\nGenerate 1,000 normally \ndistributed samples in:\n\u2022 2D\n\u2022 3D\n\u2022 100D\n\u2022 1000D\n\u2026\nCalculate the ratio of \ndistances between the \nfarthest and closest \npoints.\nApproaches 1!!\n42",
  "Curse: Volumes of a hyperspheres\n\u201cAll the volume goes to the \npeel of the orange, not the \npulp.\u201d\nUnit diameter \nhypersphere in a \nunit hypercube.\nAlso the volume ratio \nto a unit hypercube\nSee also \u201cAn Adventure in the Nth Dimension\u201d, American Scientist\n43",
  "Potential explanation:  \n\u2022 It seems that through implicit and explicit regularization (next lecture!) the \n(well trained) model tends to interpolate smoothly between training data \npoints.\n44",
  "Measuring performance\n\u2022 MNIST1D dataset model and performance\n\u2022 Noise, bias, and variance\n\u2022 Reducing variance\n\u2022 Reducing bias & bias-variance trade-off\n\u2022 Double descent\n\u2022 Curse of dimensionality & weird properties of high dimensional space\n\u2022 Choosing hyperparameters\n45",
  "Choosing hyperparameters\n\u2022 Don\u2019t know bias or variance\n\u2022 Don\u2019t know how much capacity to add \n\u2022 How do we choose capacity in practice?\n\u2022 Or model structure\n\u2022 Or training algorithm\n\u2022 Or learning rate\n\u2022 Third data set \u2013 validation set\n\u2022 Train models with different hyperparameters on training set\n\u2022 Choose best hyperparameters with validation set\n\u2022 Test once with test set\n46",
  "Feedback?\n47",
  "Regularization\nAnd other ways to improve test performance\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Regularization\n\u2022 Why is there a generalization gap between training and test data?\n\u2022 Overfitting (model describes statistical peculiarities)\n\u2022 Model unconstrained in areas where there are no training examples\n\u2022 Regularization = methods to reduce the generalization gap\n\u2022 Technically means adding terms to loss function\n\u2022 But colloquially means any method (hack) to reduce gap between \ntraining and test data",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Explicit regularization\n\u2022 Standard loss function:\n\u2022 Regularization adds and extra term\n\u2022 Favors some parameters, disfavors others.\n\u2022 \ud835\udf06>0 controls the strength\n<latexit sha1_base64=\"zikx1larubSavyQszjNw/3N0pL8\n=\">AXDniclZhbT9xGFICX9JbSG2lVXipVlHaqkoRVOnlJVICIQmBFAjXBG/Q2Dv2ThiPjT2GJdb+h6o/pm9VX/sX+k/62D\nO2dwefMzx0JdjZ832eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh8FrO",
  "32eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwL\nJD4PTVcMPz3leiFTt6cuM9xMWKxGJkGkIncz97g+ZrvwgyoZi7H19z/NZHidCnUxifiBiewnQTqNsfHTbRvonfV6kqk4Dn\nnu/Puo6VPNLHflEmJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LWf7ZN",
  "mJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LWf7ZNbnw/8QRqWCVc6lKwoj\npeXMt2vWK5FKPl41i8LnrHwlMX8GIqKJbzoV3Xyxt5tiAy8KM3hT2mvjl49omJUVwmAZgJ08MCMxN0seNSR7/0K6GyUnMVNg\n1FpfR06pmR8AYi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuOus1Y",
  "Yi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuOus1Y7HIrXGSvre9NahOaJeMN\nJbViKrlG4PG4qvhivIiB4ADEIicgVbyAOk1+gshbRhTmqgRcNRMpoL3fEyqVprHkJO9pJoUMgkH3WsVWLBUCYdZRcUz7vt\nGcB1DqMAXYUvjsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNXJrVX",
  "jsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNXJrVXc1NBFl7edfROc2LGnSdO\noIsmIRx16ojyJKwswxYwiDLbRkWe54JuJWhcKqIBNzO0+DbtuZieC5OcpgvXS9tYqk/5yhjJgArD7zLZgKeVdfTae2N0nOe\n2bAh95Qxis7iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrAn7d+",
  "iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrAn7d+BU81Ly4+8Xf+SjfrVklo3\n5R7IJFRVl5qrIhP9HRQO4luH5BRE8eKlEgweBevBSCfs7GjqW4ltIvXYQUEoJoW+RMtfxKp7TB3BnU0T1FcImHrhmwmFBjmK\nurIJGBm+4arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/JrDA5",
  "arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/JrDA5TRoMlnkJZqwHKUzJEZ0tErv\n9CwxFyrvx7ypui0Yn620bYH/YLRKcOQn51s4PGIiUdieqC2yBnXZJYjvagrul0vdqzauPVd2Rqxw7XbUpSb9tLt+1wr+kBP9\nt09HaTeMSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG1fKv",
  "MSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG1fKv0mhEVNRe0U04THSGxCWEzKrgW\n/sbIr4OLRtZoQFrcL0dVMAEsDLvEpNCEsNku4a7YxrG461E23ymQ2RGYTwuJjluCzbkJYjKkYO8VTlmVIbEIkj0OcxyHNY4al\nzCXhEckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V8P41",
  "EckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V8P41DWuGKjQBLG2RNeb5W85FuAUw2WK\n8mZQFZGE7iNnW3qTO7+gqgid3Lw4GvpJaUXl5QemjpIaW5peSJIieW0qeToLo3NJzSg8sPaC0tLSkdN/SfUojSyNKH1n6iN\nLQ0pDSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDSB5Qy",
  "DSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDSB5QySxmla5auUcotJa8OgmjF0hVKA0vJ\nsx+sNUu3Kc0szSh9aOlDSgeWkqdiuJ5ZSm5v4MJoqaR03dJ1SoWl5PktiJ5Z+ozSxNKE0qeWPqX0taWvKX1s6WNKY0vJuwG4O\n7F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n9ZKT",
  "F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n9ZKTaXW3ytonsa5GacgdrMz45muQ8UlPuYO\n3uNDma7E+RmvIh6frawfRFCqQUdvqTuYVl/BaWFg5+WFz+afHuzt2F+yvtG9qbvS96X/W+7S3fu7d7z3pbf2e2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 =",
  "2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 = argmin\n\u03c6\n\u21e5\nL[\u03c6]\n\u21e4\n= argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi]\n#",
  "Explicit regularization\n\u2022 Standard loss function:\n\u2022 Regularization adds an extra term\n\u2022 Favors some parameters, disfavors others.\n\u2022 \ud835\udf06>0 controls the strength\n<latexit sha1_base64=\"zikx1larubSavyQszjNw/3N0pL8\n=\">AXDniclZhbT9xGFICX9JbSG2lVXipVlHaqkoRVOnlJVICIQmBFAjXBG/Q2Dv2ThiPjT2GJdb+h6o/pm9VX/sX+k/62D\nO2dwefMzx0JdjZ832eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh8FrO",
  "32eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwL\nJD4PTVcMPz3leiFTt6cuM9xMWKxGJkGkIncz97g+ZrvwgyoZi7H19z/NZHidCnUxifiBiewnQTqNsfHTbRvonfV6kqk4Dn\nnu/Puo6VPNLHflEmJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LWf7ZN",
  "mJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LWf7ZNbnw/8QRqWCVc6lKwoj\npeXMt2vWK5FKPl41i8LnrHwlMX8GIqKJbzoV3Xyxt5tiAy8KM3hT2mvjl49omJUVwmAZgJ08MCMxN0seNSR7/0K6GyUnMVNg\n1FpfR06pmR8AYi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuOus1Y",
  "Yi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuOus1Y7HIrXGSvre9NahOaJeMN\nJbViKrlG4PG4qvhivIiB4ADEIicgVbyAOk1+gshbRhTmqgRcNRMpoL3fEyqVprHkJO9pJoUMgkH3WsVWLBUCYdZRcUz7vt\nGcB1DqMAXYUvjsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNXJrVX",
  "jsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNXJrVXc1NBFl7edfROc2LGnSdO\noIsmIRx16ojyJKwswxYwiDLbRkWe54JuJWhcKqIBNzO0+DbtuZieC5OcpgvXS9tYqk/5yhjJgArD7zLZgKeVdfTae2N0nOe\n2bAh95Qxis7iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrAn7d+",
  "iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrAn7d+BU81Ly4+8Xf+SjfrVklo3\n5R7IJFRVl5qrIhP9HRQO4luH5BRE8eKlEgweBevBSCfs7GjqW4ltIvXYQUEoJoW+RMtfxKp7TB3BnU0T1FcImHrhmwmFBjmK\nurIJGBm+4arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/JrDA5",
  "arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/JrDA5TRoMlnkJZqwHKUzJEZ0tErv\n9CwxFyrvx7ypui0Yn620bYH/YLRKcOQn51s4PGIiUdieqC2yBnXZJYjvagrul0vdqzauPVd2Rqxw7XbUpSb9tLt+1wr+kBP9\nt09HaTeMSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG1fKv",
  "MSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG1fKv0mhEVNRe0U04THSGxCWEzKrgW\n/sbIr4OLRtZoQFrcL0dVMAEsDLvEpNCEsNku4a7YxrG461E23ymQ2RGYTwuJjluCzbkJYjKkYO8VTlmVIbEIkj0OcxyHNY4al\nzCXhEckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V8P41",
  "EckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V8P41DWuGKjQBLG2RNeb5W85FuAUw2WK\n8mZQFZGE7iNnW3qTO7+gqgid3Lw4GvpJaUXl5QemjpIaW5peSJIieW0qeToLo3NJzSg8sPaC0tLSkdN/SfUojSyNKH1n6iN\nLQ0pDSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDSB5Qy",
  "DSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDSB5QySxmla5auUcotJa8OgmjF0hVKA0vJ\nsx+sNUu3Kc0szSh9aOlDSgeWkqdiuJ5ZSm5v4MJoqaR03dJ1SoWl5PktiJ5Z+ozSxNKE0qeWPqX0taWvKX1s6WNKY0vJuwG4O\n7F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n9ZKT",
  "F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n9ZKTaXW3ytonsa5GacgdrMz45muQ8UlPuYO\n3uNDma7E+RmvIh6frawfRFCqQUdvqTuYVl/BaWFg5+WFz+afHuzt2F+yvtG9qbvS96X/W+7S3fu7d7z3pbf2e2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 =",
  "2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 = argmin\n\u03c6\n\u21e5\nL[\u03c6]\n\u21e4\n= argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi]\n#\n<latexit sha1_base64=\"eoOVoDAQYf5iqhHG6TNOW3FC4\nlI=\">AW8niclZhbT9xGFICX9JYmaUtalZe+WEWRqjZFUKWXl0gJhCQEUiCwQI3aOwdeyeMx8YewxJr/0jfqr72D1Xqj\n+kZ27uDzxkeuhLs7P",
  "WXl0gJhCQEUiCwQI3aOwdeyeMx8YewxJr/0jfqr72D1Xqj\n+kZ27uDzxkeuhLs7Pk+z4zPXHwJMikKvbz8z9yNDz786ONPbn56/adz7/Yv7ulwdFWuYh74epTPOjgBVcCsX7WmjJj7\nKcsySQ/DA4XTP8JznhUjVvr7M+CBhsRKRCJmG0Mn8uT9iuvKDKBuJifQ81keJ0KdTEO+5JE+9osyOanEw5XJ2pj4nMp\n4dfkGJyxKdyHwqUpDLwfPF9C6",
  "ifQ81keJ0KdTEO+5JE+9osyOanEw5XJ2pj4nMp\n4dfkGJyxKdyHwqUpDLwfPF9C60Pm+eEw1Z6fBOm4imsX6hv4uYhHenAyv7i8tFx/PFpYaQuLvfazc3L36E/TMy4UqHk\nhXF8cpypgcVy7UIJZ/c8suCZyw8ZTE/hqJiCS8GVZ2giXcPIkMvSnP4U9qro1ePqFhSFJdJAGbC9KjAzARd7LjU0W+DSqi\ns1FyFTUNRKT2deib3lDkPNTyEgoszA",
  "ePqFhSFJdJAGbC9KjAzARd7LjU0W+DSqi\ns1FyFTUNRKT2deib3lDkPNTyEgoszAX01QtHLGehjG5St+EaZJwtSw8lfXdyeQdR4LVfGzsh6fyaTrNcOh+J1xurG\n/qwWoXki3nNSa2YSq4ReDypKr4UL2EgOACxAlIFS+gTpOfIPJWEIX5KAFXzVyAieC9mpCqleYx5KSjvSEaFDLJx1rjV\ngwlElH2QPF8+5BnCdwyhAV+GLozHYy5iaTI/TfKz",
  "CqleYx5KSjvSEaFDLJx1rjV\ngwlElH2QPF8+5BnCdwyhAV+GLozHYy5iaTI/TfKzpCpMDLeQMxXzugk45ZBJc0ZdQ5VSwqFhx/odW6+YOm0Tl2Z1V3M\nTQdZ+3nV0TvOihl2njiALJmHcteoIspr1mzDIcluG3SBPBNxq0JhVZCJuZOnQbftzETw3BxnsF63npF0n/OUEZMAFaf+\nRZMhbyr6Uz25sm57z2TYGPvREMVveQZhu80gicVRubULPOF",
  "npF0n/OUEZMAFaf+\nRZMhbyr6Uz25sm57z2TYGPvREMVveQZhu80gicVRubULPOFTJptiCUpxd0/TGofJMdE/QBPCiK3Ohoiva/boEU9aE/f\ntwqnkp+fGPSz/z8aBaNsvG/CPZhIqKMnNVZML/o6IhXK/w/IHrxUosGDQD14qYT9HQ0dy/HENpF67KAgFJNCX6LlL2LV\nPaO4M6mCeorBEy98M2EQoMcRV3ZBIwM3DldUygEJ1k2JxjKNOizDnZ",
  "CX6LlL2LV\nPaO4M6mCeorBEy98M2EQoMcRV3ZBIwM3DldUygEJ1k2JxjKNOizDnZ/NB8hkitm20xF+Zi1d1QpRG6+waXs6OgDBeHc\n37N4QHKaNDkM0hLNWQ5SubYDOn4rV9oWGKu1V8PeVN0WjE/2zbg37B6JRhyM9ONvF4xMSijkR1wa2Osy5JLEd7UNdsul7\ntWbX59nsytWOH6zYlqbftpdt2uNf0gJ9tOXq7RTxiUeiutoeUo9YjvagLnce",
  "ul7\ntWbX59nsytWOH6zYlqbftpdt2uNf0gJ9tOXq7RTxiUeiutoeUo9YjvagLncet1xn4XDdpiT1TvPotB3uzETP9ofc3M\nbVIqh+a2L5V+E8KipqJ2imnCYyQ2ISwmZdeC31jZE3Dx6FpNCIs7hehqJoClIZf4FJoQFpsl3DXbGFa3HOqW2UyGyGzCW\nHxGUvwWTchLMZUjJ3iKcsyJDYhkscRzuOI5jHDUuaS8IhkjhEhU8o1ofJR2pVMAEtj1",
  "UvwWTchLMZUjJ3iKcsyJDYhkscRzuOI5jHDUuaS8IhkjhEhU8o1ofJR2pVMAEtj1NrY0Rj0QKYKNdgGsVzQmVc4Z5\nCs1jRWdx3Ndy/pmHNUIUmgKVtsY8f9u5yAKcYrjNciU5E8jKaAJ3sLNDnendXxBV5E4OnowtvaT0wtILSg8tPaQ0t5Q8E\nQTRK0vJ0kQnVt6TumBpQeUlpaWlPYt7VMaWRpR+tTSp5SGloaUrlm6Rqm2lNyRwhXB0n1KR5",
  "kQnVt6TumBpQeUlpaWlPYt7VMaWRpR+tTSp5SGloaUrlm6Rqm2lNyRwhXB0n1KR5aOKD2y9IjS15a+pvS5pc\n8pfWPpG0rfW/qe0seWPqaUWcoXbd0nVJuKXl1ESrlq5SGlhKnv1grVm6Q2lmaUbpE0ufUDq0lDwVw/XMUnJ7AxdGSyWl\nG5ZuUCosJc9vQfTS0peUJpYmlL6w9AWl7yx9R+kzS59RGltK3g3A3Ymle5Tat0BVQemupbuUnl65n4",
  "TS0peUJpYmlL6w9AWl7yx9R+kzS59RGltK3g3A3Ymle5Tat0BVQemupbuUnl65n4vwGfDGLgm5ratY\nJvS1NKU0k1LyZMC3EpYekruJyPV7mrTt01kX4vUjDtYm/Hp0STnkZpxB2t3p+nRZH+K1IyPSNfXD2YvUiClsNOfzC+u4Le\nwtHDw09LKL0sPdh8sPlpt39De7H3T+7b3XW+l92vUe95b6fX74W9f+duzN2eu7OgF/5Y+HPhr0a9Mdce81",
  "lpt39De7H3T+7b3XW+l92vUe95b6fX74W9f+duzN2eu7OgF/5Y+HPhr0a9Mdce81Wv81n4+z+5\nGP2f</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi] + \u03bb \u00b7 g[\u03c6]\n#",
  "Explicit regularization\n\u2022 Standard loss function:\n\u2022 Regularization adds an extra term\n\u2022 Where g[\ud835\udf19] is smaller for preferred parameters\n\u2022 \u03bb > 0 controls the strength of influence\n<latexit sha1_base64=\"zikx1larubSavyQszjNw/3N0pL8\n=\">AXDniclZhbT9xGFICX9JbSG2lVXipVlHaqkoRVOnlJVICIQmBFAjXBG/Q2Dv2ThiPjT2GJdb+h6o/pm9VX/sX+k/62D\nO2dwefMzx0JdjZ832eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh",
  "djZ832eGZ+5+BJkUhR6aemfmRtvf3Ou+/dfH/2gw8/+viTuVufHhRpmYd8P0xlmh8FrOBSKL6vhZb8KMs5SwL\nJD4PTVcMPz3leiFTt6cuM9xMWKxGJkGkIncz97g+ZrvwgyoZi7H19z/NZHidCnUxifiBiewnQTqNsfHTbRvonfV6kqk4Dn\nnu/Puo6VPNLHflEmJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LW",
  "HflEmJ5W4tzx+Va2PfS4l/KprGpnCHShcmkLfz0U81P2TuYWlxaX649HCcltY6LWf7ZNbnw/8QRqWCVc6lKwoj\npeXMt2vWK5FKPl41i8LnrHwlMX8GIqKJbzoV3Xyxt5tiAy8KM3hT2mvjl49omJUVwmAZgJ08MCMxN0seNSR7/0K6GyUnMVNg\n1FpfR06pmR8AYi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuO",
  "mR8AYi56GWl1BgYS6gr14ZDkLNYzXrK/4RZgmCVODyl9Z2xlDUnksVMXPynrsxuOus1Y7HIrXGSvre9NahOaJeMN\nJbViKrlG4PG4qvhivIiB4ADEIicgVbyAOk1+gshbRhTmqgRcNRMpoL3fEyqVprHkJO9pJoUMgkH3WsVWLBUCYdZRcUz7vt\nGcB1DqMAXYUvjsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNX",
  "XYUvjsZgN2NqPDlO85HOk6owMdxCzlTM6ybglEMmzRl1DVKCYeGHetXbD1n6rRNXJrVXc1NBFl7edfROc2LGnSdO\noIsmIRx16ojyJKwswxYwiDLbRkWe54JuJWhcKqIBNzO0+DbtuZieC5OcpgvXS9tYqk/5yhjJgArD7zLZgKeVdfTae2N0nOe\n2bAh95Qxis7iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrA",
  "xis7iHNLnelETirNjamZp0rZNJsQShPL7qm6Y1D5ZnonqAJ4EVX5kJFV7Q7dQmrAn7d+BU81Ly4+8Xf+SjfrVklo3\n5R7IJFRVl5qrIhP9HRQO4luH5BRE8eKlEgweBevBSCfs7GjqW4ltIvXYQUEoJoW+RMtfxKp7TB3BnU0T1FcImHrhmwmFBjmK\nurIJGBm+4arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/J",
  "Bm+4arsmEAhOsmwOcdQpkWZc7L5ofkMkVo32IuzMWqu6FKI3T3DS6nR0EZLg7n/JrDA5TRoMlnkJZqwHKUzJEZ0tErv\n9CwxFyrvx7ypui0Yn620bYH/YLRKcOQn51s4PGIiUdieqC2yBnXZJYjvagrul0vdqzauPVd2Rqxw7XbUpSb9tLt+1wr+kBP9\nt09HaTeMSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG",
  "HaTeMSijkR1tT2kHrEc7UFd7jxus7C4bpNSeqd5NFpO9ypiaZ/tDfkmpnbpFQOzG1fKv0mhEVNRe0U04THSGxCWEzKrgW\n/sbIr4OLRtZoQFrcL0dVMAEsDLvEpNCEsNku4a7YxrG461E23ymQ2RGYTwuJjluCzbkJYjKkYO8VTlmVIbEIkj0OcxyHNY4al\nzCXhEckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V",
  "zCXhEckcI0KmlGtC5cO0K5kAlkaotZGjMeiBTBVqsA1iuaAzr3DOPIVmsaKzeN/V8P41DWuGKjQBLG2RNeb5W85FuAUw2WK\n8mZQFZGE7iNnW3qTO7+gqgid3Lw4GvpJaUXl5QemjpIaW5peSJIieW0qeToLo3NJzSg8sPaC0tLSkdN/SfUojSyNKH1n6iN\nLQ0pDSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDS",
  "LQ0pDSVUtXKdWkjtSuCJYukfp0NIhpUeWHlH6wtIXlD6x9AmlLy19SekbS9Q+sDSB5QySxmla5auUcotJa8OgmjF0hVKA0vJ\nsx+sNUu3Kc0szSh9aOlDSgeWkqdiuJ5ZSm5v4MJoqaR03dJ1SoWl5PktiJ5Z+ozSxNKE0qeWPqX0taWvKX1s6WNKY0vJuwG4O\n7F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n",
  "4O\n7F0l1L7FqgqKN2xdIfSM0vP3O8F+HQYA9fE3LIVbFGaWpSumEpeVKAWwlLT8n9ZKTaXW3ytonsa5GacgdrMz45muQ8UlPuYO\n3uNDma7E+RmvIh6frawfRFCqQUdvqTuYVl/BaWFg5+WFz+afHuzt2F+yvtG9qbvS96X/W+7S3fu7d7z3pbf2e2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 =",
  "2Hv35kvZ76\nZ+Xb+t/k/5v+c/6tRb8y0x3zW63zm/4PoeEJ0g=</latexit>\u02c6\u03c6 = argmin\n\u03c6\n\u21e5\nL[\u03c6]\n\u21e4\n= argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi]\n#\n<latexit sha1_base64=\"eoOVoDAQYf5iqhHG6TNOW3FC4\nlI=\">AW8niclZhbT9xGFICX9JYmaUtalZe+WEWRqjZFUKWXl0gJhCQEUiCwQI3aOwdeyeMx8YewxJr/0jfqr72D1Xqj\n+kZ27uDzxkeuhLs7P",
  "WXl0gJhCQEUiCwQI3aOwdeyeMx8YewxJr/0jfqr72D1Xqj\n+kZ27uDzxkeuhLs7Pk+z4zPXHwJMikKvbz8z9yNDz786ONPbn56/adz7/Yv7ulwdFWuYh74epTPOjgBVcCsX7WmjJj7\nKcsySQ/DA4XTP8JznhUjVvr7M+CBhsRKRCJmG0Mn8uT9iuvKDKBuJifQ81keJ0KdTEO+5JE+9osyOanEw5XJ2pj4nMp\n4dfkGJyxKdyHwqUpDLwfPF9C6",
  "ifQ81keJ0KdTEO+5JE+9osyOanEw5XJ2pj4nMp\n4dfkGJyxKdyHwqUpDLwfPF9C60Pm+eEw1Z6fBOm4imsX6hv4uYhHenAyv7i8tFx/PFpYaQuLvfazc3L36E/TMy4UqHk\nhXF8cpypgcVy7UIJZ/c8suCZyw8ZTE/hqJiCS8GVZ2giXcPIkMvSnP4U9qro1ePqFhSFJdJAGbC9KjAzARd7LjU0W+DSqi\ns1FyFTUNRKT2deib3lDkPNTyEgoszA",
  "ePqFhSFJdJAGbC9KjAzARd7LjU0W+DSqi\ns1FyFTUNRKT2deib3lDkPNTyEgoszAX01QtHLGehjG5St+EaZJwtSw8lfXdyeQdR4LVfGzsh6fyaTrNcOh+J1xurG\n/qwWoXki3nNSa2YSq4ReDypKr4UL2EgOACxAlIFS+gTpOfIPJWEIX5KAFXzVyAieC9mpCqleYx5KSjvSEaFDLJx1rjV\ngwlElH2QPF8+5BnCdwyhAV+GLozHYy5iaTI/TfKz",
  "CqleYx5KSjvSEaFDLJx1rjV\ngwlElH2QPF8+5BnCdwyhAV+GLozHYy5iaTI/TfKzpCpMDLeQMxXzugk45ZBJc0ZdQ5VSwqFhx/odW6+YOm0Tl2Z1V3M\nTQdZ+3nV0TvOihl2njiALJmHcteoIspr1mzDIcluG3SBPBNxq0JhVZCJuZOnQbftzETw3BxnsF63npF0n/OUEZMAFaf+\nRZMhbyr6Uz25sm57z2TYGPvREMVveQZhu80gicVRubULPOF",
  "npF0n/OUEZMAFaf+\nRZMhbyr6Uz25sm57z2TYGPvREMVveQZhu80gicVRubULPOFTJptiCUpxd0/TGofJMdE/QBPCiK3Ohoiva/boEU9aE/f\ntwqnkp+fGPSz/z8aBaNsvG/CPZhIqKMnNVZML/o6IhXK/w/IHrxUosGDQD14qYT9HQ0dy/HENpF67KAgFJNCX6LlL2LV\nPaO4M6mCeorBEy98M2EQoMcRV3ZBIwM3DldUygEJ1k2JxjKNOizDnZ",
  "CX6LlL2LV\nPaO4M6mCeorBEy98M2EQoMcRV3ZBIwM3DldUygEJ1k2JxjKNOizDnZ/NB8hkitm20xF+Zi1d1QpRG6+waXs6OgDBeHc\n37N4QHKaNDkM0hLNWQ5SubYDOn4rV9oWGKu1V8PeVN0WjE/2zbg37B6JRhyM9ONvF4xMSijkR1wa2Osy5JLEd7UNdsul7\ntWbX59nsytWOH6zYlqbftpdt2uNf0gJ9tOXq7RTxiUeiutoeUo9YjvagLnce",
  "ul7\ntWbX59nsytWOH6zYlqbftpdt2uNf0gJ9tOXq7RTxiUeiutoeUo9YjvagLncet1xn4XDdpiT1TvPotB3uzETP9ofc3M\nbVIqh+a2L5V+E8KipqJ2imnCYyQ2ISwmZdeC31jZE3Dx6FpNCIs7hehqJoClIZf4FJoQFpsl3DXbGFa3HOqW2UyGyGzCW\nHxGUvwWTchLMZUjJ3iKcsyJDYhkscRzuOI5jHDUuaS8IhkjhEhU8o1ofJR2pVMAEtj1",
  "UvwWTchLMZUjJ3iKcsyJDYhkscRzuOI5jHDUuaS8IhkjhEhU8o1ofJR2pVMAEtj1NrY0Rj0QKYKNdgGsVzQmVc4Z5\nCs1jRWdx3Ndy/pmHNUIUmgKVtsY8f9u5yAKcYrjNciU5E8jKaAJ3sLNDnendXxBV5E4OnowtvaT0wtILSg8tPaQ0t5Q8E\nQTRK0vJ0kQnVt6TumBpQeUlpaWlPYt7VMaWRpR+tTSp5SGloaUrlm6Rqm2lNyRwhXB0n1KR5",
  "kQnVt6TumBpQeUlpaWlPYt7VMaWRpR+tTSp5SGloaUrlm6Rqm2lNyRwhXB0n1KR5aOKD2y9IjS15a+pvS5pc\n8pfWPpG0rfW/qe0seWPqaUWcoXbd0nVJuKXl1ESrlq5SGlhKnv1grVm6Q2lmaUbpE0ufUDq0lDwVw/XMUnJ7AxdGSyWl\nG5ZuUCosJc9vQfTS0peUJpYmlL6w9AWl7yx9R+kzS59RGltK3g3A3Ymle5Tat0BVQemupbuUnl65n4",
  "TS0peUJpYmlL6w9AWl7yx9R+kzS59RGltK3g3A3Ymle5Tat0BVQemupbuUnl65n4vwGfDGLgm5ratY\nJvS1NKU0k1LyZMC3EpYekruJyPV7mrTt01kX4vUjDtYm/Hp0STnkZpxB2t3p+nRZH+K1IyPSNfXD2YvUiClsNOfzC+u4Le\nwtHDw09LKL0sPdh8sPlpt39De7H3T+7b3XW+l92vUe95b6fX74W9f+duzN2eu7OgF/5Y+HPhr0a9Mdce81",
  "lpt39De7H3T+7b3XW+l92vUe95b6fX74W9f+duzN2eu7OgF/5Y+HPhr0a9Mdce81Wv81n4+z+5\nGP2f</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi] + \u03bb \u00b7 g[\u03c6]\n#",
  "Explicit regularization\nLoss function for Gabor model \nof Lecture 6 and Chapter 6.\n    denotes local minima",
  "Explicit regularization\nExample of a regularization \nfunction that prefers \nparameters close to 0.",
  "Explicit regularization\ndenotes local minima\nFewer local minima and the \nabsolute minimum has \nmoved.",
  "\u2022 Maximum likelihood:\n\u2022 Regularization is equivalent to adding a prior over parameters\n\u2026 what you know about parameters before seeing the data \nProbabilistic interpretation\n<latexit sha1_base64=\"kW0bG9g/Xv5Q2\npReTVZwgV8/vpY=\">AW3XiclZhJb9w2FICVdEvTzWlRX3oRagRIi9Swi3S5BEjsOIl\njpx7Ha+JxBpSG0jCmKJmi7HUOfZW9Nqf1N/QH9Fre+2jZmH0Hn2IgXTY931c9LhoiQo\npSrO09PeVq+8+97H1z78PpH3/y6WdzNz7fL/NKx3wvzmWuDyNWcikU3zPC",
  "hoiQo\npSrO09PeVq+8+97H1z78PpH3/y6WdzNz7fL/NKx3wvzmWuDyNWcikU3zPCSH5YaM6\nySPKD6GTV8oMzrkuRq1zUfDjKVKJCJmBkK9OdYdMFN3o6QYiF4N+wynWZs2JuGupI\nn5qhb6Lzfq8Xd5dHLen3U0bcAX0Bg9CsUhrZwe1zhmzGzpa4W6cAc9+YWlhaXmr+QFpY\nnhYVg8tfp3fiy3+3ncZVxZWLJyvJoeakwxzXTRsSj653q5IXL",
  "+YWlhaXmr+QFpY\nnhYVg8tfp3fiy3+3ncZVxZWLJyvJoeakwxzXTRsSj653q5IXLD5hKT+ComIZL4/rJhe\nj8CZE+mGSa/inTNhE36xRs6wsL7IzIyZQYmZDfrYUWSn49roYrKcBWPO0oqGZo8tIk\nN+0Lz2MgLKLBYCxhrGA+YZrGB9F/vKn4e51nGVL/urqxtjyDBPBWq5qdVMxWjUdtZaxw\nOxcuMlfXdWSvC8Ey85qSRrGNXCLwdFTXfDFdx",
  "qxtjyDBPBWq5qdVMxWjUdtZaxw\nOxcuMlfXdWSvC8Ey85qSRrGNXCLwdFTXfDFdxEBwAGKRE5ArXkKbNj9REi4jCktPAgY\ne5UO7hsJnI9K0MjyFnLS0F0SDQiH5sGWtEgumMmspO6CE4c3QAm40zAIMFX4moOdgqn\nRtJ7hQ6OzurQx3INmKuVNF3DJMZP2itqGqSEqnHL+gVbz5g6mSQuL5qhahtB1q5uO0b\nTvKh+2kiyIJFmLatJoIsCQdFn",
  "itqGqSEqnHL+gVbz5g6mSQuL5qhahtB1q5uO0b\nTvKh+2kiyIJFmLatJoIsCQdFn8F+H03LsPF1FtqIXxUKq4IszI7Oo3bfhY3gtTksYL+\n0vbWapP+MoYzYAOw+yuYinlbX81ndjhNzlnj2wIfhgOYrHYVe+I1lzXtBK5qEhtRs8k\nVMm2IKTz87ZpR+NReSHaF2gDeNVWqjkDe12U4Ila8Pd23CpupL86LvFH/jwuF6y28b\n+h2QTGiqrwteQD",
  "SHaF2gDeNVWqjkDe12U4Ila8Pd23CpupL86LvFH/jwuF6y28b\n+h2QTGiqrwteQDb9FQ324NeH1BRE8eblEkweBZvJyCec7mjqm8cK2kWbuoCAUk8JcoO0\nvUtWu0TwYPMjRUCtl34ZUKhSU6StmwDVoZfuMl6FlCMLjIeX2Ms87LSnBx+aD1DpNH\ntsaiFvVm1D1Rphfa5weWsFpTh5nDGL6keoYxG43xGeaX6TKNkDu2UDl92SwNbzLf7myk\nf",
  "Vm1D1Rphfa5weWsFpTh5nDGL6keoYxG43xGeaX6TKNkDu2UDl92SwNbzLf7myk\nfF71Wyk83Jv3BuGB2qjmp70NPB8psagjUVvwVONtSxL0x+0NVub46s3nj5LVnaqcf\n1m5K0Oxml3/a4l4yAn256RrtJPGJR6K2JiOkHrE8/UFb/jxu+q7C4/pNSdqd5tFre9y\nZiZ/sjvghtnHpFz27WNfLrvjEBYNFY1XzDOeInEcwmJWtS34f6zsCLh5tK1",
  "re9y\nZiZ/sjvghtnHpFz27WNfLrvjEBYNFY1XzDOeInEcwmJWtS34f6zsCLh5tK1xCIudUrQ\n1G8BSn0t8CeMQFsdbuG1OYljd9KibfpXJYoDMcQiLj1iGr3ocwmJKxdQrnrCiQOI4RPI\n4wHkc0DwWCp8Ep6RwjMjZEn5FpQe5G3JBrA0RL0NPZ3BCGSuUIeTIJZLuvJK78pTaBU\nruor3fB3vXdKxYahBG8DSFtljYXfLu8kinGJ4zPIluRDIK",
  "eTIJZLuvJK78pTaBU\nruor3fB3vXdKxYahBG8DSFtljYXfLu8kinGJ4zPIluRDIKmgCO9jpUGf69BclNXmSg1d\niRy8oPXf0nNIDRw8o1Y6SN4IoeYoeTuJkjNHzyjd3Sf0srRitI9R/coTRxNKH3o6EN\nKY0djSlcdXaXUOEqeSOGO4OgupQNHB5QeOnpI6XNHn1P62NHlL5w9AWlrx19Tel9R+9\nTyhxlK45ukYpd5R8OoiSFUdXKI0cJe9+sN",
  "XNHn1P62NHlL5w9AWlrx19Tel9R+9\nTyhxlK45ukYpd5R8OoiSFUdXKI0cJe9+sNc7VBaOFpQ+sDRB5T2HSVvxXA/c5Q83sC\nN0VFJ6bqj65QKR8n7W5Q8dfQpZmjGaVPH1C6StHX1H6yNFHlKaOkm8D8HTi6A6l7it\nQXVK67eg2paeOnvq/C/DZNEa+hbnlGtiNHc0p3TDUfKmAI8Sjp6Q58lETU616dcmcq4\nlasY9bJLxaW2S80TNuIdNTqd",
  "lGtiNHc0p3TDUfKmAI8Sjp6Q58lETU616dcmcq4\nlasY9bJLxaW2S80TNuIdNTqdpbXI+JWrGB2Toa/uzDymQUjpe3MLy/grLC3sf7+4/OP\nine07C/dWJl9orwVfBV8Ht4Ll4KfgXvA46AR7QRz8FfwT/Bv8N9+b/23+9/k/xurVK5M\n6XwStv/k/wfV+PeL</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi, \u03c6)Pr(\u03c6)\n#\n<latexit sha1_base64=\"vIMNo",
  "L</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi, \u03c6)Pr(\u03c6)\n#\n<latexit sha1_base64=\"vIMNoLnkB0Kv4\nMRp2+fDjWn7tMQ=\">AW03iclZhJU9xGFIBlZ3OcDScVLrmoQrnKSTkUpJzl4iobjG0MD\noNhAMNgqVpadq0WqLVgsHKXFK5iflh+Sca/If8lqztPVecwhVznTe9/Wi14uWqJCiNE\ntLf127/s673/wY0Pb3708SefjZ36/O9Mq90zLtxLnN9ELG",
  "nTe9/Wi14uWqJCiNE\ntLf127/s673/wY0Pb3708SefjZ36/O9Mq90zLtxLnN9ELGS6F41wgj+UGhOcsiyfe\nj01XL98+5LkWuds1lwY8zliqRiJgZCJ3M7fcGzNS9KCkGYhTeD3tMpxkbnkxDPckTc9Qrd\nN4/qcX95dGren3U0XcAX0Jg9CsUhrZwd1zhm54W6cAcn8wtLC0uNX8hLSxPCgvB5K9zcu\nvLfq+fx1XGlYklK8uj5aXCHNdMGxFLP",
  "4W6cAcn8wtLC0uNX8hLSxPCgvB5K9zcu\nvLfq+fx1XGlYklK8uj5aXCHNdMGxFLPrZq0pesPiUpfwIioplvDyumwyMwtsQ6YdJruG\nfMmETfbtGzbKyvMwiMDNmBiVmNuhjR5VJfj6uhSoqw1U87ipZGjy0KYz7AvNYyMvocBiL\nWCsYTxgmsUGkn6zp/hFnGcZU/26t7K2PYK08lSomp9VzQSMRm1nrXE4FK8yVtZ3Z60Iwz\nPxhpNGsU2coXA0",
  "GcZU/26t7K2PYK08lSomp9VzQSMRm1nrXE4FK8yVtZ3Z60Iwz\nPxhpNGsU2coXA01Fd8V0EQPBAYhFTkCueAlt2vxESbiMKCw4CRh4lA/tyglfjEjTyvA\nUctLSDokGhULyYctaJRZMZdZSdkAJw9uhBdxomAUYKvxwNAc7BVOjaT3Dh0ZndWljuAfNV\nMqbLuCSYybtFbUNVUkJVeOW9Qu2XjB1OklcXjRD1TaCrF3doymeVH9tNEkAWLMG1bTQ",
  "LuCSYybtFbUNVUkJVeOW9Qu2XjB1OklcXjRD1TaCrF3doymeVH9tNEkAWLMG1bTQ\nRZEo6HPoNdPpqWYbvrLQRvyoUVgVZmB2dR+2+CxvBa3NYwH5pe2s1Sf85QxmxAdh9lc\nwFfO2vprP7HCanPGtwU+DAcwWe0q9pxrLmvaCVzVJDaiZpMrZNJsQUjnF23Tjsaj8kK0L\n9AG8KartFDJW9rdpgRL1oZ7d+FSdSX50XeLP/Dhcb1kt439D8kmNF",
  "Tjsaj8kK0L\n9AG8KartFDJW9rdpgRL1oZ7d+FSdSX50XeLP/Dhcb1kt439D8kmNFRWha8hG/4fDfXho\nTXF0Tw5OUSTR4EmsnLJZzvaOqYxgvbRpq5g4JQTApziba/SFW7ThPBg80zNFYI2Hbhlwm\nFJjlJ2rINWBl+4dbqWUAxush4fI2xzMtKc3L4ofUMkUa3x6IW9mbVPlClFdrnBpezWlCGm\n8M5v6J6hDIajfMZ5ZXqM42SObRTOnzVKw1s",
  "Ua3x6IW9mbVPlClFdrnBpezWlCGm\n8M5v6J6hDIajfMZ5ZXqM42SObRTOnzVKw1sMd/ub6Z8XPRaKT/bmPQH4LZqeKYn51s4P\nlIiUdidqCZxlvW5JYnv6grdlyfXtk9carb8nSTj2u35Sk3cko/bHvWIE/GzTM9pN4hG\nLOhK1NRkh9Yjl6Q/a8udx03cVHtdvStLuNI9e2+POTLT8k90BN8w+JuWybx/7ctkbh7Boq\nGi8Yp7xFInjEBazqm3B",
  "tdvStLuNI9e2+POTLT8k90BN8w+JuWybx/7ctkbh7Boq\nGi8Yp7xFInjEBazqm3B/2NlR8DNo2NQ1jslKt2QCW+lziSxiHsDjewm1zEsPqpkfd9K\ntMFgNkjkNYfMIyfNXjEBZTKqZe8ZQVBRLHIZLHAc7jgOaxwFLhk/CMFJ4ZIUvKt6D0IG9\nLNoClIept6OkMRiBzhTqcBLFc0pVXeleQqtY0VXc9XcvaJjw1CDNoClLbLHwt6Wd5NFO\nMXwm",
  "6OkMRiBzhTqcBLFc0pVXeleQqtY0VXc9XcvaJjw1CDNoClLbLHwt6Wd5NFO\nMXwmOVLciGQVdAEdrDToc706S9KavIkBy/Cjl5SeuHoBaX7ju5Tqh0lbwR8sJR8nYSJe\neOnlO65+gepZWjFaVdR7uUJo4mlD529DGlsaMxpauOrlJqHCVPpHBHcHSX0oGjA0oPHD2\ng9KWjLyl96uhTSg8dPaT0jaNvKH3o6ENKmaOM0jVH1yjljpJPB1Gy4ugK",
  "0oPHD2\ng9KWjLyl96uhTSg8dPaT0jaNvKH3o6ENKmaOM0jVH1yjljpJPB1Gy4ugKpZGj5N0P9pqjH\nUoLRwtKHzn6iNK+o+StGO5njpLHG7gxOiopXd0nVLhKHl/i5Lnj6nNHM0o/SZo8ofe\n3oa0qfOPqE0tR8m0Ank4c3aHUfQWqS0q3Hd2m9MzRM/93AT6bxsi3MLdcA1uU5o7mlG4\nSt4U4FHC0VPyPJmoyak2/dpEzrVEzbiHTI+rU1ynqg",
  "6bxsi3MLdcA1uU5o7mlG4\nSt4U4FHC0VPyPJmoyak2/dpEzrVEzbiHTI+rU1ynqgZ97DJ6TStTc6nRM34gAx9bW/2I\nQVSCif9ydzCMv4KSwt73y8u/7h4b/vewoOVyRfaG8FXwdfBnWA5+Cl4EDwNOkE3iIM/g7\n+Df4J/57vz9fxv87+P1evXJnW+CFp/83/8B4s3868=</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi, \u03c6)\n#\nMaximum a posteriori or \nMAP criterion",
  "Equivalence\n\u2022 Explicit regularization:\n\u2022 Probabilistic interpretation:\n\u2022 Converting to Negative Log Likelihood (e.g. \u2212 log \u22c5  ):\n<latexit sha1_base64=\"kW0bG9g/Xv5Q2\npReTVZwgV8/vpY=\">AW3XiclZhJb9w2FICVdEvTzWlRX3oRagRIi9Swi3S5BEjsOIl\njpx7Ha+JxBpSG0jCmKJmi7HUOfZW9Nqf1N/QH9Fre+2jZmH0Hn2IgXTY931c9LhoiQo\npSrO09PeVq+8+97H1z78PpH3/y6WdzNz7fL/NKx3wvzmWuDyNWcikU3zP",
  "LhoiQo\npSrO09PeVq+8+97H1z78PpH3/y6WdzNz7fL/NKx3wvzmWuDyNWcikU3zPCSH5YaM6\nySPKD6GTV8oMzrkuRq1zUfDjKVKJCJmBkK9OdYdMFN3o6QYiF4N+wynWZs2JuGupI\nn5qhb6Lzfq8Xd5dHLen3U0bcAX0Bg9CsUhrZwe1zhmzGzpa4W6cAc9+YWlhaXmr+QFpY\nnhYVg8tfp3fiy3+3ncZVxZWLJyvJoeakwxzXTRsSj653q5IX",
  "9+YWlhaXmr+QFpY\nnhYVg8tfp3fiy3+3ncZVxZWLJyvJoeakwxzXTRsSj653q5IXLD5hKT+ComIZL4/rJhe\nj8CZE+mGSa/inTNhE36xRs6wsL7IzIyZQYmZDfrYUWSn49roYrKcBWPO0oqGZo8tIk\nN+0Lz2MgLKLBYCxhrGA+YZrGB9F/vKn4e51nGVL/urqxtjyDBPBWq5qdVMxWjUdtZaxw\nOxcuMlfXdWSvC8Ey85qSRrGNXCLwdFTXfDFd",
  "rqxtjyDBPBWq5qdVMxWjUdtZaxw\nOxcuMlfXdWSvC8Ey85qSRrGNXCLwdFTXfDFdxEBwAGKRE5ArXkKbNj9REi4jCktPAgY\ne5UO7hsJnI9K0MjyFnLS0F0SDQiH5sGWtEgumMmspO6CE4c3QAm40zAIMFX4moOdgqn\nRtJ7hQ6OzurQx3INmKuVNF3DJMZP2itqGqSEqnHL+gVbz5g6mSQuL5qhahtB1q5uO0b\nTvKh+2kiyIJFmLatJoIsCQdF",
  "2itqGqSEqnHL+gVbz5g6mSQuL5qhahtB1q5uO0b\nTvKh+2kiyIJFmLatJoIsCQdFn8F+H03LsPF1FtqIXxUKq4IszI7Oo3bfhY3gtTksYL+\n0vbWapP+MoYzYAOw+yuYinlbX81ndjhNzlnj2wIfhgOYrHYVe+I1lzXtBK5qEhtRs8k\nVMm2IKTz87ZpR+NReSHaF2gDeNVWqjkDe12U4Ila8Pd23CpupL86LvFH/jwuF6y28b\n+h2QTGiqrwteQ",
  "eSHaF2gDeNVWqjkDe12U4Ila8Pd23CpupL86LvFH/jwuF6y28b\n+h2QTGiqrwteQDb9FQ324NeH1BRE8eblEkweBZvJyCec7mjqm8cK2kWbuoCAUk8JcoO0\nvUtWu0TwYPMjRUCtl34ZUKhSU6StmwDVoZfuMl6FlCMLjIeX2Ms87LSnBx+aD1DpNH\ntsaiFvVm1D1Rphfa5weWsFpTh5nDGL6keoYxG43xGeaX6TKNkDu2UDl92SwNbzLf7myk",
  "vVm1D1Rphfa5weWsFpTh5nDGL6keoYxG43xGeaX6TKNkDu2UDl92SwNbzLf7myk\nfF71Wyk83Jv3BuGB2qjmp70NPB8psagjUVvwVONtSxL0x+0NVub46s3nj5LVnaqcf\n1m5K0Oxml3/a4l4yAn256RrtJPGJR6K2JiOkHrE8/UFb/jxu+q7C4/pNSdqd5tFre9y\nZiZ/sjvghtnHpFz27WNfLrvjEBYNFY1XzDOeInEcwmJWtS34f6zsCLh5tK",
  "Fre9y\nZiZ/sjvghtnHpFz27WNfLrvjEBYNFY1XzDOeInEcwmJWtS34f6zsCLh5tK1xCIudUrQ\n1G8BSn0t8CeMQFsdbuG1OYljd9KibfpXJYoDMcQiLj1iGr3ocwmJKxdQrnrCiQOI4RPI\n4wHkc0DwWCp8Ep6RwjMjZEn5FpQe5G3JBrA0RL0NPZ3BCGSuUIeTIJZLuvJK78pTaBU\nruor3fB3vXdKxYahBG8DSFtljYXfLu8kinGJ4zPIluRDI",
  "IeTIJZLuvJK78pTaBU\nruor3fB3vXdKxYahBG8DSFtljYXfLu8kinGJ4zPIluRDIKmgCO9jpUGf69BclNXmSg1d\niRy8oPXf0nNIDRw8o1Y6SN4IoeYoeTuJkjNHzyjd3Sf0srRitI9R/coTRxNKH3o6EN\nKY0djSlcdXaXUOEqeSOGO4OgupQNHB5QeOnpI6XNHn1P62NHlL5w9AWlrx19Tel9R+9\nTyhxlK45ukYpd5R8OoiSFUdXKI0cJe9+s",
  "6XNHn1P62NHlL5w9AWlrx19Tel9R+9\nTyhxlK45ukYpd5R8OoiSFUdXKI0cJe9+sNc7VBaOFpQ+sDRB5T2HSVvxXA/c5Q83sC\nN0VFJ6bqj65QKR8n7W5Q8dfQpZmjGaVPH1C6StHX1H6yNFHlKaOkm8D8HTi6A6l7it\nQXVK67eg2paeOnvq/C/DZNEa+hbnlGtiNHc0p3TDUfKmAI8Sjp6Q58lETU616dcmcq4\nlasY9bJLxaW2S80TNuIdNTq",
  "nlGtiNHc0p3TDUfKmAI8Sjp6Q58lETU616dcmcq4\nlasY9bJLxaW2S80TNuIdNTqdpbXI+JWrGB2Toa/uzDymQUjpe3MLy/grLC3sf7+4/OP\nine07C/dWJl9orwVfBV8Ht4Ll4KfgXvA46AR7QRz8FfwT/Bv8N9+b/23+9/k/xurVK5M\n6XwStv/k/wfV+PeL</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi, \u03c6)Pr(\u03c6)\n#\n<latexit sha1_base64=\"eoOV",
  "eL</latexit>\n\u02c6\u03c6 = argmax\n\u03c6\n\" IY\ni=1\nPr(yi|xi, \u03c6)Pr(\u03c6)\n#\n<latexit sha1_base64=\"eoOVoDAQYf5iq\nhHG6TNOW3FC4lI=\">AW8niclZhbT9xGFICX9JYmaUtalZe+WEWRqjZFUKWXl0gJhCQ\nEUiCwQI3aOwdeyeMx8YewxJr/0jfqr72D1Xqj+kZ27uDzxkeuhLs7Pk+z4zPXHwJMik\nKvbz8z9yNDz786ONPbn56/adz7/Yv7ulwdFWuYh74epTPOj",
  "s7Pk+z4zPXHwJMik\nKvbz8z9yNDz786ONPbn56/adz7/Yv7ulwdFWuYh74epTPOjgBVcCsX7WmjJj7KcsyS\nQ/DA4XTP8JznhUjVvr7M+CBhsRKRCJmG0Mn8uT9iuvKDKBuJifQ81keJ0KdTEO+5JE\n+9osyOanEw5XJ2pj4nMp4dfkGJyxKdyHwqUpDLwfPF9C60Pm+eEw1Z6fBOm4imsX6hv\n4uYhHenAyv7i8tFx/PFpYaQuLvfazc3L36E/T",
  "F9C60Pm+eEw1Z6fBOm4imsX6hv\n4uYhHenAyv7i8tFx/PFpYaQuLvfazc3L36E/TMy4UqHkhXF8cpypgcVy7UIJZ/c8su\nCZyw8ZTE/hqJiCS8GVZ2giXcPIkMvSnP4U9qro1ePqFhSFJdJAGbC9KjAzARd7LjU0W+\nDSqis1FyFTUNRKT2deib3lDkPNTyEgoszAX01QtHLGehjG5St+EaZJwtSw8lfXdye\nQdR4LVfGzsh6fyaTrNcOh+J1xur",
  "goszAX01QtHLGehjG5St+EaZJwtSw8lfXdye\nQdR4LVfGzsh6fyaTrNcOh+J1xurG/qwWoXki3nNSa2YSq4ReDypKr4UL2EgOACxAl\nIFS+gTpOfIPJWEIX5KAFXzVyAieC9mpCqleYx5KSjvSEaFDLJx1rjVgwlElH2QPF8+5\n5BnCdwyhAV+GLozHYy5iaTI/TfKzpCpMDLeQMxXzugk45ZBJc0ZdQ5VSwqFhx/odW6+\nYOm0Tl2Z1V3MTQdZ+",
  "TI/TfKzpCpMDLeQMxXzugk45ZBJc0ZdQ5VSwqFhx/odW6+\nYOm0Tl2Z1V3MTQdZ+3nV0TvOihl2njiALJmHcteoIspr1mzDIcluG3SBPBNxq0JhVZC\nJuZOnQbftzETw3BxnsF63npF0n/OUEZMAFaf+RZMhbyr6Uz25sm57z2TYGPvREMVve\nQZhu80gicVRubULPOFTJptiCUpxd0/TGofJMdE/QBPCiK3Ohoiva/boEU9aE/ftwqnk\np+fGPS",
  "RubULPOFTJptiCUpxd0/TGofJMdE/QBPCiK3Ohoiva/boEU9aE/ftwqnk\np+fGPSz/z8aBaNsvG/CPZhIqKMnNVZML/o6IhXK/w/IHrxUosGDQD14qYT9HQ0dy/H\nENpF67KAgFJNCX6LlL2LVPaO4M6mCeorBEy98M2EQoMcRV3ZBIwM3DldUygEJ1k2Jx\njKNOizDnZ/NB8hkitm20xF+Zi1d1QpRG6+waXs6OgDBeHc37N4QHKaNDkM0hLNWQ",
  "jKNOizDnZ/NB8hkitm20xF+Zi1d1QpRG6+waXs6OgDBeHc37N4QHKaNDkM0hLNWQ5Sub\nYDOn4rV9oWGKu1V8PeVN0WjE/2zbg37B6JRhyM9ONvF4xMSijkR1wa2Osy5JLEd7UNd\nsul7tWbX59nsytWOH6zYlqbftpdt2uNf0gJ9tOXq7RTxiUeiutoeUo9YjvagLncet1x\nn4XDdpiT1TvPotB3uzETP9ofc3MbVIqh+a2L5V+E8KipqJ2imnC",
  "YjvagLncet1x\nn4XDdpiT1TvPotB3uzETP9ofc3MbVIqh+a2L5V+E8KipqJ2imnCYyQ2ISwmZdeC31j\nZE3Dx6FpNCIs7hehqJoClIZf4FJoQFpsl3DXbGFa3HOqW2UyGyGzCWHxGUvwWTchLMZ\nUjJ3iKcsyJDYhkscRzuOI5jHDUuaS8IhkjhEhU8o1ofJR2pVMAEtj1NrY0Rj0QKYKNdg\nGsVzQmVc4Z5Cs1jRWdx3Ndy/pmHNUIUmgKVtsY8",
  "R2pVMAEtj1NrY0Rj0QKYKNdg\nGsVzQmVc4Z5Cs1jRWdx3Ndy/pmHNUIUmgKVtsY8f9u5yAKcYrjNciU5E8jKaAJ3sLN\nDnendXxBV5E4OnowtvaT0wtILSg8tPaQ0t5Q8EQTRK0vJ0kQnVt6TumBpQeUlpaWlPY\nt7VMaWRpR+tTSp5SGloaUrlm6Rqm2lNyRwhXB0n1KR5aOKD2y9IjS15a+pvS5pc8pfWP\npG0rfW/qe0seWPqaUWcoXbd0nVJ",
  "RwhXB0n1KR5aOKD2y9IjS15a+pvS5pc8pfWP\npG0rfW/qe0seWPqaUWcoXbd0nVJuKXl1ESrlq5SGlhKnv1grVm6Q2lmaUbpE0ufUDq\n0lDwVw/XMUnJ7AxdGSyWlG5ZuUCosJc9vQfTS0peUJpYmlL6w9AWl7yx9R+kzS59RGlt\nK3g3A3Ymle5Tat0BVQemupbuUnl65n4vwGfDGLgm5ratYJvS1NKU0k1LyZMC3EpYekr\nuJyPV7mrTt01kX4",
  "mupbuUnl65n4vwGfDGLgm5ratYJvS1NKU0k1LyZMC3EpYekr\nuJyPV7mrTt01kX4vUjDtYm/Hp0STnkZpxB2t3p+nRZH+K1IyPSNfXD2YvUiClsNOfzC+\nu4LewtHDw09LKL0sPdh8sPlpt39De7H3T+7b3XW+l92vUe95b6fX74W9f+duzN2eu7O\ngF/5Y+HPhr0a9Mdce81Wv81n4+z+5GP2f</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi] +",
  "HPhr0a9Mdce81Wv81n4+z+5GP2f</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\" I\nX\ni=1\n`i[xi, yi] + \u03bb \u00b7 g[\u03c6]\n#\n<latexit sha1_base64=\"+32o\n3Z8YObLBiG84adcJ1rUvCIc=\">AWrniclZhb9s2FIDV7tZ1t\n3TD8rIXYUGBblgDp+guLwPapOkt6eI01zZ2DUqmZDYUpUhU4lTw\n/9iv2ev2F/ZvdijJZnUO8zADqdnzfeLlkJRoBZkUhe71/r12/Y\nMP/r4kxuf3v",
  "w\n/9iv2ev2F/ZvdijJZnUO8zADqdnzfeLlkJRoBZkUhe71/r12/Y\nMP/r4kxuf3vzs8y+/Grp1teHRVrmIT8IU5nmxwEruBSKH2ihJ\nT/Ocs6SQPKj4HTD8KNznhciVfv6MuPDhMVKRCJkGkKjpXsDCfKY\n+YNwnGp/kATptIpnJ4MgyiZi6P/u3x3IND7p53ea0A/D0dJKb7V\nXf3xaWGsLK176Y9ufTsejNOwTLjSoWRFcbLWy/SwYrkWoeSzm",
  "3ea0A/D0dJKb7V\nXf3xaWGsLK176Y9ufTsejNOwTLjSoWRFcbLWy/SwYrkWoeSzm4\nOy4BkLT1nMT6CoWMKLYVUPbubfhsjYj9Ic/pT26+j7V1QsKYrLJ\nAzYXpSYGaCLnZS6ui3YSVUVmquwqahqJS+Tn2TKX8sch5qeQkF\nFuYC+uqHE5azUEM+bw4UvwjTJGFqXA3WN3dn1SDgsVAVPyvr3M\n5mXWezdjgUrzLWn+0vahGaJ+IdJ5XUiqnkCo",
  "TJGFqXA3WN3dn1SDgsVAVPyvr3M\n5mXWezdjgUrzLWn+0vahGaJ+IdJ5XUiqnkCoHs6riq/EqBoIDE\nKucgFTxAuo0+Qkifw1RWEsScNUsDFgC/sZqVpHkNOtprokEh\nk3zasTaIBVOZdJQ9UHz/tm8A1znMAnQVvjiag72Mqdn8Os2nOk+\nqwsRwCzlTMa+bgCGHTJoRdQ1VSgmXh3rD2y9ZOq0TVya1V3NTQ\nRZ+3nX0TnNixp3nTqCLFiEcde",
  "bgCGHTJoRdQ1VSgmXh3rD2y9ZOq0TVya1V3NTQ\nRZ+3nX0TnNixp3nTqCLFiEcdeqI8hqNnPCIMteQDTnwTcatCY\nVWQhdnP06DbdmYieG1OM9gvXW+zIuk/ZygjJgC7z3wLpkLe1TfS\nhe3Pk3Ne+6bAp/4EJqt7CcvjZljzRmBUbWxGzTpXyKTZglCeXn\nRN0xuHyjPRHaAJ4E1X5kJF72k/1SVYsiY8+AmGmpeSn9xd/ZlPh\n1XPbBvzD8kmV",
  "RN0xuHyjPRHaAJ4E1X5kJF72k/1SVYsiY8+AmGmpeSn9xd/ZlPh\n1XPbBvzD8kmVFSUmasiE/4fFY3hWYPXF0Tw5KUSTR4E6slLJdzf\n0dSxHC9sE6nDgpCMSn0Jdr+Ilbda+oI7myaoL5CwNQL30woNMl\nR1JVNwMjwDU9NxwIK0SDZoyhTIsy5+Tmh9YzRGrd3BZzYR5W3R\nuqNEL3vsHl4iow8PhnF9xeYAyGjT5DNJSjVmOkjk1Uzp9Myg0b",
  "Grd3BZzYR5W3R\nuqNEL3vsHl4iow8PhnF9xeYAyGjT5DNJSjVmOkjk1Uzp9Myg0b\nDHX7q+nvCk6rZifbXtQb9gdsow5GejLTwfMbGoI1FdcEx1iWJ\n5WgP6los1/d7Vm29+ZEs7djhuk1J6m176bYd7hU94Gfbjt5uE4\n9Y1JGoraH1COWoz2oy53HbdcoHK7blKTeR6dtsNdmGj5R/sTr\npk5JqVybI59qRw0ISxqKmqnmCY8RmITwmJSdi34P",
  "K7blKTeR6dtsNdmGj5R/sTr\npk5JqVybI59qRw0ISxqKmqnmCY8RmITwmJSdi34P1b2BDw8ulYT\nwmK/EF3NBLA05hIPoQlhsdnCXbONYXboW67VSazCTKbEBafsAS\nPuglhMaZi7BRPWZYhsQmRPE5wHic0jxmWMpeEZyRzAhZUq4FlU\n/SrmQCWJqi1qaOxqAHMlWowTaI5YKuvMK58hRaxYqu4gNXwdXN\nKwZqtAEsLRD9pg/2HFusgCnGI5Z",
  "xqAHMlWowTaI5YKuvMK58hRaxYqu4gNXwdXN\nKwZqtAEsLRD9pg/2HFusgCnGI5ZriRnAlkZTWAfO3qzE9/QVSR\nk1wQXVp6SemFpReUHl6RGluKflFEQvLSW/ToLo3NJzSg8tPa\nS0tLSk9MDSA0ojSyNKH1v6mNLQ0pDSDUs3KNWkhMpPBEs3ad0Y\numE0mNLjyl9ZekrSp9a+pTS15a+pvSdpe8ofWjpQ0qZpYzSTUs3\nKeWklcHQbRu6Tqlg",
  "mNLjyl9ZekrSp9a+pTS15a+pvSdpe8ofWjpQ0qZpYzSTUs3\nKeWklcHQbRu6TqlgaXktx/sNUv7lGaWZpQ+svQRpWNLya9ieJ5\nZSo438GC0VFL6zNJnlApLye+3IHph6QtKE0sTSp9b+pzSt5a+pf\nSJpU8ojS0l7wbgdGLpHqX2LVBVULpr6S6lZ5aeud8L8MU0Bq6Fu\nWMr2KE0tTSldMtS8ksBjhKWnpLzZKTau9r8bRO5r0VqwR2szfj\n8a",
  "L8MU0Bq6Fu\nWMr2KE0tTSldMtS8ksBjhKWnpLzZKTau9r8bRO5r0VqwR2szfj\n8apLzSC24g7V3p/nV5P4UqQWfkK5vHi5epEBK4U4/WlpZw29hae\nHw3uraL6v3d+vPFhv39De8L7zvfueGver94D76nX9w680PvT+\n8v72/tnubd8uDxcHjXq9WvtNd94nc/y5D/oveJL</latexit>\u03bb \u00b7 g[\u03c6] = \u2212 log[Pr(\u03c6)]",
  "L2 Regularization\n\u2022 Most common regularizer is L2 regularization \n\u2022 Favors smaller parameters (like in previous example)\n\u2022 Also called Tikhonov regularization, ridge regression\n\u2022 In neural networks, usually just for weights\n<latexit sha1_b\nase64=\"yLcdtLvFHAjmNureiWm0K\np+NsoI=\">AW7niclZhbT9xGFICX\n9JI0vZFW5aUvVlGkqUIovTyUimBk\nBukQGCBJPV2Dv2ThiPjT2GJdb+j\nb5Vfe1Pav9M1TO2dwefMzx0JeLJ+T\n7PjM9cfAkyKQq9svLP3I3v/gw5u\n3Pr98Sefvb5/J0vDoq0zEPeD1",
  "Mzx0JeLJ+T\n7PjM9cfAkyKQq9svLP3I3v/gw5u\n3Pr98Sefvb5/J0vDoq0zEPeD1O\nZ5kcBK7gUive10JIfZTlnSD5YXC6\nbvjhOc8Lkap9fZnxk4TFSkQiZBpCg\n/ncHzFd+UGUjcTE+9XzWR4nQg2mI\nV/ySB/7SZCOq63JcRNd8nzDx4NKTJ\nagcGkK/uTE+97zJbQ9ZJ5flMmgejv\nxQTfHN9W9iZ+LeKRPBvOLK8sr9c+\njhdW2sNhrfzuD",
  "TE+97zJbQ9ZJ5flMmgejv\nxQTfHN9W9iZ+LeKRPBvOLK8sr9c+\njhdW2sNhrfzuDO18N/WEalglXOpSs\nKI5XVzJ9UrFci1DyW2/LHjGwlMW8\n2MoKpbw4qSqkzPx7kJk6EVpDn9Ke\n3X06hkVS4riMgnATJgeFZiZoIsdlz\nr65aQSKis1V2HTUFRKT6eybQ3FDk\nPtbyEAgtzAX31whHLWahPG7il+\nEaZIwNaz8tY3dCWSUx0JV/Kysx2Yy\n6",
  "Q3FDk\nPtbyEAgtzAX31whHLWahPG7il+\nEaZIwNaz8tY3dCWSUx0JV/Kysx2Yy\n6TobtcOheJ2x9mx/VovQPBHvOKmkV\nkwl1wg8nlQVX46XMRAcgFjmBKSKF\n1CnyU8QeauIwlyUgKtmBsFc8V5OSN\nVK8xhy0tFeEw0KmeTjrVOLBjKpK\nPsgeJ5dz0DuM5hFKCrcOBoDPYypib\nT8zQf6zypChPDLeRMxbxuAi45ZNJc\nUdQpZRwatixfsPWS",
  "5hFKCrcOBoDPYypib\nT8zQf6zypChPDLeRMxbxuAi45ZNJc\nUdQpZRwatixfsPWS6ZO28SlWd3V\n3ESQtZ93HZ3TvKh16kjyIJGHetO\noKsZvUmDLclmEnyBPRNyqUFgVZG\nLu5GnQbTszETw3xmsl63UZH0nz\nOUEROA1WeOgqmQd/X1dGZ70+Sc174\np8LE3gsHqntJsgVcagatqYxNq1rlC\nJs0WhPL0omua3jhUnonuBZoAXnRl\nLlR0RVuq",
  "E3gsHqntJsgVcagatqYxNq1rlC\nJs0WhPL0omua3jhUnonuBZoAXnRl\nLlR0RVuqSzBlTdhfgkvNS8mPf1j+k\nY9PqhWzbMw/JtQUVFmropM+H9UNI\nR7FZ5fEMGDl0o0eBCoBy+VsL+joW\nM5ntgmUo8dFIRiUuhLtPxFrLrn1BH\nc2TRBfYWAqReOTCg0yFHUlU3AyHCE\nu65jAoXoIsPmGkOZFmXOyeaH5jNE\nat1si7kwN6vuhiqN0N03uJy",
  "FHUlU3AyHCE\nu65jAoXoIsPmGkOZFmXOyeaH5jNE\nat1si7kwN6vuhiqN0N03uJydBW4O\nZza04PUEaDJp9BWqohy1Eyx2ZIx2\n/8QsMSc63+esibotOK+dlm2x70C0\nanDEN+NtjE4xETizoS1QWPOc6JLE\nc7UFds+l6tWfV5pvyNSOHa7blKTe\ntpdu2+Fe0wN+tuXo7RbxiEUdiepq\ne0g9Yjnag7rcedxyXYXDdZuS1DvNo\n9N2uDMTf9of",
  "Fe0wN+tuXo7RbxiEUdiepq\ne0g9Yjnag7rcedxyXYXDdZuS1DvNo\n9N2uDMTf9of8Q1M49JqRyax75U+\nk0Ii5qK2imCY+R2ISwmJRdC/6PlT\n0BN4+u1YSwuFOIrmYCWBpyiS+hCWG\nxWcJds41hdcuhbrlVJrMRMpsQFp+\nwBF91E8JiTMXYKZ6yLENiEyJ5HOE8\njmgeMyxlLgmPSOYETKlXBMqH6Vdy\nQSwNEatjR2NQ9kqlCDbRDLBZ15h",
  "J5HOE8\njmgeMyxlLgmPSOYETKlXBMqH6Vdy\nQSwNEatjR2NQ9kqlCDbRDLBZ15h\nXPmKTSLFZ3FfVfD/Wsa1gxVaAJY2i\nZrzPO3nYswCmGxyxXkjOBrIwmcAc\n7O9SZPv0FUWe5ODN2NJLSi8svaD\n0NJDSnNLyRtBEL20lLydBNG5peU\nHlh6QGlpaUlp39I+pZGlEaWPLX1Ma\nWhpSOm6peuUakvJEyncESzdp3Rk6\nYjSI0uPKH1l6StKn1r",
  "9I+pZGlEaWPLX1Ma\nWhpSOm6peuUakvJEyncESzdp3Rk6\nYjSI0uPKH1l6StKn1r6lNLXlr6m9J\n2l7yh9aOlDSpmljNINSzco5ZaSTwd\nBtGbpGqWBpeTdD9apTuUZpZmlD6\ny9BGlQ0vJWzHczywljzdwY7RUvrM\n0meUCkvJ+1sQvbD0BaWJpQmlzy19T\nulbS9S+sTSJ5TGlpJvA/B0Yukep\nfYrUFVQumvpLqVnlp65vwvw2TAGro\nm5bSv",
  "ulbS9S+sTSJ5TGlpJvA/B0Yukep\nfYrUFVQumvpLqVnlp65vwvw2TAGro\nm5bSvYpjS1NKV01LypgCPEpaekuf\nJSLW72vRrE9nXIjXjDtZmfHo2yXm\nkZtzB2t1pejbZnyI14yPS9Y2D2YcU\nSCns9IP5xVX8FZYWDu4tr/60fH/3\n/uKDtfYL7a3e171vet/2Vns/9x70n\nvZ2ev1e2Pu79+/czblbC9nC7wt/LP\nzZqDfm2nO+7HV+C3/9B3",
  "et/2Vns/9x70n\nvZ2ev1e2Pu79+/czblbC9nC7wt/LP\nzZqDfm2nO+7HV+C3/9B3Gj/Po=</\nlatexit>\n\u02c6\u03c6 = argmin\n\u03c6\n2\n4L[\u03c6, {xi, yi}] + \u03bb\nX\nj\n\u03c62\nj\n3\n5",
  "Why does L2 regularization help?\n\u2022 Discourages fitting excessively to the training data (overfitting)\n\u2022 Encourages smoothness between datapoints",
  "L2 regularization (simple net from last lecture)",
  "PyTorch Explicit L2 Regularizer\nhttps://pytorch.org/docs/stable/generated/torch.optim.SGD.html \nhttps://pytorch.org/docs/stable/generated/torch.optim.Adam.html",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Implicit regularization\nGradient descent approximates a \ndifferential equation \n(infinitesimal step size)\nApproximate implicit \nregularization added to \ncontinuous gradient descent\nAdd in regularization to D.E of \n~\n\"\n!\"\n!#\n$ and differential \nequation converges to same place\nGoing to infinitesimal (continuous) \nstep, change in \ud835\udf19 governed by: \n!\"\n!# = \u2212 !$\n!\"",
  "Implicit regularization\n\u2022 Gradient descent disfavors areas where gradients are steep\n\u2022 SGD likes all batches to have similar gradients\n\u2022 Depends on learning rate \u2013 perhaps why larger learning rates generalize better.\n<latexit sha1_base64=\"AHdfomwX/HC\nsyTnvfCJdX/0ABPc=\">AW/3iclZhb9xEFIA3XEu5NAURgXixiCohKFSlcsLUps\n0Tduk5LpJ2jiNxt6xd5rx2LHyaWHxA/hjfEKz+Fn8C/4IzXuxOfM3kgUtnhfJ/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuhe",
  "/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuheF\n8LflhlnOWBJIfBKcrh+c87wQqdrTlxk/TlisRCRCpiF0Mvu752shB7zykyAdVRt\n1fVKtPaqP/CDKhuLY+8WbgGnoO8+PchZWPpPZkNXV/drzJY+0L/d5rluWsVwLJr2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQo",
  "r2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQoWVEcLS\n1m+rgy7YS1zf9suAZC09ZzI+gqFjCi+OqyVbt3YHIwIvSHP4p7TXRq1dULCmKyQA\nM2F6WGBmgi52VOro5+NKqKzUXIXjhqJSejr1TOq9gch5qOUlFiYC+irFw4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8",
  "w4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8QbTipFP\nJNQKP6riC/ECBoIDEAucgFTxAuo0+QkibwlRWJwScLveYEl4OzWpWmkeQ0462kuiQ\nSGTfNSxVogFU5l0lF1QPO+OZwDXOcwCdBV+OJqD3YypenKd5iOdJ1VhYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoO",
  "hYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoOs0EWTBIoy7VhNB\nloSjZMASBluycw4MQzEbcqFYFWZhbeRp0285MBK/NUQb7peutViT95wxlxARg9\n5lfwVTIu/pKOrW9SXLOG98U+MgbwmR1L2F5PB7WpBEYVRurqdnkCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4",
  "kCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4WzbYx/yHZhIqKMnNV\nZML/o6IB3Lzw+oInrxUosmDQDN5qYTzHU0dy/HCNpFm7qAgFJNCX6LtL2LVvaJ4M\n6mCeorBEy98MuEQpMcRV3ZBIwMv3AbdiygEA0yHI8xlGlR5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5Sub",
  "R5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5SubITOnolV9o2GKu3d9M+bjot\nGJ+t62B/2C2SnDkJ+drOP5iIlFHYnqguceZ12SWI72oK7pcr3as2r91bdkacO12\n1KUm/bS7ftcK/pAT/bcPR2g3jEo5EdbU9pB6xHO1BXe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0g",
  "Xe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0gch7CYlF0L/h8ruwJuHl1rHMLiV\niG6mglgacAlHsI4hMXxFu6abQyrGw51w62LwxXzXEIi2swaMeh7AYUzF2iqcsy5A\n4DpE8DnEehzSPGZYyl4RnJHPMCFlSrgWVD9OuZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vm",
  "uZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vmbzk0W4BTDY5YryZlAVkYTuIWdLepMnv\n6CqCJPckF0aeklpReWXlB6YOkBpbml5I0giHYsJW8nQXRu6Tml+5buU1paWlLat7RP\naWRpROljSx9TGloaUrpi6Qql2lLyRAp3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKu",
  "Ap3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKuaXk0EQLVu6TGlgKXn3g71m6RalmaUZpY8sfUT\npwFLyVgz3M0vJ4w3cGC2VlD619CmlwlLy/hZEzy19TmliaULpM0ufUfra0teUrlm6\nRmlsKfk2AE8nlu5Sar8CVQWl25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXM",
  "25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXMtUlPuYG3GJ1eTnEdqyh2sPZ0mV5PzKVJTPiRdX92fki\n/O3Jr5YubLud/m/pj7c+6vsfrWTHvNZ73O39zf/wGIjQK+</latexit>BlMJfzI7v4S/wtLC/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] +",
  "/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\nSee derivation at end of UDL Ch. 9",
  "\u2022 Gradient descent disfavors areas where gradients are steep\n\u2022 SGD likes all batches to have similar gradients\n\u2022 Depends on learning rate \u2013 perhaps why larger learning rates generalize better.\nImplicit regularization\n<latexit sha1_base64=\"AHdfomwX/HC\nsyTnvfCJdX/0ABPc=\">AW/3iclZhb9xEFIA3XEu5NAURgXixiCohKFSlcsLUps\n0Tduk5LpJ2jiNxt6xd5rx2LHyaWHxA/hjfEKz+Fn8C/4IzXuxOfM3kgUtnhfJ/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuhe",
  "/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuheF\n8LflhlnOWBJIfBKcrh+c87wQqdrTlxk/TlisRCRCpiF0Mvu752shB7zykyAdVRt\n1fVKtPaqP/CDKhuLY+8WbgGnoO8+PchZWPpPZkNXV/drzJY+0L/d5rluWsVwLJr2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQo",
  "r2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQoWVEcLS\n1m+rgy7YS1zf9suAZC09ZzI+gqFjCi+OqyVbt3YHIwIvSHP4p7TXRq1dULCmKyQA\nM2F6WGBmgi52VOro5+NKqKzUXIXjhqJSejr1TOq9gch5qOUlFiYC+irFw4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8",
  "w4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8QbTipFP\nJNQKP6riC/ECBoIDEAucgFTxAuo0+QkibwlRWJwScLveYEl4OzWpWmkeQ0462kuiQ\nSGTfNSxVogFU5l0lF1QPO+OZwDXOcwCdBV+OJqD3YypenKd5iOdJ1VhYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoO",
  "hYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoOs0EWTBIoy7VhNB\nloSjZMASBluycw4MQzEbcqFYFWZhbeRp0285MBK/NUQb7peutViT95wxlxARg9\n5lfwVTIu/pKOrW9SXLOG98U+MgbwmR1L2F5PB7WpBEYVRurqdnkCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4",
  "kCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4WzbYx/yHZhIqKMnNV\nZML/o6IB3Lzw+oInrxUosmDQDN5qYTzHU0dy/HCNpFm7qAgFJNCX6LtL2LVvaJ4M\n6mCeorBEy98MuEQpMcRV3ZBIwMv3AbdiygEA0yHI8xlGlR5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5Sub",
  "R5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5SubITOnolV9o2GKu3d9M+bjot\nGJ+t62B/2C2SnDkJ+drOP5iIlFHYnqguceZ12SWI72oK7pcr3as2r91bdkacO12\n1KUm/bS7ftcK/pAT/bcPR2g3jEo5EdbU9pB6xHO1BXe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0g",
  "Xe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0gch7CYlF0L/h8ruwJuHl1rHMLiV\niG6mglgacAlHsI4hMXxFu6abQyrGw51w62LwxXzXEIi2swaMeh7AYUzF2iqcsy5A\n4DpE8DnEehzSPGZYyl4RnJHPMCFlSrgWVD9OuZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vm",
  "uZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vmbzk0W4BTDY5YryZlAVkYTuIWdLepMnv\n6CqCJPckF0aeklpReWXlB6YOkBpbml5I0giHYsJW8nQXRu6Tml+5buU1paWlLat7RP\naWRpROljSx9TGloaUrpi6Qql2lLyRAp3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKu",
  "Ap3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKuaXk0EQLVu6TGlgKXn3g71m6RalmaUZpY8sfUT\npwFLyVgz3M0vJ4w3cGC2VlD619CmlwlLy/hZEzy19TmliaULpM0ufUfra0teUrlm6\nRmlsKfk2AE8nlu5Sar8CVQWl25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXM",
  "25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXMtUlPuYG3GJ1eTnEdqyh2sPZ0mV5PzKVJTPiRdX92fki\n/O3Jr5YubLud/m/pj7c+6vsfrWTHvNZ73O39zf/wGIjQK+</latexit>BlMJfzI7v4S/wtLC/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] +",
  "/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n<latexit sha1_base64=\"r\nrKZAMUeDGcwur29Xw20pAvpE0=\">AYQHiczZhbT9x\nGFIA39JbSW9Kq4qEvVlGqKhcEUXp5iZRACEkgZbksEDC\nsxt6xd8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8OnjNI7Vt\nXgp093+e5nJnxLcg4y9Xi4u8",
  "d8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8OnjNI7Vt\nXgp093+e5nJnxLcg4y9Xi4u83Zt57/4MP7r58ewn372\n+Re3bn+5l6eFDGkvTHkqDwKSU84E7SmOD3IJCVJwOl+\ncLqi+f45lTlLxa6zOhxQmLBIhYSBaH+7Ru/eb5ifEBL\nPwnSUblRVf1yZ+1ZdeQHUTZkx953jx3GFeGeH0kSlj7h\n2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYh",
  "UTZkx953jx3GFeGeH0kSlj7h\n2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYhUjHBvA\n6SqMr/HNVbeA8WHZIvWTxUvtT1npQPK1+kokgCKj3fn\n6173XR3OpJ7ntVqOT6v2LJv8vQ+/fml9cWKw/Hi4sN\nYX5TvPp9m9/PfAHaVgkVKiQkzw/WlrM1HGp2wk5rWb9I\nqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJM\nnzyQAMy",
  "p2wk5rWb9I\nqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJM\nnzyQAMyFqmNtMB13sqFDRz8clE1mhqAjHDUF91Tq6b\nXrDZikoeKXUChZNBXLxwSJeCFT7rC3oRpklCxKD0l1\ne3IHEBjZko6VlRr/aqajurtUOheJ2x/HJ3WgtTNGHvK\nqkVnQl1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3\nNATe7EZaEt12hqoWiMeSkpR",
  "l1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3\nNATe7EZaEt12hqoWiMeSkpR0iDQoZp6OWtYIsmMqkpey\nA4nl3PA2okjAL0FX4otYc7GREVJPjFB0pmZS5jtktSCJ\niWjcBQw5hoW/bhig4h0PDlvWLbW0TcdokLs3qrkodsax\nd2XaUxHkRg7ZTRywLFmHctuqIZXE4Fw9IQiDLTbkPA048\nHXGrTNgqQwuzK9Og3XamI/baHGWwX9reao",
  "LFmHctuqIZXE4Fw9IQiDLTbkPA048\nHXGrTNgqQwuzK9Og3XamI/baHGWwX9reaonSf06sjOgA\n7D79zYgIaVtfSae2N0nOe3rAh15Q5is9iFExuNhTRqB\nUTWxCpt1riwTZwtCMr1om7o3DpVmrD1AHbA3XSGZiK5o\n9+sSLFkd9u/DUGXB6dGDhR/o6Lhc1NtG/0PZhIryInNV\npMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSz\nx",
  "1NtG/0PZhIryInNV\npMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSz\nx0UmCcqUtr+7NYtI+pI3Zn08TqKwR0vfBNmLAmOYras\ng5oGb7hPsaxgEJrkOF4jCFP80JSdPKz1jNEal2fFiXTF\n6v2CZVroX3eoHx6FJTh4nBOrzk8sDIajPMZpIUYEGklc\n6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez\n5iZGHW3XBjaOzLo",
  "Gklc\n6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez\n5iZGHW3XBjaOzLo4sR3tQ13S5Xu1ZuX5yFy3t2OG6TY\n7qbXrpth3uNT2gZxuO3m4gD1nY4VZdTQ+xhyxHe1CXO4\n8brlE4XLfJUb2TPDpthzs1reUf7Q6pIvo2KeUDfduXcn\n8cskWFReU04TGljgO2WJStC34bSs7DC4ebWscsVuztq\naDtjSgHJ7COQLY63cNtsYra64VA3G",
  "04TGljgO2WJStC34bSs7DC4ebWscsVuztq\naDtjSgHJ7COQLY63cNtsYra64VA3GrzoHVHIdscY0\nk9qjHIVuMsRg7xVOSZY4DqE8Du08DnEeM1vKXJI9I5l\njRtCSci0oOUzbkg7Y0shqbeRoDHrAU2E12ARtOcrL3e\nuPGtYoFXc/VcO+ahWxKtQBW9pEe8zN52bLBTDLdZ\nriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h\n+5hKQ",
  "8zN52bLBTDLdZ\nriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h\n+5hKQ9ETQRBtG4qeToLo3NBzTPcM3cO0MLTAtGdoD9PI\n0AjT54Y+xzQ0NMR0xdAVTJWh6I4UrgiG7mI6NHSI6YGh\nB5i+MfQNpi8MfYHpoaGHmL4z9B2mTw19ikxlGC6augqp\ntRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUP\nRXD9cxQdHsDF0ZDOa",
  "p\ntRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUP\nRXD9cxQdHsDF0ZDOaYvDX2JKTMUPb8F0WtDX2OaGJpg+\nsrQV5i+NfQtpmuGrmEaG4reDcDdiaE7mJq3QGWO6ZahW\n5ieGXrmfi9Ap9MYuBbmpqlgE9PU0BTdUPRkwLcSh6iu\n4nI9Gc1SZvm9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56\ndITPkQdX1b/oiBVIKZ/r+rfkl+y0sL",
  "9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56\ndITPkQdX1b/oiBVIKZ/r+rfkl+y0sLuw9XFj6ceHR1q\nP5J8vNG9qbnW863a+7yx1fuo86bzodDu9Tjhzd6Y782\nit>bmcO7XuT/m/pz7a6zO3GiO+arT+sz9/Q8KAH6+</latex\n\u02dcLSGD[\u03c6] = \u02dcLGD[\u03c6] + \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+",
  "+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\nWant the batch variance to be small, \nrather than some batches fitting well \nand others not well\u2026\nSee derivation at end of UDL Ch. 9\nWhere",
  "Implicit regularization\n\u2022 Gradient descent disfavors areas where gradients are steep\n\u2022 SGD likes all batches to have similar gradients\n\u2022 Depends on learning rate \u2013 perhaps why larger learning rates generalize better.\n<latexit sha1_base64=\"AHdfomwX/HC\nsyTnvfCJdX/0ABPc=\">AW/3iclZhb9xEFIA3XEu5NAURgXixiCohKFSlcsLUps\n0Tduk5LpJ2jiNxt6xd5rx2LHyaWHxA/hjfEKz+Fn8C/4IzXuxOfM3kgUtnhfJ/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuhe",
  "/n\ncubiS5BJUejFxX9m3nr7nXfe/GBzc/OjT27N3v50v0jLPOT9MJVpfhiwgkuheF\n8LflhlnOWBJIfBKcrh+c87wQqdrTlxk/TlisRCRCpiF0Mvu752shB7zykyAdVRt\n1fVKtPaqP/CDKhuLY+8WbgGnoO8+PchZWPpPZkNXV/drzJY+0L/d5rluWsVwLJr2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQo",
  "r2N2\npbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQoWVEcLS\n1m+rgy7YS1zf9suAZC09ZzI+gqFjCi+OqyVbt3YHIwIvSHP4p7TXRq1dULCmKyQA\nM2F6WGBmgi52VOro5+NKqKzUXIXjhqJSejr1TOq9gch5qOUlFiYC+irFw4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8",
  "w4ZpEvDBN\n30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJqu85q43AoXmcsP92b1iI0T8QbTipFP\nJNQKP6riC/ECBoIDEAucgFTxAuo0+QkibwlRWJwScLveYEl4OzWpWmkeQ0462kuiQ\nSGTfNSxVogFU5l0lF1QPO+OZwDXOcwCdBV+OJqD3YypenKd5iOdJ1VhYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoO",
  "hYriFnKmYN0\n3AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoOs0EWTBIoy7VhNB\nloSjZMASBluycw4MQzEbcqFYFWZhbeRp0285MBK/NUQb7peutViT95wxlxARg9\n5lfwVTIu/pKOrW9SXLOG98U+MgbwmR1L2F5PB7WpBEYVRurqdnkCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4",
  "kCpk0WxDK04uaXr\njUHkmugM0Abzpylyo6Ip2tynBkjVh/y4MNS8lP/p+4Qc+Oq4WzbYx/yHZhIqKMnNV\nZML/o6IB3Lzw+oInrxUosmDQDN5qYTzHU0dy/HCNpFm7qAgFJNCX6LtL2LVvaJ4M\n6mCeorBEy98MuEQpMcRV3ZBIwMv3AbdiygEA0yHI8xlGlR5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5Sub",
  "R5pwcfmg9Q6TRzbGYC3O\nz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5SubITOnolV9o2GKu3d9M+bjot\nGJ+t62B/2C2SnDkJ+drOP5iIlFHYnqguceZ12SWI72oK7pcr3as2r91bdkacO12\n1KUm/bS7ftcK/pAT/bcPR2g3jEo5EdbU9pB6xHO1BXe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0g",
  "Xe48brhG4XDdpiT1TvLotB3u\n1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtFNOEx0gch7CYlF0L/h8ruwJuHl1rHMLiV\niG6mglgacAlHsI4hMXxFu6abQyrGw51w62LwxXzXEIi2swaMeh7AYUzF2iqcsy5A\n4DpE8DnEehzSPGZYyl4RnJHPMCFlSrgWVD9OuZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vm",
  "uZAJYGqHWRo7GoAcyVajBNojlgq68w\nrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vmbzk0W4BTDY5YryZlAVkYTuIWdLepMnv\n6CqCJPckF0aeklpReWXlB6YOkBpbml5I0giHYsJW8nQXRu6Tml+5buU1paWlLat7RP\naWRpROljSx9TGloaUrpi6Qql2lLyRAp3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKu",
  "Ap3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlp\nS8pfWPpG0ofWvqQUmYpo3TV0lVKuaXk0EQLVu6TGlgKXn3g71m6RalmaUZpY8sfUT\npwFLyVgz3M0vJ4w3cGC2VlD619CmlwlLy/hZEzy19TmliaULpM0ufUfra0teUrlm6\nRmlsKfk2AE8nlu5Sar8CVQWl25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXM",
  "25ZuU3pm6Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCv\nAoYekpeZ6MVHuqTb42kXMtUlPuYG3GJ1eTnEdqyh2sPZ0mV5PzKVJTPiRdX92fki\n/O3Jr5YubLud/m/pj7c+6vsfrWTHvNZ73O39zf/wGIjQK+</latexit>BlMJfzI7v4S/wtLC/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] +",
  "/r2FpR8X7m/fn3+w3H6hvdH7qvd175veUu+n3oPek95Wr98Le\n\u02dcLGD[\u03c6] = L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n<latexit sha1_base64=\"r\nrKZAMUeDGcwur29Xw20pAvpE0=\">AYQHiczZhbT9x\nGFIA39JbSW9Kq4qEvVlGqKhcEUXp5iZRACEkgZbksEDC\nsxt6xd8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8OnjNI7Vt\nXgp093+e5nJnxLcg4y9Xi4u8",
  "d8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8OnjNI7Vt\nXgp093+e5nJnxLcg4y9Xi4u83Zt57/4MP7r58ewn372\n+Re3bn+5l6eFDGkvTHkqDwKSU84E7SmOD3IJCVJwOl+\ncLqi+f45lTlLxa6zOhxQmLBIhYSBaH+7Ru/eb5ifEBL\nPwnSUblRVf1yZ+1ZdeQHUTZkx953jx3GFeGeH0kSlj7h\n2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYh",
  "UTZkx953jx3GFeGeH0kSlj7h\n2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYhUjHBvA\n6SqMr/HNVbeA8WHZIvWTxUvtT1npQPK1+kokgCKj3fn\n6173XR3OpJ7ntVqOT6v2LJv8vQ+/fml9cWKw/Hi4sN\nYX5TvPp9m9/PfAHaVgkVKiQkzw/WlrM1HGp2wk5rWb9I\nqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJM\nnzyQAMy",
  "p2wk5rWb9I\nqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJM\nnzyQAMyFqmNtMB13sqFDRz8clE1mhqAjHDUF91Tq6b\nXrDZikoeKXUChZNBXLxwSJeCFT7rC3oRpklCxKD0l1\ne3IHEBjZko6VlRr/aqajurtUOheJ2x/HJ3WgtTNGHvK\nqkVnQl1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3\nNATe7EZaEt12hqoWiMeSkpR",
  "l1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3\nNATe7EZaEt12hqoWiMeSkpR0iDQoZp6OWtYIsmMqkpey\nA4nl3PA2okjAL0FX4otYc7GREVJPjFB0pmZS5jtktSCJ\niWjcBQw5hoW/bhig4h0PDlvWLbW0TcdokLs3qrkodsax\nd2XaUxHkRg7ZTRywLFmHctuqIZXE4Fw9IQiDLTbkPA048\nHXGrTNgqQwuzK9Og3XamI/baHGWwX9reao",
  "LFmHctuqIZXE4Fw9IQiDLTbkPA048\nHXGrTNgqQwuzK9Og3XamI/baHGWwX9reaonSf06sjOgA\n7D79zYgIaVtfSae2N0nOe3rAh15Q5is9iFExuNhTRqB\nUTWxCpt1riwTZwtCMr1om7o3DpVmrD1AHbA3XSGZiK5o\n9+sSLFkd9u/DUGXB6dGDhR/o6Lhc1NtG/0PZhIryInNV\npMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSz\nx",
  "1NtG/0PZhIryInNV\npMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSz\nx0UmCcqUtr+7NYtI+pI3Zn08TqKwR0vfBNmLAmOYras\ng5oGb7hPsaxgEJrkOF4jCFP80JSdPKz1jNEal2fFiXTF\n6v2CZVroX3eoHx6FJTh4nBOrzk8sDIajPMZpIUYEGklc\n6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez\n5iZGHW3XBjaOzLo",
  "Gklc\n6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez\n5iZGHW3XBjaOzLo4sR3tQ13S5Xu1ZuX5yFy3t2OG6TY\n7qbXrpth3uNT2gZxuO3m4gD1nY4VZdTQ+xhyxHe1CXO4\n8brlE4XLfJUb2TPDpthzs1reUf7Q6pIvo2KeUDfduXcn\n8cskWFReU04TGljgO2WJStC34bSs7DC4ebWscsVuztq\naDtjSgHJ7COQLY63cNtsYra64VA3G",
  "04TGljgO2WJStC34bSs7DC4ebWscsVuztq\naDtjSgHJ7COQLY63cNtsYra64VA3GrzoHVHIdscY0\nk9qjHIVuMsRg7xVOSZY4DqE8Du08DnEeM1vKXJI9I5l\njRtCSci0oOUzbkg7Y0shqbeRoDHrAU2E12ARtOcrL3e\nuPGtYoFXc/VcO+ahWxKtQBW9pEe8zN52bLBTDLdZ\nriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h\n+5hKQ",
  "8zN52bLBTDLdZ\nriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h\n+5hKQ9ETQRBtG4qeToLo3NBzTPcM3cO0MLTAtGdoD9PI\n0AjT54Y+xzQ0NMR0xdAVTJWh6I4UrgiG7mI6NHSI6YGh\nB5i+MfQNpi8MfYHpoaGHmL4z9B2mTw19ikxlGC6augqp\ntRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUP\nRXD9cxQdHsDF0ZDOa",
  "p\ntRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUP\nRXD9cxQdHsDF0ZDOaYvDX2JKTMUPb8F0WtDX2OaGJpg+\nsrQV5i+NfQtpmuGrmEaG4reDcDdiaE7mJq3QGWO6ZahW\n5ieGXrmfi9Ap9MYuBbmpqlgE9PU0BTdUPRkwLcSh6iu\n4nI9Gc1SZvm9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56\ndITPkQdX1b/oiBVIKZ/r+rfkl+y0sL",
  "9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56\ndITPkQdX1b/oiBVIKZ/r+rfkl+y0sLuw9XFj6ceHR1q\nP5J8vNG9qbnW863a+7yx1fuo86bzodDu9Tjhzd6Y782\nit>bmcO7XuT/m/pz7a6zO3GiO+arT+sz9/Q8KAH6+</latex\n\u02dcLSGD[\u03c6] = \u02dcLGD[\u03c6] + \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+",
  "+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2",
  "<latexit sha1_base64=\"AHdfomwX/HCsyTnvfCJdX/0ABPc=\">AW/3iclZhb9\nxEFIA3XEu5NAURgXixiCohKFSlcsLUps0Tduk5LpJ2jiNxt6xd5rx2LHyaWHxA/hjfEKz+Fn8C/4IzXuxOfM3kgUtnhfJ/ncubiS5BJUejFxX9m3nr7nXfe/GBzc/\nOjT27N3v50v0jLPOT9MJVpfhiwgkuheF8LflhlnOWBJIfBKcrh+c87wQqdrTlxk/TlisRCR",
  "v50v0jLPOT9MJVpfhiwgkuheF8LflhlnOWBJIfBKcrh+c87wQqdrTlxk/TlisRCRCpiF0Mvu752shB7zykyAdVRt1fVKtPaqP/CDKhuLY+8WbgGnoO8+PchZW\nPpPZkNXV/drzJY+0L/d5rluWsVwLJr2N2pbHFYCdi3io/dzor6p79cns/OLCYvPn0cJSW5jvtX9bJ7c/H/iDNCwTrnQoWVEcLS1m+rgy7YS1zf9suAZC09ZzI+gqFjCi+O\nqyV",
  "jvtX9bJ7c/H/iDNCwTrnQoWVEcLS1m+rgy7YS1zf9suAZC09ZzI+gqFjCi+O\nqyVbt3YHIwIvSHP4p7TXRq1dULCmKyQAM2F6WGBmgi52VOro5+NKqKzUXIXjhqJSejr1TOq9gch5qOUlFiYC+irFw4ZpEvDBN30Fb8I0yRhalD5y6vbkLiAx0JV/KxsJq\nu85q43AoXmcsP92b1iI0T8QbTipFPJNQKP6riC/ECBoIDEAucgFTxAuo0+QkibwlR",
  "85q43AoXmcsP92b1iI0T8QbTipFPJNQKP6riC/ECBoIDEAucgFTxAuo0+QkibwlRWJwScLveYEl4OzWpWmkeQ0462kuiQSGTfNSxVogFU5l0lF1QPO+OZwDXOcwCd\nBV+OJqD3YypenKd5iOdJ1VhYriFnKmYN03AkENY6DvYUKWUcGnYsX7F1g5Tp23i0qzpam4iyNrLu47OaV7UoOs0EWTBIoy7VhNBloSjZMASBluycw4MQzEbcqFYFWZhb\ne",
  "pam4iyNrLu47OaV7UoOs0EWTBIoy7VhNBloSjZMASBluycw4MQzEbcqFYFWZhb\neRp0285MBK/NUQb7peutViT95wxlxARg95lfwVTIu/pKOrW9SXLOG98U+MgbwmR1L2F5PB7WpBEYVRurqdnkCpk0WxDK04uaXrjUHkmugM0Abzpylyo6Ip2tynBkjVh/y4M\nNS8lP/p+4Qc+Oq4WzbYx/yHZhIqKMnNVZML/o6IB3Lzw+oInrxUosmDQDN5qY",
  "4M\nNS8lP/p+4Qc+Oq4WzbYx/yHZhIqKMnNVZML/o6IB3Lzw+oInrxUosmDQDN5qYTzHU0dy/HCNpFm7qAgFJNCX6LtL2LVvaJ4M6mCeorBEy98MuEQpMcRV3ZBIwMv3Abdiy\ngEA0yHI8xlGlR5pwcfmg9Q6TRzbGYC3Oz6h6o0gjdc4PL6VQhpvDOb/m8gBlNBjnM0hLNWA5SubITOnolV9o2GKu3d9M+bjotGJ+t62B/2C2SnDkJ+drOP5iIlF",
  "lNBjnM0hLNWA5SubITOnolV9o2GKu3d9M+bjotGJ+t62B/2C2SnDkJ+drOP5iIlFHYnqgu\nceZ12SWI72oK7pcr3as2r91bdkacO121KUm/bS7ftcK/pAT/bcPR2g3jEo5EdbU9pB6xHO1BXe48brhG4XDdpiT1TvLotB3u1ETLP9obcs3MY1IqB+axL5X+OIRFTUXtF\nNOEx0gch7CYlF0L/h8ruwJuHl1rHMLiViG6mglgacAlHsI4hMXxFu6a",
  "IRFTUXtF\nNOEx0gch7CYlF0L/h8ruwJuHl1rHMLiViG6mglgacAlHsI4hMXxFu6abQyrGw51w62LwxXzXEIi2swaMeh7AYUzF2iqcsy5A4DpE8DnEehzSPGZYyl4RnJHPMCFlSrgWV\nD9OuZAJYGqHWRo7GoAcyVajBNojlgq68wrnyFrFiq7ivqvh/jUNa4YqNAEsbZI95vmbzk0W4BTDY5YryZlAVkYTuIWdLepMnv6CqCJPckF0aeklpReWXl",
  "YqNAEsbZI95vmbzk0W4BTDY5YryZlAVkYTuIWdLepMnv6CqCJPckF0aeklpReWXlB6YOkBpbml5I0\ngiHYsJW8nQXRu6Tml+5buU1paWlLat7RPaWRpROljSx9TGloaUrpi6Qql2lLyRAp3BEv3KB1aOqT0NJDSl9Y+oLSJ5Y+ofSlpS8pfWPpG0ofWvqQUmYpo3TV0lVKuaXk0\nEQLVu6TGlgKXn3g71m6RalmaUZpY8sfUTpwFLyVgz3M0vJ4w",
  "Ypo3TV0lVKuaXk0\nEQLVu6TGlgKXn3g71m6RalmaUZpY8sfUTpwFLyVgz3M0vJ4w3cGC2VlD619CmlwlLy/hZEzy19TmliaULpM0ufUfra0teUrlm6RmlsKfk2AE8nlu5Sar8CVQWl25ZuU3pm6\n3H6hvdH7qvd175veUu+n3oPek95Wr98Le/O3Jr5YubLud/m/pj7c+6vsfrWTHvNZ73O39zf/wGIjQK+</latexit>Zn7uwCfTmPgWpibtoJN",
  "d/m/pj7c+6vsfrWTHvNZ73O39zf/wGIjQK+</latexit>Zn7uwCfTmPgWpibtoJNSlNLU0rXLSVvCvAoYekpeZ6MVHuqTb42kXMtUlPuYG3GJ1eTnEdqyh2sPZ0mV5PzKVJTPiRdX92fkiBlMJfzI7v4S/wtLC/r2FpR8X7m/fn3+w\n\u02dcLGD[\u03c6] = L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\nX\n1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212",
  "\u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\nX\n1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n<latexit sha1_base64=\"rKZAMUeDGcwur29Xw20pAvpE0=\">AYQHiczZhbT9xGFIA39JbSW9Kq4qEvVlGqKhcEUXp5iZRACEkgZbksEDCsxt6xd8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8Onj\nNI7VtXgp093+e5nJnxLcg4y9Xi4u83Zt57/4MP7r",
  "5z7W/ov+gb1Vf+9QzXu8Onj\nNI7VtXgp093+e5nJnxLcg4y9Xi4u83Zt57/4MP7r58ewn372+Re3bn+5l6eFDGkvTHkqDwKSU84E7SmOD3IJCVJwOl+cLqi+f45lTlLxa6zOhxQmLBIhYSBaH+7Ru/eb5ifEBLPwnSUblRVf1yZ+1ZdeQHUTZkx953jx3GFeGeH0kSlj7h2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYhUjHBvA6SqMr/HNV",
  "GeH0kSlj7h2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYhUjHBvA6SqMr/HNVbeA8WHZIvWTxUvtT1npQPK1+kokgCKj3fn6173XR3OpJ7ntVqOT6v2LJv8vQ\n+/fml9cWKw/Hi4sNYX5TvPp9m9/PfAHaVgkVKiQkzw/WlrM1HGp2wk5rWb9IqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJMnzyQAMyFqmNtMB13sqFDRz8clE",
  "TI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJMnzyQAMyFqmNtMB13sqFDRz8clE1mhqAjHDUF91Tq6bXrDZikoeKXUChZNBXLxwSJeCFT7rC3oRpklCxKD0l1e3IHEBjZko6VlRr/aqajurtUOheJ2x/HJ3WgtTNGHvKqkVnQl1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3NATe7EZaEt12hqoWi\nMeSkpR0iDQoZp6OWtYIsmMqkpeyA",
  "KSC5lCnzk8QeUsWhd3NATe7EZaEt12hqoWi\nMeSkpR0iDQoZp6OWtYIsmMqkpeyA4nl3PA2okjAL0FX4otYc7GREVJPjFB0pmZS5jtktSCJiWjcBQw5hoW/bhig4h0PDlvWLbW0TcdokLs3qrkodsaxd2XaUxHkRg7ZTRywLFmHctuqIZXE4Fw9IQiDLTbkPA048HXGrTNgqQwuzK9Og3XamI/baHGWwX9reaonSf06sjOgA7D79zYgIaVtfSae2",
  "A048HXGrTNgqQwuzK9Og3XamI/baHGWwX9reaonSf06sjOgA7D79zYgIaVtfSae2N0nOe3rAh15Q5is9iFExuNhTRqBUTWxCpt1riwTZwtCMr1om7o3DpVmrD1AHbA3XSGZiK5o9+sSLFk\nd9u/DUGXB6dGDhR/o6Lhc1NtG/0PZhIryInNVpMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSzx0UmCcqUtr+7NYtI+pI3Zn08TqKwR0v",
  "9QURe/JSbk0eBOrJSzmc362pI9Je2DpSzx0UmCcqUtr+7NYtI+pI3Zn08TqKwR0vfBNmLAmOYrasg5oGb7hPsaxgEJrkOF4jCFP80JSdPKz1jNEal2fFiXTF6v2CZVroX3eoHx6FJTh4nBOrzk8sDIajPMZpIUYEGklc6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez5iZGHW3XBjaOzLo4sR3tQ13S5Xu1ZuX5yFy3t2OG\n6TY7qbXr",
  "1Bv2B2ijCkZ/1ez5iZGHW3XBjaOzLo4sR3tQ13S5Xu1ZuX5yFy3t2OG\n6TY7qbXrpth3uNT2gZxuO3m4gD1nY4VZdTQ+xhyxHe1CXO48brlE4XLfJUb2TPDpthzs1reUf7Q6pIvo2KeUDfduXcn8cskWFReU04TGljgO2WJStC34bSs7DC4ebWscsVuztqaDtjSgHJ7COQLY63cNtsYra64VA3GrzoHVHIdscY0k9qjHIVuMsRg7xVOSZY4DqE8D",
  "tjSgHJ7COQLY63cNtsYra64VA3GrzoHVHIdscY0k9qjHIVuMsRg7xVOSZY4DqE8Du08DnEeM1vKXJI9I5ljRtCSci0oOUzbkg7Y0shqbeRoDHrAU2E12ARtOcrL3euPGtYoFXc/VcO+ahWxKtQBW9pEe8\n4reDcDdiaE7mJq3QGWO6ZahW5ieGXrmfi9Ap9MYuBbmpqlgE9PU0BTdUPRkwLcSh6iu4nI9Gc1SZvm9B5LRJT7mBNxidHo5xHYsodr",
  "YuBbmpqlgE9PU0BTdUPRkwLcSh6iu4nI9Gc1SZvm9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56dITPkQdX1b/oiBVIKZ/r+rfkl+y0sLuw9XFj6ceHR1qP5J8vNG9qbnW863a+7yx1fuo86bzodDu9Tjhzd6Y782bmcO7XuT/m/pz7a6zO3GiO+arT+sz9/Q8KAH6+</latexit>zN52bLBTDLdZriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heG",
  "KAH6+</latexit>zN52bLBTDLdZriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h+5hKQ9ETQRBtG4qeToLo3NBzTPcM3cO0MLTAtGdoD9PI0AjT54Y+xzQ0NMR0xdAVTJWh6I4UrgiG7mI6NHSI6YGhB5i+MfQNpi8MfYHpoaGHmL4z9B2mTw19ikxlGC6augqptRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUPRXD9cxQdH",
  "kxlGC6augqptRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUPRXD9cxQdHsDF0ZDOaYvDX2JKTMUPb8F0WtDX2OaGJpg+srQV5i+NfQtpmuGrmEaG\n\u02dcLSGD[\u03c6] = \u02dcLGD[\u03c6] + \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n<latexit sha1_base64=\"rKZAMUeDGcwur29Xw20pA",
  "\u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n<latexit sha1_base64=\"rKZAMUeDGcwur29Xw20pAvpE0=\">AYQHiczZhbT9xGFIA39JbSW9Kq4qEvVlGqKhcEUXp5iZRACEkgZbksEDCsxt6xd8J4bMZjWGL5z7W/ov+gb1Vf+9QzXu8OnjNI7VtXgp093+\ne5nJnxLcg4y9Xi4u83Zt57/4MP7r58ewn372+Re3bn+5l6eFDGkvTHkqDwKSU84E7SmOD3IJCVJwOl+c",
  "83Zt57/4MP7r58ewn372+Re3bn+5l6eFDGkvTHkqDwKSU84E7SmOD3IJCVJwOl+cLqi+f45lTlLxa6zOhxQmLBIhYSBaH+7Ru/eb5ifEBLPwnSUblRVf1yZ+1ZdeQHUTZkx953jx3GFeGeH0kSlj7h2ZBU5SNvufLzIumXweOl6qSEX5xGyud7VKpGzYhUjHBvA6SqMr/HNVbeA8WHZIvWTxUvtT1npQPK1+kokgCKj3fn6173XR3OpJ7ntVqOT6",
  "BvA6SqMr/HNVbeA8WHZIvWTxUvtT1npQPK1+kokgCKj3fn6173XR3OpJ7ntVqOT6v2LJv8vQ+/fml9cWKw/Hi4sNYX5TvPp9m9/PfAHaVgkV\nKiQkzw/WlrM1HGp2wk5rWb9IqcZCU9JTI+gKEhC8+OyXm6VdwciAy9KJfwJ5dXRq0eUJMnzyQAMyFqmNtMB13sqFDRz8clE1mhqAjHDUF91Tq6bXrDZikoeKXUChZNBXLxwSJeCFT7rC3oRpklC",
  "13sqFDRz8clE1mhqAjHDUF91Tq6bXrDZikoeKXUChZNBXLxwSJeCFT7rC3oRpklCxKD0l1e3IHEBjZko6VlRr/aqajurtUOheJ2x/HJ3WgtTNGHvKqkVnQl1wg0rsqSLsQLNmAUAFugCKSC5lCnzk8QeUsWhd3NATe7EZaEt12hqoWiMeSkpR0iDQoZp6OWtYIsmMqkpeyA4nl3PA2okjAL0FX4otYc7GREVJPjFB0\npmZS5jtktSCJiWjcBQw5h",
  "YIsmMqkpeyA4nl3PA2okjAL0FX4otYc7GREVJPjFB0\npmZS5jtktSCJiWjcBQw5hoW/bhig4h0PDlvWLbW0TcdokLs3qrkodsaxd2XaUxHkRg7ZTRywLFmHctuqIZXE4Fw9IQiDLTbkPA048HXGrTNgqQwuzK9Og3XamI/baHGWwX9reaonSf06sjOgA7D79zYgIaVtfSae2N0nOe3rAh15Q5is9iFExuNhTRqBUTWxCpt1riwTZwtCMr1om7o3D",
  "zYgIaVtfSae2N0nOe3rAh15Q5is9iFExuNhTRqBUTWxCpt1riwTZwtCMr1om7o3DpVmrD1AHbA3XSGZiK5o9+sSLFkd9u/DUGXB6dGDhR/o6Lhc1NtG/0PZhIryInNVpMP/oaIBXP3t9QURe/JSbk0eBOrJSzmc362pI9Je2DpSzx\n0UmCcqUtr+7NYtI+pI3Zn08TqKwR0vfBNmLAmOYrasg5oGb7hPsaxgEJrkOF4jCFP80JSdPKz1jNEal2fF",
  "3Zn08TqKwR0vfBNmLAmOYrasg5oGb7hPsaxgEJrkOF4jCFP80JSdPKz1jNEal2fFiXTF6v2CZVroX3eoHx6FJTh4nBOrzk8sDIajPMZpIUYEGklc6SndHTi5wq2mGv31M+LjqtmJ6tN+1Bv2B2ijCkZ/1ez5iZGHW3XBjaOzLo4sR3tQ13S5Xu1ZuX5yFy3t2OG6TY7qbXrpth3uNT2gZxuO3m4gD1nY4VZdTQ+xhyxHe1CXO48brlE4XLfJUb2T",
  "2OG6TY7qbXrpth3uNT2gZxuO3m4gD1nY4VZdTQ+xhyxHe1CXO48brlE4XLfJUb2TPDpthzs1reUf7Q6pIvo2KeUDfduXcn8cskWFReU04TGl\njgO2WJStC34bSs7DC4ebWscsVuztqaDtjSgHJ7COQLY63cNtsYra64VA3GrzoHVHIdscY0k9qjHIVuMsRg7xVOSZY4DqE8Du08DnEeM1vKXJI9I5ljRtCSci0oOUzbkg7Y0shqbeRoDHrAU2E12",
  "xVOSZY4DqE8Du08DnEeM1vKXJI9I5ljRtCSci0oOUzbkg7Y0shqbeRoDHrAU2E12ARtOcrL3euPGtYoFXc/VcO+ahWxKtQBW9pEe8zN52bLBTDLdZriRnzLIynMCu7XSxM7n7C6IS3ckF0aWhl5heGHqB6b6h+5hKQ9ETQRBtG4qeToLo3NBzTPcM3cO0MLTAtGdoD9PI0AjT54Y+xzQ0NMR0xdAVTJWh6I4U\nHR1qP5J8vNG9qbnW863a+7yx1f",
  "AtGdoD9PI0AjT54Y+xzQ0NMR0xdAVTJWh6I4U\nHR1qP5J8vNG9qbnW863a+7yx1fuo86bzodDu9Tjhzd6Y782bmcO7XuT/m/pz7a6zO3GiO+arT+sz9/Q8KAH6+</latexit>rgiG7mI6NHSI6YGhB5i+MfQNpi8MfYHpoaGHmL4z9B2mTw19ikxlGC6augqptRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUPRXD9cxQdHsDF0ZDOaYv",
  "ptRQ9OogiJYNXcY0MBQ9+8FeM7SLaWZohukzQ59hOjAUPRXD9cxQdHsDF0ZDOaYvDX2JKTMUPb8F0WtDX2OaGJpg+srQV5i+NfQtpmuGrmEaG4reDcDdiaE7mJq3QGWO6ZahW5ieGXrmfi9Ap9MYuBbmpqlgE9PU0BTdUPRkwLcSh6iu4nI9Gc1SZvm9B5LRJT7mBNxidHo5xHYsodrDk7TY5G56dITPkQdX1b/oiBVIKZ/r+rfkl+y0sLuw9XFj",
  "JT7mBNxidHo5xHYsodrDk7TY5G56dITPkQdX1b/oiBVIKZ/r+rfkl+y0sLuw9XFj6ce\n\u02dcLSGD[\u03c6] = \u02dcLGD[\u03c6] + \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n= L[\u03c6] + \u21b5\n4\n\ufffd\ufffd\ufffd\ufffd\n@L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\n+ \u21b5\n4B\nB\nX\nb=1\n\ufffd\ufffd\ufffd\ufffd\n@Lb\n@\u03c6 \u2212 @L\n@\u03c6\n\ufffd\ufffd\ufffd\ufffd\n2\nOriginal Gabor Model\nLoss",
  "Generally, performance is \n\u2022 best for larger learning rates\n\u2022 best with smaller batches",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Early stopping\n\u2022 If we stop training early,  weights don\u2019t have time to overfit to noise\n\u2022 Weights start small, don\u2019t have time to get large\n\u2022 Reduces effective model complexity\n\u2022 Known as early stopping\n\u2022 Don\u2019t have to re-train with different hyper-parameters \u2013 just \n\u201dcheckpoint\u201d regularly and pick the model with lowest validation loss",
  "",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Ensembling\n\u2022 Average together several models \u2013 an ensemble\n\u2022 Can take mean or median\n\u2022 Before softmax for classification\n\u2022 Simply different initializations or even different models \n\u2022 Or train with different subsets of the data resampled with \nreplacements -- bagging",
  "",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Dropout\nRandomly clamp ~50% of hidden units to 0 on each \niteration.",
  "Dropout\n\u2022\nMakes the network less dependent on any given hidden unit.\n\u2022\nPrevents situations where subsequent hidden units correct for excessive swings from earlier hidden \nunits\n\u2022\nCan eliminate kinks in function that are far from data and don\u2019t contribute to training loss\n\u2022\nMust use weight scaling inference rule \u2013 multiple weights by (1 \u2013 dropout probability)",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Adding noise\n\u2022 to inputs \u2013 induces weight regularization (see Exercise 9.3 in UDL)\n\u2022 to weights \u2013 makes robust to small weight perturbations\n\u2022 to outputs (labels) \u2013 reduces \u201coverconfident\u201d probability for target class\nAdding noise to input with different variances.",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Transfer Learning\nAssume we have lots of \nsegmentation training data\n(1) Train the model for \nsegmentation\n(2) Replace the final layers to \nmatch the new task and \nAssume we limited \ndepth training data\n(3) Either:\na)\nFreeze the rest of the layers and \ntrain the final layers\nb)\nFine tune the entire model",
  "Multi-Task Learning\n\u2022 Train the model for 2 or more tasks simultaneously\n\u2022 Weighted combo of loss fncs\n\u2022 Less likely to overfit to training data of one task\n\u2022 Can be harder to get training to converge. Might have to vary the \nindividual task loss weightings, \ud835\udefc and \ud835\udefd.\n\ud835\udc3f#$#%& = \ud835\udefc \u22c5 \ud835\udc3f'()*(+#%,#$+ + \ud835\udefd \u22c5 \ud835\udc3f-(.#/",
  "Self-Supervised Learning\n\u2022 Mask out part of the training data\n\u2022 Train model to try to infer missing data\n\u2022 masked data is the target\n\u2022 \u00e8 Model learns characteristics of the data\n\u2022 Then apply transfer learning\nThe animal didn\u2019t cross the street because it was too tired.",
  "Regularization\n\u2022 Explicit regularization\n\u2022 Implicit regularization\n\u2022 Early stopping\n\u2022 Ensembling\n\u2022 Dropout\n\u2022 Adding noise\n\u2022 Transfer learning, multi-task learning, self-supervised learning\n\u2022 Data augmentation",
  "Data augmentation",
  "Regularization overview",
  "Feedback?\nLink",
  "Convolutional Networks\nDL4DS \u2013 Spring 2024\n1\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n2",
  "\u2022 Multiclass classification problem (discrete classes, >2 possible classes)\n\u2022 Convolutional network\nImage classification\n3",
  "Object detection\n4",
  "\u2022 Multivariate binary classification problem (many outputs, two discrete classes)\n\u2022 Convolutional encoder-decoder network\nImage segmentation\n5",
  "Networks for images\n\u2022 Problems with fully-connected networks\n1. Size\n\u2022 224x224 RGB image = 150,528 dimensions\n\u2022 Hidden layers generally larger than inputs\n\u2022 One hidden layer = 150,520x150,528 weights -- 22 billion\n2. Nearby pixels statistically related\n\u2022 But could permute pixels and relearn and get same results with FC\n3. Should be stable under transformations\n\u2022 Don\u2019t want to re-learn appearance at different parts of image\n6",
  "Convolutional networks\n\u2022 Parameters only look at local image patches\n\u2022 Share parameters across image\n7",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n8",
  "Invariance\n\u2022 A function f[x] is invariant to a transformation t[] if:\ni.e., the function output is the same even after the transformation is \napplied.\n<latexit sha1_base64=\"wmUy\nLfaVtz8aqxmyJSpV0iFmuk=\">AWvXiclZhbT9xGFICd9JamN\n9KqvPTFKopUVekKqvTyUjWBkBukLIEFArtBY+/YO2E8NvYlj7\ne/pr+tq/6ZnbO9OfM7w0JWIJ+f7PJczM74FmRSFXl398bN9\n7/4MOPbn18+5NP/v8i6U7Xx4UaZmHfBCmMs2PAlZwKRQfaKElP",
  "FmRSFXl398bN9\n7/4MOPbn18+5NP/v8i6U7Xx4UaZmHfBCmMs2PAlZwKRQfaKElP\n8pyzpJA8sPgbMPwueFyJV+/oq46OExUpEImQaQqdLD/1hEqT\nahEfjQbSh7pExvRsxM4TEfDXMQTPfJ/69gNO1aWe2t1j+fFtb\naworX/vqnd74eD8dpWCZc6VCyojhZW830qGK5FqHks9vDsuAZC8\n9YzE+gqFjCi1FVj3Xm34XI2I/SHP6U9uvou2dULC",
  "ZW830qGK5FqHks9vDsuAZC8\n9YzE+gqFjCi1FVj3Xm34XI2I/SHP6U9uvou2dULCmKqyQAM2F6U\nmBmgi52Uuro1ElVFZqrsKmoaiUvk59kzh/LHIeankFBRbmAvrq\nhxOWs1BDem8PFb8M0yRhalwN1zd3Z5AjHgtV8fOyTvVs1nU2a4\ndD8Tpj/dn+ohaheSLeclJrZhKrhF4PKsq3ot7GAgOQPQ4AaniB\ndRp8gOTvIYoLC0JuL4OWMVK0jyE",
  "eSLeclJrZhKrhF4PKsq3ot7GAgOQPQ4AaniB\ndRp8gOTvIYoLC0JuL4OWMVK0jyEnHe2YaFDIJ92rA1iwVQm\nHWUPFN+/6xvAdQ6zAF2FA0dzsJcxNZufp/lU50lVmBhuIWcq5nU\nTMOSQSTOirqFKeHUsGP9ga2XTJ21iUuzuqu5iSBrP+86Oqd5Ue\nOuU0eQBYsw7lp1BFkSLgRjljDIcls+hQEnvom4VaGwKsjC7Odp0\nG07MxG8NqcZ7Jeut",
  "eQBYsw7lp1BFkSLgRjljDIcls+hQEnvom4VaGwKsjC7Odp0\nG07MxG8NqcZ7Jeut1mR9F8wlBETgN1njoKpkHf1jXRh+/PkXNS+\nKfCpP4HJ6p7C8rgZ1rwRGFUbm1GzhUyabYglKeXdP0xqHyTH\nQHaAJ405W5UNE72r26BEvWhIf3YKh5KfnJD72f+HRUrZptY/4h2\nYSKijJzVWTC/6OiMdx68PqCJ68VKLJg0A9eamE6zuaOpbjhW0i\n9dxB",
  "ZptY/4h2\nYSKijJzVWTC/6OiMdx68PqCJ68VKLJg0A9eamE6zuaOpbjhW0i\n9dxBQSgmhb5C21/EqntOHcGdTRPUVwiYeuHIhEKTHEVd2QSMDEe\n4iToWUIgGTZjDGValDknFz+0niFS6+aymAtzs+peUKURutcNLh\ndnQRluDhf8mtMDlNGgyWeQlmrMcpTMqZnS6ethoWGLuXZ/PeVN0\nWnF/HyrbQ/6BbNThiE/P93C8xETizoS1QVPLc6JLE",
  "MqZnS6ethoWGLuXZ/PeVN0\nWnF/HyrbQ/6BbNThiE/P93C8xETizoS1QVPLc6JLEc7UFdi+X6\nbs+qrdfk6UdO1y3KUm9bS/dtsO9pgf8fNvR23iEYs6EtXV9p\nB6xHK0B3W587jtGoXDdZuS1DvPo9N2uAsTLf9of8I1M49JqRybx\n75UDpsQFjUVtVNMEx4jsQlhMSm7FvwfK3sCbh5dqwlhsV+IrmYC\nWBpziYfQhLDYbOGu2cawu1Qt90qk9",
  "4jsQlhMSm7FvwfK3sCbh5dqwlhsV+IrmYC\nWBpziYfQhLDYbOGu2cawu1Qt90qk9kEmU0Ii09YgkfdhLAYUzF\n2imcsy5DYhEgeJziPE5rHDEuZS8IzkjlmhCwp14LKJ2lXMgEsTV\nFrU0dj0AOZKtRgG8RyQVde4Vx5Cq1iRVfxwNXw4JqGNUMVmgCWd\nsge84c7zk0W4BTDY5YryZlAVkYT2MdOnzrzp78gqsiTXBdWXpF\n6aWl5QeWnpIaW4p",
  "e84c7zk0W4BTDY5YryZlAVkYT2MdOnzrzp78gqsiTXBdWXpF\n6aWl5QeWnpIaW4peSMIopeWkreTILqw9ILSA0sPKC0tLSkdWD\nqgNLI0ovSxpY8pDS0NKd2wdINSbSl5IoU7gqX7lE4snVB6ZOkRp\na8sfUXpU0ufUnps6TGlby19S+lDSx9SyixlG5aukpt5R8Ogi\ndUvXKQ0sJe9+sNcs7VOaWZpR+sjSR5SOLSVvxXA/s5Q83sCN0VJ\nJ6TN",
  "t5R8Ogi\ndUvXKQ0sJe9+sNcs7VOaWZpR+sjSR5SOLSVvxXA/s5Q83sCN0VJ\nJ6TNLn1EqLCXvb0H0wtIXlCaWJpQ+t/Q5pW8sfUPpE0ufUBpbSr\n4NwNOJpXuU2q9AVUHprqW7lJ5beu7+LsAX0xi4FuaOrWCH0tTSl\nNItS8mbAjxKWHpGnicj1V7V7EfOGTYW3MHajM/PJjmP1I7WHt\n1mp9Nrk+RWvAJ6frmweJDCqQUrvSnSytr+CsLRz82F",
  "TYW3MHajM/PJjmP1I7WHt\n1mp9Nrk+RWvAJ6frmweJDCqQUrvSnSytr+CsLRz82Fv7uXd/9/\n7Kg/X2C+0t7xvW+87b837xXvgPfX63sALvT+9v7y/vX+Wf1/my\n3JZNerNG+05X3md3/Llf+396YA=</latexit>f [t[x]] = f[x]\n9",
  "Invariance example\ne.g., Image classification\n\u2022 Image has been translated, but we want our classifier to give the same result \n10",
  "Equivariance\n\u2022 A function f[x] is equivariant to a transformation t[] if:\ni.e. the output is transformed in the same way as the input\n<latexit sha1_base64=\"3N9g\nchi9eQHzUGINVq4oZjiE8Ek=\">AW1XiclZhJb9w2FICVdEvTz\nWlRX3oRagQoinRgF+lyKZDYcTY79Tje45kYlIbSMKYoWaLscYS5\nFb32J/V39Af02v6FPkqaofUefegAjpj3feLySGoLMikKvbz814\n2b7z73vsf3Prw9kcf/LpZwt3Pt8v0jIP+V6YyjQ/DFjBp",
  "feLySGoLMikKvbz814\n2b7z73vsf3Prw9kcf/LpZwt3Pt8v0jIP+V6YyjQ/DFjBpVB8T\nwst+WGWc5YEkh8Ep2uGH5zvBCp2tWXGR8mLFYiEiHTEDpZOBok\nQTqpBkHkR9OB5JE+thE9PYbDZDjIRTzWQ/8X/yrDdtS1TxaWlnv\nL9c+nhZW2sOS1v/7JnS9Hg1EalglXOpSsKI5XljM9rFiuRSj59P\nagLHjGwlMW82MoKpbwYljVOZj6dyEy8q",
  "Hg1EalglXOpSsKI5XljM9rFiuRSj59P\nagLHjGwlMW82MoKpbwYljVOZj6dyEy8qM0hz+l/Tp69YyKJUVxm\nQRgJkyPC8xM0MWOSx39PKyEykrNVdg0FJXS16lvEuqPRM5DLS+h\nwMJcQF/9cMxyFmpI+2B4hdhmiRMjarB6vr2FLFY6EqflbWUz\nCdp312uFQvM5YfbY7r0Vonoi3nFRSK6aSawQeT6uK9+IeBoIDE\nD1OQKp4AXWa/MBcryAKS",
  "FQvM5YfbY7r0Vonoi3nFRSK6aSawQeT6uK9+IeBoIDE\nD1OQKp4AXWa/MBcryAKS04CruxqeDklVSvNY8hJR3tFNChk86\n1hqxYCqTjrIDiu/f9Q3gOodZgK7CgaM52MmYms7O03yi86QqTAy\n3kDMV87oJGHLIpBlR1ClHBq2LF+xdZLpk7bxKVZ3dXcRJC1m3\ncdndO8qFHXqSPIgkUYd606giwJF4gRSxhkuS2fwIAT30TcqlBYF\nWRh9vM06",
  "1m3\ncdndO8qFHXqSPIgkUYd606giwJF4gRSxhkuS2fwIAT30TcqlBYF\nWRh9vM06LadmQhem5M9kvXW69I+s8ZyogJwO4zR8FUyLv6Wjq3\n/VlyzmvfFPjEH8NkdU9hedwMa9YIjKqNTalZ5wqZNFsQytOLrm\nl641B5JroDNAG86cpcqOiKdq8uwZI14cE9GpeSn78Xe8HPhlWy\n2bmH9INqGiosxcFZnw/6hoBLckvL4gicvlWjyIFBPXirh+o",
  "GpeSn78Xe8HPhlWy\n2bmH9INqGiosxcFZnw/6hoBLckvL4gicvlWjyIFBPXirh+o6m\njuV4YZtIPXdQEIpJoS/R9hex6p5TR3Bn0wT1FQKmXjgyodAkR1F\nXNgEjwxFuro4FKJBhs0YQ5kWZc7JxQ+tZ4jUurks5sLcrLoXVG\nmE7nWDy/lZUIabwzm/5vQAZTRo8hmkpRqxHCVzYqZ08npQaNhir\nt1fT3lTdFoxP9to24N+weyUYcjPTjbwfMT",
  "ZTRo8hmkpRqxHCVzYqZ08npQaNhir\nt1fT3lTdFoxP9to24N+weyUYcjPTjbwfMTEo5EdcHTjLMuSxH\ne1DXfLle7Vm18fpbsrRjh+s2Jam37aXbdrjX9ICfbTp6u0k8Yl\nFHoraHlKPWI72oC53Hjdo3C4blOSemd5dNoOd26i5R/tjrlm5\njEplSPz2JfKQRPCoqaidopwmMkNiEsJmXgv9jZUfAzaNrNSEs\n9gvR1UwASyMu8RCaEBabLdw12",
  "QRPCoqaidopwmMkNiEsJmXgv9jZUfAzaNrNSEs\n9gvR1UwASyMu8RCaEBabLdw12xhWNx3qpltlMhsjswlh8QlL8Ki\nbEBZjKsZO8ZRlGRKbEMnjGOdxTPOYSlzSXhGMseMkCXlWlD5O\n1KJoClCWpt4mgMeiBThRpsg1gu6MornCtPoVWs6CreczW8d03Dm\nqEKTQBLW2SP+YMt5yYLcIrhMcuV5EwgK6MJ7GOnT53Z018QVeRJ\nLoguLb2k9ML",
  "qEKTQBLW2SP+YMt5yYLcIrhMcuV5EwgK6MJ7GOnT53Z018QVeRJ\nLoguLb2k9MLSC0oPLD2gNLeUvBE0UtLydtJEJ1bek7pvqX7lJ\naWlpTuWbpHaWRpROljSx9TGloaUrpm6Rql2lLyRAp3BEt3KR1bO\nqb0NJDSo8sPaL0qaVPKX1l6StK31r6ltKHlj6klFnKF23dJ1S\nbin5dBEq5auUhpYSt79YK9Z2qc0szSj9JGljygdWUreiuF+Zil",
  "klFnKF23dJ1S\nbin5dBEq5auUhpYSt79YK9Z2qc0szSj9JGljygdWUreiuF+Zil\n5vIEbo6WS0meWPqNUWEre34LohaUvKE0sTSh9bulzSt9Y+obSJ5\nY+oTS2lHwbgKcTS3cotV+BqoLSbUu3KT2z9Mz9XYDPpzFwLcwtW\n8EWpamlKaUblpI3BXiUsPSUPE9Gqr2q2W+dU2zMuYO1GZ+dTXI\neqTl3sPbqNDubXJ8iNedj0vX1/fmHFEgpXOlP",
  "E9Gqr2q2W+dU2zMuYO1GZ+dTXI\neqTl3sPbqNDubXJ8iNedj0vX1/fmHFEgpXOlPFpZW8FdYWtj/vr\nfyY+/+9v2lB6vtF9pb3lfe1943or3k/fAe+r1vT0v9P70/vb+8\nf5dPFicLv62+Huj3rzRnvOF1/kt/vEf+tb0Sw=</latexit>f [t[x]] = t [f[x]]\n11",
  "Equivariance example\ne.g., Image segmentation\n\u2022 Image has been translated and we want segmentation to translate with it\n12",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n13",
  "\u2022 Input vector x:\n\u2022 Output is weighted sum of neighbors:\n\u2022 Convolutional kernel or filter:\nConvolution* in 1D\n<latexit sha1_base64=\"fRS4\nankXprOks0aLs32QghryuY=\">AWnHiclZhb9s2FIDV7tZ1t\n3TD8jJgEBYUGIbUiIvu8jKgTZq2adLFaeIkbZwalEzJbChKkajE\nqeDX/Zq9bv9l/2aHsmxW5zAPM5CYPt8nXg5J3YJMikKvrf174+\nYH3708Se3Pr392edfPnV0p2vD4u0zEPeD1OZ5scBK7gUive10\nJIfZ",
  "kKvrf174+\nYH3708Se3Pr392edfPnV0p2vD4u0zEPeD1OZ5scBK7gUive10\nJIfZzlnSD5UXC2YfjRBc8LkaoDfZXx04TFSkQiZBpCwyV/EST\n308mw6o7XYX/96erAzlKdbHqw6+t6elwaWts1Z/fFroNoUVr/n\n0hne+HQ1GaVgmXOlQsqI46a5l+rRiuRah5NPbg7LgGQvPWMxPoK\nhYwovTqh7K1L8LkZEfpTn8Ke3X0fePqFhSFdJAGbC9L",
  "ah5NPbg7LgGQvPWMxPoK\nhYwovTqh7K1L8LkZEfpTn8Ke3X0fePqFhSFdJAGbC9LjAzARd7\nKTU0W+nlVBZqbkKZw1FpfR16pu8+COR81DLKyiwMBfQVz8cs5yF\nGrJ3e6D4ZgmCVOjarC+uTetBgGPhar4eVlncjptO5u1w6F4nb\nG+dbCoRWieiHecVFIrpJrB5Pq4p34g4GgMQHU5AqngBdZr8B\nJHfRWjgQMPEgn0LnIfzklVSvNY8hJS3tN",
  "pJrB5Pq4p34g4GgMQHU5AqngBdZr8B\nJHfRWjgQMPEgn0LnIfzklVSvNY8hJS3tNChk9a1gaxYCqT\nlrIPiu/f9Q3gOodZgK7CF0dzsJ8xNZ0fp/lE50lVmBhuIWcq5nU\nTMOSQSTOitqFKeHQsGX9ga2XTJ01iUuzuqu5iSDrIG87Oqd5Ua\nO2U0eQBYswblt1BFkS9vmIJQy3JSHMODENxG3KhRWBVmYvTwN2\nm1nJoLX5iSD/dL2NiuS/guGM",
  "lt1BFkS9vmIJQy3JSHMODENxG3KhRWBVmYvTwN2\nm1nJoLX5iSD/dL2NiuS/guGMmICsPvMt2Aq5G19I13Y/jw5F7Vv\nCnzij2Gy2oewPJ4Na94IjKqJTalZ5wqZNFsQytPLtml641B5Jt\noDNAG86cpcqOg9bUuwZI14cEqDUvJT+51/mZT06rNbNtzD+ST\naioKDNXRSb8PyoawZUFry+I4MlLJZo8CNSTl0o4v6OpYzle2CZS\nzx0UhGJS6Cu",
  "aioKDNXRSb8PyoawZUFry+I4MlLJZo8CNSTl0o4v6OpYzle2CZS\nzx0UhGJS6Cu0/UWs2sfUEdzZNEF9hYCpF76ZUGiSo6gtm4CR4Ru\nukY4FKJBhrMxhjItypyTkx9azxCpdXNazIW5WLVPqNI7fMGl4\nujoAwXhwt+zeEBymgwy2eQlmrEcpTMiZnSyZtBoWGLuXZ/PeWzo\ntOK+fl20x70C2anDEN+PtzG8xETizoS1QU3Jc6JLEc7UFdi+X",
  "BoWGLuXZ/PeWzo\ntOK+fl20x70C2anDEN+PtzG8xETizoS1QU3Jc6JLEc7UFdi+X6\nfs+q7Tc/kaUdO1y3KUm9TS/dtsO9pgf8fMfR2x3iEYs6EtXV9J\nB6xHK0B3W587jGoXDdZuS1DvPo9N2uAsTLf/oYMw1M7dJqRyZ2\n75UDmYhLGoqaqeYJjxG4iyExaRsW/AbK/sCLh5taxbCYq8Qbc0E\nsDTiEg9hFsLibAu3zSaG1R2HuNWmczGyJyFs",
  "aRsW/AbK/sCLh5taxbCYq8Qbc0E\nsDTiEg9hFsLibAu3zSaG1R2HuNWmczGyJyFsPiUJXjUsxAWYyr\nGTvGMZRkSZyGSxzHO45jmMcNS5pLwjGSOGSFLyrWg8nHalkwASx\nPU2sTRGPRApgo12ASxXNCVzhXnkKrWNFV3Hc13L+mYc1QhSaAp\nV2yx/zBrnOTBTjFcJvlSnImkJXRBPaw06PO/O4viCpyJxdEV5Ze\nUXp6SWlR5YeUZpbSp4Igui",
  "TBTjFcJvlSnImkJXRBPaw06PO/O4viCpyJxdEV5Ze\nUXp6SWlR5YeUZpbSp4IguilpeTpJIguL2g9NDSQ0pLS0tK+5\nb2KY0sjSh9YukTSkNLQ0o3LN2gVFtK7kjhimDpAaVjS8eUHlt6T\nOkrS19R+szSZ5S+tvQ1pe8sfUfpI0sfUcosZRuWrpJKbeUvDoI\nonVL1ykNLCXPfrDXLO1RmlmaUfrY0seUjiwlT8VwPbOU3N7AhdF\nSemWpVuUCk",
  "I\nonVL1ykNLCXPfrDXLO1RmlmaUfrY0seUjiwlT8VwPbOU3N7AhdF\nSemWpVuUCkvJ81sQvbD0BaWJpQmlzy19TulbS9S+tTSp5TGlp\nJ3A3B3Yuk+pfYtUFVQumfpHqXnlp673wvwxTQGroW5ayvYpTS1N\nKV021LypAC3EpaekfvJSDVntfnbJnJei9SCO1iT8fnRJOeRWnA\nHa85O86PJ+SlSCz4mXd8XLxIgZTCmX64tNLFb2Fp4fB+p/tL5",
  "1iT8fnRJOeRWnA\nHa85O86PJ+SlSCz4mXd8XLxIgZTCmX64tNLFb2Fp4fB+p/tL58\nHeg5WH680b2lved94P3o9e1/vVe+g983pe3wu9P72/vL+9f5a/X\n368vL38YqbevNEc843X+iwf/gdNFdte</latexit>x = [x1, x2, . . . , xI]\n<latexit sha1\n_base64=\"H6glfS/Nw5hjZi1E\nwP+pcpXOMQ=\">AWuniclZhJ\nc9s2FICZdEvTzWmnvTC",
  "64=\"H6glfS/Nw5hjZi1E\nwP+pcpXOMQ=\">AWuniclZhJ\nc9s2FICZdEvTzWmnvTCqSczn\nSbRWGm6HNpOYsfZ7NSrbCeWowE\npkEIMgjQJ2rI5+jf9Nb2l/6b\nPlCUEL4H6oZR8j7PmJ5AEiIQS\nZFoZeX/712/b3P/jwoxsf3/z\nk08+/2Lh1pf7RVrmIe+FqUzw\n4AVXArFe1poyQ+znLMkPwgOF\nk1/OCM54VI1Z6+yPhxwmIlIhE\nyDaHBwu",
  "FqUzw\n4AVXArFe1poyQ+znLMkPwgOF\nk1/OCM54VI1Z6+yPhxwmIlIhE\nyDaHBwu/+5aASE/83v58mPGaDq\njsZQ+Red3JnFrlfRyb+nbnzQx\n25050MFpaWO8v1x6eFblNY8prP\n1uDW18P+MA3LhCsdSlYUR93lT\nB9XLNcilHxys18WPGPhCYv5ERQ\nVS3hxXNUDnfi3ITL0ozSHP6X9\nOvruFRVLiuIiCcBMmB4VmJmgi\nx2VOvrluBIqKzV",
  "3hxXNUDnfi3ITL0ozSHP6X9\nOvruFRVLiuIiCcBMmB4VmJmgi\nx2VOvrluBIqKzVX4bShqJS+Tn2\nTNX8och5qeQEFuYC+uqHI5az\nUENub/YVPw/TJGFqWPVX1rYnVT\n/gsVAVPy3rPE8mbWetdjgUrzJ\nWnu/NaxGaJ+KSk0pqxVRyhcDjS\nVXxTtzBQHAosMJSBUvoE6Tny\nDyu4jCupKAgQfpGDoX+TsTUrX\nSPIactLTXRINCJvm4Za0SC6",
  "osMJSBUvoE6Tny\nDyu4jCupKAgQfpGDoX+TsTUrX\nSPIactLTXRINCJvm4Za0SC6Yya\nSm7oPj+bd8ArnOYBegqfHE0B7\nsZU5PZdZqPdZ5UhYnhFnKmYl43\nAUMOmTQjahuqlBIuDVvWH9jaY\neqkSVya1V3NTQRZe3nb0TnNixq\n2nTqCLFiEcduqI8iScBcYsoRB\nlpvyAac+CbiVoXCqiALcytPg\n3bmYngtTnOYL+0vbWKpP+MoYy\nYAOw+",
  "BcYsoRB\nlpvyAac+CbiVoXCqiALcytPg\n3bmYngtTnOYL+0vbWKpP+MoYy\nYAOw+8y2YCnlbX03ntj9Lzlnt\nmwIf+yOYrPYlLI+nw5o1AqNqYh\nNq1rlCJs0WhPL0vG2a3jhUnon\n2AE0Ab7oyFyp6R7tbl2DJmnD/L\ngw1LyU/utf5kY+Pq2Wzbcw/J\ntQUVFmropM+H9UNITnDl5fEMG\nTl0o0eRCoJy+VcH9HU8dyvLBNp\nJ47KAjFpNAXaP",
  "UVFmropM+H9UNITnDl5fEMG\nTl0o0eRCoJy+VcH9HU8dyvLBNp\nJ47KAjFpNAXaPuLWLWvqSO4s2\nmC+goBUy98M6HQJEdRWzYBI8M3\nPEdCyhEgwynYwxlWpQ5Jzc/t\nJ4hUuvmtpgL87Bq31ClEdr3DS7\nnV0EZHg5n/IrLA5TRYJrPIC3V\nkOUomWMzpeM3/ULDFnPt/nrKp\n0WnFfPT9aY96BfMThmG/HSwjuc\njJhZ1JKoLjizOuiSxHO1B",
  "M3/ULDFnPt/nrKp\n0WnFfPT9aY96BfMThmG/HSwjuc\njJhZ1JKoLjizOuiSxHO1BXfPl\n+m7PqvU35OlHTtctylJvU0v3b\nbDvaIH/HTD0dsN4hGLOhLV1fS\nQesRytAd1ufO4RqFw3WbktQ7y\n6PTdrhzEy3/aG/ENTPHpFQOzb\nEvlf1pCIuaitop1ufbtjgNYTE\np2xb8Hyu7Ah4ebWsawuJWIdqaC\nWBpyCUewjSExekWbptNDKsbDn\nXDrT",
  "jgNYTE\np2xb8Hyu7Ah4ebWsawuJWIdqaC\nWBpyCUewjSExekWbptNDKsbDn\nXDrTKZjZA5DWHxKUvwqKchLMZU\njJ3iCcsyJE5DJI8jnMcRzWOGp\ncwl4RnJHDNClpRrQeWjtC2ZAJb\nGqLWxozHogUwVarAJYrmgK69w\nrjyFVrGiq7jnarh3RcOaoQpNA\nEubZI/5/U3nJgtwiuGY5UpyJpC\nV0QRuYWeLOrPTXxBV5CQXRBeW\nXlB6buk5pQe",
  "EubZI/5/U3nJgtwiuGY5UpyJpC\nV0QRuYWeLOrPTXxBV5CQXRBeW\nXlB6buk5pQeWHlCaW0p+EQTRjq\nXk10kQnVl6Rum+pfuUlpaWlPY\ns7VEaWRpR+sTSJ5SGloaUrlq6S\nqm2lJxI4Ylg6R6lI0tHlB5aek\njpK0tfUfrM0meUvrb0NaWXl5\nS+sjSR5QySxmla5auUcotJa8Og\nmjF0hVKA0vJbz/Ya5ZuUZpZml\nH62NLHlA4tJb+K4XlmK",
  "ySxmla5auUcotJa8Og\nmjF0hVKA0vJbz/Ya5ZuUZpZml\nH62NLHlA4tJb+K4XlmKTnewIPR\nUknpc0ufUyosJb/fguilpS8pT\nSxNKH1h6QtK31r6ltKnlj6lNLa\nUvBuA04mlu5Tat0BVQem2pduU\nnlp6n4vwOfTGLgW5qatYJPS1\nNKU0nVLyS8FOEpYekLOk5Fq7mq\nzt03kvhapOXewJuOzq0nOIzXn\nDtbcnWZXk/tTpOZ8RLq+tj9/kQ",
  "kLOk5Fq7mq\nzt03kvhapOXewJuOzq0nOIzXn\nDtbcnWZXk/tTpOZ8RLq+tj9/kQ\nIphTv9YGpi9/C0sL+/U73p86\nD7QdLD1eaN7Q3vG+8b73vK73\ns/fQe+ZteT0v9P70/vL+9v5Z/H\nUxWBSLJ1P1+rXmq+81mdR/wd\nXgudT</latexit>zi = !1xi\u22121 + !2xi + !3xi+1\n<latexit sha1_base64=\"0nQ5\nE3qPKtIiw+mOdIpqjLrMKhI=\">AWrXiclZ",
  "+ !3xi+1\n<latexit sha1_base64=\"0nQ5\nE3qPKtIiw+mOdIpqjLrMKhI=\">AWrXiclZhb9s2FIDV7tZ1t\n3TD8rIXYUGBYei8uOsuLwPapOkt6eI0cZImTgxKpmQ2FKVIVOJU\n8O/Yr9nr9hv2b3Yoy2Z1DvMwA6nZ83i5ZCUaAWZFIVeXf3xs\n3v/gw49ufXz7k08/+/yLpTtf7hdpmYe8H6YyzQ8DVnApFO9ro\nSU/zHLOkDyg+Bs3fCDC54XIlV7+",
  "08/+/yLpTtf7hdpmYe8H6YyzQ8DVnApFO9ro\nSU/zHLOkDyg+Bs3fCDC54XIlV7+irjJwmLlYhEyDSEhkvdQRCl\nCY+Z/7t/PKhLw6o7vTcv3rfFn6Ynp9XedLi0stpZrT8+LXSbwor\nXfHrDO1+PBqM0LBOudChZURx3VzN9UrFci1Dy6e1BWfCMhWcs5s\ndQVCzhxUlVj23q34XIyI/SHP6U9uvou1dULCmKqyQAM2F6XGBmg\ni52XOrot5NKq",
  "dQVCzhxUlVj23q34XIyI/SHP6U9uvou1dULCmKqyQAM2F6XGBmg\ni52XOrot5NKqKzUXIWzhqJS+jr1TaL8kch5qOUVFiYC+irH45Z\nzkIN6bw9UPwyTJOEqVE1WNvYmVaDgMdCVfy8rFM7nbadjdrhUL\nzOWHu+t6hFaJ6It5xUiumkmsEHk+rinfiDgaCAxAdTkCqeAF1m\nvwEkd9FJaSBAw8SCfQuch/NSVK81jyElLOyIaFDLJy1rnVgw\nlU",
  "xAdTkCqeAF1m\nvwEkd9FJaSBAw8SCfQuch/NSVK81jyElLOyIaFDLJy1rnVgw\nlUlL2QXF9+/6BnCdwyxAV+GLoznYzZiazq/TfKLzpCpMDLeQMxX\nzugkYcsikGVHbUKWUcGnYsv7A1iumzprEpVnd1dxEkLWXtx2d07\nyoUdupI8iCRi3rTqCLAkbf8QSBluykMYcOKbiFsVCquCLMxen\ngbtjMTwWtzksF+aXsbFUn/BUMZMQHYfeZbMBXytr",
  "luykMYcOKbiFsVCquCLMxen\ngbtjMTwWtzksF+aXsbFUn/BUMZMQHYfeZbMBXytr6eLmx/npyL\n2jcFPvHMFntS1gez4Y1bwRG1cSm1KxzhUyaLQjl6WXbNL1xqD\nwT7QGaAN50ZS5U9I52ry7BkjXhwT0Yal5KfvxD52c+OalWzbYx/\n5BsQkVFmbkqMuH/UdEIHjV4fUET14q0eRBoJ68VML9HU0dy/HC\nNpF67qAgFJNCX6HtL2LVvqaO4M6m",
  "UdEIHjV4fUET14q0eRBoJ68VML9HU0dy/HC\nNpF67qAgFJNCX6HtL2LVvqaO4M6mCeorBEy98M2EQpMcRW3ZBIw\nM3/DQdCygEA0ynI0xlGlR5pzc/NB6hkitm9tiLszDqn1DlUZo3z\ne4XFwFZXg4XPBrLg9QRoNZPoO0VCOWo2ROzJROTgeFhi3m2v31l\nM+KTivm5tNe9AvmJ0yDPn5cBPR0ws6khUF5xSnHVJYjnag7oW\ny/XdnlWbp9+TpR",
  "M+KTivm5tNe9AvmJ0yDPn5cBPR0ws6khUF5xSnHVJYjnag7oW\ny/XdnlWbp9+TpR07XLcpSb1NL92w72mB/x8y9HbLeIRizoS1d\nX0kHrEcrQHdbnzuOUahcN1m5LUO8+j03a4CxMt/2hvzDUzx6RUj\nsyxL5WDWQiLmoraKdan27Y4C2ExKdsW/B8ruwIeHm1rFsJirxBt\nzQSwNOISD2EWwuJsC7fNJobVLYe65VaZzMbInIWw+JQleNSzEBZ",
  "m1rFsJirxBt\nzQSwNOISD2EWwuJsC7fNJobVLYe65VaZzMbInIWw+JQleNSzEBZ\njKsZO8YxlGRJnIZLHMc7jmOYxw1LmkvCMZI4ZIUvKtaDycdqWTA\nBLE9TaxNEY9ECmCjXYBLFc0JVXOFeQqtY0VXcdzXcv6ZhzVCFJ\noClbLH/MG2c5MFOMVwzHIlORPIymgCe9jpUWd+guipzkgujK\n0itKLy29pPTA0gNKc0vJL4IgemUp+XUSRBeWXlC6",
  "IymgCe9jpUWd+guipzkgujK\n0itKLy29pPTA0gNKc0vJL4IgemUp+XUSRBeWXlC6b+k+paWlJa\nV9S/uURpZGlD6x9AmloaUhpeuWrlOqLSUnUngiWLpH6djSMaWHl\nh5S+trS15Q+s/QZpUeWHlH61tK3lD6y9BGlzFJG6YalG5RyS8mr\ngyBas3SN0sBS8tsP9pqlPUozSzNKH1v6mNKRpeRXMTzPLCXHG3g\nwWiopfW7pc0qFpeT3WxC9tPQl",
  "tsP9pqlPUozSzNKH1v6mNKRpeRXMTzPLCXHG3g\nwWiopfW7pc0qFpeT3WxC9tPQlpYmlCaUvLH1B6RtL31D61NKnlM\naWkncDcDqxdJdS+xaoKijdsXSH0nNLz93vBfhiGgPXwty2FWxTm\nlqaUrpKfmlAEcJS8/IeTJSzV1t/raJ3NciteAO1mR8fjXJeaQ\nW3MGau9P8anJ/itSCj0nXN/YXL1IgpXCnHy6tdPFbWFrYv9/p/t\nR+X+8uD5dOZ",
  "W3MGau9P8anJ/itSCj0nXN/YXL1IgpXCnHy6tdPFbWFrYv9/p/t\nR+X+8uD5dOZevNGc81XuzHP8Hxw/iZg=</latexit>J5sPNg5eFa84b2lveN963ndf1fvUes+8ntf3Qu9P7y/vb+f5\n! = [!1, !2, !3]T\nKernel size = 3\n* Not really technically convolution\n14",
  "Convolution with kernel size 3\n15",
  "Convolution with kernel size 3\n16",
  "Convolution with kernel size 3\nEquivariant to translation of input\n<latexit sha1_base64=\"3N9gchi9eQHzUGINVq4oZjiE8Ek=\">AW1Xic\nlZhJb9w2FICVdEvTzWlRX3oRagQoinRgF+lyKZDYcTY79Tje45kYlIbSMKYoWaLscYS5Fb32J/V39Af02v6FPkqaofUefegAjpj3feLySGoLMikKvbz814\n2b7z73vsf3Prw9kcf/LpZwt3Pt8v0jIP+V6YyjQ/DFjBpVB8Twst+WGWc5YEkh8Ep2u",
  "z73vsf3Prw9kcf/LpZwt3Pt8v0jIP+V6YyjQ/DFjBpVB8Twst+WGWc5YEkh8Ep2uGH5zvBCp2tWXGR8mLFYiEiHTEDpZOBokQTqpBkHkR9OB5JE+t\nhE9PYbDZDjIRTzWQ/8X/yrDdtS1TxaWlnvL9c+nhZW2sOS1v/7JnS9Hg1EalglXOpSsKI5XljM9rFiuRSj59PagLHjGwlMW82MoKpbwYljVOZj6dyEy8qM\n0hz+l/Tp69YyKJUVxmQRgJk",
  "uRSj59PagLHjGwlMW82MoKpbwYljVOZj6dyEy8qM\n0hz+l/Tp69YyKJUVxmQRgJkyPC8xM0MWOSx39PKyEykrNVdg0FJXS16lvEuqPRM5DLS+hwMJcQF/9cMxyFmpI+2B4hdhmiRMjarB6vr2FLFY6EqflbWUz\nCdp312uFQvM5YfbY7r0Vonoi3nFRSK6aSawQeT6uK9+IeBoIDED1OQKp4AXWa/MBcryAKS04CruxqeDklVSvNY8hJR3tFNChk",
  "awQeT6uK9+IeBoIDED1OQKp4AXWa/MBcryAKS04CruxqeDklVSvNY8hJR3tFNChk861hqxYCqTjrIDiu/f9\nQ3gOodZgK7CgaM52MmYms7O03yi86QqTAy3kDMV87oJGHLIpBlR1ClHBq2LF+xdZLpk7bxKVZ3dXcRJC1m3cdndO8qFHXqSPIgkUYd606giwJF4gRSxh\nkuS2fwIAT30TcqlBYFWRh9vM06LadmQhem5M9kvXW69I+s8ZyogJwO4",
  "JF4gRSxh\nkuS2fwIAT30TcqlBYFWRh9vM06LadmQhem5M9kvXW69I+s8ZyogJwO4zR8FUyLv6Wjq3/VlyzmvfFPjEH8NkdU9hedwMa9YIjKqNTalZ5wqZNFsQytOLrm\nl641B5JroDNAG86cpcqOiKdq8uwZI14cE9GpeSn78Xe8HPhlWy2bmH9INqGiosxcFZnw/6hoBLckvL4gicvlWjyIFBPXirh+o6mjuV4YZtIPXdQEIpJ\noS/R9hex6p5T",
  "FZnw/6hoBLckvL4gicvlWjyIFBPXirh+o6mjuV4YZtIPXdQEIpJ\noS/R9hex6p5TR3Bn0wT1FQKmXjgyodAkR1FXNgEjwxFuro4FKJBhs0YQ5kWZc7JxQ+tZ4jUurks5sLcrLoXVGmE7nWDy/lZUIabwzm/5vQAZTRo8hmkpRq\nxHCVzYqZ08npQaNhirt1fT3lTdFoxP9to24N+weyUYcjPTjbwfMTEo5EdcHTjLMuSxHe1DXfLle7Vm18fpbsr",
  "T3lTdFoxP9to24N+weyUYcjPTjbwfMTEo5EdcHTjLMuSxHe1DXfLle7Vm18fpbsrRjh+s2Jam37aXbdrjX9ICfbTp6u0k8Y\nlFHoraHlKPWI72oC53Hjdo3C4blOSemd5dNoOd26i5R/tjrlm5jEplSPz2JfKQRPCoqaidopwmMkNiEsJmXgv9jZUfAzaNrNSEs9gvR1UwASyMu8RCa\nEBabLdw12xhWNx3qpltlMhsjswlh8QlL8KibEBZjKsZO",
  "SEs9gvR1UwASyMu8RCa\nEBabLdw12xhWNx3qpltlMhsjswlh8QlL8KibEBZjKsZO8ZRlGRKbEMnjGOdxTPOYSlzSXhGMseMkCXlWlD5O1KJoClCWpt4mgMeiBThRpsg1gu6MornC\ntPoVWs6CreczW8d03DmqEKTQBLW2SP+YMt5yYLcIrhMcuV5EwgK6MJ7GOnT53Z018QVeRJLoguLb2k9MLSC0oPLD2gNLeUvBE0UtLydtJEJ1bek7pvqX7l",
  "GOnT53Z018QVeRJLoguLb2k9MLSC0oPLD2gNLeUvBE0UtLydtJEJ1bek7pvqX7l\nJaWlpTuWbpHaWRpROljSx9TGloaUrpm6Rql2lLyRAp3BEt3KR1bOqb0NJDSo8sPaL0qaVPKX1l6StK31r6ltKHlj6klFnKF23dJ1Sbin5dBEq5auUhp\nYSt79YK9Z2qc0szSj9JGljygdWUreiuF+Zil5vIEbo6WS0meWPqNUWEre34LohaUvKE0sTSh9bul",
  "0szSj9JGljygdWUreiuF+Zil5vIEbo6WS0meWPqNUWEre34LohaUvKE0sTSh9bulzSt9Y+obSJ5Y+oTS2lHwbgKcTS3cotV+BqoLSbUu3KT\n2z9Mz9XYDPpzFwLcwtW8EWpamlKaUblpI3BXiUsPSUPE9Gqr2q2W+dU2zMuYO1GZ+dTXIeqTl3sPbqNDubXJ8iNedj0vX1/fmHFEgpXOlPFpZW8FdYWtj/v\nrfyY+/+9v2lB6vtF9pb3lfe1943o",
  "8iNedj0vX1/fmHFEgpXOlPFpZW8FdYWtj/v\nrfyY+/+9v2lB6vtF9pb3lfe1943or3k/fAe+r1vT0v9P70/vb+8f5dPFicLv62+Huj3rzRnvOF1/kt/vEf+tb0Sw=</latexit>f [t[x]] = t [f[x]]17",
  "Zero padding\nTreat positions that are beyond end of the input as zero.\n18",
  "\u201cValid\u201d convolutions\nOnly process positions where kernel falls in image (smaller output).\n19",
  "Stride, kernel size, and dilation\n\u2022 Stride = shift by k positions for each output\n\u2022 Decreases size of output relative to input\n\u2022 Kernel size = weight a different number of inputs for each output\n\u2022 Combine information from a larger area\n\u2022 But kernel size 5 uses 5 parameters\n\u2022 Dilated or atrous convolutions = intersperse kernel values with zeros\n\u2022 Combine information from a larger area\n\u2022 Fewer parameters\n20",
  "1\n21",
  "1\n1\n22",
  "1\n1\n1\n23",
  "1\n1\n1\n2\n24",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n25",
  "Convolutional layer\n<latexit sha1_base64=\"p8kAci0rsfxNTdm\n+eDTUSvXBwSE=\">AXKHiclZhbU9w2FICXlN6I+mUl754yqTaRKGTdM2L5lJIOQGKRCuCS\nY7slf2CmTZyDIs8fgPdfpj+tbJa39Jj2zvCuIzpSZdNXzfdblSLJlBxlnuVpaej/zwYcfz\nJp9c+m/38iy+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1KS",
  "+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1KSCxYxEKiIDSY+9MbDUpWeT8PwkSMclqXxOI3XoB1QR75bnpwmNyaDsV2MQ7/SrW5PI3\nTpSXJ+riO3+pUvWTxSR75IRZEVPr+7H+0kBfJoDx+0K/eQg3Tyo4r67t+M7dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9Yuc",
  "X2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9YucZiQ8ITE9hKIgCc2Pyjq\ntlXcTIkMvSiX8E8qro5evKEmS5xdJAGZC1Ci3mQ62GhovtHJRNZoagIm4aignsq9fQceUMm\naj4BRIKBn01QtHRJQwUzO+oKeh2mSEDEs/eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog0Y",
  "VraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog0YBcAWKQKpoDnUqfMTRF7forCKOeCyWTQ+GK8qVLVQNI\nacdLQ3SINCxum4Y60gC6Yy6SjboHjeTU8DqiTMAnQVfqg1B9sZEdXkOkXHSiZlrmN2C5KImN\nZNwJBDwvWIuoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZko",
  "YoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZkoRAltvyAaceDriVpmwVYW5qZMg27bmY7Ya3OcwX7peqslSv8ZsTKiA7D79C8jIqRdfSWd\n2t4kOWe1rwt07I1gsrqXEBk3w5o0AqNqYxU261xZJs4WhGR63jV1bxwqzVh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId/h",
  "3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId/h8VDeEpZ68viNiTl3Jr8iBQT1\n7K4f5uTR2R9sLWkXruoMAE4UxdWNufxaJ7TR2xO5smVl8hoOuFX8KENclR1JV1QMvwC89rxwI\nKrUGzRhDnuaFpOjmZ61niNS6vi1Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+nv",
  "h9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+nvCk6rZierXtQb9gdowpKeDNXs+YmRh1t1wQHJWRdHlqM9qGu6X\nC/3rFx7+xNa2rHDdZsc1dv20m073Ct6QE/XHb1dRx6ysMOtutoeYg9ZjvagLnce12jcLhuk6\nN6J3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8Oh",
  "02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8OhaTcg\nWN3PW1XTAloaU20NoQrbYbOGu2cZsd2hrtVwrORZTYhW3xKEnvUTcgWYyzGTvGEZJklNiGU\nx5GdxHOY2ZLmUuyZyRzAhaUq4FJUdpV9IBWxpbrY0djUEPeCqsBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8XcyM",
  "tugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8XcyMpzATdvZxM7k9BdEJTrJBdGFoReYnh\nt6jum+ofuYSkPRG0EQvTIUvZ0E0ZmhZ5juGbqHaWFogemuobuYRoZGmD4x9AmoaEhpiuGrmC\nqDEUnUngiGLqD6cjQEaYHh5g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0bsf",
  "trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0bsf7DVDNzHNDM0wfWzoY0yHhqK3YnieGYqON/BgNJRj+tzQ5gyQ9H7WxC9N\nPQlpomhCaYvDH2B6bGhx5g+NfQprGh6NsAnE4M3cbUfAUqc0y3DN3C9NTQU/d3ATqdxsC1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7d5",
  "D\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7d5pcje5PkZj\nyEer6t70QwqkFO70g7mFv0VFhf27i72f128t3Vv4eFy+4X2Wu+73ve9H3v93m+9h71nvc3e\nbi+cuTFzf+bRzPL8H/N/zf89/75RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2",
  "5RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2 +\n3\nX\nj=1\n!jxi+j\u22122\n3\n5\n26",
  "Special case of fully-connected network\n<latexit sha1_base64=\"p8kAci0rsfxNTdm\n+eDTUSvXBwSE=\">AXKHiclZhbU9w2FICXlN6I+mUl754yqTaRKGTdM2L5lJIOQGKRCuCS\nY7slf2CmTZyDIs8fgPdfpj+tbJa39Jj2zvCuIzpSZdNXzfdblSLJlBxlnuVpaej/zwYcfz\nJp9c+m/38iy+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1",
  "+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1KSCxYxEKiIDSY+9MbDUpWeT8PwkSMclqXxOI3XoB1QR75bnpwmNyaDsV2MQ7/SrW5PI3\nTpSXJ+riO3+pUvWTxSR75IRZEVPr+7H+0kBfJoDx+0K/eQg3Tyo4r67t+M7dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9Y",
  "dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9YucZiQ8ITE9hKIgCc2Pyjq\ntlXcTIkMvSiX8E8qro5evKEmS5xdJAGZC1Ci3mQ62GhovtHJRNZoagIm4aignsq9fQceUMm\naj4BRIKBn01QtHRJQwUzO+oKeh2mSEDEs/eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog",
  "eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog0YBcAWKQKpoDnUqfMTRF7forCKOeCyWTQ+GK8qVLVQNI\nacdLQ3SINCxum4Y60gC6Yy6SjboHjeTU8DqiTMAnQVfqg1B9sZEdXkOkXHSiZlrmN2C5KImN\nZNwJBDwvWIuoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZ",
  "uoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZkoRAltvyAaceDriVpmwVYW5qZMg27bmY7Ya3OcwX7peqslSv8ZsTKiA7D79C8jIqRdfSWd\n2t4kOWe1rwt07I1gsrqXEBk3w5o0AqNqYxU261xZJs4WhGR63jV1bxwqzVh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId",
  "Vh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId/h8VDeEpZ68viNiTl3Jr8iBQT1\n7K4f5uTR2R9sLWkXruoMAE4UxdWNufxaJ7TR2xO5smVl8hoOuFX8KENclR1JV1QMvwC89rxwI\nKrUGzRhDnuaFpOjmZ61niNS6vi1Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+",
  "Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+nvCk6rZierXtQb9gdowpKeDNXs+YmRh1t1wQHJWRdHlqM9qGu6X\nC/3rFx7+xNa2rHDdZsc1dv20m073Ct6QE/XHb1dRx6ysMOtutoeYg9ZjvagLnce12jcLhuk6\nN6J3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8",
  "3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8OhaTcg\nWN3PW1XTAloaU20NoQrbYbOGu2cZsd2hrtVwrORZTYhW3xKEnvUTcgWYyzGTvGEZJklNiGU\nx5GdxHOY2ZLmUuyZyRzAhaUq4FJUdpV9IBWxpbrY0djUEPeCqsBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8Xc",
  "sBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8XcyMpzATdvZxM7k9BdEJTrJBdGFoReYnh\nt6jum+ofuYSkPRG0EQvTIUvZ0E0ZmhZ5juGbqHaWFogemuobuYRoZGmD4x9AmoaEhpiuGrmC\nqDEUnUngiGLqD6cjQEaYHh5g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0b",
  "g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0bsf7DVDNzHNDM0wfWzoY0yHhqK3YnieGYqON/BgNJRj+tzQ5gyQ9H7WxC9N\nPQlpomhCaYvDH2B6bGhx5g+NfQprGh6NsAnE4M3cbUfAUqc0y3DN3C9NTQU/d3ATqdxsC1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7",
  "1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7d5pcje5PkZj\nyEer6t70QwqkFO70g7mFv0VFhf27i72f128t3Vv4eFy+4X2Wu+73ve9H3v93m+9h71nvc3e\nbi+cuTFzf+bRzPL8H/N/zf89/75RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2",
  "5RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2 +\n3\nX\nj=1\n!jxi+j\u22122\n3\n5\n<latexit sha1_base64=\"KZBqmLp19s93\n3dVIEAqOjCngwU=\">AWx3iclZhbc9w0FIBdrqXcUhjywouHTBkGSiZhyuWlM23StE\n2TkqS5tnG6I3tlrxJZdmw52dTjB34Sv4bhDf4JR/buqj5HeSAzZcX5PutyJNmyw1yKU\ni8t/X3jn",
  "tlrxJZdmw52dTjB34Sv4bhDf4JR/buqj5HeSAzZcX5PutyJNmyw1yKU\ni8t/X3jnXfe/+D29+dOvjTz797PO5218clFlVRHw/ymRWHIWs5FIovq+FlvwoLzhL\nQ8kPw7NVw8veFGKTO3pq5yfpCxRIhYR0xAazG34o0EtGv/b+36Qhtm4Zk0geayPg5Br\n1qIf/KCs0kF9en+5eV0/avwgS3li2GnjyHeBIVIRvpkMLewtLjU/vm0sDwpLHiTv+3",
  "If/KCs0kF9en+5eV0/avwgS3li2GnjyHeBIVIRvpkMLewtLjU/vm0sDwpLHiTv+3\nB7a+GwTCLqpQrHUlWlsfLS7k+qVmhRSR5cyuoSp6z6Iwl/BiKiqW8PKnbUTf+HYgM/Tg\nr4J/Sfht9+4qapWV5lYZgpkyPSsxM0MWOKx3/dlILlVeaq6hrK6krzPfpNAfioJHWl\n5BgUWFgL760YgVLNKQ6FuB4pdRlqZMDetgZW2nqSGViVA1P6/apDdN3",
  "AfioJHWl\n5BgUWFgL760YgVLNKQ6FuB4pdRlqZMDetgZW2nqSGViVA1P6/apDdN31lrHQ7F64yV9\nb1ZLULzVLzhpJWMZVcI/CkqWu+mCxiIDgAscgJyBQvoU6TnzD2lxGFRSYB192qCcB40\nZCqleYJ5KSnvSIaFHLJxz1rlVgwlWlP2QXF9+/4BnBdwCxAV+GHoznYzZlqptdpPtZF\nWpcmhlsomEp42wQMOWLSjKhvqEpKuDTqWb9j6wVTZ5P",
  "V+GHoznYzZlqptdpPtZF\nWpcmhlsomEp42wQMOWLSjKhvqEpKuDTqWb9j6wVTZ5PEZXnb1cJEkLVX9B1d0LyoYd9\npI8iCRZj0rTaCLAm3hCFLGWR5Uh7AgFPfRNyqUFgVZGFuF1nYbzs3Ebw2xznsl763VpP\n0XzCUEROA3Wd+BVMR7+ur2cz2p8m5aH1T4GN/BJPVv4QVSTesaSMwqkmsoWabK2TSbE\nGoyC7pumNQ+W56A/QBPCmqwqh4re0u",
  "GN/BJPVv4QVSTesaSMwqkmsoWabK2TSbE\nGoyC7pumNQ+W56A/QBPCmqwqh4re0u20JlqwJB3dhqEUl+fGPiz/z8Um9ZLaN+Q/Jl\nRUVrmrIhP+HxUN4SGE1xdE8ORlEk0eBNrJyTc39HUsQIvbBNp5w4KQjEp9BXa/iJR/\nWvaCO5slqK+QsDUC79MKDTJcdyXTcDI8AuPU8cCitAgo26MkczKquDk5ofWM0Ra3dwW\nC2EeVv0bqjRC/7B5ewqK",
  "dyXTcDI8AuPU8cCitAgo26MkczKquDk5ofWM0Ra3dwW\nC2EeVv0bqjRC/7B5ewqKMPD4YJfc3mIMhp2+QyzSg1ZgZI5NlM6fh2UGraYa/e3U94V\nnVbCzcm7UG/YHaqKOLngw08HwmxqCNRXB+cdYlieVoD+qaLde3e1ZvP6eLO3E4bp\nNSeqd9NJtO9xresDPNx293SQesagjUV2THlKPWI72oC53Hjdo3C4blOSeqd5dNoOd2a\ni5R/vjeDAao",
  "DPNx293SQesagjUV2THlKPWI72oC53Hjdo3C4blOSeqd5dNoOd2a\ni5R/vjeDAao5JmRyaY18mgy6ERU1F7RTbY25f7EJYTKu+Bf+PlV0BD4+1YWwuF2Kvm\nYCWBpyiYfQhbDYbeG+OYlhdOhbrpVJvMRMrsQFp+wFI+6C2ExoWLiFM9YniOxC5E8j\nnAeRzSPOZyl4RnJHfMCFlSrgVjLK+ZAJYGqPWxo7GoAcyU6jBSRDLJV15pXPlKbSKF\nV3",
  "SPOZyl4RnJHfMCFlSrgVjLK+ZAJYGqPWxo7GoAcyU6jBSRDLJV15pXPlKbSKF\nV3F+6G969pWDNUoQlgaYvsMT/Ycm6yEKfYb19R6XQJZOU0gdvY2abO9PQXxjU5yYXx\nlaVXlF5aeknpoaWHlBaWkjeCMH5hKXk7CeMLSy8oPbD0gNLK0orSfUv3KY0tjSl9bOl\njSiNLI0pXLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSlpS8pfWrpU0pfWfqK0j",
  "9bOl\njSiNLI0pXLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSlpS8pfWrpU0pfWfqK0jeWvqH0oaU\nPKWMkrXLF2jlFtKPh2E8YqlK5SGlpJ3P9hrlm5TmluaU/rI0keUDi0lb8XwPLOUHG\n/gwWipHTd0nVKhaXk/S2Mn1v6nNLU0pTSZ5Y+o/TU0lNKn1j6hNLEUvJtAE4nlu5Sar\n8C1SWlO5buUHpu6bn7uwCfTWPoWphbtoItSjNLM0o3LCVvCn",
  "EUvJtAE4nlu5Sar\n8C1SWlO5buUHpu6bn7uwCfTWPoWphbtoItSjNLM0o3LCVvCnCUsPSMnCdjNbmrTb82k\nftarGbcwSYZn15Nch6rGXewyd1pejW5P8Vqxkek62sHsw8pkFK40w/mFpbxV1haOPhp\ncfmXxXs79xYerEy+0N70va+8b7zlr1fvQfeU2/b2/ci70/vL+8f79/59fls/mJ+3Knv\n3Jhc86X+5v/4z9uze1w</latexit>\nhi =",
  "/b2/ci70/vL+8f79/59fls/mJ+3Knv\n3Jhc86X+5v/4z9uze1w</latexit>\nhi = a\n2\n4\u03b2i +\nD\nX\nj=1\n!ijxj\n3\n5\nFully connected network:\nConvolutional network:\n27",
  "Special case of fully-connected network\n<latexit sha1_base64=\"p8kAci0rsfxNTdm\n+eDTUSvXBwSE=\">AXKHiclZhbU9w2FICXlN6I+mUl754yqTaRKGTdM2L5lJIOQGKRCuCS\nY7slf2CmTZyDIs8fgPdfpj+tbJa39Jj2zvCuIzpSZdNXzfdblSLJlBxlnuVpaej/zwYcfz\nJp9c+m/38iy+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1",
  "+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1KSCxYxEKiIDSY+9MbDUpWeT8PwkSMclqXxOI3XoB1QR75bnpwmNyaDsV2MQ7/SrW5PI3\nTpSXJ+riO3+pUvWTxSR75IRZEVPr+7H+0kBfJoDx+0K/eQg3Tyo4r67t+M7dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9Y",
  "dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9YucZiQ8ITE9hKIgCc2Pyjq\ntlXcTIkMvSiX8E8qro5evKEmS5xdJAGZC1Ci3mQ62GhovtHJRNZoagIm4aignsq9fQceUMm\naj4BRIKBn01QtHRJQwUzO+oKeh2mSEDEs/eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog",
  "eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog0YBcAWKQKpoDnUqfMTRF7forCKOeCyWTQ+GK8qVLVQNI\nacdLQ3SINCxum4Y60gC6Yy6SjboHjeTU8DqiTMAnQVfqg1B9sZEdXkOkXHSiZlrmN2C5KImN\nZNwJBDwvWIuoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZ",
  "uoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZkoRAltvyAaceDriVpmwVYW5qZMg27bmY7Ya3OcwX7peqslSv8ZsTKiA7D79C8jIqRdfSWd\n2t4kOWe1rwt07I1gsrqXEBk3w5o0AqNqYxU261xZJs4WhGR63jV1bxwqzVh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId",
  "Vh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId/h8VDeEpZ68viNiTl3Jr8iBQT1\n7K4f5uTR2R9sLWkXruoMAE4UxdWNufxaJ7TR2xO5smVl8hoOuFX8KENclR1JV1QMvwC89rxwI\nKrUGzRhDnuaFpOjmZ61niNS6vi1Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+",
  "Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+nvCk6rZierXtQb9gdowpKeDNXs+YmRh1t1wQHJWRdHlqM9qGu6X\nC/3rFx7+xNa2rHDdZsc1dv20m073Ct6QE/XHb1dRx6ysMOtutoeYg9ZjvagLnce12jcLhuk6\nN6J3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8",
  "3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8OhaTcg\nWN3PW1XTAloaU20NoQrbYbOGu2cZsd2hrtVwrORZTYhW3xKEnvUTcgWYyzGTvGEZJklNiGU\nx5GdxHOY2ZLmUuyZyRzAhaUq4FJUdpV9IBWxpbrY0djUEPeCqsBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8Xc",
  "sBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8XcyMpzATdvZxM7k9BdEJTrJBdGFoReYnh\nt6jum+ofuYSkPRG0EQvTIUvZ0E0ZmhZ5juGbqHaWFogemuobuYRoZGmD4x9AmoaEhpiuGrmC\nqDEUnUngiGLqD6cjQEaYHh5g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0b",
  "g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0bsf7DVDNzHNDM0wfWzoY0yHhqK3YnieGYqON/BgNJRj+tzQ5gyQ9H7WxC9N\nPQlpomhCaYvDH2B6bGhx5g+NfQprGh6NsAnE4M3cbUfAUqc0y3DN3C9NTQU/d3ATqdxsC1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7",
  "1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7d5pcje5PkZj\nyEer6t70QwqkFO70g7mFv0VFhf27i72f128t3Vv4eFy+4X2Wu+73ve9H3v93m+9h71nvc3e\nbi+cuTFzf+bRzPL8H/N/zf89/75RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2",
  "5RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2 +\n3\nX\nj=1\n!jxi+j\u22122\n3\n5\n<latexit sha1_base64=\"KZBqmLp19s93\n3dVIEAqOjCngwU=\">AWx3iclZhbc9w0FIBdrqXcUhjywouHTBkGSiZhyuWlM23StE\n2TkqS5tnG6I3tlrxJZdmw52dTjB34Sv4bhDf4JR/buqj5HeSAzZcX5PutyJNmyw1yKU\ni8t/X3jn",
  "tlrxJZdmw52dTjB34Sv4bhDf4JR/buqj5HeSAzZcX5PutyJNmyw1yKU\ni8t/X3jnXfe/+D29+dOvjTz797PO5218clFlVRHw/ymRWHIWs5FIovq+FlvwoLzhL\nQ8kPw7NVw8veFGKTO3pq5yfpCxRIhYR0xAazG34o0EtGv/b+36Qhtm4Zk0geayPg5Br\n1qIf/KCs0kF9en+5eV0/avwgS3li2GnjyHeBIVIRvpkMLewtLjU/vm0sDwpLHiTv+3",
  "If/KCs0kF9en+5eV0/avwgS3li2GnjyHeBIVIRvpkMLewtLjU/vm0sDwpLHiTv+3\nB7a+GwTCLqpQrHUlWlsfLS7k+qVmhRSR5cyuoSp6z6Iwl/BiKiqW8PKnbUTf+HYgM/Tg\nr4J/Sfht9+4qapWV5lYZgpkyPSsxM0MWOKx3/dlILlVeaq6hrK6krzPfpNAfioJHWl\n5BgUWFgL760YgVLNKQ6FuB4pdRlqZMDetgZW2nqSGViVA1P6/apDdN3",
  "AfioJHWl\n5BgUWFgL760YgVLNKQ6FuB4pdRlqZMDetgZW2nqSGViVA1P6/apDdN31lrHQ7F64yV9\nb1ZLULzVLzhpJWMZVcI/CkqWu+mCxiIDgAscgJyBQvoU6TnzD2lxGFRSYB192qCcB40\nZCqleYJ5KSnvSIaFHLJxz1rlVgwlWlP2QXF9+/4BnBdwCxAV+GHoznYzZlqptdpPtZF\nWpcmhlsomEp42wQMOWLSjKhvqEpKuDTqWb9j6wVTZ5P",
  "V+GHoznYzZlqptdpPtZF\nWpcmhlsomEp42wQMOWLSjKhvqEpKuDTqWb9j6wVTZ5PEZXnb1cJEkLVX9B1d0LyoYd9\npI8iCRZj0rTaCLAm3hCFLGWR5Uh7AgFPfRNyqUFgVZGFuF1nYbzs3Ebw2xznsl763VpP\n0XzCUEROA3Wd+BVMR7+ur2cz2p8m5aH1T4GN/BJPVv4QVSTesaSMwqkmsoWabK2TSbE\nGoyC7pumNQ+W56A/QBPCmqwqh4re0u",
  "GN/BJPVv4QVSTesaSMwqkmsoWabK2TSbE\nGoyC7pumNQ+W56A/QBPCmqwqh4re0u20JlqwJB3dhqEUl+fGPiz/z8Um9ZLaN+Q/Jl\nRUVrmrIhP+HxUN4SGE1xdE8ORlEk0eBNrJyTc39HUsQIvbBNp5w4KQjEp9BXa/iJR/\nWvaCO5slqK+QsDUC79MKDTJcdyXTcDI8AuPU8cCitAgo26MkczKquDk5ofWM0Ra3dwW\nC2EeVv0bqjRC/7B5ewqK",
  "dyXTcDI8AuPU8cCitAgo26MkczKquDk5ofWM0Ra3dwW\nC2EeVv0bqjRC/7B5ewqKMPD4YJfc3mIMhp2+QyzSg1ZgZI5NlM6fh2UGraYa/e3U94V\nnVbCzcm7UG/YHaqKOLngw08HwmxqCNRXB+cdYlieVoD+qaLde3e1ZvP6eLO3E4bp\nNSeqd9NJtO9xresDPNx293SQesagjUV2THlKPWI72oC53Hjdo3C4blOSeqd5dNoOd2a\ni5R/vjeDAao",
  "DPNx293SQesagjUV2THlKPWI72oC53Hjdo3C4blOSeqd5dNoOd2a\ni5R/vjeDAao5JmRyaY18mgy6ERU1F7RTbY25f7EJYTKu+Bf+PlV0BD4+1YWwuF2Kvm\nYCWBpyiYfQhbDYbeG+OYlhdOhbrpVJvMRMrsQFp+wFI+6C2ExoWLiFM9YniOxC5E8j\nnAeRzSPOZyl4RnJHfMCFlSrgVjLK+ZAJYGqPWxo7GoAcyU6jBSRDLJV15pXPlKbSKF\nV3",
  "SPOZyl4RnJHfMCFlSrgVjLK+ZAJYGqPWxo7GoAcyU6jBSRDLJV15pXPlKbSKF\nV3F+6G969pWDNUoQlgaYvsMT/Ycm6yEKfYb19R6XQJZOU0gdvY2abO9PQXxjU5yYXx\nlaVXlF5aeknpoaWHlBaWkjeCMH5hKXk7CeMLSy8oPbD0gNLK0orSfUv3KY0tjSl9bOl\njSiNLI0pXLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSlpS8pfWrpU0pfWfqK0j",
  "9bOl\njSiNLI0pXLV2lVFtKTqTwRLB0j9KRpSNKjyw9ovSlpS8pfWrpU0pfWfqK0jeWvqH0oaU\nPKWMkrXLF2jlFtKPh2E8YqlK5SGlpJ3P9hrlm5TmluaU/rI0keUDi0lb8XwPLOUHG\n/gwWipHTd0nVKhaXk/S2Mn1v6nNLU0pTSZ5Y+o/TU0lNKn1j6hNLEUvJtAE4nlu5Sar\n8C1SWlO5buUHpu6bn7uwCfTWPoWphbtoItSjNLM0o3LCVvCn",
  "EUvJtAE4nlu5Sar\n8C1SWlO5buUHpu6bn7uwCfTWPoWphbtoItSjNLM0o3LCVvCnCUsPSMnCdjNbmrTb82k\nftarGbcwSYZn15Nch6rGXewyd1pejW5P8Vqxkek62sHsw8pkFK40w/mFpbxV1haOPhp\ncfmXxXs79xYerEy+0N70va+8b7zlr1fvQfeU2/b2/ci70/vL+8f79/59fls/mJ+3Knv\n3Jhc86X+5v/4z9uze1w</latexit>\nhi =",
  "/b2/ci70/vL+8f79/59fls/mJ+3Knv\n3Jhc86X+5v/4z9uze1w</latexit>\nhi = a\n2\n4\u03b2i +\nD\nX\nj=1\n!ijxj\n3\n5\nFully connected network:\nConvolutional network:\n3 weights, 1 bias\n\ud835\udc37! weights, D biases\n28",
  "Special case of fully-connected network\nFully connected network\n29\nWeight\nMatrix\nBias is \nimplied",
  "Special case of fully-connected network\nFully connected network\nConvolution, kernel 3, \nstride 1, dilation 1\n30\nWeight\nMatrices\nBias is \nimplied",
  "Special case of fully-connected network\nFully connected network\nConvolution, size 3, stride 1,\ndilation 1, zero padding\nConvolution, size 3, stride 2,\ndilation 1, zero padding\n31\nWeight\nMatrices\nBias is \nimplied",
  "Question 1\n\u2022 Kernel size?\n\u2022 Stride?\n\u2022 Dilation?\n\u2022 Zero padding / valid?\n32\nBias is \nimplied",
  "33\nConvolution Configuration\n\u24d8 Start presenting to display the poll results on this slide.",
  "Question 2\n\u2022 Kernel size?\n\u2022 Stride?\n\u2022 Dilation?\n\u2022 Zero padding / valid?\n34\nBias is \nimplied",
  "35\nConv Config 2\n\u24d8 Start presenting to display the poll results on this slide.",
  "Question 3\n\u2022 Kernel size?\n\u2022 Stride?\n\u2022 Dilation?\n\u2022 Zero padding / valid?\n36\nBias is \nimplied",
  "37\nConv Config 3\n\u24d8 Start presenting to display the poll results on this slide.",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n38",
  "Channels\n\u2022 The convolutional operation averages together the inputs\n\u2022 Plus passes through ReLU function\n\u2022 Result is loss of information\n\u2022 Solution: \n\u2022 apply several convolutions and stack them in channels\n\u2022 Sometimes also called feature maps\n39",
  "Two output channels, one input channel\n40",
  "Two output channels, one input channel\n41",
  "Two input channels, one output channel\n42",
  "How many parameters?\n\u2022 If there are \ud835\udc36! input channels and kernel size K\n\u2022 If there are \ud835\udc36! input channels and \ud835\udc36\" output channels \n<latexit sha1\n_base64=\"AxEdXUlG9Ar0tM8N9\nvzMeSlitM=\">AWoXiclZhb\nb9s2FIDV7tZ1t3TD8rIXYUGBY\neiMZOguj23StE2TLs7FSdo4NSi\nZktlQlCJRiVPBv2C/Zq/bL9m/\n2aEkm9U5zMpGbP94mXQ1KiFW\nRSFHp19d9btz/48KOP7nz6d3\nPv/iy6+W7n19VKRlHvJBmMo0P\nwlY",
  "4mXQ1KiFW\nRSFHp19d9btz/48KOP7nz6d3\nPv/iy6+W7n19VKRlHvJBmMo0P\nwlYwaVQfKCFlvwkyzlLAsmPg/\nMNw48veV6IVB3q64yfJSxWIhI\nh0xAaLd0fBtFuwmM2FGqYMD0Jg\nmp/9qbaGFViNtQi4YW/PRstra\nz2VuPTwtrbWHFaz/90b1vx8Nx\nGpYJVzqUrChO1YzfVaxXItQ8\ntndYVnwjIXnLOanUFQMGjqr6vH\nM/PsQGftRmsO",
  "GpYJVzqUrChO1YzfVaxXItQ8\ntndYVnwjIXnLOanUFQMGjqr6vH\nM/PsQGftRmsOf0n4df+KiVF\ncZ0EYJouF5iZoIudljr6/awSK\nis1V2HTUFRKX6e+SY4/FjkPtby\nGAgtzAX31wnLWaghXeHil+F\naZIwNa6G65t7s2oY8Fioil+UdT\npns6zWTscijcZ61uHi1qE5ol\n4x0kltWIquUHg8ayqeC/uYSA4A\nNHjBKSKF1BnPd2Rv4YoLB8JG",
  "Z61uHi1qE5ol\n4x0kltWIquUHg8ayqeC/uYSA4A\nNHjBKSKF1BnPd2Rv4YoLB8JGH\niQTqFzkb8/I1UrzWPISUd7TQ\noZJPO9YGsWAqk45yAIrv3/cN4\nDqHWYCuwhdHc3CQMTWbX6f5VO\ndJVZgYbiFnKuZ1EzDkEkzoq6h\nSinh0rBj/YGtfabO28SlWd3V3\nESQdZh3HZ3TvKhx16kjyIJFGHe\ntOoIsCZt9zBIGW7LIxhw4puI\nWxUKq4Isz",
  "V3\nESQdZh3HZ3TvKhx16kjyIJFGHe\ntOoIsCZt9zBIGW7LIxhw4puI\nWxUKq4IszH6eBt2MxPBa3Oaw\nX7pepsVSf8lQxkxAdh95lswFfK\nuvpEubH+enMvaNwU+9ScwWd1L\nWB43w5o3AqNqYzNq1rlCJs0WhP\nL0qmua3jhUnonuAE0Ab7oyFyp\n6T3tQl2DJmvDwAQw1LyU/an3C\n5+eVatm25h/SDahoqLMXBWZ8P\n+oaAyPF7y+InL5Vo8i",
  "2DJmvDwAQw1LyU/an3C\n5+eVatm25h/SDahoqLMXBWZ8P\n+oaAyPF7y+InL5Vo8iBQT14\nq4f6Opo7leGbSD13UBCKSaGv0\nfYXsepeU0dwZ9ME9RUCpl74Zk\nKhSY6irmwCRoZveFA6FlCIBhk2\nYwxlWpQ5Jzc/tJ4hUuvmtpgL8\n7Dq3lClEbr3DS4XV0EZHg6X/Ib\nLA5TRoMlnkJZqzHKUzKmZ0umb\nYaFhi7l2fz3lTdFpxfxiu20P+\ngW",
  "EZHg6X/Ib\nLA5TRoMlnkJZqzHKUzKmZ0umb\nYaFhi7l2fz3lTdFpxfxiu20P+\ngWzU4Yhvxht4/mIiUdieqCk4m\nzLksR3tQ12K5vt+zavNj2Rp\nxw7XbUpSb9tLt+1wb+gBv9hx9H\naHeMSijkR1tT2kHrEc7UFd7jz\nuEbhcN2mJPXO8+i0He7CRMs/O\npxwzcwxKZVjc+xL5bAJYVFTUT\nvF1Jx4u2ITwmJSdi34P1YOBDw\n8ulYTwmK/EF3N",
  "xwzcwxKZVjc+xL5bAJYVFTUT\nvF1Jx4u2ITwmJSdi34P1YOBDw\n8ulYTwmK/EF3NBLA05hIPoQlhs\ndnCXbONYXHoe64VSazCTKbEB\nafsQSPuglhMaZi7BTPWZYhsQmR\nPE5wHic0jxmWMpeEZyRzAhZU\nq4FlU/SrmQCWJqi1qaOxqAHMlW\nowTaI5YKuvMK58hRaxYqu4oGr\n4cENDWuGKjQBLO2SPeYPd52bL\nMAphmOWK8mZQFZGE9jHTp8",
  "MK58hRaxYqu4oGr\n4cENDWuGKjQBLO2SPeYPd52bL\nMAphmOWK8mZQFZGE9jHTp8689N\nfEFXkJBdE15ZeU3pl6RWlx5Ye\nU5pbSn4RBNG+peTXSRBdWnpJ6Z\nGlR5SWlpaUDiwdUBpZGlH61NK\nnlIaWhpRuWLpBqbaUnEjhiWDpI\naUTSyeUnlh6QukrS19R+tzS5\nS+tvQ1pe8sfUfpY0sfU8osZR\nuWrpJKbeUvDoIonVL1ykNLCW/\nWCvWd",
  "9R+tzS5\nS+tvQ1pe8sfUfpY0sfU8osZR\nuWrpJKbeUvDoIonVL1ykNLCW/\nWCvWdqnNLM0o/SJpU8oHVtKfh\nXD8xScryB6OlktItS7coFZaS\n329B9NLSl5QmliaUvrD0BaVvL\nX1L6TNLn1EaW0reDcDpxNIDSu1\nboKqgdM/SPUovL1wvxfgi2kM\nXAtz1awS2lqaUrptqXklwIcJ\nSw9J+fJSLV3tfnbJnJfi9SCO1i\nb8fnVJOeRWnAHa+9",
  "1awS2lqaUrptqXklwIcJ\nSw9J+fJSLV3tfnbJnJfi9SCO1i\nb8fnVJOeRWnAHa+9O86vJ/SlS\nCz4hXd8WrxIgZTCnX60tLKG38\nLSwtHPvbVfew/3Hq48Wm/f0N7\nxvO+937w1rzfvEfec6/vDbzQ\n+9P7y/vb+2d5ZXlrub+836i3b7\nit>XfON1Psun/wE0N5W</latex\n\u2326 2 RCi\u21e5K\n<latexit sha1\n_base64=\"ET5sPe+tdB59cuG67\nebe",
  "E0N5W</latex\n\u2326 2 RCi\u21e5K\n<latexit sha1\n_base64=\"ET5sPe+tdB59cuG67\nebe858Qh6s=\">AWrXiclZjZ\ncts2FECZrm6Oe3UL3h1JOZT\nidVrU6PCZ2nM1Ovcp2YtoakAI\npxCBIk6Ath6Pv6Nf0tf2G/k0v\nKEoI74UfqhlHyD2HWC4AEmKYS1\nHq1dV/b73/gcfvTx7U/ufPr\nZ5198uXT3q8Myq4qID6JMZsVxy\nEouheIDLbTkx3nBWR",
  "V/b73/gcfvTx7U/ufPr\nZ5198uXT3q8Myq4qID6JMZsVxy\nEouheIDLbTkx3nBWRpKfhSerx\nt+dMmLUmTqQF/n/DRliRKxiJi\nG0HCpH4TxdsoTFgVpEyPw7Dem\n57V68NaTAMtUl76UM7m5c3pcG\nltbfafHxa6LeFa/97AzvfjMK\nRlUpVzpSLKyPOmv5vq0ZoUWk\neTO0FV8pxF5yzhJ1BUDBo6rZu\nxTf17EBn5cVbAn9J+E3ipql\nZXmdh",
  "vq0ZoUWk\neTO0FV8pxF5yzhJ1BUDBo6rZu\nxTf17EBn5cVbAn9J+E3ipql\nZXmdhmCa7peYmaCLnVQ6/v20F\niqvNFfRrKG4kr7OfJMofyQKHml\n5DQUWFQL6kdjVrBIQzrvBIpf\nRVmaMjWqg7WN3WkdhDwRquYXVZ\nPa6bTrbDQOh+JNxtrzg0UtQvN\nUvOWkYxldwg8GRa17yX9DAQH\nIDocQIyxUuos5n62O8jCktJAg\nYeZhPoXOzvTUnVS",
  "OWkYxldwg8GRa17yX9DAQH\nIDocQIyxUuos5n62O8jCktJAg\nYeZhPoXOzvTUnVSvMEctLRXhM\nNCrnk461TiyYyrSj7IPi+/d8A\n7guYBagq/DF0Rzs50xN59dpPt\nFWpcmhlsomEp40wQMOWLSjKhr\nqEpKuDTqWH9ga4+p8zZxWd50t\nTARZB0UXUcXNC9q1HWaCLJgESZ\ndq4kgS8LGH7GUQZb8hAGnPom\n4laFwqogC3OnyMJu27mJ4LU5y",
  "9q1HWaCLJgESZ\ndq4kgS8LGH7GUQZb8hAGnPom\n4laFwqogC3OnyMJu27mJ4LU5y\nWG/dL2NmqT/kqGMmADsPvMtmIp\n4V1/PFrY/T85l45sCn/hjmKzu\nJaxIZsOaNwKjamNTaja5QibNFo\nSK7Kprmt4VJ6L7gBNAG+6qhA\nqfke735RgyZpwcB+GWlSn/zY+\n4VPTutVs23MPySbUFZ5a6KTP\nh/VDSCRw1eXxDBk5dJNHkQaCY\nvk3B/R1PH",
  "zY+\n4VPTutVs23MPySbUFZ5a6KTP\nh/VDSCRw1eXxDBk5dJNHkQaCY\nvk3B/R1PHCrywTaSZOygIxaTQ1\n2j7i0R1r2kiuLNZivoKAVMvfD\nOh0CTHcVc2ASPDNzw0HQsoQoOM\nZmOMZFZWBSc3P7SeIdLo5rZYC\nPOw6t5QpRG69w0uF1dBGR4Ol/y\nGy0OU0XCWzCr1IgVKJkTM6WT\ns6DUsMVcu7+Z8lnRaSX8YrNtD\n/oFs1NFEb8YbuL5SI",
  "U0XCWzCr1IgVKJkTM6WT\ns6DUsMVcu7+Z8lnRaSX8YrNtD\n/oFs1NFEb8YbuL5SIhFHYnqglO\nKsy5JLEd7UNdiub7bs3rz7Aey\ntBOH6zYlqbftpdt2uDf0gF9sOX\nq7RTxiUeiutoeUo9YjvagLnc\net1yjcLhuU5J653l02g53YaLlH\nx+MuWbmJTJkTn2ZTKYhbCoqa\nidYmZOv1xFsJiWnUt+D9W9gU\n8PLrWLITFnVJ0NRPA0ohLPIRZC",
  "2ZTKYhbCoqa\nidYmZOv1xFsJiWnUt+D9W9gU\n8PLrWLITFnVJ0NRPA0ohLPIRZC\nIuzLdw12xhWtxzqltlMh8jcx\nbC4lOW4lHPQlhMqJg4xXOW50ic\nhUgexziPY5rHEu5S8Izkjtmh\nCwp14IqxlXMgEsTVBrE0dj0AO\nZKdRgG8RySVde6Vx5Cq1iRVfx\nwNXw4IaGNUMVmgCWtske84Nt5\nyYLcYrhmOVKci6QldME7mBnhzr\nz018Y1+QkF",
  "wNXw4IaGNUMVmgCWtske84Nt5\nyYLcYrhmOVKci6QldME7mBnhzr\nz018Y1+QkF8bXl5TemXpFaVH\nlh5RWlhKfhGE8Z6l5NdJGF9aek\nnpoaWHlFaWVpQOLB1QGlsaU/r\nE0ieURpZGlK5buk6ptpScSOGJY\nOkBpWNLx5QeW3pM6StLX1H6zN\nJnlL629DWlby19S+kjSx9Ryix\nlG5YukEpt5S8OgjNUvXKA0tJ\nb/9YK9ZukNpbmlO6WNLH",
  "DWlby19S+kjSx9Ryix\nlG5YukEpt5S8OgjNUvXKA0tJ\nb/9YK9ZukNpbmlO6WNLH1M6sp\nT8KobnmaXkeAMPRkslpc8tfU6p\nsJT8fgvjl5a+pDS1NKX0haUvK\nH1j6RtKn1r6lNLEUvJuAE4nlu5\nTat8C1SWlu5buUnph6YX7vQBf\nTGPoWpjbtoJtSjNLM0o3LSW/F\nOAoYek5OU/Gqr2rzd82kftarBb\ncwdqMz68mOY/VgjtYe3eaX03u\nT",
  "M0o3LSW/F\nOAoYek5OU/Gqr2rzd82kftarBb\ncwdqMz68mOY/VgjtYe3eaX03u\nT7Fa8DHp+sbh4kUKpBTu9MOlT\n5+C0sLhz/3+r/2Huw+WHm41r6\nhve1963nfe/1vd+8h94zb8cb\neJH3p/eX97f3z/JPy4PlYPlspr\natexit>53q73ma6/zWU7+A1BK43U=</l\n\u2326 2 RCi\u21e5Co\u21e5K\n<latexit sha1_base64=\"2Wk7WQhs8kxMDVgZOYpluHv",
  "43U=</l\n\u2326 2 RCi\u21e5Co\u21e5K\n<latexit sha1_base64=\"2Wk7WQhs8kxMDVgZOYpluHvKc/g=\"\n>AWmHiclZhbU9w2FICd9Jamt6Sd8tD2wdNMZjqdAc6eWRQAghkLJcFkhYGSv7FWQZWPLsMSzL/01fW3/Tf9Nj2zvKj5HPHRny\nCrn+6zLkWRrHWRSFHpx8d9bt97/4MP7rz8d1Pv3s8y/u3f/yoEjLPOSDMJVpfhSwgkuh+EALflRlnOWBJIfBuer",
  "t97/4MP7rz8d1Pv3s8y/u3f/yoEjLPOSDMJVpfhSwgkuh+EALflRlnOWBJIfBuerh9e8rwQqdr\nX1xk/SVisRCRCpiF0du+7YRAFXLOhUMOE6XEQVLvT02r1rEqn07N7DxZ7i/XHp4WltvDAaz/9s/tfj4ajNCwTrnQoWVEcLy1m+qRi\nuRah5NO7w7LgGQvPWcyPoahYwouTqh7G1H8IkZEfpTn8Ke3X0XevqFhSFNdJAKbpaoGZCbrYcamj308",
  "PWcyPoahYwouTqh7G1H8IkZEfpTn8Ke3X0XevqFhSFNdJAKbpaoGZCbrYcamj308qobJScxU2DUWl9HXqm5z4I5\nHzUMtrKLAwF9BXPxyznIUaMnd3qPhVmCYJU6NquLK2M62GAY+FqvhFWdxOu06a7XDoXiTsbKxP69FaJ6It5xUiumkhsEHk+rivfi\nHgaCAxA9TkCqeAF1tMc+UuIwqRgIEH6Q6F/m7U1K10jyGnHS010SDQib5pGOtEgumMu",
  "CAxA9TkCqeAF1tMc+UuIwqRgIEH6Q6F/m7U1K10jyGnHS010SDQib5pGOtEgumMukoe6D4/kPfAK5zmAXoKnxNAd7GVPT2X\nWaT3SeVIWJ4RZypmJeNwFDpk0I+oaqpQSLg071h/Y2mXqvE1cmtVdzU0EWft519E5zYsadZ06gixYhHXqiPIkrDHRyxhkOW2fAY\nDTnwTcatCYVWQhdnP06DbdmYieG1OMtgvXW+tIum/ZCgjJgC7z3wLpkLe1Vf",
  "fAY\nDTnwTcatCYVWQhdnP06DbdmYieG1OMtgvXW+tIum/ZCgjJgC7z3wLpkLe1VfTue3PknNZ+6bAJ/4YJqt7CcvjZlizRmBUbWxKzTpXy\nKTZglCeXnVN0xuHyjPRHaAJ4E1X5kJF72iP6hIsWRMePoKh5qXkxz/1fuGTk2rRbBvzD8kmVFSUmasiE/4fFY3gqYLXF0Tw5KUSTR4\nE6slLJdzf0dSxHC9sE6nDgpCMSn0Ndr+Ilbda+oI7myaoL",
  "3gqYLXF0Tw5KUSTR4\nE6slLJdzf0dSxHC9sE6nDgpCMSn0Ndr+Ilbda+oI7myaoL5CwNQL30woNMlR1JVNwMjwDc9HxwIK0SDZoyhTIsy5+Tmh9YzRGrd\n3BZzYR5W3RuqNEL3vsHl/Cow8Phkt9weYAyGjT5DNJSjViOkjkxUzo5HRYatphr9dT3hSdVswvNtv2oF8wO2UY8ouzTwfMbGoI1\nFdcCBx1iWJ5WgP6pov13d7Vm2e/kiWduxw3aYk",
  "Ntv2oF8wO2UY8ouzTwfMbGoI1\nFdcCBx1iWJ5WgP6pov13d7Vm2e/kiWduxw3aYk9ba9dNsO94Ye8IstR2+3iEcs6khUV9tD6hHL0R7U5c7jlmsUDtdtSlLvLI9O2+HO\nTbT8o/0xnEvNMSmVI3PsS+WwCWFRU1E7xThMRKbEBaTsmvB/7GyJ+Dh0bWaEBb7hehqJoClEZd4CE0Ii80W7ptDKtbDnXLrTKZjZ\nHZhLC4zhI86iaExZiKsVM8Z1mG",
  "hehqJoClEZd4CE0Ii80W7ptDKtbDnXLrTKZjZ\nHZhLC4zhI86iaExZiKsVM8Z1mGxCZE8jGeRzTPGZYylwSnpHMSNkSbkWVD5Ou5IJYGmCWps4GoMeyFShBtsglgu68grnylNoFSu\n6igeuhgc3NKwZqtAEsLRN9pg/3HZusgCnGI5ZriRnAlkZTWAfO3qzE5/QVSRk1wQXVt6TemVpVeUHlp6SGluKflFES7lpJfJ0F0a\neklpQeWHlBaWlpSOr",
  "5/QVSRk1wQXVt6TemVpVeUHlp6SGluKflFES7lpJfJ0F0a\neklpQeWHlBaWlpSOrB0QGlkaUTpM0ufURpaGlK6aukqpdpSciKFJ4Kl+5SOLR1TemTpEaWvLH1F6XNLn1P62tLXlL619C2lTyx9Qim\nzlFG6ZukapdxS8uogiFYsXaE0sJT89oO9Zmf0szSjNKnlj6ldGQp+VUMzNLyfEGHoyWSko3LN2gVFhKfr8F0UtLX1KaWJpQ+sLS\nF5S+sf",
  "Knlj6ldGQp+VUMzNLyfEGHoyWSko3LN2gVFhKfr8F0UtLX1KaWJpQ+sLS\nF5S+sfQNpeuWrlMaW0reDcDpxNI9Su1boKqgdMfSHUovL1wvxfg82kMXAtz21awTWlqaUrpqXklwIcJSw9J+fJSLV3tdnbJnJfi9\nC70/vb+8v71/Fr5ZWF5YX9ho1Nu32mu+8jqfhd3/AB5x2sw=</latexit>ScO1ib8dnVJOeRmnMHa+9Os6vJ/SlScz4mXV87m",
  "jqfhd3/AB5x2sw=</latexit>ScO1ib8dnVJOeRmnMHa+9Os6vJ/SlScz4mXV87mL9IgZQe1O9l/BbWFo4+Lm39Gv8c7jB8sr7RvaO963vfeD96S95u37D3+t7A\n\u03b2 2 RCo\n<latexit sha1_base64=\"san5CUx9PvrfJQpu1I4F6t/gSsQ=\"\n>AWm3iclZhbT9xGFICd9JbSG2lVXqVlGkqkpXUKWXxwRCSAIpy2WBhCVo7B17J4zHxh7DEmsf+2v62v",
  "FICd9JbSG2lVXqVlGkqkpXUKWXxwRCSAIpy2WBhCVo7B17J4zHxh7DEmsf+2v62v6Y/puesb078TnDQ1cin\npzv81zOzPgWZFIUemXl31u3v/gw8/uvPxwiefvb5F4t3vzws0jIP+SBMZofB6zgUig+0EJLfpzlnCWB5EfB+brhR5c8L0SqDvR\n1xk8TFisRiZBpCJ0tfjcMUjkqrhM4DAOumT8Uyh8mTI+DoNqbni0ur/RW6p9PC6tYdlrf/2zu",
  "iZBpCJ0tfjcMUjkqrhM4DAOumT8Uyh8mTI+DoNqbni0ur/RW6p9PC6tYdlrf/2zu1+PhqM0LBOudChZUZysrmT6tGK5\nFqHk04VhWfCMhecs5idQVCzhxWlVj2Tq34PIyI/SHP6U9uvou2dULClMZ8E0XSwM0EXOyl19PtpJVRWaq7CpqGolL5OfZMWfyRyHm\np5DQUW5gL6odjlrNQ/IWhopfhWmSMDWqhmsbu9MKchULVfGLsk7kdNp1NmqHQ/",
  "m\np5DQUW5gL6odjlrNQ/IWhopfhWmSMDWqhmsbu9MKchULVfGLsk7kdNp1NmqHQ/EmY+3ZwbwWoXki3nJSa2YSm4QeDytKt6LexgI\nDkD0OAGp4gXUWU9v5K8iCgtHAgYepBPoXOTvTUnVSvMYctLRXhENCpnk461TiyYyqSj7IPi+/d8A7jOYRagq3DgaA72M6ams/M0n+\ng8qQoTwy3kTMW8bgKGHDJpRtQ1VCklnBp2rD+wtcfUeZu4NKu7mp",
  "2M6ams/M0n+\ng8qQoTwy3kTMW8bgKGHDJpRtQ1VCklnBp2rD+wtcfUeZu4NKu7mpsIsg7yrqNzmhc16jp1BFmwCOuVUeQJWGbj1jCIMt+QwGnPg\nm4laFwqogC7Ofp0G37cxE8NqcZLBfut5GRdJ/yVBGTAB2nzkKpkLe1dfTue3PknNZ+6bAJ/4YJqt7CsvjZlizRmBUbWxKzTpXyKTZg\nlCeXnVN0xuHyjPRHaAJ4E1X5kJF72j36xIsWRMe3o",
  "jZlizRmBUbWxKzTpXyKTZg\nlCeXnVN0xuHyjPRHaAJ4E1X5kJF72j36xIsWRMe3oeh5qXkJz/1fuGT02rFbBvzD8kmVFSUmasiE/4fFY3gxoLXF0Tw5KUSTR4E6sl\nLJVzf0dSxHC9sE6nDgpCMSn0Ndr+Ilbdc+oI7myaoL5CwNQLRyYUmuQo6somYGQ4wi3SsYBCNMiwGWMo06LMObn4ofUMkVo3l8Vc\nmJtV94IqjdC9bnA5PwvKcHO45DecH",
  "3SsYBCNMiwGWMo06LMObn4ofUMkVo3l8Vc\nmJtV94IqjdC9bnA5PwvKcHO45DecHqCMBk0+g7RUI5ajZE7MlE5eDwsNW8y1+spb4pOK+YXW2170C+YnTIM+cXZFp6PmFjUkagueC\nZx1iWJ5WgP6pov13d7Vm29/pEs7djhuk1J6m176bYd7g094Bfbjt5uE49Y1JGoraH1COWoz2oy53HbdcoHK7blKTeWR6dtsOdm2j5\nRwdjeCI1j0n2GbUJY",
  "49Y1JGoraH1COWoz2oy53HbdcoHK7blKTeWR6dtsOdm2j5\nRwdjeCI1j0n2GbUJYVFTUTvFNOExEpsQFpOya8H/sbIv4ObRtZoQFvuF6GomgKURl3gITQiLzRbum0Mq9sOdutMpmNkdmEsLjJEj\nzqJoTFmIqxUzxnWYbEJkTyOMZ5HNM8ZljKXBKekcwxI2RJuRZUPk67kglgaYJamzgagx7IVKEG2yCWC7ryCufKU2gVK7qKB6GBzc\n0rBmq0A",
  "RJuRZUPk67kglgaYJamzgagx7IVKEG2yCWC7ryCufKU2gVK7qKB6GBzc\n0rBmq0ASwtEP2mD/cW6yAKfYr9B6XQJZGU0gX3s9Kkze/oLo8yQXRtaXlF5ZekXpkaVHlOaWkjeCINqzlLydBNGlpZeUHlp6S\nGlpaUnpwNIBpZGlEaVPLH1CaWhpSOm6peuUakvJEyncESw9oHRs6ZjSY0uPKX1p6UtKn1r6lNJXlr6i9K2lbyl9ZOkjSpmljNINSz",
  "vJEyncESw9oHRs6ZjSY0uPKX1p6UtKn1r6lNJXlr6i9K2lbyl9ZOkjSpmljNINSzc\no5ZaSTwdBtGbpGqWBpeTdD/apX1KM0szSh9b+pjSkaXkrRjuZ5aSxu4MVoqKX1m6TNKhaXk/S2IXlj6gtLE0oTS5Y+p/SNpW8o\n3bR0k9LYUvJtAJ5OLN2n1H4FqgpKdy3dpfTC0gv3dwE+n8bAtTB3bAU7lKaWpRuWUreFOBRwtJz8jwZqfaqNvaRK5rk",
  "Kdy3dpfTC0gv3dwE+n8bAtTB3bAU7lKaWpRuWUreFOBRwtJz8jwZqfaqNvaRK5rkZpzB2szPj\nub5DxSc+5g7dVpdja5PkVqzsek6xuH8w8pkFK40p8tLq/ir7C0cPhzb/X3oPdB8sP19ovtHe8b7zvR+8Ve8376H31Ot7Ay/0/vT+\n8v72/ln6dml96fnSdqPevtWe85X+S0N/gPElduW</latexit>\u03b2 2 R\n43",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n44",
  "Receptive fields\n45\n\u211d!!\u00d7!\"\u00d7#",
  "Receptive fields\n46\n\u211d!!\u00d7!\"\u00d7#\n\ud835\udf14\", \ud835\udf14#, \ud835\udf14$ \u2297 \ud835\udf14\", \ud835\udf14#, \ud835\udf14$ = [\ud835\udf14\", \ud835\udf14#, \ud835\udf14$, \ud835\udf14%, \ud835\udf14&]",
  "Receptive fields\n47\n\u211d!!\u00d7!\"\u00d7#\n\ud835\udf14\", \ud835\udf14#, \ud835\udf14$, \ud835\udf14%, \ud835\udf14& \u2297 \ud835\udf14\", \ud835\udf14#, \ud835\udf14$ = [\ud835\udf14\", \ud835\udf14#, \ud835\udf14$, \ud835\udf14%, \ud835\udf14&, \ud835\udf14', \ud835\udf14(]",
  "Receptive fields\n48\n\u211d!!\u00d7!\"\u00d7#",
  "Convolutional networks\n\u2022 Networks for images\n\u2022 Invariance and equivariance\n\u2022 1D convolution\n\u2022 Convolutional layers\n\u2022 Channels\n\u2022 Receptive fields\n\u2022 Convolutional network for MNIST 1D\n49",
  "MNIST 1D Dataset\n50",
  "MNIST-1D results for fully-connected network\n51",
  "Fully connected network\n\u2022 Exactly same number of layers and hidden units\n\u2022 All fully-connected layers\n\u2022 Total parameters = 150,185\n52",
  "Convolutional network\n\u2022 Four hidden layers\n\u2022 Three convolutional layers\n\u2022 One fully-connected layer\n\u2022 Softmax at end\n\u2022 Total parameters = 2050\n\u2022 Trained for 100,000 steps with \nSGD, LR = 0.01, batch size 100\n53",
  "MNIST-1D convolutional network\n54",
  "Performance\n55",
  "Why?\n\u2022 Better inductive bias\n\u2022 Forced the network to process each location similarly\n\u2022 Shares information across locations\n\u2022 Search through a smaller family of input/ouput mappings, all of which \nare plausible\n56",
  "2D Convolution\n57",
  "Convolution #2\n\u2022 2D Convolution\n\u2022 Downsampling and upsampling, 1x1 convolution\n\u2022 Image classification\n\u2022 Object detection\n\u2022 Semantic segmentation\n\u2022 Residual networks\n\u2022 U-Nets and hourglass networks \n58",
  "2D Convolution\n\u2022 Convolution in 2D \n\u2022 Weighted sum over a K x K region\n\u2022 K x K weights\n\u2022 Build into a convolutional layer by adding bias and passing through \nactivation function \n<latexit sha1_b\nase64=\"5l87xucmc7F6NuYlBrDVn\nEfAzT0=\">AW3niclZhbU9w2FICd\nXtP0RtopL3xlEmn05AdNk0vL5lJI\nOQGKUtgQSTreyVvQJZNrYMSzx+7\nVunr/1J/Qv9E31tH3tke1exjnjozs\nBqz/dZlyPZlu2nOVyZeWvK2+9/c6\n71/9YNrH3708SefLlz/bC9Piy",
  "1exjnjozs\nBqz/dZlyPZlu2nOVyZeWvK2+9/c6\n71/9YNrH3708SefLlz/bC9Piy\ngwyDhSXbgk5xyJuhQMsnpQZpREvuc\n7vsna4rvn9EsZ4nYlRcpPYpJFjI\nAiIhNFrw3cmoZMvHlfv1XdeL/WRak\nsrjNJSHnk8lcW+6Xl7EozK+269eld\n9VzS/R/nK9JKYRAbwsKncKVd2Mb9\n1ePr4pbt2uvIxFE3k0Wlha6a3UHxc\nX+m1hyWk/g9H1L",
  "9JKYRAbwsKncKVd2Mb9\n1ePr4pbt2uvIxFE3k0Wlha6a3UHxc\nX+m1hyWk/g9H1L8beOAmKmAoZcJL\nnh/2VB6VJMs4LS65hU5TUlwQiJ6\nCEVBYpoflXUyKvcGRMZumGTwJ6RbR\n98oiRxnl/EPpgxkZPcZCpoY4eFD\nH86KplIC0lF0DQUFtyViasy645ZRg\nPJL6BAgoxBX91gQjISMj/NU/Q8yC\nJYyLGpbe6vl2VkN+IiZKeFvVcVFX\nXWa",
  "Rg\nPJL6BAgoxBX91gQjISMj/NU/Q8yC\nJYyLGpbe6vl2VkN+IiZKeFvVcVFX\nXWa8dCsXLjNUnu/NamKQxe01RJbWi\nKrlEoFVlrQX9UzAKADWowgkguZQ\np8qPH7p9g8La4DLZil5YDyvUNVC0\nghy0tFeIg0KafTjrWGLJjKuKPsgO\nK6N1wFqMxgFqCr8EWNOdhJiahmx0\nk6lVlc5ipmtpAREdG6CRhyQLgaUdc\nQBedwaNCxfjat50SctIl",
  "EWNOdhJiahmx0\nk6lVlc5ipmtpAREdG6CRhyQLgaUdc\nQBedwaNCxfjat50SctIlL0rqrmYo\nY1m7WdWSG8yLGXaeOGBYswqhr1RHD\n4nClGJOYQJb8gGHLsqYleZMFWGF\nuYgS/xu26mKmGtzmsL50vXWS5T+M\n2JkRAXg7FPfjIiAdvW1ZG67s+Sc1b\n4q0Kk7gcnqHkKyqBnWrBEYVRursFn\nnyjBxtiCUJedU/XGotKUdQeoAuZ\nJV2RMhG9",
  "0Kk7gcnqHkKyqBnWrBEYVRursFn\nnyjBxtiCUJedU/XGotKUdQeoAuZ\nJV2RMhG9oy3UJlqwKe8sw1Kzg9PBW\n73s6PSpX1Gmj/qFsQkV5kdoqUuH/\nUdEY7k3m+oKIOXkJNyYPAvXkJRyu7\n8bUkcxc2CpSzx0UmCcyQvj9GeR6B\n5TR8zOJrHRVwioeuGbMGFMch2ZR\nVQMnzDXdaygAJjkEzxoAneZFRdPE\nz1jNEal1dFjOmblbdCypXQve6",
  "MGFMch2ZR\nVQMnzDXdaygAJjkEzxoAneZFRdPE\nz1jNEal1dFjOmblbdCypXQve6Qfn8\nKCjDzeGMXnK4b2TUb/LpJ4UYk8xI\n5lRN6fSVl0s4xWxnfz3lTdFqRfR0o\n20P+gWzUwQBPR1tmPMRIQs73KgLt\njXWujiyLO1BXfPl+mbPyo1X36KlHV\nlcu8lRvW0v7bFvaQH9HT0tN5CE\nLO9yoq+0h9pBlaQ/qsudx0zYKi2s\n3Oap3lkerbXHn",
  "RvW0v7bFvaQH9HT0tN5CE\nLO9yoq+0h9pBlaQ/qsudx0zYKi2s\n3Oap3lkerbXHnprH8w90J7GLVNinh\nY7XtS7jXhExRYlFaxXrH2xWbkCnG\nRdeC36ayw+Dm0bWakCkOctbVMCUx\npSbQ2hCpticwl2zjZnqpkXdtKuEpx\nPDbEKm+IjE5qibkClGWIys4glJU0\nNsQiPEzOPE5zH1JRSm2TOSGqZEbS\nkbAsqmyRdSQVMaWq0NrU0Bj3giTAa",
  "glJU0\nNsQiPEzOPE5zH1JRSm2TOSGqZEbS\nkbAsqmyRdSQVMaWq0NrU0Bj3giTAa\nbIOmnOVl1tXnjBWscCreGhreHhJ\nw5IYFaqAKW2hc8z1tqwnmW+m2K2fW\n/F0McNKcQIHpjPAzmz354cl2sn54\nYWmF5iea3qO6b6m+5hmqInAj98ri\nl6OvHDM03PMN3TdA/TQtMC06GmQ0x\nDTUNMH2r6ENA0wDTNU3XMJWaoh0\np3BE03cV0oukE0wND",
  "N3TdA/TQtMC06GmQ0x\nDTUNMH2r6ENA0wDTNU3XMJWaoh0\np3BE03cV0oukE0wNDzB9oekLTB9r\n+hjTl5q+xPS1pq8xva/pfUyJpgT\ndU3XMaWaolcHfriq6Sqmvqbo2Q/ON\nU0HmKapg+0PQBpmN0VMx3M80Rd\nsbuDFqyjF9oukTJm6PnND59p+g\nzTWNMY06eaPsX0WNjTB9p+gjTSFP\n0bgB2J5ruYKrfApU5ptuabmN6qump\n/b0AnU+jb",
  "MY06eaPsX0WNjTB9p+gjTSFP\n0bgB2J5ruYKrfApU5ptuabmN6qump\n/b0AnU+jb1uYW7qCLUwTRNMNzRF\nTwqwldD0BO0nQ9Fe1WZvm9B1LRzb\nmFtxmdHo5yHYs4trL06zY5G16dQz\nPkEdX19b/4iBVIKV/rRwlLfAuLC3\nu3e/0fene27yzdW23f0F51vnS+cr5\nx+s6Pzj3nsTNwhk7g/On87fzj/Lv\n4y+Kvi78t/t6ob1pj/nc6XwW/gP",
  "vnS+cr5\nx+s6Pzj3nsTNwhk7g/On87fzj/Lv\n4y+Kvi78t/t6ob1pj/nc6XwW/gP\nYKz1Qw=</latexit>\nhi,j = a\n\"\n\u03b2 +\n3\nX\nm=1\n3\nX\nn=1\n!m,nxi+m\u22122,j+n\u22122\n#\n59",
  "2D Convolution\n60",
  "2D Convolution with Zero Padding\n61",
  "Channels in 2D convolution\nKernel size, stride, dilation all \nwork as you would expect\n62",
  "How many parameters?\n\u2022 If there are \ud835\udc36! input channels and kernel size K x K\n\u2022 If there are \ud835\udc36! input channels and \ud835\udc36\" output channels \n<latexit sha1_base64=\"2Wk7WQhs8kxMDVgZOYpluHvKc/g=\"\n>AWmHiclZhbU9w2FICd9Jamt6Sd8tD2wdNMZjqdAc6eWRQAghkLJcFkhYGSv7FWQZWPLsMSzL/01fW3/Tf9Nj2zvKj5HPHRny\nCrn+6zLkWRrHWRSFHpx8d9bt97/4MP7rz8d1Pv3s8y/u3f/yoEjLPOSDMJVpfhSwgkuh+E",
  "LkWRrHWRSFHpx8d9bt97/4MP7rz8d1Pv3s8y/u3f/yoEjLPOSDMJVpfhSwgkuh+EALflRlnOWBJIfBuerh9e8rwQqdr\nX1xk/SVisRCRCpiF0du+7YRAFXLOhUMOE6XEQVLvT02r1rEqn07N7DxZ7i/XHp4WltvDAaz/9s/tfj4ajNCwTrnQoWVEcLy1m+qRi\nuRah5NO7w7LgGQvPWcyPoahYwouTqh7G1H8IkZEfpTn8Ke3X0XevqFhSFNdJAK",
  "i\nuRah5NO7w7LgGQvPWcyPoahYwouTqh7G1H8IkZEfpTn8Ke3X0XevqFhSFNdJAKbpaoGZCbrYcamj308qobJScxU2DUWl9HXqm5z4I5\nHzUMtrKLAwF9BXPxyznIUaMnd3qPhVmCYJU6NquLK2M62GAY+FqvhFWdxOu06a7XDoXiTsbKxP69FaJ6It5xUiumkhsEHk+rivfi\nHgaCAxA9TkCqeAF1tMc+UuIwqRgIEH6Q6F/m7U1K10jyGnHS01",
  "mkhsEHk+rivfi\nHgaCAxA9TkCqeAF1tMc+UuIwqRgIEH6Q6F/m7U1K10jyGnHS010SDQib5pGOtEgumMukoe6D4/kPfAK5zmAXoKnxNAd7GVPT2X\nWaT3SeVIWJ4RZypmJeNwFDpk0I+oaqpQSLg071h/Y2mXqvE1cmtVdzU0EWft519E5zYsadZ06gixYhHXqiPIkrDHRyxhkOW2fAY\nDTnwTcatCYVWQhdnP06DbdmYieG1OMtgvXW+tIum/ZC",
  "XqiPIkrDHRyxhkOW2fAY\nDTnwTcatCYVWQhdnP06DbdmYieG1OMtgvXW+tIum/ZCgjJgC7z3wLpkLe1VfTue3PknNZ+6bAJ/4YJqt7CcvjZlizRmBUbWxKzTpXy\nKTZglCeXnVN0xuHyjPRHaAJ4E1X5kJF72iP6hIsWRMePoKh5qXkxz/1fuGTk2rRbBvzD8kmVFSUmasiE/4fFY3gqYLXF0Tw5KUSTR4\nE6slLJdzf0dSxHC9sE6nDgpCMSn0N",
  "8kmVFSUmasiE/4fFY3gqYLXF0Tw5KUSTR4\nE6slLJdzf0dSxHC9sE6nDgpCMSn0Ndr+Ilbda+oI7myaoL5CwNQL30woNMlR1JVNwMjwDc9HxwIK0SDZoyhTIsy5+Tmh9YzRGrd\n3BZzYR5W3RuqNEL3vsHl/Cow8Phkt9weYAyGjT5DNJSjViOkjkxUzo5HRYatphr9dT3hSdVswvNtv2oF8wO2UY8ouzTwfMbGoI1\nFdcCBx1iWJ5WgP6pov13d",
  "Yatphr9dT3hSdVswvNtv2oF8wO2UY8ouzTwfMbGoI1\nFdcCBx1iWJ5WgP6pov13d7Vm2e/kiWduxw3aYk9ba9dNsO94Ye8IstR2+3iEcs6khUV9tD6hHL0R7U5c7jlmsUDtdtSlLvLI9O2+HO\nTbT8o/0xnEvNMSmVI3PsS+WwCWFRU1E7xThMRKbEBaTsmvB/7GyJ+Dh0bWaEBb7hehqJoClEZd4CE0Ii80W7ptDKtbDnXLrTKZjZ\nHZhLC4zhI",
  "B/7GyJ+Dh0bWaEBb7hehqJoClEZd4CE0Ii80W7ptDKtbDnXLrTKZjZ\nHZhLC4zhI86iaExZiKsVM8Z1mGxCZE8jGeRzTPGZYylwSnpHMSNkSbkWVD5Ou5IJYGmCWps4GoMeyFShBtsglgu68grnylNoFSu\n6igeuhgc3NKwZqtAEsLRN9pg/3HZusgCnGI5ZriRnAlkZTWAfO3qzE5/QVSRk1wQXVt6TemVpVeUHlp6SGluKflFES7lpJfJ0F0a",
  "riRnAlkZTWAfO3qzE5/QVSRk1wQXVt6TemVpVeUHlp6SGluKflFES7lpJfJ0F0a\neklpQeWHlBaWlpSOrB0QGlkaUTpM0ufURpaGlK6aukqpdpSciKFJ4Kl+5SOLR1TemTpEaWvLH1F6XNLn1P62tLXlL619C2lTyx9Qim\nzlFG6ZukapdxS8uogiFYsXaE0sJT89oO9Zmf0szSjNKnlj6ldGQp+VUMzNLyfEGHoyWSko3LN2gVFhKfr8F0UtLX1",
  "sJT89oO9Zmf0szSjNKnlj6ldGQp+VUMzNLyfEGHoyWSko3LN2gVFhKfr8F0UtLX1KaWJpQ+sLS\nF5S+sfQNpeuWrlMaW0reDcDpxNI9Su1boKqgdMfSHUovL1wvxfg82kMXAtz21awTWlqaUrpqXklwIcJSw9J+fJSLV3tdnbJnJfi9\nC70/vb+8v71/Fr5ZWF5YX9ho1Nu32mu+8jqfhd3/AB5x2sw=</latexit>ScO1ib8dnVJOeRmnMHa+9O",
  "WF5YX9ho1Nu32mu+8jqfhd3/AB5x2sw=</latexit>ScO1ib8dnVJOeRmnMHa+9Os6vJ/SlScz4mXV87mL9IgZQe1O9l/BbWFo4+Lm39Gv8c7jB8sr7RvaO963vfeD96S95u37D3+t7A\n\u03b2 2 RCo\n<latexit sha1_base64=\"eR0v\noIQCln3fK+kp18PIzXvcBI=\">AWtHiclZhJb9w2FICVdEvTz\nWlRX3oRagQoinRgF2maY2LH2ezU42VsJ5YzoDS",
  "\">AWtHiclZhJb9w2FICVdEvTz\nWlRX3oRagQoinRgF2maY2LH2ezU42VsJ5YzoDSUhjFyRJljyPM\nX+mv6bW9/0UaMZRu/Rhw7giHnfJy6PpLYwl6LUq6v/3rj50c\nefPrZrc9vf/HlV19/s3Tn28Myq4qID6JMZsVxyEouheIDLbTkx\n3nBWRpKfhSebRh+dMGLUmTqQF/l/DRliRKxiJiG0HDpYRBmclRe\npXAIspQnzA+E8oOU6XEY1nvTt/X",
  "dMGLUmTqQF/l/DRliRKxiJiG0HDpYRBmclRe\npXAIspQnzA+E8oOU6XEY1nvTt/XGsBbTQIuUl/7W/DgdLq2s9la\nbn08La21hxWt/eGd70fBKIuqlCsdSVaWJ2uruT6tWaFJPn0dl\nCVPGfRGUv4CRQVg4ZO62aIU/8uREZ+nBXwp7TfRD8o2ZpaUYBp\nul6iZkJuthJpeOHp7VQeaW5imYNxZX0deabfPkjUfBIysosKgQ\n0Fc/GrOCRqyejtQ",
  "6iZkJuthJpeOHp7VQeaW5imYNxZX0deabfPkjUfBIysosKgQ\n0Fc/GrOCRqyejtQ/DLK0pSpUR2sb+5O6yDkiVA1P6+aDE+nXW\nezcTgUrzPWXxwsahGap+I9J5U0iqnkGoEn07rmvaSHgeARI8Tk\nCleQp3NtMf+GqKwoiRg4GE2gc7F/t6UVK0TyAnHe0N0aCQSz7p\nWBvEgqlMO8o+KL5/1zeA6wJmAboKB47mYD9najo/T/OJLtK6NDH\ncQs",
  "0aCQSz7p\nWBvEgqlMO8o+KL5/1zeA6wJmAboKB47mYD9najo/T/OJLtK6NDH\ncQsFUwpsmYMgRk2ZEXUNVUsKpUcf6A1t7TJ21icvypquFiSDroO\ng6uqB5UaOu0SQBYsw6VpNBFkS9v+IpQy3JaHMODUNxG3KhRWB\nVmY/SILu23nJoLX5iSH/dL1NmuS/guGMmICsPvMUTAV8a6+kS1s\nf56ci8Y3BT7xzBZ3VNYkcyGNW8ERtXGptRscoVMmi",
  "uGMmICsPvMUTAV8a6+kS1s\nf56ci8Y3BT7xzBZ3VNYkcyGNW8ERtXGptRscoVMmi0IFdl1z\nS9cag8F90BmgDedFUhVPyBdq8pwZI14eAeDLWoJD/5pfcbn5zWq\n2bmH9INqGispdFZnw/6hoBHcvL4gicvk2jyINBMXibh+o6m\njhV4YZtIM3dQEIpJoa/Q9heJ6p7TRHBnsxT1FQKmXjgyodAkx3F\nXNgEjwxHunY4FKFBRrMxRjIrq4KTix9a",
  "9heJ6p7TRHBnsxT1FQKmXjgyodAkx3F\nXNgEjwxHunY4FKFBRrMxRjIrq4KTix9azxBpdHNZLIS5WXUvqN\nI3esGl4uzoAw3hwt+zekhymg4y2eYVWrECpTMiZnSydug1LDFX\nLu/mfJZ0Wkl/HyrbQ/6BbNTRE/H27h+UiIR2J6oKHFWdkliO\n9qCuxXL9sGf1tufydJOHK7blKTetpdu2+Fe0wN+vu3o7TbxiE\nUdiepqe0g9Yjnag7rcedx2j",
  "Gf1tufydJOHK7blKTetpdu2+Fe0wN+vu3o7TbxiE\nUdiepqe0g9Yjnag7rcedx2jcLhuk1J6p3n0Wk73IWJln98MOam\nck+/A6C2FRU1E7xebZtyvOQlhMq64F/8fKvoCbR9eahbDYL0VX\nMwEsjbjEQ5iFsDjbwl2zjWF126Fu1Um8zEyZyEsPmMpHvUshMW\nEiolTPGN5jsRZiORxjPM4pnMsZS7JDwjuWNGyJyLahinHUlE8\nDSBLU2cTQGPZ",
  "W\nEiolTPGN5jsRZiORxjPM4pnMsZS7JDwjuWNGyJyLahinHUlE8\nDSBLU2cTQGPZCZQg2QSyXdOWVzpWn0CpWdBUPXA0PrmlYM1ShC\nWBph+wxP9hxbrIQpxges1xJzgWycprAPnb61Jk/YVxTZ7kwvjK\n0itKLy29pPTI0iNKC0vJG0EY71lK3k7C+MLSC0oPLT2ktLK0on\nRg6YDS2NKY0qeWPqU0sjSidMPSDUq1peSJFO4Ilh5QOrZ0TOmxp",
  "oPLT2ktLK0on\nRg6YDS2NKY0qeWPqU0sjSidMPSDUq1peSJFO4Ilh5QOrZ0TOmxp\nceUvrb0NaXPLX1O6RtL31D63tL3lD629DGlzFJG6alm5RyS8mn\ngzBet3Sd0tBS8u4He83SPqW5pTmlTyx9QunIUvJWDPczS8njDdw\nYLZWUvrD0BaXCUvL+FsavLH1FaWpSulLS19S+s7Sd5Q+s/QZpY\nml5NsAPJ1Yuk+p/QpUl5TuWrpL6bml5+7vAnw",
  "FaWpSulLS19S+s7Sd5Q+s/QZpY\nml5NsAPJ1Yuk+p/QpUl5TuWrpL6bml5+7vAnwxjaFrYe7YCnYoz\nSzNKN2ylLwpwKOEpWfkeTJW7Vt/rWJXNditeAO1mZ8fjbJeaw\nW3MHaq9P8bHJ9itWCj0nXNw8XH1IgpXClHy6trOGvsLRw+Gtv7U\nD9YDpajZT5Tb95oz/nO6/yW1X8LQuYo</latexit>Hv/u79lUfr7RfaW94P3o/eT96a97v3yHvu",
  "b95oz/nO6/yW1X8LQuYo</latexit>Hv/u79lUfr7RfaW94P3o/eT96a97v3yHvu9b2BF3l/en95f3v/L\n! 2 RCi\u21e5K\u21e5K\n<latexit sha1\n_base64=\"IMPdy1SuBnZoYGe/A\nuklNweaOxU=\">AWwHiclZhb\nb9s2FIDV7tZ1t3TD8rIXYUGBY\neiMZOguj23S9JZ0ce5p49SgZEp\nmQ1GyRCVOBP+h/Zq9dr9mh5Js\nVucwDzOQij3fJ14OKYlSk",
  "S9JZ0ce5p49SgZEp\nmQ1GyRCVOBP+h/Zq9dr9mh5Js\nVucwDzOQij3fJ14OKYlSkElR6N\nXV97duf/TxJ59+dufzu198+dX\nX3yzd+/aoSMs85IdhKtP8JGAFl\n0LxQy205CdZzlkSH4cnG8Yfn\nzB80Kk6kBfZfwsYbESkQiZhtB\nw6ckgSOWouErgMEgTHjN/IJQ/S\nJgeB0G1N3tbQwrMRtokfDCh3\nI6L2/Nj7Ph0spqb7X+bSw1hZW\nvPbX",
  "jN/IJQ/S\nJgeB0G1N3tbQwrMRtokfDCh3\nI6L2/Nj7Ph0spqb7X+bSw1hZW\nvPbXH97fjQYpWGZcKVDyYrid\nG0102cVy7UIJZ/dHZQFz1h4zmJ\n+CkXFoKGzqh7uzL8PkZEfpTn8\nKe3X0Q/PqFhSmBGBaYZRYGaCL\nnZa6ujPs0qorNRchU1DUSl9nfo\nmd/5I5DzU8goKLMwF9NUPxyxn\noYM3x0ofhmScLUqBqsb+7Oqk\nHAY6EqPinrbM9m",
  "d/5I5DzU8goKLMwF9NUPxyxn\noYM3x0ofhmScLUqBqsb+7Oqk\nHAY6EqPinrbM9mXWezdjgUbzL\nWXxwsahGaJ+Kak0pqxVRyg8DjW\nVXxXtzDQHAoscJSBUvoM56CU\nT+GqKwuiRg4E6hc5F/t6MVK0\n0jyEnHe0N0aCQST7tWBvEgqlMO\nso+KL5/3zeA6xmAboKB47mYD\n9jajY/T/OpzpOqMDHcQs5UzOsm\nYMghk2ZEXUOVUsKpYcf6C1t7",
  "xmAboKB47mYD\n9jajY/T/OpzpOqMDHcQs5UzOsm\nYMghk2ZEXUOVUsKpYcf6C1t7T\nJ23iUuzuqu5iSDrIO86Oqd5UaO\nuU0eQBYsw7lp1BFkS7gUjljDI\nclsewoAT30TcqlBYFWRh9vM06\nLadmQhem9Mrpeut1mR9F8wlBE\nTgKvPHAVTIe/qG+nC9ufJuah9\nU+BTfwyT1T2F5XEzrHkjMKo2Nq\nNmnStk0mxBKE8vu6bpjUPlmeg\nO0ATwR",
  "uah9\nU+BTfwyT1T2F5XEzrHkjMKo2Nq\nNmnStk0mxBKE8vu6bpjUPlmeg\nO0ATwRVfmQkUfaA/qEixZEx48g\nKHmpeSnv/R+49OzatVcNuYfk\n2oqCgzV0Um/D8qGsHTB68viOD\nJSyWaPAjUk5dKuL+jqWM5XtgmU\ns8dFIRiUugrdPmLWHXPqSO4s2\nmC+goBUy8cmVBokqOoK5uAkeEI\nz1HAgrRIMNmjKFMizLn5OaH1\njNEat3cFnNhHlbd",
  "goBUy8cmVBokqOoK5uAkeEI\nz1HAgrRIMNmjKFMizLn5OaH1\njNEat3cFnNhHlbdG6o0Qve+weX\niLCjDw+GC3B6gDIaNPkM0lKN\nWI6SOTVTOn07KDRcYq6rv57yp\nui0Yj7ZatuDfsHslGHIJ8MtPB8\nxsagjUV2wcXHWJYnlaA/qWizX\nD3tWb39mSzt2OG6TUnqbXvpth\n3uDT3gk21Hb7eJRyzqSFRX20P\nqEcvRHtTlzuO2axQO121KUu8",
  "OG6TUnqbXvpth\n3uDT3gk21Hb7eJRyzqSFRX20P\nqEcvRHtTlzuO2axQO121KUu8j\n07b4S5MtPyjgzHXzGyT7Ea2CW\nFRU1E7xXof3BWbEBaTsmvB/7G\nyL+Dh0bWaEBb7hehqJoClEZd4C\nE0Ii80l3DXbGFa3Heq2W2UyGy\nOzCWHxGUvwqJsQFmMqxk7xnGUZ\nEpsQyeMY53FM85hKXNJeEYyx\n4yQJeVaUPk47UomgKUpam3qaAx\n6IFOFG",
  "xnGUZ\nEpsQyeMY53FM85hKXNJeEYyx\n4yQJeVaUPk47UomgKUpam3qaAx\n6IFOFGmyDWC7oyiucK0+hVazo\nKj50NXx4Q8OaoQpNAEs75BrzB\nzvOiyzAKYZtlivJmUBWRhPYx06\nfOvPdXxBVZCcXRFeWXlF6aekl\npceWHlOaW0reCIJoz1LydhJEF5\nZeUHpk6RGlpaUlpYeWHlIaWRp\nR+tTSp5SGloaUbli6Qam2lOxI4\nYlg6QGlY0vHlJ",
  "UHpk6RGlpaUlpYeWHlIaWRp\nR+tTSp5SGloaUbli6Qam2lOxI4\nYlg6QGlY0vHlJ5YekLpa0tfU/\nrc0ueUvrH0DaXl5T+tjSx5Q\nySxmlm5ZuUsotJZ8Ogmjd0nVKA\n0vJux9ca5b2Kc0szSh9YukTSk\neWkrdieJ5ZSrY38GC0VFL6wtIX\nlApLyftbEL2y9BWliaUJpS8tf\nUnpO0vfUfrM0meUxpaSbwOwO7F\n0n1L7FagqKN21dJfSiaUT9",
  "y9BWliaUJpS8tf\nUnpO0vfUfrM0meUxpaSbwOwO7F\n0n1L7FagqKN21dJfSiaUT93cB\nvpjGwLUwd2wFO5SmlqaUblK3\nhRgK2HpOdlPRq9q82/NpH7WqQ\nW3MHajM/PJjmP1I7WHt3mp9N\n7k+RWvAx6frm0eJDCqQU7vTDpZ\nU1/BWFo5+7a393nu4+3Dl0Xr\n7hfaO94P3o/eTt+b94T3ynt9\n79ALvb+9f7z3r/L68vj5XR50q\natexit>i3",
  "0Xr\n7hfaO94P3o/eTt+b94T3ynt9\n79ALvb+9f7z3r/L68vj5XR50q\natexit>i3b7XnfOd1fsvX/wGvwOtH</l\n! 2 RCi\u21e5Co\u21e5K\u21e5K\n<latexit sha1_base64=\"san5CUx9PvrfJQpu1I4F6t/gSsQ=\"\n>AWm3iclZhbT9xGFICd9JbSG2lVXqVlGkqkpXUKWXxwRCSAIpy2WBhCVo7B17J4zHxh7DEmsf+2v62v6Y/puesb078TnDQ1cin\npzv8",
  "WXxwRCSAIpy2WBhCVo7B17J4zHxh7DEmsf+2v62v6Y/puesb078TnDQ1cin\npzv81zOzPgWZFIUemXl31u3v/gw8/uvPxwiefvb5F4t3vzws0jIP+SBMZofB6zgUig+0EJLfpzlnCWB5EfB+brhR5c8L0SqDvR\n1xk8TFisRiZBpCJ0tfjcMUjkqrhM4DAOumT8Uyh8mTI+DoNqbni0ur/RW6p9PC6tYdlrf/2zu1+PhqM0LBOudChZUZysrmT6t",
  "mT8Uyh8mTI+DoNqbni0ur/RW6p9PC6tYdlrf/2zu1+PhqM0LBOudChZUZysrmT6tGK5\nFqHk04VhWfCMhecs5idQVCzhxWlVj2Tq34PIyI/SHP6U9uvou2dULClMZ8E0XSwM0EXOyl19PtpJVRWaq7CpqGolL5OfZMWfyRyHm\np5DQUW5gL6odjlrNQ/IWhopfhWmSMDWqhmsbu9MKchULVfGLsk7kdNp1NmqHQ/EmY+3ZwbwWoXki3nJSa2YSm4",
  "pfhWmSMDWqhmsbu9MKchULVfGLsk7kdNp1NmqHQ/EmY+3ZwbwWoXki3nJSa2YSm4QeDytKt6LexgI\nDkD0OAGp4gXUWU9v5K8iCgtHAgYepBPoXOTvTUnVSvMYctLRXhENCpnk461TiyYyqSj7IPi+/d8A7jOYRagq3DgaA72M6ams/M0n+\ng8qQoTwy3kTMW8bgKGHDJpRtQ1VCklnBp2rD+wtcfUeZu4NKu7mpsIsg7yrqNzmhc16jp1BFmwCO",
  "W8bgKGHDJpRtQ1VCklnBp2rD+wtcfUeZu4NKu7mpsIsg7yrqNzmhc16jp1BFmwCOuVUeQJWGbj1jCIMt+QwGnPg\nm4laFwqogC7Ofp0G37cxE8NqcZLBfut5GRdJ/yVBGTAB2nzkKpkLe1dfTue3PknNZ+6bAJ/4YJqt7CsvjZlizRmBUbWxKzTpXyKTZg\nlCeXnVN0xuHyjPRHaAJ4E1X5kJF72j36xIsWRMe3oeh5qXkJz/1fuGT02rFbBvzD8",
  "CeXnVN0xuHyjPRHaAJ4E1X5kJF72j36xIsWRMe3oeh5qXkJz/1fuGT02rFbBvzD8kmVFSUmasiE/4fFY3gxoLXF0Tw5KUSTR4E6sl\nLJVzf0dSxHC9sE6nDgpCMSn0Ndr+Ilbdc+oI7myaoL5CwNQLRyYUmuQo6somYGQ4wi3SsYBCNMiwGWMo06LMObn4ofUMkVo3l8Vc\nmJtV94IqjdC9bnA5PwvKcHO45DecHqCMBk0+g7RUI5ajZE7MlE5eD",
  "UMkVo3l8Vc\nmJtV94IqjdC9bnA5PwvKcHO45DecHqCMBk0+g7RUI5ajZE7MlE5eDwsNW8y1+spb4pOK+YXW2170C+YnTIM+cXZFp6PmFjUkagueC\nZx1iWJ5WgP6pov13d7Vm29/pEs7djhuk1J6m176bYd7g094Bfbjt5uE49Y1JGoraH1COWoz2oy53HbdcoHK7blKTeWR6dtsOdm2j5\nRwdjeCI1j0n2GbUJYVFTUTvFNOExEpsQFpOya8H/s",
  "coHK7blKTeWR6dtsOdm2j5\nRwdjeCI1j0n2GbUJYVFTUTvFNOExEpsQFpOya8H/sbIv4ObRtZoQFvuF6GomgKURl3gITQiLzRbum0Mq9sOdutMpmNkdmEsLjJEj\nzqJoTFmIqxUzxnWYbEJkTyOMZ5HNM8ZljKXBKekcwxI2RJuRZUPk67kglgaYJamzgagx7IVKEG2yCWC7ryCufKU2gVK7qKB6GBzc\n0rBmq0ASwtEP2mD/cW6yAKfYr9B6XQJ",
  "7IVKEG2yCWC7ryCufKU2gVK7qKB6GBzc\n0rBmq0ASwtEP2mD/cW6yAKfYr9B6XQJZGU0gX3s9Kkze/oLo8yQXRtaXlF5ZekXpkaVHlOaWkjeCINqzlLydBNGlpZeUHlp6S\nGlpaUnpwNIBpZGlEaVPLH1CaWhpSOm6peuUakvJEyncESw9oHRs6ZjSY0uPKX1p6UtKn1r6lNJXlr6i9K2lbyl9ZOkjSpmljNINSzc\no5ZaSTwdBtGbpGqWBpeTdD",
  "1p6UtKn1r6lNJXlr6i9K2lbyl9ZOkjSpmljNINSzc\no5ZaSTwdBtGbpGqWBpeTdD/apX1KM0szSh9b+pjSkaXkrRjuZ5aSxu4MVoqKX1m6TNKhaXk/S2IXlj6gtLE0oTS5Y+p/SNpW8o\n3bR0k9LYUvJtAJ5OLN2n1H4FqgpKdy3dpfTC0gv3dwE+n8bAtTB3bAU7lKaWpRuWUreFOBRwtJz8jwZqfaqNvaRK5rkZpzB2szPj\nub5DxSc+5g7dVp",
  "3bAU7lKaWpRuWUreFOBRwtJz8jwZqfaqNvaRK5rkZpzB2szPj\nub5DxSc+5g7dVpdja5PkVqzsek6xuH8w8pkFK40p8tLq/ir7C0cPhzb/X3oPdB8sP19ovtHe8b7zvR+8Ve8376H31Ot7Ay/0/vT+\n8v72/ln6dml96fnSdqPevtWe85X+S0N/gPElduW</latexit>\u03b2 2 R\n63",
  "Convolution #2\n\u2022 2D Convolution\n\u2022 Downsampling and upsampling, 1x1 convolution\n\u2022 Image classification\n\u2022 Object detection\n\u2022 Semantic segmentation\n\u2022 Residual networks\n\u2022 U-Nets and hourglass networks \n64",
  "Downsampling\nSample every other \nposition (equivalent to \nstride two)\n65",
  "Downsampling\nSample every other \nposition (equivalent to \nstride two)\nMax pooling\n(partial invariance to \ntranslation)\n66",
  "Downsampling\nSample every other \nposition (equivalent to \nstride two)\nMax pooling\n(partial invariance to \ntranslation)\nMean pooling\n67",
  "Upsampling\nDuplicate\n68",
  "Upsampling\nDuplicate\nMax-upsampling\n69",
  "Upsampling\nDuplicate\nMax-upsampling\nBilinear interpolation\n70",
  "Transposed convolutions\nKernel size 3, Stride 2 convolution\n71",
  "Transposed convolutions\nKernel size 3, Stride 2 convolution\nTransposed convolution\n72",
  "1x1 convolution \n\u2022 Mixes channels\n\u2022 Can change number of channels\n\u2022 Equivalent to running same fully connected network at each position\n73",
  "Convolution #2\n\u2022 2D Convolution\n\u2022 Downsampling and upsampling, 1x1 convolution\n\u2022 Image classification\n\u2022 Object detection\n\u2022 Semantic segmentation\n\u2022 Residual networks\n\u2022 U-Nets and hourglass networks \n74",
  "ImageNet database\n\u2022 224 x 224 images\n\u2022 1,281,167 training images, 50,000 validation images, and 100,000 test images\n\u2022 1000 classes\n75",
  "AlexNet (2012) \nAlmost all the 60 million \nparameters\n parameters are in fully \nconnected layers\n76",
  "Data augmentation\n\u2022 Data augmentation a factor of 2048 using (i) spatial transformations\nand (ii) modifications of the input intensities. \n77",
  "Dropout\n\u2022 Dropout was applied in the fully connected layers\n78",
  "Details\n\u2022 At test time average results from five different cropped and\nmirrored versions of the image\n\u2022 SGD with a momentum coe\ufb00icient of 0.9 and batch size of 128. \n\u2022 L2 (weight decay) regularizer used. \n\u2022 This system achieved a 16.4% top-5 error rate and a 38.1%\ntop-1 error rate. \n79",
  "VGG (2015)\n80",
  "Details\n\u2022 19 hidden layers\n\u2022 144 million parameters\n\u2022 6.8% top-5 error rate, 23.7% top-1 error rate\n81",
  "ImageNet History\n82",
  "Convolution #2\n\u2022 2D Convolution\n\u2022 Downsampling and upsampling, 1x1 convolution\n\u2022 Image classification\n\u2022 Object detection\n\u2022 Semantic segmentation\n\u2022 Residual networks\n\u2022 U-Nets and hourglass networks \n83",
  "You Only Look Once (YOLO)\n\u2022 Network similar to VGG (448x448 input)\n\u2022 7\u00d77 grid of locations\n\u2022 Predict class at each location\n\u2022 Predict 2 bounding boxes at each location\n\u2022 Five parameters \u2013x,y, height, width, and confidence\n\u2022 Momentum, weight decay, dropout, and data augmentation\n\u2022 Heuristic at the end to threshold and decide final boxes \u2013 \n(non maximum suppression) \n84",
  "Object detection (YOLO)\n85",
  "Transfer learning\nTransfer learning from ImageNet classification\n86",
  "Results\n87",
  "Convolution #2\n\u2022 2D Convolution\n\u2022 Downsampling and upsampling, 1x1 convolution\n\u2022 Image classification\n\u2022 Object detection\n\u2022 Semantic segmentation\n\u2022 Residual networks\n\u2022 U-Nets and hourglass networks \n88",
  "Semantic Segmentation (2015)\nEncoder                                       Decoder\n89",
  "Semantic segmentation results\n90",
  "https://cs231n.github.io/understanding-cnn/ \nAlexNet\n91\n1st Layer\nCat image input\n(not actual image)\n5th Layer\nActivations\n(feature maps)\nFilter Kernels\n\u2026\n\u2026\n2nd Layer",
  "https://poloclub.github.io/cnn-explainer/ \n92",
  "Feedback?\nLink\n93",
  "Residual Networks\nDL4DS \u2013 Spring 2024\n1\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited",
  "Where we are\n=== Foundational Concepts ===\n\u00fc 02 -- Supervised learning refresher\n\u00fc 03 -- Shallow networks and their representation capacity\n\u00fc 04 -- Deep networks and depth efficiency\n\u00fc 05 -- Loss function in terms of maximizing likelihoods\n\u00fc 06 \u2013 Fitting models with different optimizers\n\u00fc 07a \u2013 Gradients on deep models and backpropagation\n\u00fc 07b \u2013 Initialization to avoid vanishing and exploding weights & \ngradients\n\u00fc 08 \u2013 Measuring performance, test sets, overfitting and double \ndescent\n\u00fc 09 \u2013 Regularization to improve fitting on test sets and unseen data\n=== Network Architectures and Applications ===\n\u00fc 10 \u2013 Convolutional Networks\n\u2022 11 \u2013 Residual Networks and Recurrent Neural Networks\n\u2022 12 \u2013 Transformers\n\u2022 Large Language and other Foundational Models\n\u2022 Generative Models\n\u2022 Graph Neural Networks\n\u2022 \u2026\n2",
  "Topics\n\u2022 Residual connections and residual blocks\n\u2022 Exploding gradients in residual networks\n\u2022 Batch normalization\n\u2022 Common residual architectures\n3",
  "Topics\n\u2022 Residual connections and residual blocks\n\u2022 Exploding gradients in residual networks\n\u2022 Batch normalization\n\u2022 Common residual architectures\n4",
  "Previously we saw a sequential network:\n<latexit s\nha1_base64=\"sqOpnO6\nnY7k0+ZnjUC4ZXNbdf\nk=\">AXd3iclZhbT9x\nGFICXlN6I61aHvpQqy\nhRVNEVC/TyUimBkBukQ\nGCBhN2sxt6xd8J4bHyB\nJdY+97X9ef0pV96xvb\nu4HOGSkVKd3q+b86Mz4\nyvbixFmq2s/DX3zrv\nf/Bh7c+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8+ML",
  "+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8+MLnqQiUofZV\ncz7IQuU8IXHMgNFv5x\neqEbjYue6zujyaDj3P3\n1WsSfDIrO5BSa4+V5B3\n7ikdCRfk9FKg9dnji93\nnwzxSpNsVqmGOmey9Mk\nq/+ZI0mWZsmWTVJ1kg\nS17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9PTz",
  "S17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9PTzkqc9QuWZMKTfDLf\ny1MeM+MBfwUmoqFPO0\nX5SpNnDsQGTp+lMA/lT\nl9HqPgoVpehW6YIYsG\n6WY6aCNneaZ/0u/ECrO\nM68aiA/l04WOXrJnaF\nIuJfJK2gwLxEwV8cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx9HC",
  "cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx9HC\nWRWQ8FG85SVIqOskNAg\n8mRcHbQRsDwQGINicgU\njyFnLo+sO4dROGkIAL\nszNeTEhqlfEAatLQXhE\nNGrHk4a1SxYyrChHI\nDiOHcDXiWwCrAVOGHo\nzU4iJmaTPtlfJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQdZh0",
  "fJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQdZh0nSyh\nNZFDZtOGUEWbMKgaZUR\nZEm4hA1ZyKDKdXsABxw\n6OmJXhcKqIBtzL4nc5t\nixjuC9OY7hfGl6WwUp/\nwVDFdEBOPv0r2DK4019\nM5rZzrQ4F6WvG3zsjGC\nxml1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi2rw",
  "1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi2rw71lONQkl/z\n0h/aPfNwvVvRpo/9Dqg\nmJ0jy2JdLh/5FoCDdNv\nL8ghcvkmjxIFAuXiTh\n+o6WjiV4Y+tIuXbQEIp\nJkV2h018EqtmnjODJRi\nGaKwR0XvhlQqF9v2mr\nANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz\n1gj",
  "ANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz\n1gjbcHC74Dd1dVFG3q\ncb5WrIElTMsV7S8etem\nsEpZjv7yWvmlYr4Ofb\n9XgwL1id3P4+WAbr0d\nALOpIlAuet6y5JLEs40\nGu2Xa9PrNi+/X3ZGsHF\ntduSpK3nqXdtrg3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/uHI",
  "3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/uHI\n54x/ZgUyaF+7Itkrwph\nMaNiZhWjkAdIrEJYDPO\nmBf+PlQMBN4+mVYWwuJ\neKpqYDWBpyiQ+hCmGxO\noWbZh3D6o5F3bGrTMYj\nZFYhLD5mIT7qKoTFgIq\nBVTxjcYzEKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6yCWU",
  "EKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6yCWU7rzUuv\nOU2gXK7qLu7aBuzcMnD\nGUAewtEvOMae3az3JX\nFxieMyFTkWyIpAfew\ns0ed6dOf6xfkSQ7eig2\n9ovTS0EtKjw09pjQxlL\nwRuP4LQ8nbietfGHpB6\nZGhR5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLoCaU",
  "5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLoCaUvDX1J6RNDn\n1D6ytBXlL419C2lDwx9\nQCkzlFG6ZegWpdxQ8un\nA9TcM3aDUNZS8+8G5Zu\ngepbGhMaUPDX1I6dBQ8\nlYM9zNDyeMN3BgNlZQ+\nNfQpcJQ8v7m+s8NfU5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn\n9u",
  "5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn\n9u8CfLaMrm1j7poEu5R\nGhkaUbhtK3hTgUcLQM/\nI86av6qma+e06wMeMWVl\nd82pvU3FczbmH1cl8w\nCXGjI/I1LeOZh9SoKRw\npR8sLHXwV1jaOFptd35\nqr+vL93fqL/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25us+X",
  "/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25us+Xrcb\nfYudfruop4g=</late\nxit>h1 = f1[x, \u03c61]\nh2 = f2[h1, \u03c62]\nh3 = f3[h2, \u03c63]\ny = f4[h3, \u03c64]\n5\n<latexit sha1_base64=\"p8kAci0rsfxNTdm\n+eDTUSvXBwSE=\">AXKHiclZhbU9w2FICXlN6I+mUl754yqTaRKGTdM2L5lJIOQGKRCuCS\nY7slf2CmTZyDIs8fgP",
  "bU9w2FICXlN6I+mUl754yqTaRKGTdM2L5lJIOQGKRCuCS\nY7slf2CmTZyDIs8fgPdfpj+tbJa39Jj2zvCuIzpSZdNXzfdblSLJlBxlnuVpaej/zwYcfz\nJp9c+m/38iy+/+nru+o29PC1kSHfDlKfyICA5UzQXcUpweZpCQJON0PTlY03z+jMmep2FEX\nGT1KSCxYxEKiIDSY+9MbDUpWeT8PwkSMclqXxOI3XoB1QR75bnpwmNyaDsV2MQ7/SrW",
  "KSCxYxEKiIDSY+9MbDUpWeT8PwkSMclqXxOI3XoB1QR75bnpwmNyaDsV2MQ7/SrW5PI3\nTpSXJ+riO3+pUvWTxSR75IRZEVPr+7H+0kBfJoDx+0K/eQg3Tyo4r67t+M7dSX2DuYWlxa\nX6z8OFfltY6LV/m4Pr3w79YRoWCRUq5CTPD/tLmToqiVQs5LSa9YucZiQ8ITE9hKIgCc2Pyjq\ntlXcTIkMvSiX8E8qro5evKEmS5xdJAGZC1Ci3mQ62G",
  "ucZiQ8ITE9hKIgCc2Pyjq\ntlXcTIkMvSiX8E8qro5evKEmS5xdJAGZC1Ci3mQ62GhovtHJRNZoagIm4aignsq9fQceUMm\naj4BRIKBn01QtHRJQwUzO+oKeh2mSEDEs/eXVraqETMZMlPS0qGe1qrOau1QKF5lLD/fm\ndbCFE3YO4oqRVdyRUCjauypIvxog0YBcAWKQKpoDnUqfMTRF7forCKOeCyWTQ+GK8qVLVQNI\nacdLQ3SINCxum4Y60gC",
  "0YBcAWKQKpoDnUqfMTRF7forCKOeCyWTQ+GK8qVLVQNI\nacdLQ3SINCxum4Y60gC6Yy6SjboHjeTU8DqiTMAnQVfqg1B9sZEdXkOkXHSiZlrmN2C5KImN\nZNwJBDwvWIuoYoOIdLw471u29IuKkTVya1V2VOmJZO7LrKInzIoZdp45YFizCuGvVEcvicM8\nZkoRAltvyAaceDriVpmwVYW5qZMg27bmY7Ya3OcwX7peqslSv8ZsTKiA7D79C8jIq",
  "koRAltvyAaceDriVpmwVYW5qZMg27bmY7Ya3OcwX7peqslSv8ZsTKiA7D79C8jIqRdfSWd\n2t4kOWe1rwt07I1gsrqXEBk3w5o0AqNqYxU261xZJs4WhGR63jV1bxwqzVh3gDpgb7pCMhFd0\nm7XJViyOuzfhqHKgtPDO4u/0PFRuaS3jf4PyiZUlBeZqyId/h8VDeEpZ68viNiTl3Jr8iBQT1\n7K4f5uTR2R9sLWkXruoMAE4UxdWNufxaJ7TR2",
  "/h8VDeEpZ68viNiTl3Jr8iBQT1\n7K4f5uTR2R9sLWkXruoMAE4UxdWNufxaJ7TR2xO5smVl8hoOuFX8KENclR1JV1QMvwC89rxwI\nKrUGzRhDnuaFpOjmZ61niNS6vi1Kph9W3Rsq10L3vkH59Cow8PhjF5xeWBlNGjyGaSFGBJp\nJXOsp3T81s8VbDHX7q+nvCk6rZierXtQb9gdowpKeDNXs+YmRh1t1wQHJWRdHlqM9qGu6X\nC/3rFx7+xNa2",
  "nvCk6rZierXtQb9gdowpKeDNXs+YmRh1t1wQHJWRdHlqM9qGu6X\nC/3rFx7+xNa2rHDdZsc1dv20m073Ct6QE/XHb1dRx6ysMOtutoeYg9ZjvagLnce12jcLhuk6\nN6J3l02g53alrLP9oZwXlVH5NSPtTHvpT7TcgWFRaVU6xPul2xCdliUnQt+H9b2Wbw8OhaTcg\nWN3PW1XTAloaU20NoQrbYbOGu2cZsd2hrtVwrORZTYhW3xKEnvUTcgWYy",
  "OhaTcg\nWN3PW1XTAloaU20NoQrbYbOGu2cZsd2hrtVwrORZTYhW3xKEnvUTcgWYyzGTvGEZJklNiGU\nx5GdxHOY2ZLmUuyZyRzAhaUq4FJUdpV9IBWxpbrY0djUEPeCqsBtugLed45eXOlSesVSzwK\nt51Nbx7RcOKWBXqgC1toD3m+RvOTRbYKfbqN1Q8XcyMpzATdvZxM7k9BdEJTrJBdGFoReYnh\nt6jum+ofuYSkPRG0EQvTIUvZ0E0ZmhZ5",
  "yMpzATdvZxM7k9BdEJTrJBdGFoReYnh\nt6jum+ofuYSkPRG0EQvTIUvZ0E0ZmhZ5juGbqHaWFogemuobuYRoZGmD4x9AmoaEhpiuGrmC\nqDEUnUngiGLqD6cjQEaYHh5g+trQ15g+M/QZpm8MfYPpO0PfYfrI0EeYEkMJpquGrmJKDUWf\nDoJo2dBlTAND0bsf7DVDNzHNDM0wfWzoY0yHhqK3YnieGYqON/BgNJRj+tzQ5gyQ9H7WxC9N\nPQlpo",
  "sf7DVDNzHNDM0wfWzoY0yHhqK3YnieGYqON/BgNJRj+tzQ5gyQ9H7WxC9N\nPQlpomhCaYvDH2B6bGhx5g+NfQprGh6NsAnE4M3cbUfAUqc0y3DN3C9NTQU/d3ATqdxsC1MD\ndMBRuYpoamK4Zit4U4Ch6Ak6T0aivatNvjah+1okptzB2oxPrkY5j8SUO1h7d5pcje5PkZj\nyEer6t70QwqkFO70g7mFv0VFhf27i72f128t3Vv4eFy+4X2Wu+73",
  "d5pcje5PkZj\nyEer6t70QwqkFO70g7mFv0VFhf27i72f128t3Vv4eFy+4X2Wu+73ve9H3v93m+9h71nvc3e\nbi+cuTFzf+bRzPL8H/N/zf89/75RP5hpr/m1/mb/+dfqpUP9g=</latexit>hi = a [\u03b2 + !1xi\u22121 + !2xi + !3xi+1]\n= a\n2\n4\u03b2 +\n3\nX\nj=1\n!jxi+j\u22122\n3\n5\n<latexit \nsha1_base64=\"KZBq\nmLp19s93dVIEAqO\njCngwU=\">AWx3i\nclZhbc9w",
  "sha1_base64=\"KZBq\nmLp19s93dVIEAqO\njCngwU=\">AWx3i\nclZhbc9w0FIBdrqXc\nUhjywouHTBkGSiZhy\nuWlM23StE2TkqS5tn\nG6I3tlrxJZdmw52dT\njB34Sv4bhDf4JR/bu\nqj5HeSAzZcX5PutyJ\nNmyw1yKUi8t/X3jn\nXfe/+D29+dOvjTz\n797PO5218clFlVRHw\n/ymRWHIWs5FIovq+F\nlvwoLzhLQ8kPw7NVw\nw8veFGKTO3",
  "797PO5218clFlVRHw\n/ymRWHIWs5FIovq+F\nlvwoLzhLQ8kPw7NVw\nw8veFGKTO3pq5yfpC\nxRIhYR0xAazG34o0\nEtGv/b+36Qhtm4Zk0\ngeayPg5Br1qIf/KCs\n0kF9en+5eV0/avwgS\n3li2GnjyHeBIVIRv\npkMLewtLjU/vm0sDw\npLHiTv+3B7a+GwTCL\nqpQrHUlWlsfLS7k+\nqVmhRSR5cyuoSp6z6\nIwl/BiKiqW8PKnbUT\nf+HYgM/",
  "TCL\nqpQrHUlWlsfLS7k+\nqVmhRSR5cyuoSp6z6\nIwl/BiKiqW8PKnbUT\nf+HYgM/Tgr4J/Sfht\n9+4qapWV5lYZgpkyP\nSsxM0MWOKx3/dlILl\nVeaq6hrK6krzPfp\nNAfioJHWl5BgUWFgL\n760YgVLNKQ6FuB4pd\nRlqZMDetgZW2nqSGV\niVA1P6/apDdN31lrH\nQ7F64yV9b1ZLULzVL\nzhpJWMZVcI/CkqWu\n+mCxiIDgAscgJyBQ\nvoU6T",
  "N31lrH\nQ7F64yV9b1ZLULzVL\nzhpJWMZVcI/CkqWu\n+mCxiIDgAscgJyBQ\nvoU6TnzD2lxGFRSYB\n192qCcB40ZCqleYJ5\nKSnvSIaFHLJxz1rlV\ngwlWlP2QXF9+/4BnB\ndwCxAV+GHoznYzZlq\nptdpPtZFWpcmhlso\nmEp42wQMOWLSjKhvq\nEpKuDTqWb9j6wVTZ5\nPEZXnb1cJEkLVX9B1\nd0LyoYd9pI8iCRZj0\nrTaCLAm3hCFLGWR5U",
  "b9j6wVTZ5\nPEZXnb1cJEkLVX9B1\nd0LyoYd9pI8iCRZj0\nrTaCLAm3hCFLGWR5U\nh7AgFPfRNyqUFgVZG\nFuF1nYbzs3Ebw2xz\nnsl763VpP0XzCUERO\nA3Wd+BVMR7+ur2cz2\np8m5aH1T4GN/BJPVv\n4QVSTesaSMwqkmsoW\nabK2TSbEGoyC7pum\nNQ+W56A/QBPCmqwq\nh4re0u20JlqwJB3dh\nqEUl+fGPiz/z8Um9Z\nLaN+Q/JlRUVrmrI",
  "56A/QBPCmqwq\nh4re0u20JlqwJB3dh\nqEUl+fGPiz/z8Um9Z\nLaN+Q/JlRUVrmrIh\nP+HxUN4SGE1xdE8OR\nlEk0eBNrJyTc39HU\nsQIvbBNp5w4KQjEp9\nBXa/iJR/WvaCO5sl\nqK+QsDUC79MKDTJcd\nyXTcDI8AuPU8cCitA\ngo26MkczKquDk5ofW\nM0Ra3dwWC2EeVv0bq\njRC/7B5ewqKMPD4Y\nJfc3mIMhp2+QyzSg\n1ZgZI5NlM6fh2U",
  "Ra3dwWC2EeVv0bq\njRC/7B5ewqKMPD4Y\nJfc3mIMhp2+QyzSg\n1ZgZI5NlM6fh2UGra\nYa/e3U94VnVbCzcm\n7UG/YHaqKOLngw08H\nwmxqCNRXB+cdYlie\nVoD+qaLde3e1ZvP6\neLO3E4bpNSeqd9NJt\nO9xresDPNx293SQe\nsagjUV2THlKPWI72o\nC53Hjdo3C4blOSeq\nd5dNoOd2ai5R/vjeD\nAao5JmRyaY18mgy6E\nRU1F7RTbY25f7",
  "3Hjdo3C4blOSeq\nd5dNoOd2ai5R/vjeD\nAao5JmRyaY18mgy6E\nRU1F7RTbY25f7EJYT\nKu+Bf+PlV0BD4+1\nYWwuF2KvmYCWBpyiY\nfQhbDYbeG+OYlhdO\nhbrpVJvMRMrsQFp+w\nFI+6C2ExoWLiFM9Yn\niOxC5E8jnAeRzSPOZ\nZyl4RnJHfMCFlSrgV\nVjLK+ZAJYGqPWxo7\nGoAcyU6jBSRDLJV15\npXPlKbSKFV3F+6G9\n69pWDNUoQlga",
  "VjLK+ZAJYGqPWxo7\nGoAcyU6jBSRDLJV15\npXPlKbSKFV3F+6G9\n69pWDNUoQlgaYvsMT\n/Ycm6yEKfYb19R6XQ\nJZOU0gdvY2abO9PQX\nxjU5yYXxlaVXlF5a\neknpoaWHlBaWkjeCM\nH5hKXk7CeMLSy8oPb\nD0gNLK0orSfUv3KY0\ntjSl9bOljSiNLI0pX\nLV2lVFtKTqTwRLB0j\n9KRpSNKjyw9ovSlpS\n8pfWrpU0pfWfqK0j\neWvqH0oa",
  "pX\nLV2lVFtKTqTwRLB0j\n9KRpSNKjyw9ovSlpS\n8pfWrpU0pfWfqK0j\neWvqH0oaUPKWMkr\nXLF2jlFtKPh2E8Yql\nK5SGlpJ3P9hrlm5Tm\nluaU/rI0keUDi0lb8\nXwPLOUHG/gwWipHT\nd0nVKhaXk/S2Mn1v\n6nNLU0pTSZ5Y+o/TU\n0lNKn1j6hNLEUvJtA\nE4nlu5Sar8C1SWlO5\nbuUHpu6bn7uwCfTWP\noWphbtoItSjNLM0o3\nLCVvCn",
  "JtA\nE4nlu5Sar8C1SWlO5\nbuUHpu6bn7uwCfTWP\noWphbtoItSjNLM0o3\nLCVvCnCUsPSMnCdjN\nbmrTb82kftarGbcw\nSYZn15Nch6rGXewyd\n1pejW5P8Vqxkek62s\nHsw8pkFK40w/mFpbx\nV1haOPhpcfmXxXs79\nxYerEy+0N70va+8b\n7zlr1fvQfeU2/b2/\nci70/vL+8f79/59fl\ns/mJ+3Knv3Jhc86X\n+5v/4z9uze1w</lat\nexit",
  "U2/b2/\nci70/vL+8f79/59fl\ns/mJ+3Knv3Jhc86X\n+5v/4z9uze1w</lat\nexit>\nhi = a\n2\n4\u03b2i +\nD\nX\nj=1\n!ijxj\n3\n5\nFully connected network:\nConvolutional network (e.g. 1 channel \u00e0 1 channel):\n\ud835\udc21!\n\ud835\udc21\"\n\ud835\udc21#",
  "Previously we saw a sequential network:\n<latexit s\nha1_base64=\"sqOpnO6\nnY7k0+ZnjUC4ZXNbdf\nk=\">AXd3iclZhbT9x\nGFICXlN6I61aHvpQqy\nhRVNEVC/TyUimBkBukQ\nGCBhN2sxt6xd8J4bHyB\nJdY+97X9ef0pV96xvb\nu4HOGSkVKd3q+b86Mz4\nyvbixFmq2s/DX3zrv\nf/Bh7c+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8+ML",
  "+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8+MLnqQiUofZV\ncz7IQuU8IXHMgNFv5x\neqEbjYue6zujyaDj3P3\n1WsSfDIrO5BSa4+V5B3\n7ikdCRfk9FKg9dnji93\nnwzxSpNsVqmGOmey9Mk\nq/+ZI0mWZsmWTVJ1kg\nS17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9PTz",
  "S17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9PTzkqc9QuWZMKTfDLf\ny1MeM+MBfwUmoqFPO0\nX5SpNnDsQGTp+lMA/lT\nl9HqPgoVpehW6YIYsG\n6WY6aCNneaZ/0u/ECrO\nM68aiA/l04WOXrJnaF\nIuJfJK2gwLxEwV8cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx9HC",
  "cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx9HC\nWRWQ8FG85SVIqOskNAg\n8mRcHbQRsDwQGINicgU\njyFnLo+sO4dROGkIAL\nszNeTEhqlfEAatLQXhE\nNGrHk4a1SxYyrChHI\nDiOHcDXiWwCrAVOGHo\nzU4iJmaTPtlfJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQdZh0",
  "fJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQdZh0nSyh\nNZFDZtOGUEWbMKgaZUR\nZEm4hA1ZyKDKdXsABxw\n6OmJXhcKqIBtzL4nc5t\nixjuC9OY7hfGl6WwUp/\nwVDFdEBOPv0r2DK4019\nM5rZzrQ4F6WvG3zsjGC\nxml1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi2rw",
  "1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi2rw71lONQkl/z\n0h/aPfNwvVvRpo/9Dqg\nmJ0jy2JdLh/5FoCDdNv\nL8ghcvkmjxIFAuXiTh\n+o6WjiV4Y+tIuXbQEIp\nJkV2h018EqtmnjODJRi\nGaKwR0XvhlQqF9v2mr\nANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz\n1gj",
  "ANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz\n1gjbcHC74Dd1dVFG3q\ncb5WrIElTMsV7S8etem\nsEpZjv7yWvmlYr4Ofb\n9XgwL1id3P4+WAbr0d\nALOpIlAuet6y5JLEs40\nGu2Xa9PrNi+/X3ZGsHF\ntduSpK3nqXdtrg3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/uHI",
  "3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/uHI\n54x/ZgUyaF+7Itkrwph\nMaNiZhWjkAdIrEJYDPO\nmBf+PlQMBN4+mVYWwuJ\neKpqYDWBpyiQ+hCmGxO\noWbZh3D6o5F3bGrTMYj\nZFYhLD5mIT7qKoTFgIq\nBVTxjcYzEKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6yCWU",
  "EKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6yCWU7rzUuv\nOU2gXK7qLu7aBuzcMnD\nGUAewtEvOMae3az3JX\nFxieMyFTkWyIpAfew\ns0ed6dOf6xfkSQ7eig2\n9ovTS0EtKjw09pjQxlL\nwRuP4LQ8nbietfGHpB6\nZGhR5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLoCaU",
  "5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLoCaUvDX1J6RNDn\n1D6ytBXlL419C2lDwx9\nQCkzlFG6ZegWpdxQ8un\nA9TcM3aDUNZS8+8G5Zu\ngepbGhMaUPDX1I6dBQ8\nlYM9zNDyeMN3BgNlZQ+\nNfQpcJQ8v7m+s8NfU5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn\n9u",
  "5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn\n9u8CfLaMrm1j7poEu5R\nGhkaUbhtK3hTgUcLQM/\nI86av6qma+e06wMeMWVl\nd82pvU3FczbmH1cl8w\nCXGjI/I1LeOZh9SoKRw\npR8sLHXwV1jaOFptd35\nqr+vL93fqL/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25us+X",
  "/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25us+Xrcb\nfYudfruop4g=</late\nxit>h1 = f1[x, \u03c61]\nh2 = f2[h1, \u03c62]\nh3 = f3[h2, \u03c63]\ny = f4[h3, \u03c64]\n6\n\ud835\udc21!\n\ud835\udc21\"\n\ud835\udc21#\nCan think of as a sequence of nested functions:",
  "More layers are better\u2026 \n7\n8 layers\n18 layers\nImageNet",
  "More layers are better\u2026  up to a point \n8\nConvolutional Network on CIFAR10\nCan\u2019t even overfit in \ntraining!",
  "What\u2019s going on?\n9\n200 hidden units each layer\n200 hidden units\nNot completely understood, but\u2026\nTake a look at \ud835\udf15\ud835\udc66/\ud835\udf15\ud835\udc65 for shallow and deep networks.\nA small step in gradient descent may jump to wildly \ndifferent valued gradient!\nGradients of deeper networks \nare much less correlated!",
  "This work is subject to a Creative Commons CC-BY-NC-ND license. (C) MIT Press.\nWhat\u2019s going on?\n10\n200 hidden units each layer\n200 hidden units\nNot completely understood, but\u2026\nTake a look at \ud835\udf15\ud835\udc66/\ud835\udf15\ud835\udc65 for shallow and deep networks.\nA small step in gradient descent may jump to wildly \ndifferent valued gradient!\nGradients of deeper networks \nare much less correlated!\nThe Shattered Gradient Phenomenon",
  "This work is subject to a Creative Commons CC-BY-NC-ND license. (C) MIT Press.\nWhat\u2019s going on?\n11\nThe Shattered Gradient Phenomenon\nThe derivative of the output \ud835\udc66 w.r.t. the first layer f\" is, by the \nchain rule:\nf1 impacts f2 impacts f3, etc\u2026",
  "Solution: Residual connections\nResidual network:\n<latexit s\nha1_base64=\"sqOpnO6\nnY7k0+ZnjUC4ZXNbdf\nk=\">AXd3iclZhbT9x\nGFICXlN6I61aHvpQqy\nhRVNEVC/TyUimBkBukQ\nGCBhN2sxt6xd8J4bHyB\nJdY+97X9ef0pV96xvb\nu4HOGSkVKd3q+b86Mz4\nyvbixFmq2s/DX3zrv\nf/Bh7c+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8",
  "h7c+mv/4k08/+3zh\n9hdHaZQnHu96kYySE5e\nlXArFu5nIJD+JE85CV/\nJj92xT8+MLnqQiUofZV\ncz7IQuU8IXHMgNFv5x\neqEbjYue6zujyaDj3P3\n1WsSfDIrO5BSa4+V5B3\n7ikdCRfk9FKg9dnji93\nnwzxSpNsVqmGOmey9Mk\nq/+ZI0mWZsmWTVJ1kg\nS17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9",
  "kg\nS17+iXdenXdM1/VJf7C\nwtNJeKf8c2ujUjaVW/b\nc3uP3VsDeMvDzkKvMkS\n9PTzkqc9QuWZMKTfDLf\ny1MeM+MBfwUmoqFPO0\nX5SpNnDsQGTp+lMA/lT\nl9HqPgoVpehW6YIYsG\n6WY6aCNneaZ/0u/ECrO\nM68aiA/l04WOXrJnaF\nIuJfJK2gwLxEwV8cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx",
  "wV8cbsY\nR5GWyM+Z7il14UhkwNi\n97G1v4EysYDoQp+npeb\nZDJpOlulw6F5k7Hx9HC\nWRWQ8FG85SVIqOskNAg\n8mRcHbQRsDwQGINicgU\njyFnLo+sO4dROGkIAL\nszNeTEhqlfEAatLQXhE\nNGrHk4a1SxYyrChHI\nDiOHcDXiWwCrAVOGHo\nzU4iJmaTPtlfJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQd",
  "PtlfJwlYZHq\nGB4hYSrg5RBwyB6T+oi\nahsqlhK5ew/oNWy+YOq\nsLF8XlVBMdQdZh0nSyh\nNZFDZtOGUEWbMKgaZUR\nZEm4hA1ZyKDKdXsABxw\n6OmJXhcKqIBtzL4nc5t\nixjuC9OY7hfGl6WwUp/\nwVDFdEBOPv0r2DK4019\nM5rZzrQ4F6WvG3zsjGC\nxml1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi",
  "xml1YElSHNR0EjqOTa\nhZ1gqZtFoQSqLpqlnY\n1F5LJoHqAP4pMsTofxr\n2nLZgi2rw71lONQkl/z\n0h/aPfNwvVvRpo/9Dqg\nmJ0jy2JdLh/5FoCDdNv\nL8ghcvkmjxIFAuXiTh\n+o6WjiV4Y+tIuXbQEIp\nJkV2h018EqtmnjODJRi\nGaKwR0XvhlQqF9v2mr\nANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz",
  "2mr\nANahl+4/Vs2kIcO0quO\n0ZNRmiecXPzQfoZIqev\nLYiL0zap5QZVaF43uJz\n1gjbcHC74Dd1dVFG3q\ncb5WrIElTMsV7S8etem\nsEpZjv7yWvmlYr4Ofb\n9XgwL1id3P4+WAbr0d\nALOpIlAuet6y5JLEs40\nGu2Xa9PrNi+/X3ZGsHF\ntduSpK3nqXdtrg3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/u",
  "trg3zICf\n71hmu0M8YlFHolz1DKl\nHLMt4kMtex3bUVhcuy\nlJ3mkdrbFnZlo+/uHI\n54x/ZgUyaF+7Itkrwph\nMaNiZhWjkAdIrEJYDPO\nmBf+PlQMBN4+mVYWwuJ\neKpqYDWBpyiQ+hCmGxO\noWbZh3D6o5F3bGrTMYj\nZFYhLD5mIT7qKoTFgIq\nBVTxjcYzEKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6y",
  "cYzEKkTqOMJ1HN\nE6xliKbRJekdiyImRL2\nTZUMoqakg5gaYxG1sG\ngxnISKEB6yCWU7rzUuv\nOU2gXK7qLu7aBuzcMnD\nGUAewtEvOMae3az3JX\nFxieMyFTkWyIpAfew\ns0ed6dOf6xfkSQ7eig2\n9ovTS0EtKjw09pjQxlL\nwRuP4LQ8nbietfGHpB6\nZGhR5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLo",
  "GhR5TmhuaUdg3tUuob\n6lP6yNBHlHqGepRuGrp\nJaWYoeSKFO4Kh5SODB\n1RemLoCaUvDX1J6RNDn\n1D6ytBXlL419C2lDwx9\nQCkzlFG6ZegWpdxQ8un\nA9TcM3aDUNZS8+8G5Zu\ngepbGhMaUPDX1I6dBQ8\nlYM9zNDyeMN3BgNlZQ+\nNfQpcJQ8v7m+s8NfU5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn",
  "NfU5\npaGhI6TNDn1H6xtA3lD\n429DGlgaHk2wA8nRh6Q\nKn5ClSklO4buk/puaHn\n9u8CfLaMrm1j7poEu5R\nGhkaUbhtK3hTgUcLQM/\nI86av6qma+e06wMeMWVl\nd82pvU3FczbmH1cl8w\nCXGjI/I1LeOZh9SoKRw\npR8sLHXwV1jaOFptd35\nqr+vL93fqL/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25u",
  "fqL/Q3mp90/\nquda/Vaf3cut960tprd\nVveHJv7fe6PuT+/nvx\n28W7i/cq9Z25us+Xrcb\nfYudfruop4g=</late\nxit>h1 = f1[x, \u03c61]\nh2 = f2[h1, \u03c62]\nh3 = f3[h2, \u03c63]\ny = f4[h3, \u03c64]\n<latexit s\nha1_base64=\"J/6Tea\nBelklyMaNQkoJZdL9Ge\nU=\">AXi3iclZhb9s\n2FMed7tZla9duGPKwF\n2FBh2FLg9x2wbABbVL3\nlnRJmjhJG7sGJVMyG4p",
  "Zhb9s\n2FMed7tZla9duGPKwF\n2FBh2FLg9x2wbABbVL3\nlnRJmjhJG7sGJVMyG4p\nSdEmcCvoke90+1L7NDi\nXZrM5hHmYgMX1+fx2S5\nxSlNxYijRbWfl37sYH\nH3708Sc3P53/7PNbt7+\n4c/fLozTKE4/3vEhGyY\nnLUi6F4r1MZJKfxAlno\nSv5sXu2pfnxBU9SEan\nD7Crmg5AFSvjCYxmYhn\nfnbjt91x8Pi9XS+e4P3\nZ4Pzr90I0mBfx",
  "fnxBU9SEan\nD7Crmg5AFSvjCYxmYhn\nfnbjt91x8Pi9XS+e4P3\nZ4Pzr90I0mBfxw/FKj\nU21fgn/xWOjfg76KVB6\n6PH6/fnGw9rUQ+0Ne1\nmrvFRs5mnN6mnd+Fmjf\ntanftaMn3WLnyvjZ16\n2Zh6WS+XnKmbjXIwvL\nO4srxSfRzaWG0ai53ms\nze8+/WoP4q8POQq8yRL\n09PVlTgbFCzJhCd5Od/\nPUx4z74wF/BSaioU8HR\nRV5krnHlh",
  "WoP4q8POQq8yRL\n09PVlTgbFCzJhCd5Od/\nPUx4z74wF/BSaioU8HR\nRV5krnHlhGjh8l8Kcyp\n7K+f0XBwjS9Cl1Qhiwb\np5hpo42d5pn/6AQKs4\nzry6Iz+XThY5ugyckU\ni4l8kraDAvETBWxuz\nhHkZFMt8X/FLwpDpkZ\nFf7O7X0LceCBUwc/zqn\nDKsq3pVhoOzesUm8OZ\n15ExkPxjhMnlUQ7uUbA\ng7Io+HKwjIHgAMQyJyB\nSPAWfOj",
  "sq3pVhoOzesUm8OZ\n15ExkPxjhMnlUQ7uUbA\ng7Io+HKwjIHgAMQyJyB\nSPAWfOj6Q+FVEYaFIwI\nUpjZclca0yHkBMWrLXR\nAaNWPJS7VFVJDKsCU5\nAInj3HM04FkCWYChwh\ndHOTiImSqn12V8kiVhk\nWob7iFhKuBVFzBlj0k9\no7ZC5VLCpV5L9SdWvWT\nqrAlcFdDTbQFqQ6Tti\nZLaFzUqK2pLEgFRi0V\nZUFqSRsayMWMohy0x7C\nhEN",
  "WT\nqrAlcFdDTbQFqQ6Tti\nZLaFzUqK2pLEgFRi0V\nZUFqSRsayMWMohy0x7C\nhENHW+xSobBUkMLcSyK\n3XesLbg2JzGsl7auW\n5DwXzAUEW2A1ae/BVMe\nb8u3opnamQbnotLrBp8\n4Y0hW+xKWBPW0p3ArB\npbSZVrJCSRgtMSXTZV\nurRWKQ8Fu0JagNedHki\nlP+ebKlqQclqc38Jpr\nkp/eX/6JTwbFil42+h\n+JjhK89jmSJv/h6MR3\nEh",
  "dHki\nlP+ebKlqQclqc38Jpr\nkp/eX/6JTwbFil42+h\n+JjhK89jmSJv/h6MR3\nEhxfYEFJy+SKHlgqJI\nXSdjfUepYgtbW6rcQU\nMoJkV2hZa/CFT7msqCB\nxuFaKxg0H7hmwmFkuz7\nbE2aDF8w5HAUkAemqR\nXz9GTUZonGx+qJ7BUs\nn1tpgIfbNqb6hSC9r7B\npezq6AN4cLfs3lLoqo\nW8fTjXI1YgkK5kSndP\nKmn2awxGyrv0p53bSqA",
  "C9r7B\npezq6AN4cLfs3lLoqo\nW8fTjXI1YgkK5kSndP\nKmn2awxGyrv0p53bSqA\nn6+3fQH4Ls5J7Hz4fb\nOB8BUVGNRL7gDGb1JYn\nK0h/4mpXr+yMrt/8QE\no7sGjtSkn8NqO0qy3a\n0bAz3cso90hOqKiGol8\nNSOkOqKy9Ae+7Hcsc3\nCorUrJfE7jaNVbdHOlK\nj8/cMxz5g+JkVypI9\nkezXJizMqDCzCqOQB0h\nYm7AwzNsq+I0lBwJuH",
  "VbdHOlK\nj8/cMxz5g+JkVypI9\nkezXJizMqDCzCqOQB0h\nYm7AwzNsq+I0lBwJuHm\n1VbcLCvVS0ZdqARSMu8\nRqExbWS7itbGxYumOR\n7tilTMZjpKxNWPiEhXj\nWtQkLAyoMrMIzFsdIWJ\ntIHMc4jmMaxiLYpsIZ\nyS2ZISUlK2gknHUFmkD\nFk1QbxNLZzACGSnUYW\nPE4pRWXmqtPIWqWNEq7\ntk67l3TcaQ23Aol2y\nxpz+rnWRuTjEcMy",
  "ZzACGSnUYW\nPE4pRWXmqtPIWqWNEq7\ntk67l3TcaQ23Aol2y\nxpz+rnWRuTjEcMyBTk\nWSBXTAO5hzR7VTE9/rl\n+Qkxw8IRt6RemloZeUH\nht6TGliKHkicP2XhpKn\nE9e/MPSC0iNDjyjNDc0\np7Rnao9Q31Kf0saGPK\nfUM9SjdMnSL0sxQciKF\nO4Kh5SODR1TemLoCaW\nvDH1F6VNDn1L62tDXlL\n4z9B2lDw19SCkzlFHaN\nbRLKTeUvDp",
  "SODR1TemLoCaW\nvDH1F6VNDn1L62tDXlL\n4z9B2lDw19SCkzlFHaN\nbRLKTeUvDpw/U1DNyl1\nDSXPfrDWDN2jNDY0pvS\nRoY8oHRlKnorhfmYoOd\n7AjdFQSekzQ59RKgwlz\n2+u/8LQF5SGhoaUPjf\n0OaVvDX1L6RNDn1AaGE\nreDcDpxNADSs1boCKld\nN/QfUrPDT23vxfgszS6\ntsLcNQ52KY0MjSjdNpQ\n8KcBRwtAzcp70VbOrmR\nefJ",
  "N/QfUrPDT23vxfgszS6\ntsLcNQ52KY0MjSjdNpQ\n8KcBRwtAzcp70VbOrmR\nefJVbMuIU1EZ9eTWLuq\nxm3sGZ3ml5N9idfzfiY\nDL17NHuRAiGFnX54Z3\nEVv4WljaO15dWflzf2N\nxYfbDZvaG92vul82/m+\ns9r5pfOg87Sz1+l1vLl\n87q+5v+f+Wbi1sL7w28\nLvtfTGXHPNV53WZ6H7H\nxzhLZs=</latexit>h1 = x + f1[x,",
  "q+5v+f+Wbi1sL7w28\nLvtfTGXHPNV53WZ6H7H\nxzhLZs=</latexit>h1 = x + f1[x, \u03c61]\nh2 = h1 + f2[h1, \u03c62]\nh3 = h2 + f3[h2, \u03c63]\ny = h3 + f4[h3, \u03c64]\n12\n\ud835\udc21!\n\ud835\udc21\"\n\ud835\udc21#\n\ud835\udc21!\n\ud835\udc21\"\n\ud835\udc21#\nResidual connections\nK. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep Residual Learning for Image Recognition,\u201d arXiv:1512.03385 [cs], Dec. 2015, http://arxiv.org/abs/1512.03385\nRegular sequential network:",
  "Residual Network\n13\n<latexit s\nha1_base64=\"J/6Tea\nBelklyMaNQkoJZdL9Ge\nU=\">AXi3iclZhb9s\n2FMed7tZla9duGPKwF\n2FBh2FLg9x2wbABbVL3\nlnRJmjhJG7sGJVMyG4p\nSdEmcCvoke90+1L7NDi\nXZrM5hHmYgMX1+fx2S5\nxSlNxYijRbWfl37sYH\nH3708Sc3P53/7PNbt7+\n4c/fLozTKE4/3vEhGyY\nnLUi6F4r1MZJKfxAlno\nSv5sXu2pfn",
  "c3P53/7PNbt7+\n4c/fLozTKE4/3vEhGyY\nnLUi6F4r1MZJKfxAlno\nSv5sXu2pfnxBU9SEan\nD7Crmg5AFSvjCYxmYhn\nfnbjt91x8Pi9XS+e4P3\nZ4Pzr90I0mBfxw/FKj\nU21fgn/xWOjfg76KVB6\n6PH6/fnGw9rUQ+0Ne1\nmrvFRs5mnN6mnd+Fmjf\ntanftaMn3WLnyvjZ16\n2Zh6WS+XnKmbjXIwvL\nO4srxSfRzaWG0ai53ms\nze8+/Wo",
  "anftaMn3WLnyvjZ16\n2Zh6WS+XnKmbjXIwvL\nO4srxSfRzaWG0ai53ms\nze8+/WoP4q8POQq8yRL\n09PVlTgbFCzJhCd5Od/\nPUx4z74wF/BSaioU8HR\nRV5krnHlhGjh8l8Kcyp\n7K+f0XBwjS9Cl1Qhiwb\np5hpo42d5pn/6AQKs4\nzry6Iz+XThY5ugyckU\ni4l8kraDAvETBWxuz\nhHkZFMt8X/FLwpDpkZ\nFf7O7X0LceCBUwc/zqn\nDKsq",
  "kU\ni4l8kraDAvETBWxuz\nhHkZFMt8X/FLwpDpkZ\nFf7O7X0LceCBUwc/zqn\nDKsq3pVhoOzesUm8OZ\n15ExkPxjhMnlUQ7uUbA\ng7Io+HKwjIHgAMQyJyB\nSPAWfOj6Q+FVEYaFIwI\nUpjZclca0yHkBMWrLXR\nAaNWPJS7VFVJDKsCU5\nAInj3HM04FkCWYChwh\ndHOTiImSqn12V8kiVhk\nWob7iFhKuBVFzBlj0k9\no7ZC5VLCpV5L9SdWvWT",
  "Chwh\ndHOTiImSqn12V8kiVhk\nWob7iFhKuBVFzBlj0k9\no7ZC5VLCpV5L9SdWvWT\nqrAlcFdDTbQFqQ6Tti\nZLaFzUqK2pLEgFRi0V\nZUFqSRsayMWMohy0x7C\nhENHW+xSobBUkMLcSyK\n3XesLbg2JzGsl7auW\n5DwXzAUEW2A1ae/BVMe\nb8u3opnamQbnotLrBp8\n4Y0hW+xKWBPW0p3ArB\npbSZVrJCSRgtMSXTZV\nurRWKQ8Fu0JagNedH",
  "notLrBp8\n4Y0hW+xKWBPW0p3ArB\npbSZVrJCSRgtMSXTZV\nurRWKQ8Fu0JagNedHki\nlP+ebKlqQclqc38Jpr\nkp/eX/6JTwbFil42+h\n+JjhK89jmSJv/h6MR3\nEhxfYEFJy+SKHlgqJI\nXSdjfUepYgtbW6rcQU\nMoJkV2hZa/CFT7msqCB\nxuFaKxg0H7hmwmFkuz7\nbE2aDF8w5HAUkAemqR\nXz9GTUZonGx+qJ7BUs\nn1tpgIfbNqb6hSC9",
  "hmwmFkuz7\nbE2aDF8w5HAUkAemqR\nXz9GTUZonGx+qJ7BUs\nn1tpgIfbNqb6hSC9r7B\npezq6AN4cLfs3lLoqo\nW8fTjXI1YgkK5kSndP\nKmn2awxGyrv0p53bSqA\nn6+3fQH4Ls5J7Hz4fb\nOB8BUVGNRL7gDGb1JYn\nK0h/4mpXr+yMrt/8QE\no7sGjtSkn8NqO0qy3a\n0bAz3cso90hOqKiGol8\nNSOkOqKy9Ae+7Hcsc3\nCorUrJfE7jaNVb",
  "n8NqO0qy3a\n0bAz3cso90hOqKiGol8\nNSOkOqKy9Ae+7Hcsc3\nCorUrJfE7jaNVbdHOlK\nj8/cMxz5g+JkVypI9\nkezXJizMqDCzCqOQB0h\nYm7AwzNsq+I0lBwJuHm\n1VbcLCvVS0ZdqARSMu8\nRqExbWS7itbGxYumOR\n7tilTMZjpKxNWPiEhXj\nWtQkLAyoMrMIzFsdIWJ\ntIHMc4jmMaxiLYpsIZ\nyS2ZISUlK2gknHUFmkD\nFk1QbxNLZz",
  "AyoMrMIzFsdIWJ\ntIHMc4jmMaxiLYpsIZ\nyS2ZISUlK2gknHUFmkD\nFk1QbxNLZzACGSnUYW\nPE4pRWXmqtPIWqWNEq7\ntk67l3TcaQ23Aol2y\nxpz+rnWRuTjEcMyBTk\nWSBXTAO5hzR7VTE9/rl\n+Qkxw8IRt6RemloZeUH\nht6TGliKHkicP2XhpKn\nE9e/MPSC0iNDjyjNDc0\np7Rnao9Q31Kf0saGPK\nfUM9SjdMnSL0sxQciKF\nO4Kh5SO",
  "e/MPSC0iNDjyjNDc0\np7Rnao9Q31Kf0saGPK\nfUM9SjdMnSL0sxQciKF\nO4Kh5SODR1TemLoCaW\nvDH1F6VNDn1L62tDXlL\n4z9B2lDw19SCkzlFHaN\nbRLKTeUvDpw/U1DNyl1\nDSXPfrDWDN2jNDY0pvS\nRoY8oHRlKnorhfmYoOd\n7AjdFQSekzQ59RKgwlz\n2+u/8LQF5SGhoaUPjf\n0OaVvDX1L6RNDn1AaGE\nreDcDpxNADSs1boCKld\nN",
  "wlz\n2+u/8LQF5SGhoaUPjf\n0OaVvDX1L6RNDn1AaGE\nreDcDpxNADSs1boCKld\nN/QfUrPDT23vxfgszS6\ntsLcNQ52KY0MjSjdNpQ\n8KcBRwtAzcp70VbOrmR\nefJVbMuIU1EZ9eTWLuq\nxm3sGZ3ml5N9idfzfiY\nDL17NHuRAiGFnX54Z3\nEVv4WljaO15dWflzf2N\nxYfbDZvaG92vul82/m+\ns9r5pfOg87Sz1+l1vLl\n87q+5v+f+Wbi1s",
  "5dWflzf2N\nxYfbDZvaG92vul82/m+\ns9r5pfOg87Sz1+l1vLl\n87q+5v+f+Wbi1sL7w28\nLvtfTGXHPNV53WZ6H7H\nxzhLZs=</latexit>h1 = x + f1[x, \u03c61]\nh2 = h1 + f2[h1, \u03c62]\nh3 = h2 + f3[h2, \u03c63]\ny = h3 + f4[h3, \u03c64]\n\ud835\udc21!\n\ud835\udc21\"\n\ud835\udc21#\nResidual connections\nSubstituting in:",
  "Residual Network\n14\nWe can unravel all the possible paths\nThe output is the sum of the input plus 4 \npartial networks.",
  "Residual Network as Ensemble of Networks\n15\nEnsemble of four networks\nAnother ensemble of four \nnetworks",
  "Residual Network as Ensemble of Networks\n16\n\u2022 16 possible paths through the \nnetwork!\n\u2022 8 paths include f1\n\u2022 The influence of f1 on \ud835\udf15\ud835\udc66/\ud835\udf15\ud835\udc53\" takes \n8 different forms\n\u2022 Gradients on shorter paths \ngenerally better behaved.",
  "Residual Network as Ensemble of Networks\n17\nDuring training, the model can amplify \nor attenuate the different paths to \nachieve the best results",
  "Order of operations is important \n18\nCan only add to the residual \nbecause of the ReLU\nMore flexible approach to end \nwith linear block.\nStarting with linear block gives us \nsome flexibility on spatial \nresolution.\nNote: if we start with a ReLU, \nthen will clamp negative values \nand so do nothing",
  "This helps increase depth\nup to a point\u2026\n19",
  "Topics\n\u2022 Residual connections and residual blocks\n\u2022 Exploding gradients in residual networks\n\u2022 Batch normalization\n\u2022 Common residual architectures\n20",
  "Exploding Gradients in Residual Networks\n21\nWith He initialization we can \ncontrol the variance inside \nthe block\nBut variance doubles when \nwe add the residual back in.\nAnd then grows exponentially.",
  "Exploding Gradients in Residual Networks\n22\nCould stabilize by renormalizing after adding \neach residual.\nMore common to apply batch normalization.",
  "Topics\n\u2022 Residual connections and residual blocks\n\u2022 Exploding gradients in residual networks\n\u2022 Batch normalization\n\u2022 Common residual architectures\n23",
  "Batch Normalization (a.k.a. BatchNorm)\n\u2022 Shifts and rescales each activation so that its mean and variance \nacross the batch become values that are learned during training\n24\nS. Ioffe and C. Szegedy, \u201cBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,\u201d arXiv:1502.03167 [cs], Mar. 2015, \nhttp://arxiv.org/abs/1502.03167",
  "Batch Normalization (a.k.a. BatchNorm)\n\u2022 Shifts and rescales each activation so that its mean and variance \nacross the batch become values that are learned during training\n25\nCalculate the sample mean and standard \ndeviation for each hidden unit across \nsamples of the batch.",
  "Batch Normalization (a.k.a. BatchNorm)\n\u2022 Shifts and rescales each activation so that its mean and variance \nacross the batch become values that are learned during training\n26\nCalculate the sample mean and standard \ndeviation for each hidden unit across \nsamples of the batch.\nStandardize (normalize) to zero-mean and unit \nstandard deviation.\nScale by \ud835\udefe and shift by \ud835\udeff, which are learned parameters.\n^\n^",
  "Batch Normalization (a.k.a. BatchNorm)\n\u2022 Applied independently to each hidden unit\n\u2022 Standard FC Network with K layers, each with D hidden units:\n \nKD learned scales, \ud835\udefe , and KD learned offset, \ud835\udeff  \n\u2022 Convolutional Network with K layers, each with C channels:\n \n KC learned scales, \ud835\udefe , and KC learned offset, \ud835\udeff\n27",
  "Benefits of BatchNorm\nStable forward propagation\n\u2022 Initialize offsets \ud835\udeff to zero and scales \ud835\udefe to 1\n\u2022 Variance now increases linearly\n\u2022 kth block adds one unit of variance to variance of k\n\u2022 At initialization, later layers make smaller relative change to overall \nvariation\n\u2022 During training, the scales can increase in later layers if helpful\n \n\u00e0control the effective depth\n28",
  "Benefits of BatchNorm\nSupports higher learning rates\nMakes the loss surface smoother (reduces shattered gradients)\n29\nH. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, \u201cVisualizing the Loss Landscape of Neural Nets,\u201d arXiv.org,   \nhttps://arxiv.org/abs/1712.09913v3",
  "Benefits of BatchNorm\nRegularization via added noise\nBatchNorm injects noise since BN scale and shift depend on batch \nstatistics\n30",
  "Topics\n\u2022 Residual connections and residual blocks\n\u2022 Exploding gradients in residual networks\n\u2022 Batch normalization\n\u2022 Common residual architectures\n31",
  "ResNet (2015)\n32\nK. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep Residual Learning for Image Recognition,\u201d arXiv:1512.03385 [cs], Dec. 2015, \nhttp://arxiv.org/abs/1512.03385\nResNet Block\nBottleneck Residual\n\u00f7 4 \n\u00d74",
  "Resnet 200 (2016) for ImageNet Classification\n33\nK. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep Residual Learning for Image Recognition,\u201d arXiv:1512.03385 [cs], Dec. 2015, \nhttp://arxiv.org/abs/1512.03385",
  "ImageNet History\n34",
  "DenseNet\n35\nHuang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017b). Densely connected convolutional networks. \nIEEE/CVF Computer Vision & Pattern Recognition, 4700\u20134708.\nFigure from UDL\nFigure from paper",
  "U-Net (2016)\n36\nRonneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. International Conference on \nMedical Image Computing and ComputerAssisted Intervention, 234\u2013241.",
  "U-Net Results \n37",
  "Stacked hourglass networks for Pose Estimation\n38\nNewell, A., Yang, K., & Deng, J. (2016). Stacked hourglass networks for human pose estimation. European Conference on Computer Vision, 483\u2013499.",
  "Feature Pyramid Networks\n39\nT.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, \u201cFeature Pyramid Networks for Object Detection,\u201d in 2017 IEEE Conference \non Computer Vision and Pattern Recognition (CVPR), Honolulu, HI: IEEE, Jul. 2017, pp. 936\u2013944. doi: 10.1109/CVPR.2017.106.",
  "Feature Pyramid Networks\n40\nT.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, \u201cFeature Pyramid Networks for Object Detection,\u201d in 2017 IEEE Conference \non Computer Vision and Pattern Recognition (CVPR), Honolulu, HI: IEEE, Jul. 2017, pp. 936\u2013944. doi: 10.1109/CVPR.2017.106.",
  "Feedback?\nLink\n41",
  "Recurrent Neural Networks\nDL4DS \u2013 Spring 2024\n1\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\nDall-E: create an image of a \nrecurrent neural network \nprocessing text strings",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n2",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n3",
  "Motivation\n\u2022 We want to process a sequence of data like text, digitized speech, \nvideo frames, etc.\n\u2022 Want past samples to influence output from current sample\n4",
  "References\n1. Understanding LSTMs, Colah\u2019s blog, 2015, \nhttps://colah.github.io/posts/2015-08-Understanding-LSTMs/ \n2. Speech and Language Processing. Daniel Jurafsky & James H. \nMartin. Draft of January 5, 2024. \u2013 Chapter 9, RNNs and LSTMs, \nhttps://web.stanford.edu/~jurafsky/slpdraft/9.pdf \n3. The Unreasonable Effectiveness of LSTMs, Andrej Karpathy, 2015, \nhttps://karpathy.github.io/2015/05/21/rnn-effectiveness/ \n5",
  "Recurrent Neural Network\nUnderstanding LSTM Networks, C. Colah Blog Post\nInput at time/step t\nNeural\nNetwork\nLayers\nOutput Activation\n6\nHidden state at time/step t-1\nht-1",
  "Recurrent Neural Network \u2013 Weight Matrices\n7\nS&LP, Jurafsky & Martin",
  "Unrolled view over time\nUnderstanding LSTM Networks, C. Colah Blog Post\nRNN\nDelay \nMemory\nD\nD\nD\nUnrolled RNN\n8\nIn this case you are emitting an output for every input token\nUnrolled network is fed sequentially (not all at once)",
  "Unrolled Network\n9\nThe weights, U, W and \nV, are the same at \neach time step. Only \nthe inputs (xt, ht-1) \nchange.\nS&LP, Jurafsky & Martin",
  "Different RNN configurations\n10\nOutput\nState\nInputs\n(a)\n(b)\n(c)\n(d)\n(e)\n(a) Regular Feed Forward Network\n(b) E.g. image captioning \u2013 input 1 image, outputs sequence of words\n(c) E.g. sentiment analysis from string of words or characters\n(d) E.g. machine translation such as English to French \n(e) Synced sequence input and output, e.g. video frame-by-frame action classification or text \ngeneration\nTransforms",
  "RNN next letter prediction example\n11\nOutput is probability or likelihood over \nthe vocabulary\nOne-hot encoded input of vocabulary \nlength, e.g. (\u2018h\u2019, \u2018e\u2019, \u2018l\u2019, \u2018o\u2019) \nHidden layer encodes history, here e.g. \nlength 3.",
  "Training an RNN\n12\nimport torch.nn as nn\nclass RNN(nn.Module):\n    def __init__(self, input_size, hidden_size, output_size):\n        super().__init__()\n        self.hidden_size = hidden_size\n        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n        self.h2o = nn.Linear(hidden_size, output_size)\n        self.softmax = nn.LogSoftmax(dim=1)\n    def forward(self, input, hidden):\n        combined = torch.cat((input, hidden), 1)\n        hidden = self.i2h(combined)\n        output = self.h2o(hidden)\n        output = self.softmax(output)\n        return output, hidden\n    def initHidden(self):\n        return torch.zeros(1, self.hidden_size)\nhttps://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html \nSimple feed forward network.\nHistory and recurrence \nmanaged outside the model",
  "Training an RNN \u2013 Same Backprop as FFN\n13\n# If you set this too high, it might explode. If too low, it might not learn\nlearning_rate = 0.005\ndef train(category_tensor, line_tensor):\n    hidden = rnn.initHidden()\n    rnn.zero_grad()\n    for i in range(line_tensor.size()[0]):\n        output, hidden = rnn(line_tensor[i], hidden)\n    loss = criterion(output, category_tensor)\n    loss.backward()\n    # Add parameters' gradients to their values, multiplied by learning rate\n    for p in rnn.parameters():\n        p.data.add_(p.grad.data, alpha=-learning_rate) # in-place addition\n    return output, loss.item()\nhttps://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html \nManaging recurrence. \nSingle output for classification in \nthis case.\n\u201cBack propagation Through Time\u201d, e.g. BPTT",
  "Loss Calculation for Sequence\n14\nThe probability that the model \nassigns to the next word in the \ntraining sequence.\nS&LP, Jurafsky & Martin",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n15",
  "Problem of vanishing gradients\nUnderstanding LSTM Networks, C. Colah Blog Post\nTokens from earlier in the sequence can \ninfluence the current output\nBut for plain RNNs, the influence can \nreduce rapidly the further the sequence \ndifference\n16",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n17",
  "Redrawing RNN\nUnderstanding LSTM Networks, C. Colah Blog Post\nRedraw the RNN in slightly more \ndetail\n18",
  "First redraw RNN\n19",
  "Long Short Term Memory (LSTM)\n20",
  "LSTM \u2013 Cell State\n21",
  "LSTM -- Forgetting Gate\n22\nDecides what part of cell state to suppress",
  "LSTM \u2013 Cell state update\n23\nInput Gate Layer\nCandidate Cell State",
  "LSTM \u2013 Apply changes to cell state\n24",
  "LSTM \u2013 Output and Hidden State Update\n25\nOutput Gate\nNext hidden state and output",
  "Long Short Term Memory (LSTM)\nUnderstanding LSTM Networks, C. Colah Blog Post\nIllustrated Guide to LSTM\u2019s and GRU\u2019s, M. Phi Blog Post\n\u201cforget gate\u201d\n\u201cinput gate\u201d and \nnew cell state\nnew \noutput\n\u201coutput \ngate\u201d\nC\nt\nCt-\n1\n26",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n27",
  "Gradient Recurrent Unit\n28\n\u2022 Combines the forget and input gates into a single \u201cupdate gate.\u201d\n\u2022 Merges the cell state and hidden state\nThe resulting model is simpler than standard LSTM models.\nResults are mixed.\nK. Cho et al., \u201cLearning Phrase Representations using RNN Encoder-Decoder for Statistical \nMachine Translation.\u201d arXiv, Sep. 02, 2014. doi: 10.48550/arXiv.1406.1078.",
  "Topics\n\u2022 Plain (vanilla) Recurrent Neural Network\n\u2022 Problem of vanish gradients\n\u2022 Long Short-Term Memory\n\u2022 Gradient Recurrent Unit\n\u2022 Example applications\n29",
  "30\nTrained on complete works of Shakespeare\n3-layer RNN with 512 hidden nodes on each layer. \nTrained for a few hours on a GPU",
  "31\nTrained and \nGenerated \nWikipedia Content\nStructured Markdown\nValid XML",
  "32",
  "33\n\u2026\n\u2026",
  "2000\n34",
  "Next Up                                         Feedback?\n\u2022Transformers!\n\u2022LLMs, embeddings, multi-\nmodal, foundation \nmodels\n35\nLink",
  "Transformers\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "2000\n2",
  "A Neural Probabilistic Language Model\nBengio et al, 2000 and 2003\n\u2022 Build a probabilistic language \nmodel from NNs\n\u2022 Feed forward network with \nshared parameters, C, that create \nembeddings\n\u2022 Predicts the probability of a word \nat time t, based on the context of \nthe last n words\n\u2022 Can use shallow feed forward or \nrecurrent neural networks\n3\nY. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, \u201cA Neural Probabilistic Language Model,\u201d Journal of Machine Learning Research, vol. 3, pp. 1137-- 1155, Feb. 2003.\nC  is a \ud835\udc49 \u00d7\ud835\udc5a matrix\n\ud835\udc64! \u2208 \ud835\udc49 words in the vocabulary\n\u00df feature vectors, \ud835\udc36(\ud835\udc64!)\nOptional direct \nconnections \u00e0 \nLimited to context length of n",
  "Sequence to Sequence Learning with Neural Networks\nSutskever et al (2014)\n\u2022 Used LSTMs in an Encoder/Decoder \nstructure\n\u2022 Estimate the probability of \n\ud835\udc5d \ud835\udc66!, \u2026 , \ud835\udc66\"! \ud835\udc65!, \u2026 , \ud835\udc65\") where \ud835\udc47# \u2260 \ud835\udc47 \n\u2022 Encoder mapped sequence to a fixed \nsize token (hidden state)\n\u2022 The hidden state may not encode all \nthe information needed by the \ndecoder\n4\nI. Sutskever, O. Vinyals, and Q. V. Le, \u201cSequence to Sequence Learning with Neural Networks,\u201d in Advances in Neural Information Processing \nSystems, Curran Associates, Inc., 2014.  Link\nEncoder\nDecoder\nBottleneck\nBottleneck between Encoder \nand Decoder!\nUse LSTMs",
  "Neural Machine Translation by Jointly Learning to Align and Translate\nBahdanau, Cho & Bengio (2014-15)\n\u2022 Used bi-directional LSTMs\n\u2022 Automatically \u201csoft-search\u201d parts of \ninput that influence the output\n\u2022 Overcomes the bottleneck of a fixed \nsize hidden state between encoder \nand decoder\n\u2022 Significantly improved ability to \ncomprehend longer sequences\n5\nAdd Attention",
  "Attention is All You Need\nVaswani et al (2017)\n\u2022 Removed LSTMs and didn\u2019t use \nconvolutions\n\u2022 Only attention mechanisms and \nMLPs\n\u2022 Parallelizable by removing \nsequential hidden state \ncomputation\n\u2022 Outperformed all previous \nmodels\n6\nEncoder\nDecoder\nRemove LSTMs",
  "Transformers applied to many NLP applications\n\u2022 Translation\n\u2022 Question answering\n\u2022 Summarizing\n\u2022 Generating new text\n\u2022 Correcting spelling and grammar\n\u2022 Finding entities\n\u2022 Classifying bodies of text\n\u2022 Changing style etc.\n7",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n8",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n9",
  "Motivation\nDesign neural network to encode and process text:\n10",
  "Motivation\nDesign neural network to encode and process text:\n11\nEncode word (or word parts) in some kind of D-dimensional embedding vector.\nWe\u2019ll look at tokenization and embedding encoding later.\nFor now assume a word is a token.",
  "Motivation\nDesign neural network to encode and process text:\nx N\n12\nIn this example, we have a D-dimensional input vector for each of the 37 words \nabove.\nNormally we would represent punctuation, capitalization, spaces, etc. as well.",
  "Standard fully-connected layer\n<latexit sha1_base64\n=\"qpdQ60UJs2/a32vDagROasEJKU=\">AWpX\niclZhbU9w2FICd9JamN9JOemLp0ymt3QHOunl\npTMJhNwgZQkskLBkR/bKXgVZNrYMSz7G/pr+tr\n+jv6bHtneVXyOeOjOJBbn+6zLkWzLDjIpCr26+\nu+16+8+97H9z48OZH3/y6WdLtz4/KNIyD/k\ngTGWaHwWs4FIoPtBCS36U5ZwlgeSHwemG4YfnPC\n9Eqvb1Z",
  "y6WdLtz4/KNIyD/k\ngTGWaHwWs4FIoPtBCS36U5ZwlgeSHwemG4YfnPC\n9Eqvb1ZcZPEhYrEYmQaQiNlr4bBtHE/90fJkE6\nreAPn82O4RBwzX6A407CYwbH6cloaW1t1r/fF\npYawsrXvrj259OR6O07BMuNKhZEVxvLa6ZOK5\nVqEks9uDsuCZyw8ZTE/hqJiCS9OqnpM/82RMZ\n+lObwT2m/jr59RsWSorhMAjATpicFZiboYselj\nn47qYTK",
  "qJiCS9OqnpM/82RMZ\n+lObwT2m/jr59RsWSorhMAjATpicFZiboYselj\nn47qYTKSs1V2DQUldLXqW8S5I9FzkMtL6HAwlxA\nX/1wnIWakjzaHiF2GaJEyNq+H65u4McsZjoS\np+VtYpnc26zmbtcCheZaw/2V/UIjRPxBtOKqkV\nU8kVAo9nVcV7cQ8DwQGIHicgVbyAOk1+YNLXEIU\nlJAFXdlk8n5GqleYx5KSjvSQaFDLJpx1rg1gwl\nUlH2",
  "IHicgVbyAOk1+YNLXEIU\nlJAFXdlk8n5GqleYx5KSjvSQaFDLJpx1rg1gwl\nUlH2QPF92/7BnCdwyxAV+HA0RzsZUzN5udpPtV\n5UhUmhlvImYp53QMOWTSjKhrqFJKODXsWH9g6z\nlTp23i0qzuam4iyNrPu47OaV7UuOvUEWTBIoy7\nVh1BloQLfswSBluyMYcOKbiFsVCquCLMx+ng\nbdtjMTwWtzmsH10vU2K5L+c4YyYgJw9ZmjYCrkX\nX0",
  "yMYcOKbiFsVCquCLMx+ng\nbdtjMTwWtzmsH10vU2K5L+c4YyYgJw9ZmjYCrkX\nX0jXdj+PDntW8KfOpPYLK6p7A8boY1bwRG1cZ\nm1KxzhUyaLQjl6UXNL1xqDwT3QGaAL7oylyo6\nC3tTl2CJWvCwzsw1LyU/PjH3s98elKtmsvG/Eey\nCRUVZeaqyIT/R0VjeMTg9QURPHmpRJMHgXryUg\nn3dzR1LMcL20TquYOCUEwKfYkufxGr7jl1BHc2",
  "VjeMTg9QURPHmpRJMHgXryUg\nn3dzR1LMcL20TquYOCUEwKfYkufxGr7jl1BHc2\nTVBfIWDqhSMTCk1yFHVlEzAyHOFh6VhAIRpk2Iw\nxlGlR5pzc/NB6hkitm9tiLszDqntDlUbo3je4X\nJwFZXg4nPMrTg9QRoMmn0FaqjHLUTKnZkqnr4a\nFhkvMdfXU94UnVbMz7ba9qBfMDtlGPKz0Raej\n5hY1JGoLtidOuSxHK0B3UtluvbPau2Xn1Pln",
  "4UnVbMz7ba9qBfMDtlGPKz0Raej\n5hY1JGoLtidOuSxHK0B3UtluvbPau2Xn1Plnbs\ncN2mJPW2vXTbDveKHvCzbUdvt4lHLOpIVFfbQ+\noRy9Ee1OXO47ZrFA7XbUpS7zyPTtvhLky0/KP9\nCWxIzTYplWOz7UvlsAlhUVNRO8XU7Gy7YhPCYlJ\n2LfgbK3sCHh5dqwlhsV+IrmYCWBpziYfQhLDYX\nMJds41hduhbrtVJrMJMpsQFh+xBI+6CW",
  "sCHh5dqwlhsV+IrmYCWBpziYfQhLDYX\nMJds41hduhbrtVJrMJMpsQFh+xBI+6CWExpmL\nsFE9ZliGxCZE8TnAeJzSPGZYyl4RnJHPMCFlSrg\nWVT9KuZAJYmqLWpo7GoAcyVajBNojlgq68wrny\nFrFiq7igavhwRUNa4YqNAEs7ZBrzB/uOC+yAK\ncYtlmuJGcCWRlNYB87ferMd39BVJGdXBdWnpJ6\nYWlF5QeWnpIaW4peSMIoueWkreTIDq",
  "uJGcCWRlNYB87ferMd39BVJGdXBdWnpJ6\nYWlF5QeWnpIaW4peSMIoueWkreTIDq39JzSA0s\nPKC0tLSkdWDqgNLI0ovShpQ8pDS0NKd2wdINSb\nSnZkcITwdJ9SieWTig9svSI0heWvqD0saWPKX1p\n6UtK31j6htL7lt6nlFnKN20dJNSbin5dBE65\nauUxpYSt794FqztE9pZmlG6QNLH1A6tpS8FcPz\nzFKyvYEHo6WS0ieWPqFUWEre34Lo",
  "xpYSt794FqztE9pZmlG6QNLH1A6tpS8FcPz\nzFKyvYEHo6WS0ieWPqFUWEre34LomaXPKE0sTSh\n9aulTSl9b+prSR5Y+ojS2lHwbgN2JpXuU2q9AV\nUHprqW7lJ5Zeub+LsAX0xi4FuaOrWCH0tTSlNI\ntS8mbAmwlLD0l+8lItXe1+dcmcl+L1I7WJvx+d\nk5FacAdr707zs8n9KVILPiFd3zxYfEiBlMKd\nfrS0soa/wtLCwU+9tV96d3fvrt",
  "k5FacAdr707zs8n9KVILPiFd3zxYfEiBlMKd\nfrS0soa/wtLCwU+9tV96d3fvrtxb7/Q3vC+8r\n72vXWvF+9e95jr+8NvND70/vL+9v7Z/mb5WfL+\n8sHjXr9WnvOF17ntz6D4gk3uw=</latexit>h = a[\u03b2 + \u2326x]\n13\nAssuming D inputs and \nD hidden units.",
  "Standard fully-connected layer\n<latexit sha1_base64\n=\"qpdQ60UJs2/a32vDagROasEJKU=\">AWpX\niclZhbU9w2FICd9JamN9JOemLp0ymt3QHOunl\npTMJhNwgZQkskLBkR/bKXgVZNrYMSz7G/pr+tr\n+jv6bHtneVXyOeOjOJBbn+6zLkWzLDjIpCr26+\nu+16+8+97H9z48OZH3/y6WdLtz4/KNIyD/k\ngTGWaHwWs4FIoPtBCS36U5ZwlgeSHwemG4YfnPC\n9Eqvb1Z",
  "y6WdLtz4/KNIyD/k\ngTGWaHwWs4FIoPtBCS36U5ZwlgeSHwemG4YfnPC\n9Eqvb1ZcZPEhYrEYmQaQiNlr4bBtHE/90fJkE6\nreAPn82O4RBwzX6A407CYwbH6cloaW1t1r/fF\npYawsrXvrj259OR6O07BMuNKhZEVxvLa6ZOK5\nVqEks9uDsuCZyw8ZTE/hqJiCS9OqnpM/82RMZ\n+lObwT2m/jr59RsWSorhMAjATpicFZiboYselj\nn47qYTK",
  "qJiCS9OqnpM/82RMZ\n+lObwT2m/jr59RsWSorhMAjATpicFZiboYselj\nn47qYTKSs1V2DQUldLXqW8S5I9FzkMtL6HAwlxA\nX/1wnIWakjzaHiF2GaJEyNq+H65u4McsZjoS\np+VtYpnc26zmbtcCheZaw/2V/UIjRPxBtOKqkV\nU8kVAo9nVcV7cQ8DwQGIHicgVbyAOk1+YNLXEIU\nlJAFXdlk8n5GqleYx5KSjvSQaFDLJpx1rg1gwl\nUlH2",
  "IHicgVbyAOk1+YNLXEIU\nlJAFXdlk8n5GqleYx5KSjvSQaFDLJpx1rg1gwl\nUlH2QPF92/7BnCdwyxAV+HA0RzsZUzN5udpPtV\n5UhUmhlvImYp53QMOWTSjKhrqFJKODXsWH9g6z\nlTp23i0qzuam4iyNrPu47OaV7UuOvUEWTBIoy7\nVh1BloQLfswSBluyMYcOKbiFsVCquCLMx+ng\nbdtjMTwWtzmsH10vU2K5L+c4YyYgJw9ZmjYCrkX\nX0",
  "yMYcOKbiFsVCquCLMx+ng\nbdtjMTwWtzmsH10vU2K5L+c4YyYgJw9ZmjYCrkX\nX0jXdj+PDntW8KfOpPYLK6p7A8boY1bwRG1cZ\nm1KxzhUyaLQjl6UXNL1xqDwT3QGaAL7oylyo6\nC3tTl2CJWvCwzsw1LyU/PjH3s98elKtmsvG/Eey\nCRUVZeaqyIT/R0VjeMTg9QURPHmpRJMHgXryUg\nn3dzR1LMcL20TquYOCUEwKfYkufxGr7jl1BHc2",
  "VjeMTg9QURPHmpRJMHgXryUg\nn3dzR1LMcL20TquYOCUEwKfYkufxGr7jl1BHc2\nTVBfIWDqhSMTCk1yFHVlEzAyHOFh6VhAIRpk2Iw\nxlGlR5pzc/NB6hkitm9tiLszDqntDlUbo3je4X\nJwFZXg4nPMrTg9QRoMmn0FaqjHLUTKnZkqnr4a\nFhkvMdfXU94UnVbMz7ba9qBfMDtlGPKz0Raej\n5hY1JGoLtidOuSxHK0B3UtluvbPau2Xn1Pln",
  "4UnVbMz7ba9qBfMDtlGPKz0Raej\n5hY1JGoLtidOuSxHK0B3UtluvbPau2Xn1Plnbs\ncN2mJPW2vXTbDveKHvCzbUdvt4lHLOpIVFfbQ+\noRy9Ee1OXO47ZrFA7XbUpS7zyPTtvhLky0/KP9\nCWxIzTYplWOz7UvlsAlhUVNRO8XU7Gy7YhPCYlJ\n2LfgbK3sCHh5dqwlhsV+IrmYCWBpziYfQhLDYX\nMJds41hduhbrtVJrMJMpsQFh+xBI+6CW",
  "sCHh5dqwlhsV+IrmYCWBpziYfQhLDYX\nMJds41hduhbrtVJrMJMpsQFh+xBI+6CWExpmL\nsFE9ZliGxCZE8TnAeJzSPGZYyl4RnJHPMCFlSrg\nWVT9KuZAJYmqLWpo7GoAcyVajBNojlgq68wrny\nFrFiq7igavhwRUNa4YqNAEs7ZBrzB/uOC+yAK\ncYtlmuJGcCWRlNYB87ferMd39BVJGdXBdWnpJ6\nYWlF5QeWnpIaW4peSMIoueWkreTIDq",
  "uJGcCWRlNYB87ferMd39BVJGdXBdWnpJ6\nYWlF5QeWnpIaW4peSMIoueWkreTIDq39JzSA0s\nPKC0tLSkdWDqgNLI0ovShpQ8pDS0NKd2wdINSb\nSnZkcITwdJ9SieWTig9svSI0heWvqD0saWPKX1p\n6UtK31j6htL7lt6nlFnKN20dJNSbin5dBE65\nauUxpYSt794FqztE9pZmlG6QNLH1A6tpS8FcPz\nzFKyvYEHo6WS0ieWPqFUWEre34Lo",
  "xpYSt794FqztE9pZmlG6QNLH1A6tpS8FcPz\nzFKyvYEHo6WS0ieWPqFUWEre34LomaXPKE0sTSh\n9aulTSl9b+prSR5Y+ojS2lHwbgN2JpXuU2q9AV\nUHprqW7lJ5Zeub+LsAX0xi4FuaOrWCH0tTSlNI\ntS8mbAmwlLD0l+8lItXe1+dcmcl+L1I7WJvx+d\nk5FacAdr707zs8n9KVILPiFd3zxYfEiBlMKd\nfrS0soa/wtLCwU+9tV96d3fvrt",
  "k5FacAdr707zs8n9KVILPiFd3zxYfEiBlMKd\nfrS0soa/wtLCwU+9tV96d3fvrtxb7/Q3vC+8r\n72vXWvF+9e95jr+8NvND70/vL+9v7Z/mb5WfL+\n8sHjXr9WnvOF17ntz6D4gk3uw=</latexit>h = a[\u03b2 + \u2326x]\nProblem:\n\u2022 token (word) vectors may be 512 or 1024 dimensional\n\u2022 need to process large segment of text\n\u2022 Hence, would require a very large number of parameters\n\u2022 Can\u2019t cope with text of different lengths\nConclusion:  \n\u2022 We need a model where parameters don\u2019t increase with input length\n14",
  "Motivation\nDesign neural network to encode and process text:\nThe word their must \u201cattend to\u201d the word restaurant.\n15",
  "The word their must \u201cattend to\u201d the word restaurant.\nConclusions:\n\u2022\nThere must be connections between the words.\n\u2022\nThe strength of these connections will depend on the words themselves.\nMotivation\nDesign neural network to encode and process text:\n16",
  "Motivation\n\u2022 Need to efficiently process large strings of text\n\u2022 Need to relate words across fairly long context lengths\nSelf-Attention addresses these problems",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n18",
  "Dot-Product Self-Attention\n1. Shares parameters to cope with long input passages of different \nlengths\n2. Contains connections between word representations that depend \non the words themselves\n19",
  "Dot-product self attention\n\u2022 Takes N inputs of size Dx1 and returns N inputs of size Dx1\n\u2022 Computes N values (no ReLU)\n\u2022 N outputs are weighted sums of these values\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"s3yg\npR0XPJtRZRZ8OK/g/OJAjcM=\">AWrniclZhbU9w2FICd9JamN\n9JOemLp0xmOm3CQCa9vHQmgZAbpCyBRKW7Mhe2asgy8aWlyWe\n/R/9NX1t/0L/TY9s7yo+RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1",
  "RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1Sm+XHACi6F4n0t\nOTHWc5ZEkh+FJxtGn404XkhUnWgLzN+mrBYiUiETENouHTPHwTR\nZFipmf+7KQZcs2E1mf0E5d2Ex/Ufg4TpcRBV05kRh0sra6tr9Y9\nPC+tYcVrf3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzM",
  "3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzMvw2RkR+lOfxT2q+j719RsaQoLpMAT\nNPNAjMTdLGTUke/nVZCZaXmKmwaikrp69Q3mfJHIuehlpdQYGEu\noK9+OGY5CzXk8+ZA8YswTRKmRtVgY2tvVg0CHgtV8fOyzu1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZ",
  "1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZor9dURhLUnAwIN0Cp2L/JczUrXSPIacdLTXRINCJvm0\nY20SC6Yy6Sj7oPj+bd8ArnOYBegq/OJoDvYzpmbz6zSf6jypChP\nDLeRMxbxuAoYcMmlG1DVUKSVcGnasP7D1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+C",
  "1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+CbiVoXCq\niALs5enQbftzETw2pxmsF+63lZF0j9hKCMmALvP/BZMhbyrb6YL\n258nZ1L7psCn/hgmq3sJy+NmWPNGYFRtbEbNOlfIpNmCUJ5edE\n3TG4fKM9EdoAngTVfmQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9",
  "mQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9cXRPDkpRJNHgTqyUsl3N/R\n1LEcL2wTqecOCkIxKfQl2v4iVt1r6gjubJqgvkLA1Au/mVBokqO\noK5uAkeE3PDUdCyhEgwybMYyLcqck5sfWs8QqXVzW8yFeVh1b6\njSCN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/f",
  "CN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/fbtuDfsHslGHIz4fbeD5iYlFHorgmOKsSxL\n0R7UtViu7/es2n7zI1nascN1m5LU2/bSbTvcK3rAz3cvd0hHr\nGoI1FdbQ+pRyxHe1CXO487rlE4XLcpSb3zPDpth7sw0fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVe",
  "fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVeIbqaCWBpxCUeQhPCYrOFu2Ybw+qOQ91xq0xmY2Q2ISw+YQk\nedRPCYkzF2CmesSxDYhMieRzjPI5pHjMsZS4Jz0jmBGypFwLKh\n+nXckEsDRFrU0djUEPZKpQg20QywVdeYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR",
  "deYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR515qe/+vW\n0EtLym9sPSC0iNLjyjNLSVvBEH0lLydgLv65ZOKD209JDS0t\nKS0r6lfUojSyNKH1v6mNLQ0pDSTUs3KdWkhMpPBEsPaB0bOmY0\nmNLjyl9ZekrSp9a+pTS15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0of",
  "S15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0ofWfqI0pGl5K0YnmeWkuM\nNPBgtlZQ+s/QZpcJS8v4WRC8sfUFpYmlC6XNLn1P61tK3lD6x9A\nmlsaXk2wCcTizdp9R+BaoKSvcs3aP03NJz93cBvpjGwLUwd20Fu\n5SmlqaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWv",
  "qaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWvAx6frW4eJDCqQU7vTDpZV1/BWFg7vra\n7/snp/7/7Kg432C+0N7zve+8Hb9371XvgPfV6Xt8LvT+9v7y/v\nX+W15YPl0+Xh416/Vp7zTde52d5/B/AjePT</latexit>vn = \u03b2v + \u2326vxn\n20",
  "Dot-product self attention\n\u2022 Takes N inputs of size Dx1 and returns N inputs of size Dx1\n\u2022 Computes N values (no ReLU)\n\u2022 N outputs are weighted sums of these values\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"s3yg\npR0XPJtRZRZ8OK/g/OJAjcM=\">AWrniclZhbU9w2FICd9JamN\n9JOemLp0xmOm3CQCa9vHQmgZAbpCyBRKW7Mhe2asgy8aWlyWe\n/R/9NX1t/0L/TY9s7yo+RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1",
  "RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1Sm+XHACi6F4n0t\nOTHWc5ZEkh+FJxtGn404XkhUnWgLzN+mrBYiUiETENouHTPHwTR\nZFipmf+7KQZcs2E1mf0E5d2Ex/Ufg4TpcRBV05kRh0sra6tr9Y9\nPC+tYcVrf3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzM",
  "3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzMvw2RkR+lOfxT2q+j719RsaQoLpMAT\nNPNAjMTdLGTUke/nVZCZaXmKmwaikrp69Q3mfJHIuehlpdQYGEu\noK9+OGY5CzXk8+ZA8YswTRKmRtVgY2tvVg0CHgtV8fOyzu1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZ",
  "1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZor9dURhLUnAwIN0Cp2L/JczUrXSPIacdLTXRINCJvm0\nY20SC6Yy6Sj7oPj+bd8ArnOYBegq/OJoDvYzpmbz6zSf6jypChP\nDLeRMxbxuAoYcMmlG1DVUKSVcGnasP7D1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+C",
  "1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+CbiVoXCq\niALs5enQbftzETw2pxmsF+63lZF0j9hKCMmALvP/BZMhbyrb6YL\n258nZ1L7psCn/hgmq3sJy+NmWPNGYFRtbEbNOlfIpNmCUJ5edE\n3TG4fKM9EdoAngTVfmQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9",
  "mQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9cXRPDkpRJNHgTqyUsl3N/R\n1LEcL2wTqecOCkIxKfQl2v4iVt1r6gjubJqgvkLA1Au/mVBokqO\noK5uAkeE3PDUdCyhEgwybMYyLcqck5sfWs8QqXVzW8yFeVh1b6\njSCN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/f",
  "CN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/fbtuDfsHslGHIz4fbeD5iYlFHorgmOKsSxL\n0R7UtViu7/es2n7zI1nascN1m5LU2/bSbTvcK3rAz3cvd0hHr\nGoI1FdbQ+pRyxHe1CXO487rlE4XLcpSb3zPDpth7sw0fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVe",
  "fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVeIbqaCWBpxCUeQhPCYrOFu2Ybw+qOQ91xq0xmY2Q2ISw+YQk\nedRPCYkzF2CmesSxDYhMieRzjPI5pHjMsZS4Jz0jmBGypFwLKh\n+nXckEsDRFrU0djUEPZKpQg20QywVdeYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR",
  "deYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR515qe/+vW\n0EtLym9sPSC0iNLjyjNLSVvBEH0lLydgLv65ZOKD209JDS0t\nKS0r6lfUojSyNKH1v6mNLQ0pDSTUs3KdWkhMpPBEsPaB0bOmY0\nmNLjyl9ZekrSp9a+pTS15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0of",
  "S15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0ofWfqI0pGl5K0YnmeWkuM\nNPBgtlZQ+s/QZpcJS8v4WRC8sfUFpYmlC6XNLn1P61tK3lD6x9A\nmlsaXk2wCcTizdp9R+BaoKSvcs3aP03NJz93cBvpjGwLUwd20Fu\n5SmlqaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWv",
  "qaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWvAx6frW4eJDCqQU7vTDpZV1/BWFg7vra\n7/snp/7/7Kg432C+0N7zve+8Hb9371XvgPfV6Xt8LvT+9v7y/v\nX+W15YPl0+Xh416/Vp7zTde52d5/B/AjePT</latexit>vn = \u03b2v + \u2326vxn\n<latexit sha1_base64=\"nQjrh7ml\ninp/UHhS2MLkCIPSK+s=\">AW1n",
  "vn = \u03b2v + \u2326vxn\n<latexit sha1_base64=\"nQjrh7ml\ninp/UHhS2MLkCIPSK+s=\">AW1niclZhbT9xGFICd9Jaml5BW5aUvVlGkqk\npXUKWXl0gJhNwgYQksEFiyGnvH3gnjsfEFljuW9X/qT+jf6BvrY/oWds7w\n4+Z3joSrCz5/s8lzMzvnmJFm+vPzXtevf/Bhx/d+PjmJ59+9vmthdtf7G\nVxkfp84McyTg8lnEpFB/kIpf8IEk5izJ972TNc3z3i",
  "x/d+PjmJ59+9vmthdtf7G\nVxkfp84McyTg8lnEpFB/kIpf8IEk5izJ972TNc3z3iaiVjt5hcJP45Yq\nEQgfJZDaLRw6A4jL56W5dAL3IxV1dEwYvnEC8pNSpVdezed4dZEY3K6P5K9\naZ8WTFk3HUv/46qY6jpTBdGC0vLveX649LCSltYctpPf3T7q/FwHPtFxFXuS\n5ZlRyvLSX5csjQXvuTVzWGR8YT5JyzkR1BULOLZcVknoXLvQGTsBnEKf",
  "tFxFXuS\n5ZlRyvLSX5csjQXvuTVzWGR8YT5JyzkR1BULOLZcVknoXLvQGTsBnEKfyp36\n+jlI0oWZdlF5IGp+5thpoM2dlTkwS/HpVBJkXPlNw0FhXTz2NUZdci5X4u\nL6DA/FRAX1/wlLm5D3m0PFz/04ipgal8PV9e0KMs1DoUp+WtRzUFVdZ712\nOBSvMlaf7c5rETmPxDtOKqkVXckVAg+rsuS9sIeB4ABEjxMQK5Bnc1cuyuI\nwpqTgMtmN",
  "laf7c5rETmPxDtOKqkVXckVAg+rsuS9sIeB4ABEjxMQK5Bnc1cuyuI\nwpqTgMtmNenF9KoiVauch5CTjnZINCgk871hqxYCqjrIDiuvecTXgeQqz\nAF2FL47mYCdhqpodl/NpnkZlpmO4hZSpkNdNwJB9JvWIuoYqpIRD/Y71Elu\nvmDpExcndVdTHUHWbtp18pTmRY27Th1BFizCsGvVEWRJOEOMWcQgy215BAO\nOXB2xq0JhVZCF2U9jr9t2oi",
  "18pTmRY27Th1BFizCsGvVEWRJOEOMWcQgy215BAO\nOXB2xq0JhVZCF2U9jr9t2oiN4bU4T2C9db70k6T9jKCM6ALtPfwumfN7V1+K\n57c6Sc1b7usCn7gQmq3sIS8NmWLNGYFRtrKJmnStk0mxBKI3Pu6bujUXlieg\nOUAfwpitSoYJL2t26BEtWh4d3YahpIfnR970f+fS4XNbRv8j2YSKsiKxVa\nTD/6OiMVyT8PqCJ68WKLJg0A9ebGE8zuaO",
  "IfnR970f+fS4XNbRv8j2YSKsiKxVa\nTD/6OiMVyT8PqCJ68WKLJg0A9ebGE8zuaOpbiha0j9dxBQSgmRX6Btr8IVf\neYOoI7G0eorxDQ9cI3EwpNchB0ZR3QMnzD1dWygHw0SL8Zoy/jrEg5Ofmh9Q\nyRWtenxVToi1X3hCq10D1vcDk/CspwcTjVxzuoYx6T69uFBjlqJkTvWUTt\n8Msxy2mG31PeFK1WyE832vagXzA7he/z09EGno+QWNSRqC",
  "T69uFBjlqJkTvWUTt\n8Msxy2mG31PeFK1WyE832vagXzA7he/z09EGno+QWNSRqC64nbHWJYla\nQ/qmi/Xyz0rN958R5Z2aHtpiT1tr202xb3ih7w01LbzeJRyzqSFRX20PqE\ncvSHtRlz+OmbRQW125KUu8sj1b4s5NtPyD3QnPmb5NiuVY3/bFctiEsJhTM\nbeKcRDJDYhLEZF14LfWNkRcPHoWk0Ii/1MdDUdwNKYSzyEJoTFZgt3zTaG1\nU2",
  "M\nbeKcRDJDYhLEZF14LfWNkRcPHoWk0Ii/1MdDUdwNKYSzyEJoTFZgt3zTaG1\nU2LumlXmUwmyGxCWHzCIjzqJoTFkIqhVTxhSYLEJkTyOMF5nNA8JlhKbBKe\nkcQyI2RJ2RZUOom7kg5gaYpam1oagx7IWKEG2yCWM7ryMuvKU2gVK7qKB7aG\nB1c0nDNUoQ5gaYvsMXe4Zd1kHk4x3GbZkpwIZCU0gX3s9Kkzu/urn3MJvTD0\ngtJzQ8p3Td0n",
  "Q5gaYvsMXe4Zd1kHk4x3GbZkpwIZCU0gX3s9Kkzu/urn3MJvTD0\ngtJzQ8p3Td0n9LUPJE4AWvDCVPJ/AYbugZpXuG7lFaGFpQOjB0QGlgaEDp\nY0MfU+ob6lO6Zugapbmh5I4UrgiG7lI6MXRC6YGhB5S+NvQ1pU8NfUrpoaGH\nlL4z9B2lDw19SCkzlFG6bug6pdxQ8urAC1YNXaXUM5Q8+8FeM7RPaWJoQuk\njQx9ROjaUPBXD9cxQcnsDF0Z",
  "bug6pdxQ8urAC1YNXaXUM5Q8+8FeM7RPaWJoQuk\njQx9ROjaUPBXD9cxQcnsDF0ZDJaXPDH1GqTCUPL95wQtDX1AaGRpR+tzQ5S\n+NfQtpU8MfUJpaCh5NwB3J4buUGreApUZpduGblN6auip/b0An0+jZ1uYW6a\nCLUpjQ2NKNwlTwpwK2HoCbmfDFR7Vpu9bSLntUDNuYW1GZ8dTXIeqDm3sPb\nsNDuanJ8CNecT0vX1vfmLFEgpnOlHC0sr+C",
  "9bSLntUDNuYW1GZ8dTXIeqDm3sPb\nsNDuanJ8CNecT0vX1vfmLFEgpnOlHC0sr+C0sLez90Fv5qXdv+97Sg9X2De\n0N52vnG+dbZ8X52XngPHX6zsDxnT+dv51/nH8XDxZ/Xfxt8fdGvX6tPeZLp/\nNZ/OM/1en1kw=</latexit>\nsa[xn] =\nN\nX\nm=1\na[xn, xm]vm\n21",
  "Dot-product self attention\n\u2022 Takes N inputs of size Dx1 and returns N inputs of size Dx1\n\u2022 Computes N values (no ReLU)\n\u2022 N outputs are weighted sums of these values\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"s3yg\npR0XPJtRZRZ8OK/g/OJAjcM=\">AWrniclZhbU9w2FICd9JamN\n9JOemLp0xmOm3CQCa9vHQmgZAbpCyBRKW7Mhe2asgy8aWlyWe\n/R/9NX1t/0L/TY9s7yo+RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1",
  "RzyUmQRxvs+6HEm27CTotBra/9eu/\n7Bhx9/MmNT29+9vkX361dOvrwyIt85D3w1Sm+XHACi6F4n0t\nOTHWc5ZEkh+FJxtGn404XkhUnWgLzN+mrBYiUiETENouHTPHwTR\nZFipmf+7KQZcs2E1mf0E5d2Ex/Ufg4TpcRBV05kRh0sra6tr9Y9\nPC+tYcVrf3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzM",
  "3rDW9+OBqM0LBOudChZUZysr2X6tGK5FqHks5uDsu\nAZC89YzE+gqFjCi9OqHtzMvw2RkR+lOfxT2q+j719RsaQoLpMAT\nNPNAjMTdLGTUke/nVZCZaXmKmwaikrp69Q3mfJHIuehlpdQYGEu\noK9+OGY5CzXk8+ZA8YswTRKmRtVgY2tvVg0CHgtV8fOyzu1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZ",
  "1s1n\nW2aodD8Spj49nBohaheSLecVJrZhKrhB4PKsqvhqvYiA4ALHKC\nUgVL6DOZor9dURhLUnAwIN0Cp2L/JczUrXSPIacdLTXRINCJvm0\nY20SC6Yy6Sj7oPj+bd8ArnOYBegq/OJoDvYzpmbz6zSf6jypChP\nDLeRMxbxuAoYcMmlG1DVUKSVcGnasP7D1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+C",
  "1kqmzNnFpVnc1NxFkHe\nRdR+c0L2rUdeoIsmARxl2rjiBLws4fsYRBltvyEAac+CbiVoXCq\niALs5enQbftzETw2pxmsF+63lZF0j9hKCMmALvP/BZMhbyrb6YL\n258nZ1L7psCn/hgmq3sJy+NmWPNGYFRtbEbNOlfIpNmCUJ5edE\n3TG4fKM9EdoAngTVfmQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9",
  "mQkXvaXfqEixZEx7cgaHmpeQnd1d/5tPTa\ns1sG/MfySZUVJSZqyIT/h8VjeBZg9cXRPDkpRJNHgTqyUsl3N/R\n1LEcL2wTqecOCkIxKfQl2v4iVt1r6gjubJqgvkLA1Au/mVBokqO\noK5uAkeE3PDUdCyhEgwybMYyLcqck5sfWs8QqXVzW8yFeVh1b6\njSCN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/f",
  "CN37BpeLq6AMD4cJv+LyAGU0aPIZpKUasRwlc2qmdPpmUGjY\nq7dX095U3RaMT/fbtuDfsHslGHIz4fbeD5iYlFHorgmOKsSxL\n0R7UtViu7/es2n7zI1nascN1m5LU2/bSbTvcK3rAz3cvd0hHr\nGoI1FdbQ+pRyxHe1CXO487rlE4XLcpSb3zPDpth7sw0fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVe",
  "fKPDsZwQ\njXHpFSOzLEvlYMmhEVNRe0U3PM7YpNCItJ2bXgb6zsC3h4dK0m\nhMVeIbqaCWBpxCUeQhPCYrOFu2Ybw+qOQ91xq0xmY2Q2ISw+YQk\nedRPCYkzF2CmesSxDYhMieRzjPI5pHjMsZS4Jz0jmBGypFwLKh\n+nXckEsDRFrU0djUEPZKpQg20QywVdeYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR",
  "deYVz5Sm0ihVdxX1Xw/0rG\ntYMVWgCWNole8wf7Do3WYBTDMcsV5IzgayMJrCHnR515qe/+vW\n0EtLym9sPSC0iNLjyjNLSVvBEH0lLydgLv65ZOKD209JDS0t\nKS0r6lfUojSyNKH1v6mNLQ0pDSTUs3KdWkhMpPBEsPaB0bOmY0\nmNLjyl9ZekrSp9a+pTS15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0of",
  "S15a+pvSdpe8ofWjpQ0qZpYzSLUu3KOW\nk8HQbRh6QalgaXk3Q/2mqU9SjNLM0ofWfqI0pGl5K0YnmeWkuM\nNPBgtlZQ+s/QZpcJS8v4WRC8sfUFpYmlC6XNLn1P61tK3lD6x9A\nmlsaXk2wCcTizdp9R+BaoKSvcs3aP03NJz93cBvpjGwLUwd20Fu\n5SmlqaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWv",
  "qaUbltK3hTgKGHpGTlPRq9q82/NpH7WqQW3MHajM+vJjm\nP1I7WHt3ml9N7k+RWvAx6frW4eJDCqQU7vTDpZV1/BWFg7vra\n7/snp/7/7Kg432C+0N7zve+8Hb9371XvgPfV6Xt8LvT+9v7y/v\nX+W15YPl0+Xh416/Vp7zTde52d5/B/AjePT</latexit>vn = \u03b2v + \u2326vxn\n22\nScalar self-attention weights that \nrepresent how much attention the nth \ntoken should pay to the mth token\n\ud835\udc4e +, \ud835\udc31\"  are non-negative and sum to \none",
  "Attention as routing\n23",
  "Attention as routing\n24\nLinear Transform\nHere:\n# of inputs, N = 3\nDimension of each input, D = 4\nWe\u2019ll show how to calculate the \nself-attention weights shortly.\nSums to 1",
  "Attention as routing\n25",
  "Attention as routing\n26",
  "Attention weights\n<latexit sha1_base64=\"8sOUSya8Dvq\nTUKF1XEfy96GK2B4=\">AW8HiclZhbU9tGFIBNekvojbRTXvqiKZNOp0Y6KSXl84\nkEHKDFAgYSDxrOSVvPFqJUsrMNH4f/St09f+o/bX9Kwke9E5y0M9k3g537e3sytpL\nT+VItdra/8s3Hjv/Q8+/OjmrcWP/n0s8+Xbn9xlCdFvBukMgkO/FZzqVQvKuFlvw\nkzTiLfcmP/dGm4cfnPMtFog71ZcrP",
  "s8+Xbn9xlCdFvBukMgkO/FZzqVQvKuFlvw\nkzTiLfcmP/dGm4cfnPMtFog71ZcrPYhYpEYqAaQj1l7TX8Nxv1RT79vfTNnmvXL8\nfQHKO/GPKr+6MVMD/2wnEyN2VOJKmKfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLD\neFY6zWevf/urQW+QBEXMlQ4ky/PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+V\nXqm3h2IDLwyeCf0l4VvVqjZHG",
  "PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+V\nXqm3h2IDLwyeCf0l4VvVqjZHGeX8Y+mGacOWYm6GKnhQ5/PSuFSgvNVB3FBbS04l\nncu0NRMYDLS+hwIJMwFi9YMgyFmhYkcWe4hdBEsdMDcrextb+tOz5PBKq5OiWp3p\ntO1sVQ6H4nXGxrPDeStC81i846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9Rp764\njCbpSAgfvJBAYXei+npG",
  "846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9Rp764\njCbpSAgfvJBAYXei+npGmleQ5aWmviQaFVPJy9okFixl3FIOQPG8O54BXGewCjBU\n+OJoDQ5SpqazepPdBaXuYnhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6\nFmJoKsw6zt6IzmRQ3aThVBFmzCqG1VEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZcl\nfrv1ETw3pykcL20",
  "aThVBFmzCqG1VEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZcl\nfrv1ETw3pykcL20va2SpP+coYyYAFx95lswFfC2vpnMbW+WnPKNwU+8YawWO0qLI\nvqac06gVk1sSk1q1whk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6a\nFZKf3lv9iU/OyjVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo\n5leGObSLV2UB",
  "jVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo\n5leGObSLV2UBCKSaEv0eUvItWuU0XwYJMYjRUCpl34ZkKhRQ7DtmwCRoZveO46NlC\nAJhnUcwxkhcZJzc/tJ8hUunmtpgJ87Bq31ClEdr3DS7ntaAMD4dzfk1H2XUr/PpJ\n4UasAwlc2KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/vbeD0iYlFHorbgoON\nsSxL0R+0",
  "KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/vbeD0iYlFHorbgoON\nsSxL0R+0Nd+uV0dWbr/5nmztyOG6TUnabUbpth3uNSPg4x3HaHeIRyzqSNRWM0LqE\ncvRH7TlzuOaxYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFTUTvFxJx\nz2IdwmJctC34GysHAh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ\n2TW",
  "wmJctC34GysHAh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ\n2TWISw+YTGedR3CYkTFyCmOWJoisQ6RPA5xHoc0jymWUpeEVyR1rAjZUq4NlQ2TtmQ\nCWJqg3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQBLO2Sa8zr7Tov\nMh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pK",
  "Mh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pKfp\n34bml5QeWXpEaWFpQWnX0i6loaUhpY8tfUxpYGlA6alm5RqS8mJFJ4Ilh5SOrR0\nSOmJpSeUvrL0FaVPLX1K6WtLX1P6ztJ3lD609CGlzFJG6ZalW5RyS8mrAz/csHSDUt\n9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Qcb+DBaKmk9JmlzygVlpLf",
  "HSDUt\n9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Qcb+DBaKmk9JmlzygVlpLfb374wtIX\nlMaWxpQ+t/Q5pW8tfUvpE0ufUBpZSt4NwOnE0gNK7VugMqd039J9SseWjt3vBfh8GX\n3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE7muhmnMHazI+q01yHqo5d7Dm\n7jSrTe5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/",
  "I+q01yHqo5d7Dm\n7jSrTe5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/v79lQcbzRvam52vO9\n90vusd37pPOg87ex1up2g8+9CZ+HWwuJytvzH8p/Lf9XqjYWmzped1mf57/8ASrz\n75w=</latexit>qn = \u03b2q + \u2326qxn\nkn = \u03b2k + \u2326kxn,\n\u2022 Compute N \u201cqueries\u201d and N \u201ckeys\u201d from input\n\u2022 Calculate similarity and pass through softmax:\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"yqSqjMw0sHj\ni+98SIpgQNH",
  "N \u201ckeys\u201d from input\n\u2022 Calculate similarity and pass through softmax:\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"yqSqjMw0sHj\ni+98SIpgQNHtaE=\">AXYHiclZhZb9w2EMfX6ZW6R5wWrYH2RaiRtihSwy7S4yVAYse57\nMR2fCbWxqC0lJYxRckSZa8t7Afta9EP0qGkXVozdNEuYC93fn8OyeHwkIJMikIvLf05c+O\n9z/48KObH89+8uln9+au/3FfpGWecj3wlSm+WHACi6F4ntaMkPs5yzJD8IDhZNf",
  "9z/48KObH89+8uln9+au/3FfpGWecj3wlSm+WHACi6F4ntaMkPs5yzJD8IDhZNfzgj\nOeFSNWuvsh4P2GxEpEImQbT8dzfHjvyE6aHQVSNxseVGt+9+jMZ973v73t+EqSjqkgjnbD\nG7Ese6aPWLpLxkR9EJzUIolPjpu/nIh7qvq9SVSYBz31/tnYV5SysfD7K/rOLceUXZQLk\nh/vL47fVy/G/1v6BVL97PLewtLhUfzxaWG4LC732s3V8+6u",
  "fD7K/rOLceUXZQLk\nh/vL47fVy/G/1v6BVL97PLewtLhUfzxaWG4LC732s3V8+6uBP0jDMuFKh5IVxdHyUqb7Fc\nu1CUfz/plwTMWnrCYH0FRsYQX/aqejrF3BywDL0pz+FPaq61Xa1QsKYqLJAClCXWBmTG6\n2FGpoz/6lVBZqbkKm4aiUno69czcegOR81DLCyiwMBfQVy8cMoi3hgyY9RU/D9MkYWpQ+S\ntr2xDWgMdCVfy0rLNhPO5q1moNh",
  "DLCyiwMBfQVy8cMoi3hgyY9RU/D9MkYWpQ+S\ntr2xDWgMdCVfy0rLNhPO5q1moNh+J1ipVnu1MvQvNEXHLipJYJ9cIeDyuKr4YL2IgOACx\nyAlIFS/AZ5Om3jKikP0ScNWkBiSD92pMXCvNY4hJR/aGyKCQST7qFaJCqYy6Uh2QOJ5d\nzwDuM5hFqCr8MXRHOxkTI0n9TQf6TypCmPDLeRMxbxuAoYcMmlG1FWoUkqoGnZUL7HqFVM\nnbeDSrO5q",
  "OxkTI0n9TQf6TypCmPDLeRMxbxuAoYcMmlG1FWoUkqoGnZUL7HqFVM\nnbeDSrO5qbixItZt3NTqncVGDrqa2IBUkYdxV1RakrBXDZjZViblYxhw4hmLWyoUlgqSm\nFt5GnTbzowF5+Yog/XS1a1VJPxnDEXEGD1mW/BVMi78tV0qvYmwTmr9abAR94QJqtbheV\nxM6xJIzCq1jamyjpWSEmjBaY8Pe8qTW8cUp6J7gCNAS+6MhcquiJrzglIWP27",
  "eV\nxM6xJIzCq1jamyjpWSEmjBaY8Pe8qTW8cUp6J7gCNAS+6MhcquiJrzglIWP278JQ81Lyo\n58Xf+WjfrVklo35R6IJjoyczky5v/haACnI84vsODJSyWaPDUk5dK2N/R1LEcJ7ax1H\nMHBaGYFPoCLX8Rq26d2oI7myaor2AwfuGbCYUmOYq6YmMwYviGc96RQCEaZNiMZRpUeac\nbH4on8FSy82mAtzWHU3VGkE3X2Dy2ktKMPhcMavqR6giA",
  "96RQCEaZNiMZRpUeac\nbH4on8FSy82mAtzWHU3VGkE3X2Dy2ktKMPhcMavqR6giAZNPIO0VAOWo2COzJSO3vqFhi\nXmWv31lDdFpyrmp+te9Avc3EIQ356vI7nIyYqpHIF1ysnL4kUTnaA1/TdL3as2r97U8k\ntWOH1q2UxG/bS7faob2mB/x0w9HbDaIjKqRyFfbQ6ojKkd74Msdxw3XKBxat1ISv5M4Ot\nUO7VSJ0j/aHXLNzDUplQNz7Uul35",
  "RyFfbQ6ojKkd74Msdxw3XKBxat1ISv5M4Ot\nUO7VSJ0j/aHXLNzDUplQNz7Uul35iwUFOhdgrThMdI2JiwMCm7KviNJTsCDo+uqjFh4VY\nhujJjwKIBl3gIjQkLmyXcVbY2LN1wSDfcUiazIVI2Jix8whI86saEhTEVxk7hCcsyJGxMJ\nI5DHMchjWOGRZlLhGckc8wISlXQuXDtCsyBiwaodZGjsagBzJVqMHWiMUFzbzCmXkKZbG\niWbznanjv",
  "hGckc8wISlXQuXDtCsyBiwaodZGjsagBzJVqMHWiMUFzbzCmXkKZbG\niWbznanjvmoY1Qw6NAYs2yRrz/E3nIgtwiOGa5QpyJpAqowHcwpotqpnc/upHdEIvL2g9\nNzSc0oPLD2gNLeUPBE0StLydNJEJ1ZekbpvqX7lJaWlpTuWbpHaWRpROljSx9TGloaUrp\nq6Sql2lJyI4UTwdJdSoeWDik9tPSQ0teWvqb0qaVPKX1j6RtKLy29pPShpQ8p",
  "rp\nq6Sql2lJyI4UTwdJdSoeWDik9tPSQ0teWvqb0qaVPKX1j6RtKLy29pPShpQ8pZYyStcs\nXaOUW0peHQTRiqUrlAaWkmc/WGuWblGaWZpR+sjSR5QOLCVPxXCeWUquN3AwWiopfWbpM0\nqFpeT5LYheWPqC0sTShNLnlj6n9J2l7yh9YukTSmNLybsBuJ1YukOpfQtUFZRuW7pN6aml\np+73Anw6jYErMTetg01KU0tTStctJU8KcJWw9ITcJy",
  "YukOpfQtUFZRuW7pN6aml\np+73Anw6jYErMTetg01KU0tTStctJU8KcJWw9ITcJyPV7mqTt01kX4vUlDtYG/FJbRLzSE\n25g7W706Q2Z8iNeVD0vW1/emLFAgp7PTHcwvL+C0sLez/srj82+K97XsLD1baN7Q3e9/2\nvuv92Fvu/d570Hva2+rt9cKZ5zPZzMXM5d/zd+cvzV/u5HemGnrfNnrfOa/+QfR6ipf<\n/latexit>a[xn,",
  "t9cKZ5zPZzMXM5d/zd+cvzV/u5HemGnrfNnrfOa/+QfR6ipf<\n/latexit>a[xn, xm] = softmaxm [sim[kmqn]]\n=\nexp [sim[kmqn]]\nPN\nm0=1 exp [sim[k0mqn]]\n,\n27",
  "Attention weights\n<latexit sha1_base64=\"8sOUSya8Dvq\nTUKF1XEfy96GK2B4=\">AW8HiclZhbU9tGFIBNekvojbRTXvqiKZNOp0Y6KSXl84\nkEHKDFAgYSDxrOSVvPFqJUsrMNH4f/St09f+o/bX9Kwke9E5y0M9k3g537e3sytpL\nT+VItdra/8s3Hjv/Q8+/OjmrcWP/n0s8+Xbn9xlCdFvBukMgkO/FZzqVQvKuFlvw\nkzTiLfcmP/dGm4cfnPMtFog71ZcrP",
  "s8+Xbn9xlCdFvBukMgkO/FZzqVQvKuFlvw\nkzTiLfcmP/dGm4cfnPMtFog71ZcrPYhYpEYqAaQj1l7TX8Nxv1RT79vfTNnmvXL8\nfQHKO/GPKr+6MVMD/2wnEyN2VOJKmKfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLD\neFY6zWevf/urQW+QBEXMlQ4ky/PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+V\nXqm3h2IDLwyeCf0l4VvVqjZHG",
  "PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+V\nXqm3h2IDLwyeCf0l4VvVqjZHGeX8Y+mGacOWYm6GKnhQ5/PSuFSgvNVB3FBbS04l\nncu0NRMYDLS+hwIJMwFi9YMgyFmhYkcWe4hdBEsdMDcrextb+tOz5PBKq5OiWp3p\ntO1sVQ6H4nXGxrPDeStC81i846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9Rp764\njCbpSAgfvJBAYXei+npG",
  "846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9Rp764\njCbpSAgfvJBAYXei+npGmleQ5aWmviQaFVPJy9okFixl3FIOQPG8O54BXGewCjBU\n+OJoDQ5SpqazepPdBaXuYnhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6\nFmJoKsw6zt6IzmRQ3aThVBFmzCqG1VEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZcl\nfrv1ETw3pykcL20",
  "aThVBFmzCqG1VEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZcl\nfrv1ETw3pykcL20va2SpP+coYyYAFx95lswFfC2vpnMbW+WnPKNwU+8YawWO0qLI\nvqac06gVk1sSk1q1whk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6a\nFZKf3lv9iU/OyjVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo\n5leGObSLV2UB",
  "jVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo\n5leGObSLV2UBCKSaEv0eUvItWuU0XwYJMYjRUCpl34ZkKhRQ7DtmwCRoZveO46NlC\nAJhnUcwxkhcZJzc/tJ8hUunmtpgJ87Bq31ClEdr3DS7ntaAMD4dzfk1H2XUr/PpJ\n4UasAwlc2KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/vbeD0iYlFHorbgoON\nsSxL0R+0",
  "KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/vbeD0iYlFHorbgoON\nsSxL0R+0Nd+uV0dWbr/5nmztyOG6TUnabUbpth3uNSPg4x3HaHeIRyzqSNRWM0LqE\ncvRH7TlzuOaxYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFTUTvFxJx\nz2IdwmJctC34GysHAh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ\n2TW",
  "wmJctC34GysHAh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ\n2TWISw+YTGedR3CYkTFyCmOWJoisQ6RPA5xHoc0jymWUpeEVyR1rAjZUq4NlQ2TtmQ\nCWJqg3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQBLO2Sa8zr7Tov\nMh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pK",
  "Mh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pKfp\n34bml5QeWXpEaWFpQWnX0i6loaUhpY8tfUxpYGlA6alm5RqS8mJFJ4Ilh5SOrR0\nSOmJpSeUvrL0FaVPLX1K6WtLX1P6ztJ3lD609CGlzFJG6ZalW5RyS8mrAz/csHSDUt\n9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Qcb+DBaKmk9JmlzygVlpLf",
  "HSDUt\n9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Qcb+DBaKmk9JmlzygVlpLfb374wtIX\nlMaWxpQ+t/Q5pW8tfUvpE0ufUBpZSt4NwOnE0gNK7VugMqd039J9SseWjt3vBfh8GX\n3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE7muhmnMHazI+q01yHqo5d7Dm\n7jSrTe5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/",
  "I+q01yHqo5d7Dm\n7jSrTe5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/v79lQcbzRvam52vO9\n90vusd37pPOg87ex1up2g8+9CZ+HWwuJytvzH8p/Lf9XqjYWmzped1mf57/8ASrz\n75w=</latexit>qn = \u03b2q + \u2326qxn\nkn = \u03b2k + \u2326kxn,\n\u2022 Compute N \u201cqueries\u201d and N \u201ckeys\u201d from input\n\u2022 Take dot products and pass through softmax:\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"/B2myYgAT9aiLO\n8UOYH7s",
  "N \u201ckeys\u201d from input\n\u2022 Take dot products and pass through softmax:\n\u2022 Weights depend on the inputs themselves\n<latexit sha1_base64=\"/B2myYgAT9aiLO\n8UOYH7sf0glDw=\">AXSHiclZhbT9xGFICX9JbSG2lV+tAXqyhNVaUIovTyEimBkBskLI\nEFEkzQ2Dv2ThiPvfYlj7B6v+gf6LvlV96xl7dwefM6jqSrCz5/vm4jPj8SXIpCj0ysqf\nc9fe/+Dz+6/vH8J59+9vkXCze+3C/SMg95L0xlmh8GrOBSKN7TQkt+mOWcJYH",
  "qf\nc9fe/+Dz+6/vH8J59+9vkXCze+3C/SMg95L0xlmh8GrOBSKN7TQkt+mOWcJYHkB8HpuE\nHZzwvRKr29EXGjxMWKxGJkGkInSz84bEjP2F6ETVaHxSqfHtyz+T8bH3/T3PT4J0VBVp\nBPWhH3JI3kB9Gp+fWm2htDeWjq+7mIB/rYV6kqk4Dnvj9fNxHlLKx8Psr+s+q48osyAXj\nr3irwF2NS61armjetd7KwtLK8Un8WlidFJY6k0/35MbXf",
  "Kx8Psr+s+q48osyAXj\nr3irwF2NS61armjetd7KwtLK8Un8WlidFJY6k0/35MbXfb+fhmXClQ4lK4qj1ZVMH1cs1y\nKUfDzvlwXPWHjKYn4ERcUSXhxXdrH3k2I9L0ozeFPa+OXq5RsaQoLpIATJPSAjMTdLGj\nUke/HVdCZaXmKmw6ikrp6dQzc+j1Rc5DLS+gwMJcwFi9cMAgvxpmet5X/DxMk4SpfuWvbex\nAOgMeC1XxYVnP+njcdjZqh0Px",
  "S+gwMJcwFi9cMAgvxpmet5X/DxMk4SpfuWvbex\nAOgMeC1XxYVnP+njcdjZqh0PxKmPt6d6sFaF5It5x0kitmEauEHg8riq+HC9jIDgAscwJS\nBUvoM1mOXqriMIql4CrZm3CWvBejknTSvMYctLSXhMNCpnko5a1TiyYyqSl7ILieTc9A7j\nOYRZgqPDF0RzsZkyNp/U0H+k8qQoTwz3kTMW87gIOWTSHFHbUKWUDVsWS+w9ZKp0ni0q\nweam4i",
  "zsZkyNp/U0H+k8qQoTwz3kTMW87gIOWTSHFHbUKWUDVsWS+w9ZKp0ni0q\nweam4iyNrL247OaV5Uv+3UEWTBIozbVh1BloQ9qc/M9jEtn8ABJ56JuFWhsCrIwuzmadDu\nOzMRvDZHGZwvbW+jIuk/YygjJgBn/kWTIW8ra+nM9ubJues9k2Bj7wBTFa7Csvj5rCmnc\nBRTWJjata5QibNFoTy9LxtmtE4VJ6J9gGaAD7pylyo6JLWXA9gyZqwfxs",
  "5rCmnc\nBRTWJjata5QibNFoTy9LxtmtE4VJ6J9gGaAD7pylyo6JLWXA9gyZqwfxsONS8lP/p+Wc+O\nq5WzGlj/pFsQkNFmbkaMuH/0VAfroJ4fUET14q0eRBoJ68VML+jqaO5Xhm0g9d1AQikm\nhL9DpL2LVrlNH8GDTBI0VAqZd+GZCoUmOorZsAkaGb7ieOxZQiA4ybI4xlGlR5pxsfmg9Q\n6TWzbaYC3Oxam+o0gjtfYPLWS0ow8XhjF9RPUA",
  "OxZQiA4ybI4xlGlR5pxsfmg9Q\n6TWzbaYC3Oxam+o0gjtfYPLWS0ow8XhjF9RPUAZDZp8Bmp+ixHyRyZKR298QsNp5jr7K+n\nvCk6rZgPNyf9wbjMDUMY8uHJp6PmFjUkagtuIFytiWJ5egP2pot18sjqzbf/EiWduxw3a\nYk7U5G6bYd7hUj4Mtx2i3iEcs6kjU1mSE1COWoz9oy53HLdROFy3KUm70zw6bYc7M9Hyj\n/YGXDNzm5TKvrntS6Xf",
  "6kjU1mSE1COWoz9oy53HLdROFy3KUm70zw6bYc7M9Hyj\n/YGXDNzm5TKvrntS6XfhLCoqaidYprwGIlNCItJ2bgN1Z2BVw82lYTwmK3EG3NBLDU5xI\nfQhPCYnMKt81JDKtbDnXLrTKZDZDZhLD4mCX4qJsQFmMqxk7xlGUZEpsQyeMA53FA85hK\nXNJeEYyx4yQJeVaUPkgbUsmgKUR6m3k6AxGIFOFOpwEsVzQlVc4V5Cq1jRVdxzdy7omPN\nU",
  "x4yQJeVaUPkgbUsmgKUR6m3k6AxGIFOFOpwEsVzQlVc4V5Cq1jRVdxzdy7omPN\nUIMmgKVtco5/rbzJAtwiuE2y5XkTCArownsYqdLnendX/0oTuiFpReUnlt6TumBpQeU5p\naSJ4IgemkpeToJojNLzyjdt3Sf0tLSktKepT1KI0sjSh9Z+ojS0NKQ0nVL1ynVlpI7Urgi\nWLpH6cDSAaWHlh5S+srSV5Q+sfQJpa8tfU3pO0vfUfrA0geUMksZ",
  "ynVlpI7Urgi\nWLpH6cDSAaWHlh5S+srSV5Q+sfQJpa8tfU3pO0vfUfrA0geUMksZpRuWblDKLSWvDoJozdI\n1SgNLybMfnGuWdinNLM0ofWjpQ0r7lpKnYrieWUpub+DCaKmk9KmlTykVlpLntyB6bulzS\nhNLE0qfWfqM0reWvqX0saWPKY0tJe8G4O7E0l1K7VugqB0x9IdSoeWDt3vBfhsGgPXwty\n2DWxTmlqaUrpKXlSgFsJS0/J/WSkJrv",
  "1K7VugqB0x9IdSoeWDt3vBfhsGgPXwty\n2DWxTmlqaUrpKXlSgFsJS0/J/WSkJrva9G0T2dciNeMONsn4tDbJeaRm3MEmu9O0NtmfIj\n0ul2ep1w7s7c4RybCxZ/X/xr8e/Ffxr12tykzled1ueba/8CaLoiPQ=</latexit>XjAzL0jf3ZixRIKez0JwtLq/gtLC3s31le/WX57s7dpftrkze01zvfdr7r/NBZ7fzaud95\na[xn,",
  "RIKez0JwtLq/gtLC3s31le/WX57s7dpftrkze01zvfdr7r/NBZ7fzaud95\na[xn, xm] = softmaxm\n\u21e5\nkT\nmqn\n\u21e4\n=\nexp\n\u21e5\nkT\nmqn\n\u21e4\nPN\nm0=1 exp\n\u21e5\nkT\nm0qn\n\u21e4\n28",
  "Dot product = measure of similarity\n29\nA drawback of the dot product as similarity measure is the magnitude of each \nvector influences the value. More rigorous to divide by magnitudes.\n\ud835\udc31!\ud835\udc32 = \ud835\udc31 \ud835\udc32 cos(\ud835\udf03)\nCosine Similarity:  \n\ud835\udc31!\ud835\udc32\n\ud835\udc31 \ud835\udc32 = cos(\ud835\udf03)",
  "Conclusions:\n\u00fc We need a model where parameters don\u2019t increase with input length, e.g.\n\u00fc There must be connections between the words.\n\u00fc The strength of these connections will depend on the words themselves.\nMotivation\nDesign neural network to encode and process text:\n<latexit sha1_base64=\"Xftsz9\n71LUeLx6nHNLh+Mc6BQg=\">AW0XiclZhb9s2FIDV7tZ1t3TD8rIXYU\nGBYeiMZOguLwPapOkt6eI0cZImTg1KpmTWFCXrktgVDAx73U/aL9njXrc\n/sUNJNqNzmIcZaM2c7xMvh6REy0ukyPL19b9u3Hzn3fe/+DWh7c/+viT\nTz9bufP5",
  "sUNJNqNzmIcZaM2c7xMvh6REy0ukyPL19b9u3Hzn3fe/+DWh7c/+viT\nTz9bufP5URYXqc97fizj9MRjGZdC8V4ucslPkpSzyJP82BtvaX58wdNMx\nOownyX8PGKhEoHwWQ6hwUqv7wXJSPzSL6Hg8ZwNyov5PSjvRTxc/lGDyVU\nwuQLGV8F43p8PVtbWO+vVx6WFjaw5jSf7uDOl8P+MPaLiKvclyzLzjbW\nk/y8ZGkufMnt/tFxhPmj1nIz6C",
  "vVx6WFjaw5jSf7uDOl8P+MPaLiKvclyzLzjbW\nk/y8ZGkufMnt/tFxhPmj1nIz6CoWMSz87Ia/9y9C5GhG8Qp/FO5W0WvX\nlGyKMtmkQdmxPJRhpkO2thZkQc/n5dCJUXOlV83FBTSzWNXJ9MdipT7uZ\nxBgfmpgL6/oilzM8h5bf7il/6cRQxNSz7m9v7c8gxD4Uq+aSo0j+ft53\ntyuFQvM7YfHa4rEXkPBJvOamkUnQl1wg8nJcl74QdDAQHI",
  "gxD4Uq+aSo0j+ft53\ntyuFQvM7YfHa4rEXkPBJvOamkUnQl1wg8nJcl74QdDAQHIDqcgFjxDOrU\n+fECdwNRWG4SMHAvnupl5L6ck6pVzkPISUs7JRoUEsmnLWuLWDCVUs5A\nMV17oa8DyFWYCuwhdHc3CQMDVfXJfzaZ5GZaZjuIWUqZBXTcCQfSb1iN\nqGKqSES/2W9Su2XjI1bhIXJ1VXUx1B1mHadvKU5kUN204VQRYswrBtVRF\nkSbg5DFn",
  "KqSES/2W9Su2XjI1bhIXJ1VXUx1B1mHadvKU5kUN204VQRYswrBtVRF\nkSbg5DFnEIMtNeQADjlwdsatCYVWQhdlNY6/dqIjeG1OE9gvbW+7JOm/Y\nCgjOgC7T38Lpnze1rfipe0uknNR+brAp+4IJqt9CUvDeliLRmBUTWxOzS\npXyKTZglAaX7ZN3RuLyhPRHqAO4E1XpEIFV7R7VQmWrA7378FQ0Lys+8\n6P/Dpebmut43+j2QTKsqKxFaRDv",
  "PRHqAO4E1XpEIFV7R7VQmWrA7378FQ0Lys+8\n6P/Dpebmut43+j2QTKsqKxFaRDv+PiobwOMLrCyJ48mKJg8C1eTFEu7v\naOpYihe2jlRzBwWhmBT5DG1/Ear2NVUEdzaOUF8hoOuFbyYUmuQgaMs6o\nGX4hgerZQH5aJB+PUZfxlmRcnLzQ+sZIpWub4up0A+r9g1VaqF93+ByeR\nWU4eFwa+53EMZ9ep8enGhixFyZzqKZ2+7mc5bDHb7q+mvC",
  "A+r9g1VaqF93+ByeR\nWU4eFwa+53EMZ9ep8enGhixFyZzqKZ2+7mc5bDHb7q+mvC5arZBPdpr\n2oF8wO4Xv8lgB89HSCzqSFQXnGSsdUliWdqDupbL9WrPyp3X35KlHVpc\nuylJvU0v7bFvaYHfLJr6e0u8YhFHYnqanpIPWJZ2oO67HnctY3C4tpNS\nepd5NFqW9yliZ/cDiCU6k+JsVyqI9sezXISzmVMytYqyPtm2xDmExKto\nW/I2VAwEPj7",
  "NFqW9yliZ/cDiCU6k+JsVyqI9sezXISzmVMytYqyPtm2xDmExKto\nW/I2VAwEPj7ZVh7DYzURb0wEsDbnEQ6hDWKy3cNtsYljdtai7dpXJZITM\nOoTFJyzCo65DWAypGFrFMUsSJNYhkscRzuOI5jHBUmKT8IwklhkhS8q2o\nNJR3JZ0AEtT1NrU0hj0QMYKNdgEsZzRlZdZV5Cq1jRVdyzNdy7puGcoQ\np1AEt7ZI+5/T3rJvNwiuGYZUtyIpCV",
  "NdgEsZzRlZdZV5Cq1jRVdyzNdy7puGcoQ\np1AEt7ZI+5/T3rJvNwiuGYZUtyIpCV0AR2sdOlzuL05wUlOcl5wczQGaW\nXhl5SemzoMaWpoeQXgRe8NJT8OvGC0MvKD0y9IjSwtC0p6hPUoDQwNK\nHxv6mFLfUJ/SLUO3KM0NJSdSeCIYekjpyNARpSeGnlD6ytBXlD419Cmlp\n4aeUvrW0LeUPjT0IaXMUEbptqHblHJDyasDL9g0dJNSz1Dy2w/",
  "ytBXlD419Cmlp\n4aeUvrW0LeUPjT0IaXMUEbptqHblHJDyasDL9g0dJNSz1Dy2w/2mqFdSh\nNDE0ofGfqI0qGh5FcxPM8MJcbeDAaKil9ZugzSoWh5PebF7w9AWlkaE\nRpc8NfU7pG0PfUPrE0CeUhoaSdwNwOjH0gFLzFqjMKN03dJ/SiaET+3sBv\npxGz7Yw90wFe5TGhsaU7hKfinAUcLQMTlPBq5qy3eNpH7WqCW3MKajC\n+uJjkP1JbWHN3",
  "Yw90wFe5TGhsaU7hKfinAUcLQMTlPBq5qy3eNpH7WqCW3MKajC\n+uJjkP1JbWHN3WlxN7k+BWvIR6fr20fJFCqQU7vSDlbUN/BaWFo6+72z\n82Lm/f3/twWbzhvaW85XztfONs+H85Dxwnjpdp+f4zp/O384/zr+rB6uz\n1d9Wf6/Vmzea75wWp/VP/4D8ijy4w=</latexit>\u03c6 = {\u03b2v, \u2326v, \u03b2q, \u2326q, \u03b2k, \u2326k}\n30",
  "Ok, we defined queries, keys and values, but how \nare they used?\n31",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n32",
  "Computing Attention Weights\n33\n<latexit sha1_base64=\"/B2myYgAT9aiLO8UOYH7sf0glDw=\">AXSHiclZhbT9xGFIC\nX9JbSG2lV+tAXqyhNVaUIovTyEimBkBskLIEFEkzQ2Dv2ThiPvfYlj7B6v+gf6LvlV96xl7dwefM6jqSrCz5/vm4jPj8SXIpCj0ysqfc9fe/+Dz+6/vH8J59+9vkXCze+3C/SMg95\nL0xlmh8GrOBSKN7TQkt+mOWcJYHkB8HpuEHZzwvRKr29E",
  "9+9vkXCze+3C/SMg95\nL0xlmh8GrOBSKN7TQkt+mOWcJYHkB8HpuEHZzwvRKr29EXGjxMWKxGJkGkInSz84bEjP2F6ETVaHxSqfHtyz+T8bH3/T3PT4J0VBVpBPWhH3JI3kB9Gp+fWm2htDeWjq+7mIB/rY\nV6kqk4Dnvj9fNxHlLKx8Psr+s+q48osyAXjr3irwF2NS61armjetd7KwtLK8Un8WlidFJY6k0/35MbXfb+fhmXClQ4lK4qj1ZVMH",
  "3irwF2NS61armjetd7KwtLK8Un8WlidFJY6k0/35MbXfb+fhmXClQ4lK4qj1ZVMH1cs1yKUfDzvlwXPWHjKYn4ERcUSXhxXdrH3k2I9\nL0ozeFPa+OXq5RsaQoLpIATJPSAjMTdLGjUke/HVdCZaXmKmw6ikrp6dQzc+j1Rc5DLS+gwMJcwFi9cMAgvxpmet5X/DxMk4SpfuWvbexAOgMeC1XxYVnP+njcdjZqh0PxKmPt6d6sFa\nF5It5x0kitm",
  "5X/DxMk4SpfuWvbexAOgMeC1XxYVnP+njcdjZqh0PxKmPt6d6sFa\nF5It5x0kitmEauEHg8riq+HC9jIDgAscwJSBUvoM1mOXqriMIql4CrZm3CWvBejknTSvMYctLSXhMNCpnko5a1TiyYyqSl7ILieTc9A7jOYRZgqPDF0RzsZkyNp/U0H+k8qQoTwz3kTM\nW87gIOWTSHFHbUKWUDVsWS+w9ZKp0ni0qweam4iyNrL247OaV5Uv+3UEWTBIoz",
  "M\nW87gIOWTSHFHbUKWUDVsWS+w9ZKp0ni0qweam4iyNrL247OaV5Uv+3UEWTBIozbVh1BloQ9qc/M9jEtn8ABJ56JuFWhsCrIwuzmadDuOzMRvDZHGZwvbW+jIuk/YygjJgBn/kWTIW\n8ra+nM9ubJues9k2Bj7wBTFa7Csvj5rCmncBRTWJjata5QibNFoTy9LxtmtE4VJ6J9gGaAD7pylyo6JLWXA9gyZqwfxsONS8lP/p+Wc+Oq5WzGlj/pFs",
  "y9LxtmtE4VJ6J9gGaAD7pylyo6JLWXA9gyZqwfxsONS8lP/p+Wc+Oq5WzGlj/pFsQkNFmbkaMuH/0VAfroJ4fUE\nT14q0eRBoJ68VML+jqaO5Xhm0g9d1AQikmhL9DpL2LVrlNH8GDTBI0VAqZd+GZCoUmOorZsAkaGb7ieOxZQiA4ybI4xlGlR5pxsfmg9Q6TWzbaYC3Oxam+o0gjtfYPLWS0ow8XhjF9RP\nUAZDZp8Bmp+ixHyRyZKR298QsNp",
  "6TWzbaYC3Oxam+o0gjtfYPLWS0ow8XhjF9RP\nUAZDZp8Bmp+ixHyRyZKR298QsNp5jr7K+nvCk6rZgPNyf9wbjMDUMY8uHJp6PmFjUkagtuIFytiWJ5egP2pot18sjqzbf/EiWduxw3aYk7U5G6bYd7hUj4Mtx2i3iEcs6kjU1mSE1\nCOWoz9oy53HLdROFy3KUm70zw6bYc7M9Hyj/YGXDNzm5TKvrntS6XfhLCoqaidYprwGIlNCItJ2bgN1Z2",
  "3KUm70zw6bYc7M9Hyj/YGXDNzm5TKvrntS6XfhLCoqaidYprwGIlNCItJ2bgN1Z2BVw82lYTwmK3EG3NBLDU5xIfQhPCYnMKt81JDKtbDnXLrTKZDZDZhLD4mC\nX4qJsQFmMqxk7xlGUZEpsQyeMA53FA85hKXNJeEYyx4yQJeVaUPkgbUsmgKUR6m3k6AxGIFOFOpwEsVzQlVc4V5Cq1jRVdxzdy7omPNUIMmgKVtco5/rbzJAtwiuE2y5XkTCA",
  "IFOFOpwEsVzQlVc4V5Cq1jRVdxzdy7omPNUIMmgKVtco5/rbzJAtwiuE2y5XkTCArown\nsYqdLnendX/0oTuiFpReUnlt6TumBpQeU5paSJ4IgemkpeToJojNLzyjdt3Sf0tLSktKepT1KI0sjSh9Z+ojS0NKQ0nVL1ynVlpI7UrgiWLpH6cDSAaWHlh5S+srSV5Q+sfQJpa8tfU3\npO0vfUfrA0geUMksZpRuWblDKLSWvDoJozdI1SgNLybMfn",
  "rSV5Q+sfQJpa8tfU3\npO0vfUfrA0geUMksZpRuWblDKLSWvDoJozdI1SgNLybMfnGuWdinNLM0ofWjpQ0r7lpKnYrieWUpub+DCaKmk9KmlTykVlpLntyB6bulzShNLE0qfWfqM0reWvqX0saWPKY0tJe8G4O7E\nWX57s7dpftrkze01zvfdr7r/NBZ7fzaud950ul2ep1w7s7c4RybCxZ/X/xr8e/Ffxr12tykzled1ueba/8CaLoiPQ=</late",
  "d950ul2ep1w7s7c4RybCxZ/X/xr8e/Ffxr12tykzled1ueba/8CaLoiPQ=</latexit>0l1K7VugqB0x9IdSoeWDt3vBfhsGgPXwty2DWxTmlqaUrpKXlSgFsJS0/J/WSkJrva9G0T2dciNeMONsn4tDbJeaRm3MEmu9O0NtmfIjXjAzL0jf3ZixRIKez0JwtLq/gtLC3s31le/\na[xn, xm] = softmaxm\n\u21e5\nkT\nmqn\n\u21e4\n=\nexp\n\u21e5\nkT\nmqn\n\u21e4\nPN\nm0=1",
  "tLq/gtLC3s31le/\na[xn, xm] = softmaxm\n\u21e5\nkT\nmqn\n\u21e4\n=\nexp\n\u21e5\nkT\nmqn\n\u21e4\nPN\nm0=1 exp\n\u21e5\nkT\nm0qn\n\u21e4\n<latexit sha1_base64=\"8sOUSya8DvqTUKF1XEfy\n96GK2B4=\">AW8HiclZhbU9tGFIBNekvojbRTXvqiKZNOp0Y6KSXl84kEHKDFAgYSDxrOSVvPFqJUsrM\nNH4f/St09f+o/bX9Kwke9E5y0M9k3g537e3sytpLT+VItdra/8s3Hjv/Q",
  "qJUsrM\nNH4f/St09f+o/bX9Kwke9E5y0M9k3g537e3sytpLT+VItdra/8s3Hjv/Q8+/OjmrcWP/n0s8+Xbn9xlCdF\nFvBukMgkO/FZzqVQvKuFlvwkzTiLfcmP/dGm4cfnPMtFog71ZcrPYhYpEYqAaQj1l7TX8Nxv1RT79vfTNn\nnmvXL8fQHKO/GPKr+6MVMD/2wnEyN2VOJKmKfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLDeFY6zWevf/\nu",
  "VMD/2wnEyN2VOJKmKfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLDeFY6zWevf/\nurQW+QBEXMlQ4ky/PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+VXqm3h2IDLwyeCf0l4VvVqjZHGeX\n8Y+mGacOWYm6GKnhQ5/PSuFSgvNVB3FBbS04lncu0NRMYDLS+hwIJMwFi9YMgyFmhYkcWe4hdBEsdMDcr\nextb+tOz5PBKq5OiWp3ptO1sVQ6H4",
  "S+hwIJMwFi9YMgyFmhYkcWe4hdBEsdMDcr\nextb+tOz5PBKq5OiWp3ptO1sVQ6H4nXGxrPDeStC81i846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9R\np764jCbpSAgfvJBAYXei+npGmleQ5aWmviQaFVPJy9okFixl3FIOQPG8O54BXGewCjBU+OJoDQ5Spqaze\npPdBaXuYnhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6FmJo",
  "5Spqaze\npPdBaXuYnhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6FmJoKsw6zt6IzmRQ3aThVBFmzCqG1V\nEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZclfrv1ETw3pykcL20va2SpP+coYyYAFx95lswFfC2vpnMbW+\nWnPKNwU+8YawWO0qLIvqac06gVk1sSk1q1whk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6\na",
  "qac06gVk1sSk1q1whk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6\naFZKf3lv9iU/OyjVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo5leGObSLV2UBCKS\naEv0eUvItWuU0XwYJMYjRUCpl34ZkKhRQ7DtmwCRoZveO46NlCAJhnUcwxkhcZJzc/tJ8hUunmtpgJ87Bq\n31ClEdr3DS7ntaAMD4dzfk1H2XUr/",
  "lCAJhnUcwxkhcZJzc/tJ8hUunmtpgJ87Bq\n31ClEdr3DS7ntaAMD4dzfk1H2XUr/PpJ4UasAwlc2KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/v\nbeD0iYlFHorbgoONsSxL0R+0Nd+uV0dWbr/5nmztyOG6TUnabUbpth3uNSPg4x3HaHeIRyzqSNRWM0LqEc\nvRH7TlzuOaxYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFT",
  "M0LqEc\nvRH7TlzuOaxYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFTUTvFxJxz2IdwmJctC34GysHA\nh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ2TWISw+YTGedR3CYkTFyCmOWJoisQ6RPA5x\nHoc0jymWUpeEVyR1rAjZUq4NlQ2TtmQCWJqg3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQ\nB",
  "Uq4NlQ2TtmQCWJqg3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQ\nBLO2Sa8zr7TovMh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pKfp34\nbml5QeWXpEaWFpQWnX0i6loaUhpY8tfUxpYGlA6alm5RqS8mJFJ4Ilh5SOrR0SOmJpSeUvrL0FaVPLX1K\n6WtLX1P6ztJ3lD609CGlzFJG6ZalW5",
  "JFJ4Ilh5SOrR0SOmJpSeUvrL0FaVPLX1K\n6WtLX1P6ztJ3lD609CGlzFJG6ZalW5RyS8mrAz/csHSDUt9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Q\ncb+DBaKmk9JmlzygVlpLfb374wtIXlMaWxpQ+t/Q5pW8tfUvpE0ufUBpZSt4NwOnE0gNK7VugMqd039J9Ss\neWjt3vBfh8GX3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE",
  "qd039J9Ss\neWjt3vBfh8GX3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE7muhmnMHazI+q01yHqo5d7Dm7jSrT\ne5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/v79lQcbzRvam52vO90vusd37pPOg87ex1up2g\n8+9CZ+HWwuJytvzH8p/Lf9XqjYWmzped1mf57/8ASrz75w=</latexit>qn = \u03b2q + \u2326qxn\nkn = \u03b2k + \u2326kxn,",
  "8p/Lf9XqjYWmzped1mf57/8ASrz75w=</latexit>qn = \u03b2q + \u2326qxn\nkn = \u03b2k + \u2326kxn,\n<latexit sha1_base64=\"8sOUSya8DvqTUKF1XEfy\n96GK2B4=\">AW8HiclZhbU9tGFIBNekvojbRTXvqiKZNOp0Y6KSXl84kEHKDFAgYSDxrOSVvPFqJUsrM\nNH4f/St09f+o/bX9Kwke9E5y0M9k3g537e3sytpLT+VItdra/8s3Hjv/Q8+/OjmrcWP/n0s8+",
  "f+o/bX9Kwke9E5y0M9k3g537e3sytpLT+VItdra/8s3Hjv/Q8+/OjmrcWP/n0s8+Xbn9xlCdF\nFvBukMgkO/FZzqVQvKuFlvwkzTiLfcmP/dGm4cfnPMtFog71ZcrPYhYpEYqAaQj1l7TX8Nxv1RT79vfTNn\nnmvXL8fQHKO/GPKr+6MVMD/2wnEyN2VOJKmKfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLDeFY6zWevf/\nurQW+QBEXMlQ4ky/P",
  "KfZ16vt2jqjEj90dX6I1T/bn9pZW1rfp4tLDeFY6zWevf/\nurQW+QBEXMlQ4ky/PT9bVUn5Us0yKQfLrYK3KesmDEIn4KRcVinp+VXqm3h2IDLwyeCf0l4VvVqjZHGeX\n8Y+mGacOWYm6GKnhQ5/PSuFSgvNVB3FBbS04lncu0NRMYDLS+hwIJMwFi9YMgyFmhYkcWe4hdBEsdMDcr\nextb+tOz5PBKq5OiWp3ptO1sVQ6H4nXGxrPDeStC81i84",
  "mhYkcWe4hdBEsdMDcr\nextb+tOz5PBKq5OiWp3ptO1sVQ6H4nXGxrPDeStC81i846SRSjGNXCPwaFqWfDVaxUBwAGKVE5AonkOb9R\np764jCbpSAgfvJBAYXei+npGmleQ5aWmviQaFVPJy9okFixl3FIOQPG8O54BXGewCjBU+OJoDQ5Spqaze\npPdBaXuYnhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6FmJoKsw6zt6IzmRQ3aTh",
  "nhHjKmIl51AVMOmDQzahuqkBKqBi3rd2y9ZGrUJC5Jq6FmJoKsw6zt6IzmRQ3aThVBFmzCqG1V\nEWRJuHcMWMwgy025DxOPRNxq0JhVZCNuZclfrv1ETw3pykcL20va2SpP+coYyYAFx95lswFfC2vpnMbW+\nWnPKNwU+8YawWO0qLIvqac06gVk1sSk1q1whk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6\naFZKf3lv9iU/OyjVz",
  "hk2YLQly0TbNaBwqT0V7giaAL7oiEyq8ot2tSrBlTbh3F6\naFZKf3lv9iU/OyjVz2Zj/SDahobxIXQ2Z8P9oaABPK7y/IXL5Fo8SBQLV4i4f6Olo5leGObSLV2UBCKS\naEv0eUvItWuU0XwYJMYjRUCpl34ZkKhRQ7DtmwCRoZveO46NlCAJhnUcwxkhcZJzc/tJ8hUunmtpgJ87Bq\n31ClEdr3DS7ntaAMD4dzfk1H2XUr/PpJ4UasAwlc2KWdP",
  "c/tJ8hUunmtpgJ87Bq\n31ClEdr3DS7ntaAMD4dzfk1H2XUr/PpJ4UasAwlc2KWdPKml2u4xFxXf7XkdFpRXy83fQH4LVKYKAj/v\nbeD0iYlFHorbgoONsSxL0R+0Nd+uV0dWbr/5nmztyOG6TUnabUbpth3uNSPg4x3HaHeIRyzqSNRWM0LqEc\nvRH7TlzuOaxYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFTUTvFxJxz2IdwmJct",
  "axYO121K0u4sj07b4c5NtP3DwyEcUc0xKZEDc+xLZK8OYVFTUTvFxJxz2IdwmJctC34GysHA\nh4ebasOYXEvF23NBLA04BJPoQ5hsb6E2YTw+qOQ91xq0ymQ2TWISw+YTGedR3CYkTFyCmOWJoisQ6RPA5x\nHoc0jymWUpeEVyR1rAjZUq4NlQ2TtmQCWJqg3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQ\nBLO2Sa8zr7TovMh+n",
  "3iaOzmAEMlGowyaI5ZzuvNy58xTaxYru4q6r4+41HWuGjQ\nBLO2Sa8zr7TovMh+nGI5ZriSnAlkpTeAedvaoMzv9Vb9vCb209JLSC0svKD29JjSzFLyi8APX1pKfp34\nbml5QeWXpEaWFpQWnX0i6loaUhpY8tfUxpYGlA6alm5RqS8mJFJ4Ilh5SOrR0SOmJpSeUvrL0FaVPLX1K\n6WtLX1P6ztJ3lD609CGlzFJG6ZalW5RyS8mrAz/csHSDUt",
  "JpSeUvrL0FaVPLX1K\n6WtLX1P6ztJ3lD609CGlzFJG6ZalW5RyS8mrAz/csHSDUt9S8tsPrjVL9yhNLU0pfWTpI0oHlpJfxfA8s5Q\ncb+DBaKmk9JmlzygVlpLfb374wtIXlMaWxpQ+t/Q5pW8tfUvpE0ufUBpZSt4NwOnE0gNK7VugMqd039J9Ss\neWjt3vBfh8GX3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE7muhmnMHazI+q01y",
  "Bfh8GX3Xxty1DexSmliaULptKfmlAEcJS0fkPBmq5q42e9tE7muhmnMHazI+q01yHqo5d7Dm7jSrT\ne5PoZrzIRn61tH8RQqkFO70/aWVdfwWlhaOflxd/3n1/v79lQcbzRvam52vO90vusd37pPOg87ex1up2g\n8+9CZ+HWwuJytvzH8p/Lf9XqjYWmzped1mf57/8ASrz75w=</latexit>qn = \u03b2q + \u2326qxn\nkn = \u03b2k + \u2326kxn,",
  "This work is subject to a Creative Commons CC-BY-NC-ND license. (C) MIT Press.\nComputing Values and Self-Attention Outputs as Sparse Matrix Ops\n34",
  "\u2022 Store N input vectors in matrix X \n\u2022 Compute values, queries and keys:\n\u2022\nCombine self-attentions\nFrom Input Vector to Input Matrix\n<latexit sha1_base64=\"mvsFdVImIWGd2Vp+GA\n43VQ3KtSY=\">AXLXiclZhb9s2FICdXbvslm5Y9rAXYUGHYeuMeOguLwXapGnaJl2cJk7cxqlByZ\nTMhqJkXRyngn/TsB+zhwHDXvc3dijJZnUO8zADjdnzfbwdUhItN5YizTY3/1p56+13n3v/RsfrH740\ncefLp287OTNMoTj/e8SEZJ32Upl0LxXiY",
  "ItN5YizTY3/1p56+13n3v/RsfrH740\ncefLp287OTNMoTj/e8SEZJ32Upl0LxXiYyftxwlnoSn7qXmxrfjrlSoidZxdxfw8ZIESvBYBqH\nh2h8D1z85gz/9c+ebuw4UXJ6xYTGdQ7Hzsjiefw+Fg5AHi2B/oCKVhy5PnMFgFQKHluoTW/WJtfqep\nfqFrXoZ7N8erm1stjfLj0MLnbqw0ao/3eHNL0aDUeTlIVeZJ1manU24+y8YEkmPMnq4M85T",
  "7N8erm1stjfLj0MLnbqw0ao/3eHNL0aDUeTlIVeZJ1manU24+y8YEkmPMnq4M85THzLljA\nz6CoWMjT86LM7dy5BZGR40cJ/FOZU0bfrFGwME2vQhfMkGXjFDMdtLGzPN/PS+EivOMK6/qyM+lk0\nWOXihnJBLuZfIKCsxLBIzV8cYsYV4Gy7k6UPzSi8KQqVEx2No5nBcDlwdCFXySl0s7nzedndLhULzO2\nHp8vGxFZDwUrzlpFR0I9cIPJgXB",
  "VEx2No5nBcDlwdCFXySl0s7nzedndLhULzO2\nHp8vGxFZDwUrzlpFR0I9cIPJgXBW8HbQwEByDanIBI8RTa1PlxfaeDKGxlCRi4G81gcL7zbE6aVhk\nPICcN7QXRoBLPmtY28SCpQwbyhEojnPL0YBnCawCDBW+OFqDo5ip+aJexmdZEhapjuEeEqYCXnYBU/\naY1DNqGiqXEqp6Des3bD1j6qJOXBSXQ010BFnHSdPJEpoXNWo6ZQRZsAmDplVG",
  "/\naY1DNqGiqXEqp6Des3bD1j6qJOXBSXQ010BFnHSdPJEpoXNWo6ZQRZsAmDplVGkCXhxjNiIYMs1+Uh\nTDh0dMSuCoVQTZmN4ncZt+xjuC9OYvheml6OwVJ/5ShjOgAXH36WzDl8a+HS1tZ5GcaenrAp85Y1i\nsZhWBNW0Fp3ArOrYnJplrpBJswWhJLpsmno0FpXHojlBHcAXZ4I5b+h3S5LsGV1eHAbprkp/90\nP6Jz86LTX3Z6D8km9BQmse",
  "no0FpXHojlBHcAXZ4I5b+h3S5LsGV1eHAbprkp/90\nP6Jz86LTX3Z6D8km9BQmse2hnT4fzQ0gkcd3l8QwYsXSbR4ECgXL5Jwf0dLxK8sXWkXDsoCMWkyK7\nQ5S8C1axTRvBgoxCNFQK6XfhmQqF9v2mrANahm94aFs2kIcm6Vz9GSU5gknNz+0nyFS6vq2mAj9sG\nreUKUWmvcNLpe1oAwPhym/prqLMupW+XSjXI1YgpI50s6ezlIM7jEbFd/u",
  "Aj9sG\nreUKUWmvcNLpe1oAwPhym/prqLMupW+XSjXI1YgpI50s6ezlIM7jEbFd/ueRV0WoFfLJX9wfjgtXJ\nPY9Phnt4PQJiUeituCUZG1LEsvSH7S13K5vjqzYe/kd2dqBxbWbkrRbj9JuW9xrRsAn+5bR7hOPWNS\nRqK16hNQjlqU/aMuex3bLCyu3ZSk3UerbFXZpo+/vHYzig6mNSJEf62BfJQRXCYkbFzCpG+nTbF\nKsQFsO8acH/sXIk4O",
  "UerbFXZpo+/vHYzig6mNSJEf62BfJQRXCYkbFzCpG+nTbF\nKsQFsO8acH/sXIk4OHRtKoQFrupaGo6gKURl3gKVQiL1SXcNOsYVvct6r5dZTIeI7MKYXGXhXjWVQiL\nARUDq3jB4hiJVYjkcYzOKZ5jLEU2yS8IrFlRciWsm2oZBw1JR3A0gz1NrN0BiOQkUId1kEsp3Tnpd\nadp9AuVnQX92wd967pOGOoQR3A0gG5xpzBgfUic3GK4ZhlS3Isk",
  "Id1kEsp3Tnpd\nadp9AuVnQX92wd967pOGOoQR3A0gG5xpzBgfUic3GK4ZhlS3IskBXTBHax06XO4vTn+gU5ybn+laFXl\nF4aeknpqaGnlCaGkl8Erv/MUPLrxPWnhk4pPTH0hNLc0JzSnqE9Sn1DfUofGvqQUs9Qj9JtQ7cpzQw\nlJ1J4Ih6TOnY0DGlfUP7lD439Dmljwx9ROkLQ19Q+trQ15TeN/Q+pcxQRumOoTuUckPJqwPX3zJ0i\n1LXUPL",
  "lD439Dmljwx9ROkLQ19Q+trQ15TeN/Q+pcxQRumOoTuUckPJqwPX3zJ0i\n1LXUPLbD641Q7uUxobGlD4w9AGlI0PJr2J4nhlKjfwYDRUvrY0MeUCkPJ7zfXf2roU0pDQ0NKnxj6\nhNJXhr6idNfQXUoDQ8m7ATidGHpEqXkLVKSUHhp6SOnE0In9vQBfLqNr25gHpoEDSiNDI0r3DCW/FO\nAoYegFOU/6qr6rLd42kfuar5bcwuqML2qTnPtqyS2",
  "25gHpoEDSiNDI0r3DCW/FO\nAoYegFOU/6qr6rLd42kfuar5bcwuqML2qTnPtqyS2svjstapP7k6+WfEyGvnOyfJECKYU7/XBto4Pfw\nVus7nrcZn/d/ADuMEak=</latexit>tLCyY/tzs/tO4d3Nu5t1W9ob7S+an3d+rbVaf3Sutd61Oq2ei1v5cuVuysPV3bXf1/c/3v9X8q9a2\nV[X] = \u03b2v1T + \u2326vX\nQ[X] = \u03b2q1T + \u2326qX\nK[X] = \u03b2k1T +",
  "VuysPV3bXf1/c/3v9X8q9a2\nV[X] = \u03b2v1T + \u2326vX\nQ[X] = \u03b2q1T + \u2326qX\nK[X] = \u03b2k1T + \u2326kX,\n<latexit sha1_base64\n=\"jbGxiTNQeaC/HcwhB7SFaZKG2I=\">AW3i\nclZhJb9w2FICVrm6OS3qSy9CjQBFkQ7sIl0uB\nRI7zmanHsce24lnYlAaSsOYomSJscR5txb0Wt/\nUn9Cf0Wv7a2PkmYvUcfOoBn6Pd94vJIagsyKQq\n9uvrXtbfefufd96/s",
  "txb0Wt/\nUn9Cf0Wv7a2PkmYvUcfOoBn6Pd94vJIagsyKQq\n9uvrXtbfefufd96/sGNDz/6+JNPl25+dlCkZR\n7yQZjKND8KWMGlUHyghZb8KMs5SwLJD4PTDcMP\nz3leiFTt68uMjxIWKxGJkGkInSyF/jAJ0mlVDYP\nI32Oz2fEwYXoSRNXRbOT/DNGDY/g6Gg3Dcaobt1\nbTSCdsOhui1gaY6vRXlb7MyjstgcBzUcnSyurv\ndX649PCWltY8d",
  "6Gg3Dcaobt1\nbTSCdsOhui1gaY6vRXlb7MyjstgcBzUcnSyurv\ndX649PCWltY8dpP/+TmF+PhOA3LhCsdSlYUx2u\nrmR5VLNcilHx2Y1gWPGPhKYv5MRQVS3gxqupszP\nxbEBn7UZrDn9J+HX3ziIolRXGZBGCaoRaYmaCLH\nZc6+mlUCZWVmquwaSgqpa9T36TWH4uch1peQoGF\nuYC+uGE5SzUMAE3hopfhGmSMDWuhubuzPI4\n+FqvhZWU/G",
  "T36TWH4uch1peQoGF\nuYC+uGE5SzUMAE3hopfhGmSMDWuhubuzPI4\n+FqvhZWU/GbNZ1NmuHQ/EqY/3x/qIWoXkiXnNS\na2YSq4QeDyrKt6LexgIDkD0OAGp4gXU2SwTfw1R\nWHwScGWXyrMZqVpHkNOtoLokEhk3zasTaIBVO\nZdJQ9UHz/lm8A1znMAnQVfjiag72Mqdn8OM2nO\nk+qwsRwCzlTMa+bgCGHTJoRdQ1VSgmHh3rF2w9\nY+q0TVy",
  "fjiag72Mqdn8OM2nO\nk+qwsRwCzlTMa+bgCGHTJoRdQ1VSgmHh3rF2w9\nY+q0TVya1V3NTQRZ+3nX0TnNixp3nTqCLFiEcde\nqI8iScKoYM7OT5+UTGHDim4hbFQqrgizMfp4G3b\nYzE8Frc5rBful6mxVJ/zlDGTEB2H3mVzAV8q6+\nkS5sf56c89o3BT71JzBZ3UNYHjfDmjcCo2pjM2r\nWuUImzRaE8vSia5reOFSeie4ATQBvujIXKnpDu1",
  "JzBZ3UNYHjfDmjcCo2pjM2r\nWuUImzRaE8vSia5reOFSeie4ATQBvujIXKnpDu1\n2XYMma8PA2DUvJT/+tvc9n46qVbNtzBfJlRUl\nJmrIhP+HxWN4eKE1xdE8OSlEk0eBOrJSyWc39H\nUsRwvbBOp5w4KQjEp9CXa/iJW3WPqCO5smqC+Qs\nDUC79MKDTJUdSVTcDI8AuXWcCtEgw2aMoUyLM\nufk5IfWM0Rq3ZwWc2EuVt0TqjRC97zB5eIoKM",
  "dSVTcDI8AuXWcCtEgw2aMoUyLM\nufk5IfWM0Rq3ZwWc2EuVt0TqjRC97zB5eIoKMPF\n4ZxfcXiAMho0+QzSUo1ZjpI5NVM6fTksNGwx1+\n6vp7wpOq2Yn217UG/YHbKMORnJ1t4PmJiUeiu\nuC+xlmXJajPahrsVzf7Fm19fIbsrRjh+s2Jam3\n7aXbdrhX9ICfbTt6u08YlFHoraHlKPWI72oC5\n3Hrdo3C4blOSeud5dNoOd2Gi5R/tT7hm5jY",
  "ICfbTt6u08YlFHoraHlKPWI72oC5\n3Hrdo3C4blOSeud5dNoOd2Gi5R/tT7hm5jYpl\nWNz25fKYRPCoqaidopwmMkNiEsJmXgv+xsifg\n4tG1mhAW+4XoaiaApTGXeAhNCIvNFu6abQyr2w5\n1260ymU2Q2YSw+JAleNRNCIsxFWOneMqyDIlNiO\nRxgvM4oXnMsJS5JDwjmWNGyJyLah8knYlE8DS\nFLU2dTQGPZCpQg2QSwXdOUVzpWn0CpWdB",
  "XnMsJS5JDwjmWNGyJyLah8knYlE8DS\nFLU2dTQGPZCpQg2QSwXdOUVzpWn0CpWdBUPXA0\nPrmhYM1ShCWBph+wxf7j3GQBTjHcZrmSnAlkZT\nSBfez0qTO/+4NHZHInF0SXl5SemHpBaWHlh5Sm\nltKngiC6Jml5OkiM4tPaf0wNIDSktLS0oHlg4\nojSyNKH1g6QNKQ0tDSjcs3aBUW0ruSOGKYOk+pR\nNLJ5QeWXpE6XNLn1P6yNJHlL6w9AWl",
  "H1g6QNKQ0tDSjcs3aBUW0ruSOGKYOk+pR\nNLJ5QeWXpE6XNLn1P6yNJHlL6w9AWlry19Tek9S\n+9RyixlG5aukpt5S8OgidUvXKQ0sJc9+sNcs\n7VOaWZpRet/S+5SOLSVPxXA9s5Tc3sCF0VJ6W\nNLH1MqLCXPb0H01NKnlCaWJpQ+sfQJpa8sfUXpQ\n0sfUhpbSt4NwN2JpXuU2rdAVUHprqW7lJ5ZeuZ+\nL8AX0xi4FuaOrWCH0tTSlNItS8m",
  "UhpbSt4NwN2JpXuU2rdAVUHprqW7lJ5ZeuZ+\nL8AX0xi4FuaOrWCH0tTSlNItS8mTAtxKWHpK7ic\nj1Z7V5m+byHktUgvuYG3G50eTnEdqwR2sPTvNj\nybnp0gt+IR0fNg8SIFUgpn+pOlTX8FpYWDr7r\nrf3Qu7N7Z+XuevuG9r3pfeV97W35v3o3fUeX1\nt/Ae3U/fF</latexit>v4IXen97f3j/ev8ts+dfl35Z/b9S3rXHfO51Ps\nSa[X] =",
  "t/Ae3U/fF</latexit>v4IXen97f3j/ev8ts+dfl35Z/b9S3rXHfO51Ps\nSa[X] = V[X] \u00b7 Softmax\nh\nK[X]T Q[X]\ni\n35",
  "Scaled Dot Product Self-Attention\n\u2022 To avoid the case where a large value dominates the softmax in\n\u2022 you can scale the dot product by the square root of the dimension of \nthe query\n36",
  "Put it all together in matrix form\n37",
  "Put it all together in matrix form\n38\nScales linearly with \nsequence length, N\n# attention weights scales \nquadratically with sequence \nlength, N, but independent \nof length D of each input",
  "Put it all together in matrix form\n39\nScales linearly with \nsequence length, N\nLinear\n&\nCan be calculated \nin parallel\nNon-linear\n# attention weights scales \nquadratically with sequence \nlength, N, but independent \nof length D of each input\nLinear combination of \nweighted inputs where \nweights calculated from \nnonlinear functions",
  "Hypernetwork \u2013 1 branch calculates weights \nof other branch\n40\nScales linearly with \nsequence length, N\nLinear\n&\nCan be calculated \nin parallel\nNon-linear\nLinear combination of \nweighted inputs where \nweights calculated from \nnonlinear functions\n# attention weights scales \nquadratically with sequence \nlength, N, but independent \nof length D of each input",
  "Multi-Head Self Attention\n\u2022 Multiple self-attention heads are \nusually applied in parallel\n\u2022 \u201callows model to jointly attend to \ninfo from different \nrepresentation subspaces at \ndifferent positions\u201d\n\u2022 Original paper used 8 heads\n\u2022 All can be executed in parallel\n41\nSA outputs are \nconcatenated \nand combined \nweighted by \n\u03a9%.",
  "Equivariance to Word Order\nSelf-attention is equivariant to permuting word order. Just a bag of words.\nBut word order is important in language:\nThe man ate the fish\nvs.\nThe fish ate the man\n42\nA function f[x] is equivariant to a \ntransformation t[] if:\n<latexit sha1_base64=\"3N9gchi9eQHzUGINVq4oZjiE8Ek=\">AW1XiclZhJb9w2FICVdEvTzWlRX3oRagQoinRgF+lyKZDYcTY79Tje45kY\nlIbSMKYoWaLscYS5Fb32J/V39Af02v6FPkqaofUefegAjpj3feLySGoLMikKvbz8142b7z73vsf3Prw9kc",
  "32J/V39Af02v6FPkqaofUefegAjpj3feLySGoLMikKvbz8142b7z73vsf3Prw9kcf/LpZwt3Pt8v0jIP+V6YyjQ/DFjBpVB8Twst+WGWc5YEkh8Ep2uGH5zvBCp2tWXGR8mLFYiEiHTEDpZOBokQTqpBkHkR9OB5JE+thE9PYbDZDjIRTzWQ/8X/yrDdtS1TxaWlnvL9c+\nnhZW2sOS1v/7JnS9Hg1EalglXOpSsKI5XljM9rFiuRSj59PagLHjG",
  "xaWlnvL9c+\nnhZW2sOS1v/7JnS9Hg1EalglXOpSsKI5XljM9rFiuRSj59PagLHjGwlMW82MoKpbwYljVOZj6dyEy8qM0hz+l/Tp69YyKJUVxmQRgJkyPC8xM0MWOSx39PKyEykrNVdg0FJXS16lvEuqPRM5DLS+hwMJcQF/9cMxyFmpI+2B4hdhmiRMjarB6vr2FLFY6EqflbWUzCdp312uFQvM5YfbY7r0Vono\ni3nFRSK6aSawQeT6uK9+IeBo",
  "vr2FLFY6EqflbWUzCdp312uFQvM5YfbY7r0Vono\ni3nFRSK6aSawQeT6uK9+IeBoIDED1OQKp4AXWa/MBcryAKS04CruxqeDklVSvNY8hJR3tFNChk861hqxYCqTjrIDiu/f9Q3gOodZgK7CgaM52MmYms7O03yi86QqTAy3kDMV87oJGHLIpBlR1ClHBq2LF+xdZLpk7bxKVZ3dXcRJC1m3cdndO8qFHXqSPIgkUYd606giwJF4gRSxhkuS2fwI",
  "F+xdZLpk7bxKVZ3dXcRJC1m3cdndO8qFHXqSPIgkUYd606giwJF4gRSxhkuS2fwIAT3\n0TcqlBYFWRh9vM06LadmQhem5M9kvXW69I+s8ZyogJwO4zR8FUyLv6Wjq3/VlyzmvfFPjEH8NkdU9hedwMa9YIjKqNTalZ5wqZNFsQytOLrml641B5JroDNAG86cpcqOiKdq8uwZI14cE9GpeSn78Xe8HPhlWy2bmH9INqGiosxcFZnw/6hoBLckvL4g",
  "pcqOiKdq8uwZI14cE9GpeSn78Xe8HPhlWy2bmH9INqGiosxcFZnw/6hoBLckvL4gicvlWjyIFBPXirh+o6mjuV4YZtIPXdQ\nEIpJoS/R9hex6p5TR3Bn0wT1FQKmXjgyodAkR1FXNgEjwxFuro4FKJBhs0YQ5kWZc7JxQ+tZ4jUurks5sLcrLoXVGmE7nWDy/lZUIabwzm/5vQAZTRo8hmkpRqxHCVzYqZ08npQaNhirt1fT3lTdFoxP9to24N+w",
  "/lZUIabwzm/5vQAZTRo8hmkpRqxHCVzYqZ08npQaNhirt1fT3lTdFoxP9to24N+weyUYcjPTjbwfMTEo5EdcHTjLMuSxHe1DXfLle7Vm18fpbsrRjh+s2Jam37aX\nbdrjX9ICfbTp6u0k8YlFHoraHlKPWI72oC53Hjdo3C4blOSemd5dNoOd26i5R/tjrlm5jEplSPz2JfKQRPCoqaidopwmMkNiEsJmXgv9jZUfAzaNrNSEs9gvR1UwASyMu8R",
  "5jEplSPz2JfKQRPCoqaidopwmMkNiEsJmXgv9jZUfAzaNrNSEs9gvR1UwASyMu8RCaEBabLdw12xhWNx3qpltlMhsjswlh8QlL8KibEBZjKsZO8ZRlGRKbEMnjGOdxTPOYSlzSXhGMseMkCXlWlD5O\n1KJoClCWpt4mgMeiBThRpsg1gu6MornCtPoVWs6CreczW8d03DmqEKTQBLW2SP+YMt5yYLcIrhMcuV5EwgK6MJ7GOnT53Z018QVeRJLog",
  "eczW8d03DmqEKTQBLW2SP+YMt5yYLcIrhMcuV5EwgK6MJ7GOnT53Z018QVeRJLoguLb2k9MLSC0oPLD2gNLeUvBE0UtLydtJEJ1bek7pvqX7lJaWlpTuWbpHaWRpROljSx9TGloaUrpm6Rql2lLyRAp3BEt3KR1bOqb0NJDSo8sPaL0qaVPK\nX1l6StK31r6ltKHlj6klFnKF23dJ1Sbin5dBEq5auUhpYSt79YK9Z2qc0szSj9JGljygdWUreiu",
  "ltKHlj6klFnKF23dJ1Sbin5dBEq5auUhpYSt79YK9Z2qc0szSj9JGljygdWUreiuF+Zil5vIEbo6WS0meWPqNUWEre34LohaUvKE0sTSh9bulzSt9Y+obSJ5Y+oTS2lHwbgKcTS3cotV+BqoLSbUu3KT2z9Mz9XYDPpzFwLcwtW8EWpamlKaUblpI3BXiUsPSUPE9Gqr2q2W+dU2zMu\nYO1GZ+dTXIeqTl3sPbqNDubXJ8iNedj0vX1/fmHFEgpX",
  "SUPE9Gqr2q2W+dU2zMu\nYO1GZ+dTXIeqTl3sPbqNDubXJ8iNedj0vX1/fmHFEgpXOlPFpZW8FdYWtj/vrfyY+/+9v2lB6vtF9pb3lfe1943or3k/fAe+r1vT0v9P70/vb+8f5dPFicLv62+Huj3rzRnvOF1/kt/vEf+tb0Sw=</latexit>f [t[x]] = t [f[x]]",
  "Solution: Position Encoding\nIdea is to somehow encode absolute or \nrelative position in the inputs\n43\nEncoder\nDecoder",
  "Absolute Position encoding\nAdd some matrix, \u03a0, to the \n\ud835\udc37\u00d7\ud835\udc41 input matrix: \n44\n+ \u03a0 \n\u03a0 = \n\u03a0 can be pre-defined or learned",
  "Absolute Position encoding\n<latexit sha1_base64=\"NIfj\nl5jRW3Tli7yX+1AxbREAlmI=\">AWw3iclZhb9s2FIDV7tZ1l\n6Yblpe9CAsKDENnJEN3eRnQJnXTNuniNc2dgNKpmQ2FKVIVOJU\n0E/ar9nTgO2/7FCyzeoc5mEGUrHn+8TLIalbkElR6NXVv2/c/O\nDjz7+5Nantz/7/Isv7yzd/eqwSMs85AdhKtP8OGAFl0LxAy205\nMdZzlkSH4UnG0YfnTB80Kkal9fZX",
  "sv7yzd/eqwSMs85AdhKtP8OGAFl0LxAy205\nMdZzlkSH4UnG0YfnTB80Kkal9fZXyUsFiJSIRMQ+h0adMfJkE6\nraphEPl7rK5PhgnTkyCqjuR/ztED4fhONWt1UhpBM2BTGIt5\nU+zUcd0enSyurvdXm59PC2qyw4s1+g9O734yH4zQsE650KFlRnK\nytZnpUsVyLUPL69rAseMbCMxbzEygqlvBiVDUjrv17EBn7UZrDn\n9J+E3/jIolRXGVBG",
  "pUsVyLUPL69rAseMbCMxbzEygqlvBiVDUjrv17EBn7UZrDn\n9J+E3/jIolRXGVBGCa4RSYmaCLnZQ6+m1UCZWVmquwbSgqpa9T\n36TPH4uch1peQYGFuYC+uGE5SzUkOTbQ8UvwzRJmBpXw/X+bg\n0Z47FQFT8vm4TXdfpNw6H4nXG+rP9RS1C80S846SRjGVXCPwu\nK4q3ot7GAgOQPQ4AaniBdTZLgV/DVFYBJwZRfFy5pUrTSPIScd\n7TXRoJ",
  "VXCPwu\nK4q3ot7GAgOQPQ4AaniBdTZLgV/DVFYBJwZRfFy5pUrTSPIScd\n7TXRoJBJPu1YG8SCqUw6yh4ovn/PN4DrHGYBugoHjuZgL2Oqnp+\nn+VTnSVWYG4hZyrmTRMw5JBJM6KuoUop4dSwY/2BrZdMnc0Sl2\nZNV3MTQdZ+3nV0TvOixl2niSALFmHctZoIsiRcDsbM7Nl5+RQGn\nPgm4laFwqogC3OQp0G37cxE8NqcZrBful6/Ium/YCgj",
  "ZoIsiRcDsbM7Nl5+RQGn\nPgm4laFwqogC3OQp0G37cxE8NqcZrBful6/Ium/YCgjJgC7zxwF\nUyHv6hvpwvbnyblofFPgU38Ck9U9heVxO6x5IzCqWaymZpMrZN\nJsQShPL7um6Y1D5ZnoDtAE8KYrc6Gi97T7TQmWrAkP78NQ81Lyk\nx97P/PpqFo128b8Q7IJFRVl5qrIhP9HRWO4AeH1BRE8ealEkweB\nZvJSCd3NHUsxwvbRJq5g4JQTAp9h",
  "7IJFRVl5qrIhP9HRWO4AeH1BRE8ealEkweB\nZvJSCd3NHUsxwvbRJq5g4JQTAp9hba/iFX3nCaCO5smqK8QMPX\nCkQmFJjmKurIJGBmOcCt1LKAQDTJsxjKtChzTi5+aD1DpNHNZT\nEX5mbVvaBKI3SvG1wuzoIy3Bwu+DWnByijQZvPIC3VmOUomVMzp\ndM3w0LDFnPt/mbK26LTivn51qw96BfMThmG/Px0C89HTCzqSFQX\nPLs465LEcrQHdS",
  "3w0LDFnPt/mbK26LTivn51qw96BfMThmG/Px0C89HTCzqSFQX\nPLs465LEcrQHdS2W6/s9q7be/ECWduxw3aYk9c56bYd7jU94O\nfbjt5uE49Y1JGorlkPqUcsR3tQlzuP265ROFy3KUm98zw6bYe7M\nNHyj/YnXDPzmJTKsXnsS+WwDWFRU1E7xThMRLbEBaTsmvB/7Gy\nJ+Dm0bXaEBYHhehqJoClMZd4CG0Ii+0W7pqzGFa3Heq2W2UymyC\nz",
  "aTsmvB/7Gy\nJ+Dm0bXaEBYHhehqJoClMZd4CG0Ii+0W7pqzGFa3Heq2W2UymyC\nzDWFxkyV41G0IizEVY6d4xrIMiW2I5HGC8zihecywlLkPCOZY0\nbIknItqHySdiUTwNIUtTZ1NAY9kKlCDc6CWC7oyiucK0+hVazoK\nj5wNXxwTcOaoQpNAEs7ZI/5wx3nJgtwiuExy5XkTCArowkcYGdA\nnfnTH7wGkye5ILqy9IrS0svKT2y9IjS3FLyRhB",
  "nJgtwiuExy5XkTCArowkcYGdA\nnfnTH7wGkye5ILqy9IrS0svKT2y9IjS3FLyRhBELy0lbydBdG\nHpBaWHlh5SWlpaUnpg6QGlkaURpU8sfUJpaGlI6YalG5RqS8kTK\ndwRLN2ndGLphNJjS48pfWXpK0qfWvqU0teWvqb0naXvKH1k6SNK\nmaWM0r6lfUq5peTQRCtW7pOaWApefeDvWbpgNLM0ozSx5Y+pnR\nsKXkrhvuZpeTxBm6MlkpKn1n6",
  "peTQRCtW7pOaWApefeDvWbpgNLM0ozSx5Y+pnR\nsKXkrhvuZpeTxBm6MlkpKn1n6jFJhKXl/C6IXlr6gNLE0ofS5pc\n8pfWvpW0o3Ld2kNLaUfBuApxNL9yi1X4GqgtJdS3cpPbf03P1dg\nC+mMXAtzB1bwQ6lqaUpVuWkjcFeJSw9Iw8T0ZqdlWbf20i17V\nILbiDzTI+P5vkPFIL7mCzq9P8bHJ9itSCT0jX+4eLDymQUrjSny\n6trOGvsLRw+",
  "ILbiDzTI+P5vkPFIL7mCzq9P8bHJ9itSCT0jX+4eLDymQUrjSny\n6trOGvsLRw+FNv7Zfeg90HKw/XZ19ob3nfet953tr3q/eQ+pN\n</latexit>/AOvND70/vL+8f7d7m/fLacL+tWvXljds7Xue3XP8HensVw=\nSa[X] = V \u00b7 Softmax[KT Q]\n<latexit sha1\n_base64=\"QM5KCS2uh4GckQCS/\nhtcoiskl7E=\">AW23iclZhJ\nb9w2FICVdEvTzWlRX3oR",
  "=\"QM5KCS2uh4GckQCS/\nhtcoiskl7E=\">AW23iclZhJ\nb9w2FICVdEvTzWlRX3oRagRI2\n9Swi3S5FEjsOJudehx7bCeiUF\npKA1jipIlyh5HmFNvRa/9Sf0R\n/Q29tvc+Sph9B596AD2cN73ic\nsjqS3IpCj0yspfV6+9fY75\n37f3rH3z40cefLNz4dL9Iyzk/\nTCVaX4YsIJLoXhfCy35YZzlg\nSHwQn64YfnPG8EKna0xcZHyY\nsViISIdMQ",
  "yzk/\nTCVaX4YsIJLoXhfCy35YZzlg\nSHwQn64YfnPG8EKna0xcZHyY\nsViISIdMQOl49gdJkE6qahBE/\ni6bTo8GCdPjIKoOp0P/51sQ3v\n8G/vXEV4NwlOrGruU0gmbTI+M\ns9k6L6u9qfm90/4eHi8srSyv1\nB+fFlbwpLXfnrHNz4fDUZpWCZ\nc6VCyojhaXcn0sGK5FqHk0+uD\nsuAZC09YzI+gqFjCi2FVZ2Lq3\n4TIyI/SHP6U9uvom0dU",
  "haXcn0sGK5FqHk0+uD\nsuAZC09YzI+gqFjCi2FVZ2Lq3\n4TIyI/SHP6U9uvom0dULCmKiyQ\nA0wyzwMwEXeyo1NFPw0qorNRc\nhU1DUSl9nfomrf5I5DzU8gIKLM\nwF9NUPxyxnoYbkXx8ofh6mScL\nUqBqsbexMIYM8Fqrip2U9EdNp1\n9moHQ7Fy4y1x3vzWoTmiXjNS\nW1Yiq5RODxtKr4cryMgeAxDI\nnIFW8gDqbJeKvIgoLTwKu7CJ5N\ni",
  "zWoTmiXjNS\nW1Yiq5RODxtKr4cryMgeAxDI\nnIFW8gDqbJeKvIgoLTwKu7CJ5N\niVK81jyElHe0E0KGSTzrWOr\nFgKpOsguK79/0DeA6h1mArsIX\nR3OwmzE1nR2n+UTnSVWYG4hZ\nyrmdRMw5JBJM6KuoUop4dCwY/2\nCrWdMnbSJS7O6q7mJIGsv7zo6\np3lRo65TR5AFizDuWnUEWRJOE\nyNm9vCsfAwDTnwTcatCYVWQhdn\nL06DbdmYieG1",
  "3lRo65TR5AFizDuWnUEWRJOE\nyNm9vCsfAwDTnwTcatCYVWQhdn\nL06DbdmYieG1OMtgvXW+jIuk/\nYygjJgC7z3wLpkLe1dfTue3Pkn\nNW+6bAJ/4YJqt7CMvjZlizRmB\nUbWxKzTpXyKTZglCendN0xuHy\njPRHaAJ4E1X5kJFb2i36xIsWR\nMe3Iah5qXkR98uf8nw2rFbBv\nzj2QTKirKzFWRCf+PikZwYcLrC\nyJ48lKJg8C9eSlEs7vaOp",
  "kR98uf8nw2rFbBv\nzj2QTKirKzFWRCf+PikZwYcLrC\nyJ48lKJg8C9eSlEs7vaOpYjh\ne2idRzBwWhmBT6Am1/EavuMXUE\ndzZNUF8hYOqFbyYUmuQo6somY\nGT4hkusYwGFaJBhM8ZQpkWZc3L\nyQ+sZIrVuTou5MBer7glVGqF7\n3uByfhSU4eJwxi85PEAZDZp8B\nmpRixHyZyYKZ28HBQatphr9d\nT3hSdVsxPN9v2oF8wO2UY8tPj\nTwfMb",
  "AZDZp8B\nmpRixHyZyYKZ28HBQatphr9d\nT3hSdVsxPN9v2oF8wO2UY8tPj\nTwfMbGoI1FdcE/jrEsSy9Ee1D\nVfrm/2rNp8+TVZ2rHDdZuS1Nv\n20m073Et6wE+3HL3dIh6xqCNRX\nW0PqUcsR3tQlzuPW65ROFy3KU\nm9szw6bYc7N9Hyj/bGXDNzm5T\nKkbntS+WgCWFRU1E7xThMRKbE\nBaTsmvBb6zsCrh4dK0mhMVeIb\nqaCWBpxCUeQhPC",
  "bntS+WgCWFRU1E7xThMRKbE\nBaTsmvBb6zsCrh4dK0mhMVeIb\nqaCWBpxCUeQhPCYrOFu2Ybw+qW\nQ91yq0xmY2Q2ISw+ZAkedRPCY\nkzF2CmesCxDYhMieRzjPI5pHjM\nsZS4Jz0jmBGypFwLKh+nXckE\nsDRBrU0cjUEPZKpQg20QywVde\nYVz5Sm0ihVdxX1Xw/1LGtYMVWg\nCWNome8wfbDs3WYBTDLdZriRn\nAlkZTWAPOz3qzO7+4PGY3M",
  "xX1Xw/1LGtYMVWg\nCWNome8wfbDs3WYBTDLdZriRn\nAlkZTWAPOz3qzO7+4PGY3MkF0Y\nWlF5SeW3pO6YGlB5TmlpIngiB\n6Zil5OgmiM0vPKN23dJ/S0tKS0\nr6lfUojSyNKH1j6gNLQ0pDSdU\nvXKdWkjtSuCJYukfp2NIxpYe\nWHlL63NLnlD6y9BGlLyx9QelrS\n19Tes/Se5QySxmlG5ZuUMotJa\n8OgmjN0jVKA0vJsx/sNUt7lGaW\nZpT",
  "x9QelrS\n19Tes/Se5QySxmlG5ZuUMotJa\n8OgmjN0jVKA0vJsx/sNUt7lGaW\nZpTet/Q+pSNLyVMxXM8sJbc3c\nG0VFL62NLHlApLyfNbED219Cm\nliaUJpU8sfULpK0tfUfrQ0oeU\nxpaSdwNwd2LpLqX2LVBVULpj6\nQ6lp5aeut8L8Pk0Bq6FuW0r2KY\n0tTSldNS8qQAtxKWnpD7yUi1\nZ7XZ2yZyXovUnDtYm/HZ0STnkZ\npzB2vPTrOjyf",
  "0tTSldNS8qQAtxKWnpD7yUi1\nZ7XZ2yZyXovUnDtYm/HZ0STnkZ\npzB2vPTrOjyfkpUnM+Jl3f2J+\n/SIGUwpn+eGFpFb+FpYX975ZX\nf1i+s3Nn6e5a+4b2mveF96V3y1\nv1fvTueo+8ntf3Qu9P72/vH+/\n3/8B0Om9FI=</latexit>fxeHir4u/Lf7eqFevtMd85nU+i\nSa[X] = (V + \u21e7) \u00b7 Softmax[(K + \u21e7)T (Q + \u21e7)]\n45\nAlternatively, could be added to each layer",
  "Relative Position Encoding\nAbsolute position of a word is less important than relative position \nbetween inputs\n46\nThe panda eats shoots and leaves\nAbs Pos:  0    1    2      3    4    5    \nRel Pos: -2   -1    0      1    2    3    \nEach element of the attention matrix corresponds to \nan offset between query position a and key position b\nLearn a parameter \ud835\udf0b4,5 for each offset and modify \nAttention[a,b] in some way.",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n47",
  "Transformers\n\u2022 Multi-headed Self Attention is just one \ncomponent of the transformer \narchitecture\n48\nEncoder\nDecoder",
  "Transformers\n\u2022 Multi-headed Self Attention is just one \ncomponent of the transformer \narchitecture\n\u2022 Let\u2019s look at a transformer block (or \nlayer) from the encoder\n49\nEncoder\nDecoder",
  "Transformer Layer -- Complete\n50\n\u2022 Adds a 2-layer MLP\n\u2022 Adds residual connections around multi-head self-\nattentions and the parallels MLPs\n\u2022 Adds LayerNorm, which normalizes across all the N \ninput samples\nTransform Layer",
  "Transformer Layer -- MLP\n51\n\u2022 Ads 2-layer MLP\n\u2022\nSame network (same weights) operates \nindependently on each word\n\u2022\nLearn more complex representations and expand \nmodel capacity\nLinearDx4D \u00e0 ReLU(.) \u00e0 Linear4DxD",
  "Transformer Layer -- LayerNorm\n52\n\u2022 Normalize across same layer\n\u2022 Learned gain and offset\nD\nN\n# NLP Example\nbatch, sentence_length, embedding_dim = 20, 5, 10\nembedding = torch.randn(batch, sentence_length, embedding_dim)\nlayer_norm = nn.LayerNorm(embedding_dim)\n# Activate module\nlayer_norm(embedding)\nhttps://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html \nCalculated column-wise",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n53",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n54",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n55",
  "3 Types of Transformer Models\n1. Encoder \u2013 transforms text embeddings into representations that \nsupport variety of tasks (e.g. sentiment analysis, classification)\nv Model Example: BERT\n2. Decoder \u2013 predicts the next token to continue the input text (e.g. \nChatGPT, AI assistants)\nv Model Example: GPT4, GPT4\n3. Encoder-Decoder \u2013 used in sequence-to-sequence tasks, where one \ntext string is converted to another (e.g. machine translation)\n56",
  "Encoder Model Example: BERT (2019)\nBidirectional Encoder Representations from Transformers\n\u2022 Hyperparameters\n\u2022 30,000 token vocabulary\n\u2022 1024-dimensional word embeddings\n\u2022 24x transformer layers\n\u2022 16 heads in self-attention mechanism\n\u2022 4096 hidden units in middle of MLP\n\u2022 ~340 million parameters\n\u2022 Pre-trained in a self-supervised manner, \n\u2022 then can be adapted to task with one additional layer and fine-tuned\n57\nJ. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, \u201cBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\u201d \narXiv, May 24, 2019. doi: 10.48550/arXiv.1810.04805.",
  "Encoder Pre-Training\n\u2022 A small percentage of input embedding replaced with a generic <mask> \ntoken\n\u2022 Predict missing token from output embeddings\n\u2022 Added linear layer and softmax to generate probabilities over vocabulary\n\u2022 Trained on BooksCorpus (800M words) and English Wikipedia (2.5B words)\n58\nXT\nSpecial <cls> token \nused for aggregate \nsequence \nrepresentation for \nclassification",
  "Encoder Fine-Tuning\n\u2022 Extra layer(s) appended to convert output vectors to desired \noutput format\n\u2022 3rd Example: Text span prediction -- predict start and end \nlocation of answer to a question in passage of Wikipedia, see \nhttps://rajpurkar.github.io/SQuAD-explorer/ \n59\nSentiment \nAnalysis\nNamed Entity \nRecognition (NER)\n<cls> token position",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n60",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n61\nT. B. Brown et al., \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n\u2022 Factors the probability of the sentence: \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60 \ud835\udc53\ud835\udc62\ud835\udc5b = \n \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \u00d7 Pr \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d)",
  "Pr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d) \u00d7\n \nPr \ud835\udc56\ud835\udc60 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc53\ud835\udc62\ud835\udc5b \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n62\nT. B. Brown et al. \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul.",
  "\ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n62\nT. B. Brown et al. \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n\u2022 Factors the probability of the sentence: \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60 \ud835\udc53\ud835\udc62\ud835\udc5b = \n \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \u00d7 Pr \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d)",
  "Pr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d) \u00d7\n \nPr \ud835\udc56\ud835\udc60 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc53\ud835\udc62\ud835\udc5b \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n\u2022 More formally: Autoregressive model\nPr \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61# = Pr(\ud835\udc61!)",
  "\ud835\udc56\ud835\udc60   \n\u2022 More formally: Autoregressive model\nPr \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61# = Pr(\ud835\udc61!) 9\n$%\"\n#\nPr \ud835\udc61$ \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61$&!)\n63\nT. B. Brown et al., \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder: Masked Self-Attention\n\u2022 During training we want to maximize the log probability of the input text \nunder the autoregressive model\n\u2022 We want to make sure the model doesn\u2019t \u201ccheat\u201d during training by \nlooking ahead at the next token\n\u2022 Hence we mask the self attention weights corresponding to current and \nright context to negative infinity\n64",
  "Masked Self-Attention\n65\nX\nX\nX\nMask right context self-attention weights to zero",
  "Masked Self-Attention\n66\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e",
  "Decoder: Text Generation (Generative AI)\n67\nIgnore\nPrompt\nGenerated\n\u2022 Prompt with token string \u201c<start> It takes great\u201d\n\u2022 Generate next token for the sequence by\n\u2022 picking most likely token\n\u2022 sample from the probability distribution\n\u2022 alternative top-k sampling to avoid picking from the long tail\n\u2022 beam search \u2013 select the most likely sentence rather than greedily pick",
  "Decoder: Text Generation (Generative AI)\n68\nIgnore\nPrompt\nGenerated\nGenerated\n\u2022 Feed the output back into input",
  "Decoder: Text Generation (Generative AI)\n69\nIgnore\nPrompt\nGenerated\nGenerated\n\u2022 Feed the output back into input",
  "Technical Details\nBERT\nGPT3\nModel Architecture\nEncoder\nDecoder\nEmbedding Size\n1024\n12,288\nVocabulary\n30K tokens\nSequence Length\n2048\n# Heads\n16\n96\n# Layers\n24\n96\nQ,K,V dimensions\n64\n128\nTraining set size\n3.3B tokens\n300B+ tokens\n# Parameters\n340M\n175B\n70",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n71",
  "Encoder-Decoder Model\n\u2022 Used for machine translation, which is a sequence-to-sequence task\n72\nhttps://jalammar.github.io/illustrated-transformer/",
  "Encoder Decoder Model\n\u2022 The transformer layer in the decoder of \nthe encoder-decoder model has an \nextra stage\n\u2022 Attends to the input of the encoder \nwith cross attention using Keys and \nValues from the output of the encoder\n\u2022 Shown here on original diagram from \n\u201cAttention is all you need\u201d paper\n73\nEncoder\nDecoder",
  "Encoder Decoder Model\n\u2022 Same view per UDL book\n74",
  "Cross-Attention\n75\nKeys and Values come from the last stage of \nthe encoder",
  "Next Time\n\u2022 Tokenization and Learned Embeddings\n\u2022 Training and Fine-Tuning Transformers\n\u2022 Image Transformers\n\u2022 Multimodal Transformers\n\u2022 \u2026\n76\nLink\nFeedback",
  "Transformers \u2013 Part 2\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "Recap From Part 1\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer \nModels\n\u2022 Encoder\n2",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n3",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n4",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n5",
  "3 Types of Transformer Models\n1. Encoder \u2013 transforms text embeddings into representations that \nsupport variety of tasks (e.g. sentiment analysis, classification)\nv Model Example: BERT\n2. Decoder \u2013 predicts the next token to continue the input text (e.g. \nChatGPT, AI assistants)\nv Model Example: GPT4, GPT4\n3. Encoder-Decoder \u2013 used in sequence-to-sequence tasks, where one \ntext string is converted to another (e.g. machine translation)\n6",
  "Encoder Model Example: BERT (2019)\nBidirectional Encoder Representations from Transformers\n\u2022 Hyperparameters\n\u2022 30,000 token vocabulary\n\u2022 1024-dimensional word embeddings\n\u2022 24x transformer layers\n\u2022 16 heads in self-attention mechanism\n\u2022 4096 hidden units in middle of MLP\n\u2022 ~340 million parameters\n\u2022 Pre-trained in a self-supervised manner, \n\u2022 then can be adapted to task with one additional layer and fine-tuned\n7\nJ. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, \u201cBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.\u201d \narXiv, May 24, 2019. doi: 10.48550/arXiv.1810.04805.",
  "Encoder Pre-Training\n\u2022 A small percentage of input embedding replaced with a generic <mask> \ntoken\n\u2022 Predict missing token from output embeddings\n\u2022 Added linear layer and softmax to generate probabilities over vocabulary\n\u2022 Trained on BooksCorpus (800M words) and English Wikipedia (2.5B words)\n8\nXT\nSpecial <cls> token \nused for aggregate \nsequence \nrepresentation for \nclassification",
  "Encoder Fine-Tuning\n\u2022 Extra layer(s) appended to convert output vectors to desired \noutput format\n\u2022 3rd Example: Text span prediction -- predict start and end \nlocation of answer to a question in passage of Wikipedia, see \nhttps://rajpurkar.github.io/SQuAD-explorer/ \n9\nSentiment \nAnalysis\nNamed Entity \nRecognition (NER)\n<cls> token position",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n10",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n11\nT. B. Brown et al., \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n\u2022 Factors the probability of the sentence: \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60 \ud835\udc53\ud835\udc62\ud835\udc5b = \n \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \u00d7 Pr \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d)",
  "Pr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d) \u00d7\n \nPr \ud835\udc56\ud835\udc60 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc53\ud835\udc62\ud835\udc5b \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n12\nT. B. Brown et al. \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul.",
  "\ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n12\nT. B. Brown et al. \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder Model Example: GPT3 (2020)\nGenerative Pre-trained Transformer\n\u2022 One purpose: generate the next token in a sequence\n\u2022 By constructing an autoregressive model\n\u2022 Factors the probability of the sentence: \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60 \ud835\udc53\ud835\udc62\ud835\udc5b = \n \nPr \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \u00d7 Pr \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d)",
  "Pr \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d) \u00d7\n \nPr \ud835\udc56\ud835\udc60 \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54) \u00d7\n \nPr \ud835\udc53\ud835\udc62\ud835\udc5b \ud835\udc3f\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc51\ud835\udc52\ud835\udc52\ud835\udc5d \ud835\udc59\ud835\udc52\ud835\udc4e\ud835\udc5f\ud835\udc5b\ud835\udc56\ud835\udc5b\ud835\udc54 \ud835\udc56\ud835\udc60   \n\u2022 More formally: Autoregressive model\nPr \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61# = Pr(\ud835\udc61!)",
  "\ud835\udc56\ud835\udc60   \n\u2022 More formally: Autoregressive model\nPr \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61# = Pr(\ud835\udc61!) 8\n$%\"\n#\nPr \ud835\udc61$ \ud835\udc61!, \ud835\udc61\", \u2026 , \ud835\udc61$&!)\n13\nT. B. Brown et al., \u201cLanguage Models are Few-Shot Learners.\u201d arXiv, Jul. 22, 2020. doi: 10.48550/arXiv.2005.14165.",
  "Decoder: Masked Self-Attention\n\u2022 During training we want to maximize the log probability of the input text \nunder the autoregressive model\n\u2022 We want to make sure the model doesn\u2019t \u201ccheat\u201d during training by \nlooking ahead at the next token\n\u2022 Hence we mask the self attention weights corresponding to current and \nright context to negative infinity\n14",
  "Masked Self-Attention\n15\nX\nX\nX\nMask right context self-attention weights to zero",
  "Masked Self-Attention\n16\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e\n\u2212\u221e",
  "Decoder: Training Process \u2013 Teacher Forcing\n17\n\u2022 During training we compute loss between ground truth label input and \ngenerated output\n\u2022 We do not feed output back to input \u00e8 \u201dTeacher Forcing\u201d\nloss(it, it)\n+ loss(takes, takes) + \u2026\nGround Truth Labels\nGenerated",
  "Decoder: Text Generation (Generative AI)\n18\nIgnore\nPrompt\nGenerated\n\u2022 Prompt with token string \u201c<start> It takes great\u201d\n\u2022 Generate next token for the sequence by some strategy",
  "Decoder: Text Generation (Generative AI)\n19\nIgnore\nPrompt\nGenerated\nGenerated\n\u2022 Feed the output back into input",
  "Decoder: Text Generation (Generative AI)\n20\nIgnore\nPrompt\nGenerated\nGenerated\n\u2022 Feed the output back into input",
  "Transformers\n\u2022 Motivation\n\u2022 Dot-product self-attention\n\u2022 Applying Self-Attention\n\u2022 The Transformer Architecture\n\u2022 Three Types of NLP Transformer Models\n\u2022 Encoder\n\u2022 Decoder\n\u2022 Encoder-Decoder\n21",
  "Encoder-Decoder Model\n\u2022 Used for machine translation, which is a sequence-to-sequence task\n22\nhttps://jalammar.github.io/illustrated-transformer/",
  "Encoder Decoder Model\n\u2022 The transformer layer in the decoder of \nthe encoder-decoder model has an \nextra stage\n\u2022 (As opposed to a standalone decoder \ni.e. GPT)\n\u2022 Attends to the input of the encoder \nwith cross attention using Keys and \nValues from the output of the encoder\n\u2022 Shown here on original diagram from \n\u201cAttention is all you need\u201d paper\n23\nEncoder\nDecoder",
  "Encoder Decoder Model Training\n\u2022 Target translation is fed to \nthe decoder\n\u2022 \u201cTeacher forcing\u201d is used, \nin that, regardless of \ndecoder output, the \ncorrect word is provided \nthe decoder\n24",
  "Encoder Decoder Model Inference\n\u2022 TODO: Show inference \nprogression\n25",
  "Cross-Attention\n26\nKeys and Values come from the last stage of \nthe encoder",
  "27\nWhich model flavor do you use for Named \nEntity Recognition?\n\u24d8 Start presenting to display the poll results on this slide.",
  "28\nWhich model flavor do you use for \nlanguage translation?\n\u24d8 Start presenting to display the poll results on this slide.",
  "29\nWhich model flavor do you use for generating text, \nquestion answering, AI assistant?\n\u24d8 Start presenting to display the poll results on this slide.",
  "Next Token Selection\nRecall: output is a \ud835\udcb1 \u00d71 vector of probabilities\n\u2022 How should we pick the next token in decoder \nand encoder-decoder models?\n\u2022 Trade off between accuracy and diversity\n30",
  "Next Token Selection\nRecall: output is a \ud835\udcb1 \u00d71 vector of probabilities\n\u2022 Greedy selection\n\u2022 Top-K\n\u2022 Nucleus\n\u2022 Beam search\n31",
  "Next Token Selection \u2013 Greedy \nPick most likely token (greedy)\nSimple to implement. Just take the max().\nMight pick first token \ud835\udc66!, but then there is no \ud835\udc66\" where Pr \ud835\udc66\" \ud835\udc66!) \nis high. \nResult is generic and predictable. Same output for a given input \ncontext.\n32",
  "Next Token Selection -- Sampling\nSample from the probability distribution\nGet a bit more diversity in the output\nWill occasionally sample from the long tail of the distribution, \nproducing some unlikely word combinations\n33",
  "Next Token Selection \u2013 Top K Sampling\n1. Generate the probability vector as usual\n2. Sort tokens by likelihood\n3. Discard all but top k most probable words\n4. Renormalize the probabilities to be valid probability distribution \n(e.g. sum to 1)\n5. Sample from the new distribution\nDiversifies word selection\nDepends on the distribution. Could be low variance, reducing diversity\n34",
  "Next Token Selection \u2013 Nucleus Sampling\nInstead of keeping top-k, keep the top p percent of the \nprobability mass.\nChoose from the smallest set from the vocabulary such \nthat\nDiversifies word selection with less dependence on nature \nof distribution.\nDepends on the distribution. Could be low variance, \nreducing diversity\n35",
  "Next Token Selection \u2013 Beam Search\nCommonly used in machine \ntranslation\nMaintain multiple output choices \nand then choose best combinations \nlater via tree search\nV = {yes, ok, <eos>}\nWe want to maximize \ud835\udc5d \ud835\udc61\", \ud835\udc61#, \ud835\udc61$ .\nGreedy: 0.5\u00d70.4\u00d71.0 = 0.20\nOptimal: 0.4\u00d70.7\u00d71.0 = 0.28\n36\nD. Jurafsky and J. H. Martin, Speech and Language Processing. 2024. https://web.stanford.edu/~jurafsky/slpdraft/",
  "Next Token Selection \u2013 Beam Search\nBut we can\u2019t exhaustively search the entire vocabulary\nKeep k tokens (beam width) at each step\n37\nD. Jurafsky and J. H. Martin, Speech and Language Processing. 2024.  https://web.stanford.edu/~jurafsky/slpdraft/",
  "Next Token Selection \u2013 Beam Search\n38\nD. Jurafsky and J. H. Martin, Speech and Language Processing. 2024.  https://web.stanford.edu/~jurafsky/slpdraft/\nKeep k tokens at each step\nE.g. k = 2\nPrune to k at each step",
  "Next Token Selection \u2013 Beam Search\n39\nD. Jurafsky and J. H. Martin, Speech and Language Processing. 2024.  https://web.stanford.edu/~jurafsky/slpdraft/\nCalculated with log \nprobabilities\nand add",
  "Next Token Selection\n\u2022 Greedy selection\n\u2022 Top-K\n\u2022 Nucleus\n\u2022 Beam search\n40\nJupyter notebook exploring each of these will be \nassigned after spring break",
  "Transformers for Long Sequences\n41",
  "Context Length of LLMs\n42\nModel\nContext Length\nLlama 2\n32K\nGPT4\n32K\nGPT-4 Turbo\n128K\nClaude 2.1\n200K\nhttps://cobusgreyling.medium.com/rag-llm-context-size-6728a2f44beb",
  "Attention Matrix\n43\nN\nN\nScales quadratically with \nsequence length N, e.g. N2.",
  "Masked Attention\n44\nN\nN\n~1/2 the interactions but \nstill scales quadratically",
  "Use Convolutional Structure in Attention\n45\nEncoder\nDecoder",
  "Dilated Convolutional Structures\n46\nEncoder\nDecoder\nEncoder\nDecoder",
  "Have some tokens interact globally\n47\nEncoder\nDecoder",
  "Tokenization and Word Embedding\n48",
  "NLP Preprocessing Pipeline\n49\nTokenizer\nLearned\nEmbeddings\nTransformer\nPreprocessing: Tokenization and Embedding\nTransformers don\u2019t work on character string directly, but rather on vectors.\nThe character strings must be converted to vectors\n<Some text string>",
  "Tokenizer\nTokenizer chooses input \u201cunits\u201d, e.g. words, sub-words, characters via tokenizer \ntraining\nIn tokenizer training, commonly occurring substrings are greedily merged based on \ntheir frequency, starting with character pairs\n50\nEncode\nDecode\ncharacter (e.g. \nUnicode) \nstrings\ntoken \nIDs\ncharacter (e.g. \nUnicode) \nstrings",
  "Tokenization Issues\n\u201cA lot of the issues that may look like issues with the neural network architecture actually trace back to tokenization. Here are \njust a few examples\u201d \u2013 Andrej Karpathy\n\u2022\nWhy can't LLM spell words? Tokenization.\n\u2022\nWhy can't LLM do super simple string processing tasks like reversing a string? Tokenization.\n\u2022\nWhy is LLM worse at non-English languages (e.g. Japanese)? Tokenization.\n\u2022\nWhy is LLM bad at simple arithmetic? Tokenization.\n\u2022\nWhy did GPT-2 have more than necessary trouble coding in Python? Tokenization.\n\u2022\nWhy did my LLM abruptly halt when it sees the string \"<|endoftext|>\"? Tokenization.\n\u2022\nWhat is this weird warning I get about a \"trailing whitespace\"? Tokenization.\n\u2022\nWhy did the LLM break if I ask it about \"SolidGoldMagikarp\"? Tokenization.\n\u2022\nWhy should I prefer to use YAML over JSON with LLMs? Tokenization.\n\u2022\nWhy is LLM not actually end-to-end language modeling? Tokenization.\n\u2022\nWhat is the real root of suffering? Tokenization.",
  "Tokenization.\n\u2022\nWhy should I prefer to use YAML over JSON with LLMs? Tokenization.\n\u2022\nWhy is LLM not actually end-to-end language modeling? Tokenization.\n\u2022\nWhat is the real root of suffering? Tokenization.\n51\nhttps://github.com/karpathy/minbpe/blob/master/lecture.md",
  "Unicode Standard and UTF-8\n\u2022 Unicode \u2013 variable length character encoding standard. currently defines 149,813 \ncharacters and 161 scripts, including emoji, symbols, etc.\n\u2022 Unicode Codepoint \u2013 can represent up to 17\u00d72-. = 1,114,112 entries. e.g. \nU+0000 \u2013 U+10FFFF in hexadecimal\n\u2022 Unicode Transformation Standard (e.g. UTF-8) \u2013 is a variable length encoding \nusing one to four bytes\n\u2022 First 128 chars same as ASCII\n52\nhttps://en.wikipedia.org/wiki/Unicode \nhttps://en.wikipedia.org/wiki/UTF-8 \nCovers ASCII\nBasic Multilingual Plane including Chinese, Japanese and Korean characters\nCovers remainder of almost all Latin-script alphabets\nEmoji, historic scripts, math symbols",
  "Tokenizer\nTwo common tokenizers:\n\u2022 Byte Pair Encoding (BPE) \u2013 Used by OpenAI GPT2, GPT4, etc.\n\u2022 The BPE algorithm is \"byte-level\" because it runs on UTF-8 encoded strings.\n\u2022 This algorithm was popularized for LLMs by the GPT-2 paper and the \nassociated GPT-2 code release from OpenAI. Sennrich et al. 2015 is cited as \nthe original reference for the use of BPE in NLP applications. Today, all \nmodern LLMs (e.g. GPT, Llama, Mistral) use this algorithm to train their \ntokenizers.*\n\u2022 sentencepiece\n\u2022 (e.g. Llama, Mistral) use sentencepiece instead. Primary difference being that \nsentencepiece runs BPE directly on Unicode code points instead of on UTF-8 \nencoded bytes.\n53\n* https://github.com/karpathy/minbpe/tree/master",
  "BPE Pseudocode\nInitialize vocabulary with individual characters in \nthe text and their frequencies\nWhile desired vocabulary size not reached:\n    Identify the most frequent pair of adjacent    \n    tokens/characters in the vocabulary\n    Merge this pair to form a new token\n    Update the vocabulary with this new token\n    Recalculate frequencies of all tokens including \n    the new token\nReturn the final vocabulary\n54",
  "Enforce a Token Split Pattern\n\u2022 Do not allow tokens to merge across certain characters or patterns\n\u2022 Common contraction endings: \u2018ll, \u2018ve, \u2018re\n\u2022 Match words with a leading space\n\u2022 Match numeric sequences\n\u2022 carriage returns, new lines\n55\nGPT4_SPLIT_PATTERN = r\"\"\"'(?i:[sdmt]|ll|ve|re)|[^\\r\\n\\p{L}\\p{N}]?+\\p{L}+|\\p{N}{1,3}| \n?[^\\s\\p{L}\\p{N}]++[\\r\\n]*|\\s*[\\r\\n]|\\s+(?!\\S)|\\s+\"\"\"\nGPT2_SPLIT_PATTERN = r\"\"\"'(?:[sdmt]|ll|ve|re)| ?\\p{L}+| ?\\p{N}+| \n?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+\"\"\"",
  "GPT4 Tokenizer\n56\nhttps://tiktokenizer.vercel.app/ \ncl100k_base is the GPT4 tokenizer",
  "GPT2 Tokenizer\n57\nhttps://tiktokenizer.vercel.app/ \nYou can see some issues with the \nGPT2 tokenizer with respect to \npython code",
  "GPT4 Tokenizer\n58\nhttps://tiktokenizer.vercel.app/ \nIssues are improved with GPT4 \ntokenizer",
  "59\nByte Pair Encoding (BPE) Example",
  "60\nByte Pair Encoding (BPE) Example",
  "61\nByte Pair Encoding (BPE) Example",
  "62\nByte Pair Encoding (BPE) Example",
  "63\nByte Pair Encoding (BPE) Example",
  "64\nGenerally # of tokens increases and \nthen starts decreasing after \ncontinuing to merge tokens",
  "Learned Embeddings\n\u2022 After the tokenizer, you have an updated \u201dvocabulary\u201d indexed by token ID\n\u2022 Next step is to translate the token into an embedding vector\n\u2022 Translation is done via a linear layer which is typically learned with the rest of the \ntransformer model\n\u2022 Special layer definition, likely to exploit sparsity of input\n65\nTokenizer\nLearned\nEmbeddings:\nLinear Layer\nTransformer\n<Some text string>\nself.embedding = nn.Embedding(vocab_size, embedding_dim)",
  "Embeddings Output\n\u201dOne hot encoding\u201d\n66\nN\n\ud835\udcb1\nIn this example, we are \nassuming a token is simply a \ncomplete word\n\u2022\nTypical embedding size, D, is 1024\n\u2022\nTypical vocabulary size, \ud835\udcb1  , is 30,000\n\u2022\nSo 30M parameters just for this matrix!",
  "Next set of Jupyter Notebook assignments\n\u2022 Not due till after break\n\u2022 will likely release in the next day or two\n\u00d8self-attention\n\u00d8multi-head self-attention\n\u00d8tokenization\n\u00d8decoding strategies\n67",
  "After the break\n\u2022 Image Transformers\n\u2022 Multimodal Transformers\n\u2022 RAG pattern\n\u2022 Training and Fine-Tuning \nTransformers\n\u2022 \u2026\n68\nFeedback\nChatGPT",
  "Improving Generative LLMs:\nCognitive Architectures and RAG\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "Topics\n\u2022 Generative LLM flow and how to evaluate\n\u2022 Improve LLM performance by prompting strategies\n\u2022 Improving with retrieval augmentation\n\u2022 Building more complex systems with LLMs: \u201dCognitive Architectures\u201d\n2",
  "Topics\n\u2022 Generative LLM flow and how to evaluate\n\u2022 Improve LLM performance by prompting strategies\n\u2022 Improving with retrieval augmentation\n\u2022 Building more complex systems with LLMs: \u201dCognitive Architectures\u201d\n3",
  "LLM Generative Flow\n4\nToken\nEncoding & \nLinear \nEmbedding\nToken \nDecoding\nTransformer\nToken \nSelection\nQuery\nResponse",
  "LLM Generative Flow\n\u2022 How do we evaluate the response?\n\u2022 How do we improve the response?\n5\nToken\nEncoding & \nLinear \nEmbedding\nToken \nDecoding\nTransformer\nToken \nSelection\nQuery\nResponse",
  "Generative LLM Evaluations\nEvaluate for\n\u2022 Accuracy (is it factual or hallucinated?)\n\u2022 Relevance (is it answering the question?)\n\u2022 Bias, Toxicity (Is it fair? Or even worse is it racist or toxic?)\n\u2022 Diversity of Response (does it always give same response? or equally \nuseful diverse responses?)\n6",
  "Ways to Evaluate\n\u2022 Find a benchmark that matches your task\n\u2022 HellaSwag (which evaluates how well an LLM can complete a sentence), \n\u2022 TruthfulQA (measuring truthfulness of model responses), and \n\u2022 MMLU (which measures how well the LLM can multitask), \n\u2022 WinoGrande (commonsense reasoning), \n\u2022 GSM8K, (arithmetic reasoning), etc.\n\u2022 Evaluate with a metric, e.g.  BLEU, METEOR, ROUGE, CIDEr, SPICE, etc.\n\u2022 Pros and Cons of each metric \n\u2022 Create your own evaluation prompt/response pairs \u2013 \n\u2022 need thousands!\n\u2022 Build an LLM to evaluate your LLM!\n7\nSee: https://arize.com/blog-course/llm-evaluation-the-definitive-guide/ for a nice overview",
  "Model vs System Evals\n8\nSee: https://arize.com/blog-course/llm-evaluation-the-definitive-guide/ for a nice overview \nUseful for choosing a model or deciding when \nto switch.\nUseful for prompt tuning and monitoring over time.",
  "9\nhttps://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard \nOpen LLM Leaderboard",
  "Crowd-Sourcing Evaluations\nUser Feedback\n10\nhttps://lmsys.org/ \nhttps://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard",
  "Topics\n\u2022 Generative LLM flow and how to evaluate\n\u2022 Improve LLM performance by prompting strategies\n\u2022 Improving with retrieval augmentation\n\u2022 Building more complex systems with LLMs: \u201dCognitive Architectures\u201d\n11",
  "12\nhttps://karpathy.ai/stateofgpt.pdf",
  "13\nhttps://karpathy.ai/stateofgpt.pdf",
  "14\nhttps://karpathy.ai/stateofgpt.pdf",
  "Chain of Thought Prompting (few shot)\nCoT enhances the reasoning abilities of LLMs by generating intermediate reasoning steps before arriving at a final \nanswer\n\u25cf\nEnables models to break down \ncomplex, multi-step problems into \nclearer, intermediate steps\n\u25cf\nOffers a clear view into the model's \nthought process, highlighting areas for \ndebugging and improvement.\n\u25cf\nSuitable for a broad range of reasoning \ntasks including math, commonsense, \nand symbolic reasoning\n\u25cf\nEasily activated in LLMs  with simple \nprompts, leveraging existing \ncapabilities without  customization.\nJ. Wei et al., \u201cChain-of-Thought Prompting Elicits Reasoning in \nLarge Language Models.\u201d arXiv, Jan. 10, 2023. doi: \n10.48550/arXiv.2201.11903.",
  "Chain of Thought Prompting (zero shot)\n\u2022 LLMs are decent zero-shot reasoners by simply adding \u201cLet\u2019s think \nstep by step\u201d before each answer\nT. Kojima, S. (Shane) Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, \u201cLarge Language Models are Zero-Shot \nReasoners,\u201d Advances in Neural Information Processing Systems, vol. 35, pp. 22199\u201322213, Dec. 2022. (link)",
  "17\nhttps://karpathy.ai/stateofgpt.pdf",
  "Tree of Thought\n\u2022 ToT allows LMs to perform deliberate decision making \nby considering multiple different reasoning paths and \nself-evaluating choices to decide the next course of \naction, as well as looking ahead or backtracking when \nnecessary to make global choices. \n\u2022 Our experiments show that ToT significantly enhances \nlanguage models\u2019 problem-solving abilities on three \nnovel tasks requiring non-trivial planning or search\n18\nS. Yao et al., \u201cTree of Thoughts: Deliberate Problem Solving with Large Language Models,\u201d Advances in Neural Information Processing Systems, vol. \n36, pp. 11809\u201311822, Dec. 2023.",
  "These techniques can improve LLM \nresponse generation, but how do we \ntailor to specific knowledge bases?\n19",
  "Topics\n\u2022 Generative LLM flow and how to evaluate\n\u2022 Improve LLM performance by prompting strategies\n\u2022 Improving with retrieval augmentation\n\u2022 Building more complex systems with LLMs: \u201dCognitive Architectures\u201d\n20",
  "Retrieval-Augmented Generation (RAG)\nRAG enhances LLMs by referencing external knowledge to generate relevant \nresponses.\n\u2022 Integrates external data into LLM text generation.\n\u2022 Reduces hallucination, improves response relevance.\n\u2022 Works with\n\u2022 Unstructured data (e.g. documents)\n\u2022 Structured data (e.g. SQL data)\n\u2022 Code (e.g. python)",
  "RAG Architecture\nTypical RAG application has two main components:\n\u2022 Loading and Indexing: \n\u2022 A pipeline for ingesting data from a source and indexing it\n\u2022 Usually happens offline\n\u2022 Retrieval and Generation: \n\u2022 Takes user query at run time and retrieves relevant data from the index and \npasses it to the model\n22\nhttps://python.langchain.com/docs/use_cases/question_answering/",
  "RAG \u2013 Loading and Indexing\n23\nhttps://python.langchain.com/docs/use_cases/question_answering/",
  "RAG \u2013 Load\nLoad the data, e.g.\n\u2022 PDFs\n\u2022 HTML\n\u2022 Plain text\n\u2022 Images, video, audio\n\u2022 Structured data (SQL, CSV/TSV, \u2026)\n\u2022 JSON\n\u2022 URLs\n\u2022 \u2026\n24\nhttps://python.langchain.com/docs/use_cases/question_answering/ \nSee for example: https://python.langchain.com/docs/modules/data_connection/document_loaders/",
  "RAG \u2013 Split\nBreak large documents into \nsmaller chunks.\nEasier to:\n\u2022 index\n\u2022 pass to model\n\u2022 search\n\u2022 fit into model\u2019s context window\n25\nhttps://python.langchain.com/docs/use_cases/question_answering/ \nSee for example: https://python.langchain.com/docs/modules/data_connection/document_transformers/",
  "RAG \u2013 Embed\n\u2022 Encode (e.g. with Byte Pair \nEncoding) and \n\u2022 Transform to embedding vectors \nwith the learned embedding \nmodel.\n26\nhttps://python.langchain.com/docs/use_cases/question_answering/ \nSee for example: https://python.langchain.com/docs/modules/data_connection/text_embedding/",
  "RAG \u2013 Store\n\u2022 Store the data in some kind of Vector Store\n\u2022 e.g. Chroma, FAISS, Lance, Pinecone, etc\u2026\n27\nhttps://python.langchain.com/docs/use_cases/question_answering/ \nSee for example: https://python.langchain.com/docs/modules/data_connection/vectorstores/",
  "RAG \u2013 Retrieval and Generation\n28\nhttps://python.langchain.com/docs/use_cases/question_answering/",
  "RAG \u2013 Retrieval\n29\nhttps://python.langchain.com/docs/modules/data_connection/vectorstores/",
  "RAG \u2013 Retrieval Similarity Measure\n30\nL2 Norm*: \ud835\udc51 = \u2211! \ud835\udc34! \u2212 \ud835\udc35!\n\"\nInner Product: \ud835\udc51 = 1 \u2212 \u2211!(\ud835\udc34!\u00d7\ud835\udc35!)\nCosine Similarity: 1 \u2212\n\u2211!(%!\u00d7'!)\n\u2211! %!\n\"\n\u2211!('!\n\"\nhttps://docs.trychroma.com/usage-guide#changing-the-distance-function \n* Default on Chroma Vector Database",
  "Is simple similarity measure \nbetween query and document \nthe best approach?\n31",
  "RAG \u2013 Other Query-Document Matching Approaches\n1. BERT and Variants for Query-Document Matching\nBERT:\nDevlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. \narXiv:1810.04805. This foundational paper introduces BERT and its methodology for language understanding, which has been widely applied to \ninformation retrieval tasks.\nApplication in Information Retrieval:\nNogueira, R., & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. This work explores how BERT can be used for re-ranking search results, \ndemonstrating its effectiveness in improving information retrieval systems. https://arxiv.org/abs/1901.04085 \n2. Fine-tuning for Specific Tasks\nFine-Tuning BERT for Search:\nMacAvaney, S., Cohan, A., & Goharian, N. (2019). CEDR: Contextualized Embeddings for Document Ranking. SIGIR.",
  "Fine-tuning for Specific Tasks\nFine-Tuning BERT for Search:\nMacAvaney, S., Cohan, A., & Goharian, N. (2019). CEDR: Contextualized Embeddings for Document Ranking. SIGIR. This paper discusses fine-tuning BERT \nwith contextual embeddings specifically for document ranking, providing insights into adapting Transformer models for search tasks. \nhttps://dl.acm.org/doi/abs/10.1145/3331184.3331317 \n3. Dual-encoder and Cross-encoder Architectures\nDual-Encoders for Efficient Retrieval:\nKarpukhin, V., O\u011fuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W. (2020). Dense Passage Retrieval for Open-Domain Question Answering. \nEMNLP. This paper introduces a method using dense vector representations for passages and questions to improve open-domain question answering.",
  "(2020). Dense Passage Retrieval for Open-Domain Question Answering. \nEMNLP. This paper introduces a method using dense vector representations for passages and questions to improve open-domain question answering. \nhttps://arxiv.org/abs/2004.04906 \nCross-Encoders for Detailed Similarity Scoring:\nHumeau, S., Shuster, K., Lachaux, M. A., & Weston, J. (2019). Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate \nMulti-sentence Scoring. arXiv:1905.01969. The poly-encoder architecture introduced here incorporates aspects of both dual and cross-encoders, \noffering a balance between speed and accuracy for matching tasks. https://arxiv.org/abs/1905.01969 \n4. Semantic Search Systems\nSemantic Search with Transformers:\nGuo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2020).",
  "Semantic Search Systems\nSemantic Search with Transformers:\nGuo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2020). A Deep Look into Neural Ranking Models for Information \nRetrieval. Information Processing & Management. This review covers deep learning approaches to information retrieval, including the use of \nTransformer models for understanding query intent and document relevance in a semantic search context. \nhttps://www.sciencedirect.com/science/article/pii/S0306457319302390 \n32",
  "Evaluating RAG-based LLMs\n33\nhttps://www.trulens.org/trulens_eval/getting_started/core_concepts/rag_triad/",
  "Evaluating RAG: Context Relevance\n\u2022 Is the content retrieved from the vector \ndatabase relevant to the query?\n\u2022 Irrelevant information will be likely \nintegrated into the response, contributing \nto hallucinations\n34\nhttps://www.trulens.org/trulens_eval/getting_started/core_concepts/rag_triad/",
  "Evaluating RAG: Groundedness\n\u2022 The context was provided to the LLM as \npart of the prompt\n\u2022 Did the LLM response incorporate the \ncontext appropriately?\n\u2022 Can we support each claim in the \nresponse from the context?\n35\nhttps://www.trulens.org/trulens_eval/getting_started/core_concepts/rag_triad/",
  "Evaluating RAG: Answer Relevance\n\u2022 Is the answer relevant to the original \nquestion?\n\u2022 Prompt is augmented with context.\n\u2022 Did the context cause the LLM to stray \naway from the question?\n36\nhttps://www.trulens.org/trulens_eval/getting_started/core_concepts/rag_triad/",
  "Growing ecosystem of  \ntools to do evaluation\n37",
  "Retrieval-Augmented Generation (RAG)\nRAG systems have evolved from Naive RAG to Advanced RAG and Modular RAG. This evolution \nhas occurred to address certain limitations around performance, cost, and efficiency.\nImage source (Kojima et al., 2022)\nhttps://www.promptingguide.ai/research/rag \nPre-Retrieval Improvements\n\u2022\nEnhance indexed data quality, optimize chunk size and \noverlap.\n\u2022\nRewrite user queries for better match in vector database.\n\u2022\nUse metadata and pronoun replacement to maintain context \nin chunks.\nRetrieval Enhancements\n\u2022\nExplore alternative search methods (e.g., full-text, graph-\nbased).\n\u2022\nExperiment with different embedding models for task \nsuitability.\n\u2022\nImplement hierarchical and recursive search for precision.\nPost-Retrieval Optimization\n\u2022\nRe-rank or score chunks for relevance; compress information \nfrom multiple chunks.\n\u2022\nEmploy smaller, faster models for specific steps to reduce \nlatency.\n\u2022\nParallelize intermediate steps and use caching for common \nqueries.\nBalancing Quality and Latency\n\u2022\nOpt for parallel processing, smaller models, and caching \nstrategies.",
  "\u2022\nEmploy smaller, faster models for specific steps to reduce \nlatency.\n\u2022\nParallelize intermediate steps and use caching for common \nqueries.\nBalancing Quality and Latency\n\u2022\nOpt for parallel processing, smaller models, and caching \nstrategies.\n\u2022\nTailor RAG approach based on the complexity of user \nqueries and the nature of tasks.",
  "Topics\n\u2022 Generative LLM flow and how to evaluate\n\u2022 Improve LLM performance by prompting strategies\n\u2022 Improving with retrieval augmentation\n\u2022 Building more complex systems with LLMs: \u201dCognitive Architectures\u201d\n39",
  "Cognitive Architecture*\n\u2022 Orchestration of components of an LLM application\n\u2022 There are two main components here: \n\u2022 (1) how is context provided to the application, \n\u2022 (2) how does the application \u201creason\u201d. Both of these components make up \nthe cognitive architecture of an application.\n40\nhttps://blog.langchain.dev/openais-bet-on-a-cognitive-architecture/ \n*Perhaps coined by Flo Crivello, creator of Lindy (no code customized AI assistants).",
  "Hierarchy of Cognitive Architectures\n41\nSingle LLM call\nChain of LLM calls\nUse LLMs as a router to choose \naction\nState machine with state \ntransitions managed by LLM\nAgents -- Actions taken on \nbehalf of user by LLMs",
  "Coordinator\nVector Databases\nCogArch SW Stack & Ecosystem\n42\nUI & App Frameworks\nCoordinators and API Wrappers\nAssistant\nGPT\nModel APIs\nAPIs\nGPT \nStore\nHosted Models\nHosted Models\nModel APIs\nServer Infrastructure/Hosting\nWork in Progress",
  "43\nhttps://karpathy.ai/stateofgpt.pdf",
  "LLMs Chains & Agents\n44\nhttps://github.com/assafelovic/gpt-researcher \nhttps://karpathy.ai/stateofgpt.pdf",
  "sweep.dev \ncognitive \narchitecture\n45\nhttps://docs.sweep.dev/blogs/sweeps-core-algo",
  "46\nhttps://www.latent.space/p/agents \nhttps://www.hopkinsmedicine.org/health/condition\ns-and-diseases/anatomy-of-the-brain \nAn ecosystem view",
  "Cognitive Architectures for Language Agents\n\u2022 \u201cdraw on rich history of cognitive science and symbolic artificial intelligence\u2026\u201d\n\u2022 CoALA: Cognitive Architectures for Language Agents\n47\nT. R. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths, \u201cCognitive Architectures for Language Agents.\u201d arXiv, Mar. 15, 2024. doi: 10.48550/arXiv.2309.02427.\nA: In natural language processing (NLP), an LLM \ntakes text as input and outputs text. \nB: Language agents place the LLM in a direct \nfeedback loop with the external environment by \ntransforming observations into text and using the \nLLM to choose actions. \nC: Cognitive language agents additionally use the \nLLM to manage the agent\u2019s internal state via \nprocesses such as learning and reasoning.",
  "Theories of Intelligence\n48\nThe Society of Mind is both the title of a 1986 book and \nthe name of a theory of natural intelligence as written and \ndeveloped by Marvin Minsky.[1]\nIn his book of the same name, Minsky constructs a model \nof human intelligence step by step, built up from the \ninteractions of simple parts called agents, which are \nthemselves mindless. He describes the \npostulated interactions as constituting a \"society of mind\", \nhence the title.[2]\nhttps://en.wikipedia.org/wiki/Society_of_Mind \nhttps://web.media.mit.edu/~minsky/papers/ConfocalMemoir.html\nhttps://patents.google.com/patent/US3013467A/en",
  "Next Time\n\u2022 LLM Finetuning\n\u2022 back to book sequence on\n\u2022 unsupervised learning\n\u2022 GANs\n\u2022 VAEs\n\u2022 Diffusion Models\n49\nFeedback\nChatGPT",
  "Parameter Efficient Fine Tuning \n(PEFT) of LLMs\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "Last time\u2026\nWe looked at ways of improving LLM performance via prompting \nstrategies such as\n\u2022 Chain of Thought, Tree of Thought\nand through\n\u2022 Retrieval augmentation\n2",
  "Today\u2026\nWe look at ways to improve model performance through finetuning the \nmodel\n\u2022 full model fine tuning\n\u2022 parameter efficient fine tuning\n3",
  "Topics\n\u2022 Full finetuning\n\u2022 Low rank adaptation\n\u2022 Prompt tuning\n4",
  "Topics\n\u2022 Full finetuning\n\u2022 Low rank adaptation\n\u2022 Prompt tuning\n5",
  "Model Training in the Transformer Era\n6\nLarge-scale pretraining \non generic internet-scale \ndata\nFine-tuning to \ndownstream tasks with \nsmaller dataset\nChatGPT",
  "Model Finetuning\n\u2022 Large foundation models are pre-trained on general tasks\n\u2022 Might not do as well on specialized tasks\n\u2022 Try prompt engineering and retrieval augmentation first\n\u2022 Good news: can fine tune model with much smaller dataset to adapt \nto downstream tasks\n\u2022 Fine tuned model is same size as original. \n\u2022 Resource Intensive: Can take very large memory and compute resources to \nfine tune\n\u2022 Storage Demands: If you have n downstream tasks, you will have n copies of \nyour large model.\n7",
  "Full Finetuning Example\n8\nText classification performance on the Stanford Natural Language Inference (SNLI) Corpus. \nOrdered pairs of sentences are classified by their logical relationship: either contradicted, \nentailed (implied), or neutral. Default fine-tuning parameters were used when not otherwise \nspecified.\nhttps://learn.microsoft.com/en-us/ai/playbook/technology-guidance/generative-ai/working-with-llms/fine-tuning",
  "\ud83e\udd17 HuggingFace \u2013 Fine-tune Pretrained Model Tutorials\n\u2022 Finetune for Sentiment Analysis Example (broken??)\n\u2022 https://huggingface.co/docs/transformers/training \n\u2022 Finetune bert-base-cased (109M params, FP32, 436MB) on Yelp review \ndataset (650K reviews, 323 MB)\n\u2022 Finetune for text classification example\n\u2022 https://github.com/huggingface/notebooks/blob/main/examples/text_classifi\ncation.ipynb \n\u2022 preprocess the data and fine-tune a pretrained model on any GLUE task\n\u2022 Finetune for question answering\n\u2022 https://github.com/huggingface/notebooks/blob/main/examples/question_a\nnswering.ipynb \n\u2022 preprocess the data and fine-tune a pretrained model on SQUAD\n9",
  "Model Finetuning Drawbacks\n\u2022 Fine tuned model is same size as original. \n\u2022 Resource Intensive: Can take very large memory and compute resources to \nfine tune\n\u2022 Storage Demands: If you have n downstream tasks, you will have n copies of \nyour large model\n10",
  "Model Finetuning Drawbacks\n\u2022 Fine tuned model is same size as original. \n\u2022 Resource Intensive: Can take very large memory and compute resources to \nfine tune\n\u2022 Storage Demands: If you have n downstream tasks, you will have n copies of \nyour large model\nSolution is to update aspects of the model, rather than entire model\n\u2022 Low rank adaptation of the weight updates -- LoRA\n\u2022 Train and concatenated soft prompts -- Prompt Tuning\n11",
  "Topics\n\u2022 Full finetuning\n\u2022 Low rank adaptation\n\u2022 Prompt tuning\n12",
  "Low Rank Adaptation\n\u2022 Deploying independent instances of \ndownstream fine-tuned models can be \nprohibitive (e.g. GPT3, 175B params, \n700GB@fp32)\n\u2022 Instead, freeze the pre-trained model and \ninject trainable rank decomposition matrices \ninto each layer\n\u2022 Reduce trainable parameters by 10,000x!!\n\u2022 On-par or better than finetuning on RoBERTa, \nDeBERTa, GPT-2 and GPT-3\n13\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021. http://arxiv.org/abs/2106.09685",
  "Low Rank Adaptation\n\u2022 Aghajanyan et al show that pretrained language \nmodels have a low \u201cintrinsic dimension\u201d\n\u2022 Updates to weight matrices likely have a low \n\u201cintrinsic rank\u201d during training\n\u2022 Found that even very low rank (e.g. r=1 or2) with \nGPT-3 175B is effective where full rank \n(embedding dimension) is 12,288\n14\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021. http://arxiv.org/abs/2106.09685\nA. Aghajanyan et al., \u201cIntrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning\u201d. arXiv:2012.13255 [cs], \nDecember 2020. URL http://arxiv.org/abs/2012.13255.",
  "Reminder: Rank of a Matrix\n\u2022 The number of linearly independent rows or columns of a matrix\n\u2022 Determines the dimension of the vector space spanned by the \ncolumn vectors\n\u2022 A measure of \u201cdimensionality\u201d\n15",
  "LoRA: Method\nSay you have pre-trained weights, \n \n \n\ud835\udc4a! \u2208 \u211d\"\u00d7$ \nRepresent update with a low rank decomposition\n \n \n\ud835\udc4a! + \u2206\ud835\udc4a = \ud835\udc4a! + \ud835\udc35\ud835\udc34 , \nwhere \ud835\udc35 \u2208 \u211d\"\u00d7%, \ud835\udc34 \u2208 \u211d%\u00d7$ and the rank \ud835\udc5f \u226a\nmin \ud835\udc51, \ud835\udc58 , is much less than the full rank.\nFor updates, \n\u210e = \ud835\udc4a! + \u2206\ud835\udc4a \ud835\udc65 = \ud835\udc4a!\ud835\udc65 + \u2206\ud835\udc4a\ud835\udc65 = \ud835\udc4a!\ud835\udc65 + \ud835\udc35\ud835\udc34\ud835\udc65\nInitialize A to random gaussian and B to zero\n16\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "LoRA: Method\nLoRA can be viewed as a generalization of full \nfinetuning, since using full rank = full finetuning\nUpdates: \n\u210e = \ud835\udc4a! + \u2206\ud835\udc4a \ud835\udc65 = \ud835\udc4a!\ud835\udc65 + \u2206\ud835\udc4a\ud835\udc65 = \ud835\udc4a!\ud835\udc65 + \ud835\udc35\ud835\udc34\ud835\udc65\nGenerally only applied to \ud835\udc4a& and \ud835\udc4a' matrices.\n17\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "LoRA Results / Comparisons\n18\nGLUE benchmark \u2013 measure across 9 language tasks\nBitFit \u2013 train only the bias vectors\nAdpt \u2013 Inserts adaptation layer between self-attention and MLP module\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685\n\u2020 indicates runs configured in a setup similar to Houlsby et al. (2019) for a fair comparison.",
  "LoRA Results / Comparisons\n19\nGPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG \nChallenge. For all metrics, higher is better. LoRA outperforms several baselines with \ncomparable or fewer trainable parameters. Confidence intervals are shown for \nexperiments we ran. * indicates numbers published in prior works.\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "Understanding the Low-Rank Updates\n1. Given a parameter budget constraint, which subset of weight \nmatrices in a pre-trained Transformer should we adapt to maximize \ndownstream performance? \n2. Is the \u201coptimal\u201d adaptation matrix \u2206W really rank-deficient? If so, \nwhat is a good rank to use in practice? \n20\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "1) Which weight matrices to target?\n21\nValidation accuracy on WikiSQL and MultiNLI after applying LoRA to different types of \nattention weights in GPT-3, given the same number of trainable parameters. Adapting \nboth Wq and Wv gives the best performance overall. We find the standard deviation \nacross random seeds to be consistent for a given dataset, which we report in the first \ncolumn.\nRank of 16 on 2 matrices or even 4 on 4 matrices is sufficient.\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "2) What is the optimal rank?\n22\n\u201cValidation accuracy on WikiSQL and MultiNLI with different rank r. To our \nsurprise, a rank as small as one suffices for adapting both Wq and Wv on \nthese datasets while training Wq alone needs a larger r.\u201d\nE. J. Hu et al., \u201cLoRA: Low-Rank Adaptation of Large Language Models.\u201d arXiv, Oct. 16, 2021.  http://arxiv.org/abs/2106.09685",
  "An alternative to adapting model \nupdates is to train a set of soft \nprompt tokens\n23",
  "Topics\n\u2022 Full finetuning\n\u2022 Low rank adaptation\n\u2022 Prompt tuning\n24",
  "Prompt Tuning\n\u2022 Prompt engineering can improve LLM performance but is very brittle\n\u2022 small change in words can have drastic impact on performance\n\u2022 show example\n\u2022 Turns out you can learn a set of \u201csoft tokens\u201d that are prepended to \nthe actual prompt which improves LLM performance\n\u2022 Makes it much more robust to small changes\n25",
  "Prompt Tuning\n\u2022 P-Tuning:  employ trainable continuous prompt embeddings in \nconcatenation with discrete prompts\n26\nX. Liu et al., \u201cGPT Understands, Too.\u201d arXiv, Oct. 25, 2023. http://arxiv.org/abs/2103.10385\nResults are precision@1 on LAMA-TREx P17 with BERT-\nbase-cased.\nInstability of discrete prompts.",
  "Prompt Tuning\n\u2022 employs trainable continuous prompt embeddings in concatenation \nwith discrete prompts given a discrete prompt as the input, \n\u2022 P-Tuning concatenates continuous prompt embeddings with the \ndiscrete prompt tokens and feeds them as the input to the language \nmodel. \n\u2022 The continuous prompts are updated by backpropagation to optimize \nthe task objective.\n27\nX. Liu et al., \u201cGPT Understands, Too.\u201d arXiv, Oct. 25, 2023. http://arxiv.org/abs/2103.10385\nIncorporate a certain degree of learnability into the input, which may learn to offset \nthe effects of minor changes in discrete prompts to improve training stability",
  "p-tuning methodology\n\u2022 Let [D!] be a discrete prompt token. \n\u2022 Each prompt can be described as a template \n \n\ud835\udc47 = { D\":! , x, D !$% :& , y, D(&$%):) } \nwhich could organize the labeled data (including the inputs x and the label y) into a \nsequence of text tokens, such that the task could be reformulated as filling in the blanks \nof the input text.\n\u2022 \u201cThe capital of [INPUT] is [LABEL].\u201d \n\u2022 labeled data \u201c(Britain, London)\u201d\n\u2022 Both discrete prompts and discrete data are together mapped into input embeddings: \n \n{e \ud835\udc37\" \u2026e \ud835\udc37! , e \ud835\udc65\" , \u2026, e \ud835\udc65* , \u2026, e \ud835\udc37) } \nthrough the pretrained embedding layer, where \ud835\udc52 \u2208 \u211d \ud835\udcb1 \u00d7-.\n\u2022 we propose P-Tuning that uses continuous prompt embeddings\n28\nX. Liu et al., \u201cGPT Understands, Too.\u201d arXiv, Oct. 25, 2023. http://arxiv.org/abs/2103.10385",
  "p-tuning methodology\n\u2022 Proposes continuous prompt embeddings\n\u2022 Let [\ud835\udc43!] be the ith continuous prompt \nembedding.\n\u2022 The prompt template for P-Tuning is as \nfollows:\n\ud835\udc47 = { P\":! , x, P !$% :& , y, P(&$%):) }  \n\u2022 P-Tuning leverages an extra embedding \nfunction \ud835\udc53: P! \u2192 \u210e! to map the template to\n\u210e\", \u2026 , \u210e!, \ud835\udc52 \ud835\udc65 , \u210e!$%, \u2026 , \u210e&, \ud835\udc52 \ud835\udc66 , \u210e!$%, \u2026 , \u210e)  \n\u2022 Finally, we update the embeddings {\ud835\udc43!}!*%\n)\n to \noptimize a task loss function.\n29\nLSTM or MLP to model the \ndependency between \ncontinuous prompt \nembeddings\nX. Liu et al., \u201cGPT Understands, Too.\u201d arXiv, Oct. 25, 2023. http://arxiv.org/abs/2103.10385",
  "Discrete Prompt Searching vs P-Tuning\n30\nX. Liu et al., \u201cGPT Understands, Too.\u201d arXiv, Oct. 25, 2023. http://arxiv.org/abs/2103.10385",
  "Additional  References\n\u2022 X. Liu et al., \u201cP-Tuning v2: Prompt Tuning Can Be Comparable to Fine-\ntuning Universally Across Scales and Tasks.\u201d arXiv, Mar. 20, 2022. \nhttp://arxiv.org/abs/2110.07602\n\u2022 B. Lester, R. Al-Rfou, and N. Constant, \u201cThe Power of Scale for \nParameter-Efficient Prompt Tuning.\u201d arXiv, Sep. 02, 2021. \nhttp://arxiv.org/abs/2104.08691\n31",
  "\ud83e\udd17 HuggingFace PEFT Resources\n32",
  "HuggingFace PEFT\n\u2022 Blog: \ud83e\udd17 PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models \non Low-Resource Hardware\n\u2022 Library: https://github.com/huggingface/peft\n33",
  "\ud83e\udd17 HuggingFace PEFT Library\n34\nPrepare a model for training with PEFT method\nLoad a PEFT model for inference\nhttps://github.com/huggingface/peft?tab=readme-ov-file#quickstart \nCreate PEFT config\nGet the PEFT model based on config\nGet the PEFT model\nUse it like a regular model",
  "\ud83e\udd17 HuggingFace PEFT Library\n35\nHigh performance on consumer hardware\nConsider the memory requirements for training the following \nmodels on the ought/raft/twitter_complaints dataset with an A100 \n80GB GPU with more than 64GB of CPU RAM.",
  "\ud83e\udd17 HuggingFace PEFT Library\n35\nHigh performance on consumer hardware\nConsider the memory requirements for training the following \nmodels on the ought/raft/twitter_complaints dataset with an A100 \n80GB GPU with more than 64GB of CPU RAM.\nModel\nFull Finetuning\nPEFT-LoRA PyTorch\nPEFT-LoRA DeepSpeed with \nCPU Offloading\nbigscience/T0_3B (3B params)\n47.14GB GPU / 2.96GB CPU\n14.4GB GPU / 2.96GB CPU\n9.8GB GPU / 17.8GB CPU\nbigscience/mt0-xxl (12B params)\nOOM GPU\n56GB GPU / 3GB CPU\n22GB GPU / 52GB CPU\nbigscience/bloomz-7b1 (7B params)\nOOM GPU\n32GB GPU / 3.8GB CPU\n18.1GB GPU / 35GB CPU\nhttps://github.com/huggingface/peft?tab=readme-ov-file#high-performance-on-consumer-hardware \nSubmission Name\nAccuracy\nHuman baseline (crowdsourced)\n0.897\nFlan-T5 (fully finetuned)\n0.892\nlora-t0-3b (LoRA)\n0.863",
  "\ud83e\udd17 HuggingFace PEFT Library\n36\nDiffusers\nModel\nFull Finetuning\nPEFT-LoRA\nPEFT-LoRA with Gradient \nCheckpointing\nCompVis/stable-diffusion-v1-4\n27.5GB GPU / 3.97GB CPU\n15.5GB GPU / 3.84GB CPU\n8.12GB GPU / 3.77GB CPU\nhttps://github.com/huggingface/peft?tab=readme-ov-file#diffusers \nTake a look at the examples/lora_dreambooth/train_dreambooth.py training script \nto try training your own Stable Diffusion model with LoRA, and play around with \nthe smangrul/peft-lora-sd-dreambooth Space which is running on a T4 instance. \nLearn more about the PEFT integration in Diffusers in this tutorial.",
  "Next Time\n\u2022 back to book sequence on\n\u2022 unsupervised learning\n\u2022 GANs\n\u2022 VAEs\n\u2022 Diffusion Models\n\u2022 graph neural nets\n\u2022 etc.\n37\nFeedback\nChatGPT",
  "Unsupervised Learning\n&\nGenerative Adversarial Networks\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos \u2013 Understanding Deep Learning, Other Content Cited\n1",
  "April Dates\nSunday\nMonday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\nApril 1\n2\n3\n4\nGANs\n5\n6\n7\n8\n9\nVAEs\n10\nDiscussion\n11\nDiffusion Models\n12\n13\n14\n15\n16\nGraph Neural \nNets\n17\nDiscussion\n18\nReinforcement \nLearning\n19\n20\n21\n22\n23\nTBD/Overflow\n24\nDiscussion\n25\n\u2605 Project \nPresentations 1 \u2605 \n26\n27\n28\n29\n30\n\u2605 Project \nPresentations 2 \u2605 \nMay 1\nDiscussion??\n2\nStudy Period\n3\nStudy Period\n4\n5\n6\nFinal Exams\n7\n8\n9\n10\nFinal report\n& Repo **\n11\n2\n** Might be earlier. Depends on when grades are due.",
  "Project Presentations\nApril 25 \u2013 75 minutes\n\u2022 Slot 1\n\u2022 Slot 2\n\u2022 Slot 3\n\u2022 Slot 4\n\u2022 Slot 5\n\u2022 Slot 6\n\u2022 Slot 7\n\u2022 Slot 8\nApril 30 \u2013 75 minutes\n\u2022 Slot 9\n\u2022 Slot 10\n\u2022 Slot 11\n\u2022 Slot 12\n\u2022 Slot 13\n\u2022 Slot 14\n\u2022 Slot 15\n\u2022 Slot 16\n\u2022 Slot 17\nFormat:\n\u2264 3 minutes screencast/video\n\u2264 2 minutes additional presentation\n~2 minutes Q&A\nLooking for volunteers for April 25.\nThen I will randomly draw remainder of April 25 spots.\n3",
  "Up to this point\u2026\n\u2022 we looked at discriminative supervised learning models \n\u2022 Exceptions:\n\u2022 Transformers pretrained unsupervised (then usually finetuned \nsupervised)\n\u2022 and the Transformer decoder which generated text\nSupervised\nUnsupervised\nDiscriminative\nGenerative\n4",
  "Supervised vs. Self/Unsupervised Learning\nSupervised Learning\nData: (\ud835\udc65, \ud835\udc66)\n\ud835\udc65 is data, \ud835\udc66 is a label\nGoal: Learn function to map\n\ud835\udc65 \u2192 \ud835\udc66\nApplications: Classification, regression, \nobject detection, semantic \nsegmentation, etc.\nSelf/Unsupervised Learning\nData: \ud835\udc65\n\ud835\udc65 is data, no labels!\nGoal: Learn the hidden or underlying \nstructure of the data.\nApplications: Clustering, dimensionality \nreduction, compression, find outliers, \ngenerating new examples, denoising, \ninterpolating between data points, etc.\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n5",
  "Supervised vs. Self/Unsupervised Learning\nSupervised Learning\nData: (\ud835\udc65, \ud835\udc66)\n\ud835\udc65 is data, \ud835\udc66 is a label\nGoal: Learn function to map\n\ud835\udc65 \u2192 \ud835\udc66\nApplications: Classification, regression, \nobject detection, semantic \nsegmentation, etc.\nSelf/Unsupervised Learning\nData: \ud835\udc65\n\ud835\udc65 is data, no labels! Or labels part of the \ndata\nGoal: Learn the hidden or underlying \nstructure of the data.\nApplications: Clustering, dimensionality \nreduction, compression, find outliers, \ngenerating new examples, denoising, \ninterpolating between data points, etc.\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n6",
  "We\u2019ll consider two attributes of models\n\u2022 Probabilistic Models\n\u2022 Latent Variable Models\n7",
  "Probabilistic models\n\u2022 Maximize log likelihood of training data\n\u2022 Find the parameters, \ud835\udf19, of some parametric probability distribution \nso that the training data is most likely under that distribution\n!\ud835\udf19 = argmax\n!\n)\n\"#$\n%\nlog[Pr \ud835\udc65\" \ud835\udf19)\n8",
  "Latent variable models\nLatent variable models map a random \u201clatent\u201d variable to create a new data sample\n9",
  "Generative Modeling\nGoal: Take as input training samples from some distribution \nand learn a model that represents that distribution\nProbability Density Estimation\nSample Generations\nHow can we learn \ud835\udc43!\"#$% \ud835\udc65  similar to \ud835\udc43#&'&(\ud835\udc65)?\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n10",
  "Types of unsupervised generative model\n\u2022 Generative adversarial networks (GANs) (LV)\n\u2022 Variational auto-encoders (VAEs) (P, LV)\n\u2022 Diffusion models (P, LV)\n\u2022 Normalizing flows (P, LV)\n\u2022 Energy models (P)\n\u2022 Autoregressive models (P)\n11",
  "Decoder model: GPT3\n\u2022 One job:  predict the next word in a sequence\n\u2022 More formally builds an autoregressive probability model\n\u2022 Doesn\u2019t use latent variables, but is probabilistic and generative\n\u2022 Can generate new examples\n\u2022 Can assign a probability to new data \n<latexit sha1_base64\n=\"LwyxvEQlnerT+QB7n9FcqC2vtNg=\">AW0X\niclZhJb9tGFICZdEvTzWlRX3ohagRICkewgnS5B\nEjsOJudWI4t27HlCENqSDEeDmlyaMthBRS9if1\nl/TYa/sn+oakNOF740MF2Bq97+Msb2a4eamIcr\nWy8teVqx98+NHn1z79Ppn3/x",
  "l/TYa/sn+oakNOF740MF2Bq97+Msb2a4eamIcr\nWy8teVqx98+NHn1z79Ppn3/x5VcLN7ey5Mi8\n3nfT0SHXgs5yKSvK8iJfhBmnEWe4Lveydrmu+f\n8SyPErmrLlJ+HLNQRkHkMwWh4ULf7W31LDsTp\nfh/93p8kCMEpW78OPl9LZ7f45vD9IsGQ1Lef/u9\nA2wOi6nv1bUHCXvgDtcWFrprFQflxa6TWHJaT69\n4Y1vR4NR4hcxl8oXLM+Pui",
  "9\nA2wOi6nv1bUHCXvgDtcWFrprFQflxa6TWHJaT69\n4Y1vR4NR4hcxl8oXLM+PuiupOi5ZpiJf8On1QZ\nHzlPknLORHUJQs5vlxWY1/6t6EyMgNkgz+pHKr6\nPtHlCzO84vYAzNmapxjpoM2dlSo4JfjMpJpobj0\n64aCQrgqcXUy3VGUcV+JCygwP4ugr64/ZhnzFa\nT8+kDycz+JYyZH5WB1fXtaDjweRrLkp0WV/um07\naxXDofiZcbqs915L",
  "/ZhnzFa\nT8+kDycz+JYyZH5WB1fXtaDjweRrLkp0WV/um07\naxXDofiZcbqs915LZHicfSOk0oqRVdyicDaVn\nyTtjBIOIAog4nIJE8hzp1frzA7SIKy0ABu4lE+\nhc4L6akql4iHkpKUdEg0KqeCTlrVGLJjKuKXsg\nOK6N10NuMpgFqCr8MXRHOykTE5nxyk+UVlc5jq\nGW8iYDHnVBAzZ0KPqG3IQg41G9ZL7H1ismTJn\nFJWnU10xFk7WZtR",
  "yk+UVlc5jq\nGW8iYDHnVBAzZ0KPqG3IQg41G9ZL7H1ismTJn\nFJWnU10xFk7WZtR2U0L3LUdqoIsmARhm2riBLw\nMlhxGIGW7KQxhw7OqIXY0kViOyMHtZ4rXbTnU\nEr81JCvul7a2XJP1nDGVEB2D36e+ISZ+39bVkbr\nuz5JxVvi7wiTuGyWofwrKwHtasERhVE5tSs8oVM\nm2IJQl521T98ai8jRqD1AH8KYrskgG72nLVQm\nWrA4PlmGoWSH",
  "ERhVE5tSs8oVM\nm2IJQl521T98ai8jRqD1AH8KYrskgG72nLVQm\nWrA4PlmGoWSH40Z3Oj3xyXK7obaP/kWxCRXmR2i\nrS4f9R0QguR3h9QRPXiLQ5EGgmrxEwPkdTR3L8\nMLWkWruoBJiJ1gbZ/FMr2MVUEdzaJUV8hoOu\nFbxZJNMlB0JZ1QMvwDRdWywLy0SD9eoy+SPIi4+\nTkh9YzRCpdnxazSF+s2idUoYX2eYOL+VFQhovD\nGb/kcA9l1",
  "y0SD9eoy+SPIi4+\nTkh9YzRCpdnxazSF+s2idUoYX2eYOL+VFQhovD\nGb/kcA9l1Kvz6SWFHLEMJXOip3TyZpAr2GK23V9\nNeV20WiE/3Wjag37B7BS+z0+HG3g+QmJR6C64E\n7GWpcglqU9qGu+XN/vWbnx5geytEOLazcFqbfp\npd2uJf0gJ9uWnq7STxiUegupoeUo9YlvagLns\neN2jsLh2U5B6Z3m02hZ3bqLlH+yOuWL6NikRI3\n3bl4hB",
  "UegupoeUo9YlvagLns\neN2jsLh2U5B6Z3m02hZ3bqLlH+yOuWL6NikRI3\n3bl4hBHcKioqKyiknMQyTWISzGRduC31jZieDi\n0bqEBZ7edTWdABLIy7wEOoQFust3DabGFY3Leq\nmXWUiHSOzDmHxCYvxqOsQFkMqhlbxhKUpEusQye\nMY53FM85hiKbVJeEZSy4yQJWVbUNk4aUs6gKUJ\nam1iaQx6IBKJGmyCWM7pysutK0+iVSzpKu7bGu5\nf",
  "ZSy4yQJWVbUNk4aUs6gKUJ\nam1iaQx6IBKJGmyCWM7pysutK0+iVSzpKu7bGu5\nf0rBiqEIdwNIW2WPuYMu6yTycYrjNsiU5jZCV0\ngT2sNOjzuzuzwtKcifnBReGXlB6bug5pfuG7lOa\nGUqeCLzglaHk6cQLzgw9o3TP0D1KC0MLSvuG9ik\nNDA0ofWzoY0p9Q31K1wxdo1QZSu5I4Ypg6C6lY\n0PHlB4YekDpa0NfU/rU0KeUHhp6SOk7Q9R+",
  "0p9Q31K1wxdo1QZSu5I4Ypg6C6lY\n0PHlB4YekDpa0NfU/rU0KeUHhp6SOk7Q9R+tDQ\nh5QyQxml64auU8oNJa8OvGDV0FVKPUPJsx/sNUN\n7lKaGpQ+MvQRpSNDyVMxXM8MJbc3cGE0VFD6z\nNBnlEaGkuc3L3h6AtKY0NjSp8b+pzSt4a+pfSJ\noU8oDQ0l7wbg7sTQHUrNW6Ayp3Tb0G1KTw09tb8\nX4PNp9GwLc8tUsEVpYmhC6Yah5EkBbi",
  "l7wbg7sTQHUrNW6Ayp3Tb0G1KTw09tb8\nX4PNp9GwLc8tUsEVpYmhC6Yah5EkBbiUMPSH3k\n4Fszmqzt03kvBbIObewJuOzo0nOAznFtacnWZH\nk/NTIOd8TLq+vjd/kQIphTP9cGpi9/C0sLe3U7\n3p8697XtLD1abN7TXnO+c751bTtf52XngPHV6T\nt/xnT+dv51/nH8XdxYvFn9b/L1Wr15pjvnGaX0W\n/gPpjvwUQ=</latexit>\nPr(t1,",
  "nT+dv51/nH8XdxYvFn9b/L1Wr15pjvnGaX0W\n/gPpjvwUQ=</latexit>\nPr(t1, t2, . . . tN) = Pr(t1)\nN\nY\nn=2\nPr(tn|t1 . . . tn\u22121)\n12",
  "Amini et al, \u201cUncovering and Mitigating Algorithmic Bias through Learned Latent Structure,\u201d 2019\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n13",
  "A. Amini et al, \u201cVariational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing,\u201d 2018\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n14",
  "More outlier examples\nYouTube Video, Feb. 2020 -- https://www.youtube.com/watch?v=hx7BXih7zx8&t=514s \n15",
  "Why generative models? \nimage, video and audio creation\nWrite a short pop song about \nstudents wanting to learn about \nneural networks and do great \nthings with them.\nA teenage superhero fighting crime in an urban \nsetting shown in the style of claymation.\n16",
  "Encoder\nDecoder\n\ud835\udc67\n\ud835\udc65\n)\ud835\udc65\nLatent Variable Models\nDiscriminator\nGenerator\nAutoencoders and \nVariational Autoencoders (VAEs)\nGenerative Adversarial Networks\n\ud835\udc67\n\ud835\udc65\n)\ud835\udc65\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n17",
  "Generative = can generate \nnew examples\nProbabilistic = can assign \nprobability to data examples\n18",
  "19",
  "20",
  "What makes a good model?\n\u2022 E\ufb00icient sampling: Generating samples from the model should be computation-\nally inexpensive and take advantage of the parallelism of modern hardware.\n\u2022 High-quality sampling: The samples should be indistinguishable from the real\ndata that the model was trained with.\n\u2022 Coverage: Samples should represent the entire training distribution. It is insufficient to \nonly generate samples that all look like a subset of the training data.\n\u2022 Well-behaved latent space: Every latent variable z should correspond to a\nplausible data example x and smooth changes in z should correspond to smooth\nchanges in x.\n\u2022 Interpretable latent space: Manipulating each dimension of z should correspond to \nchanging an interpretable property of the data. For example, in a model\nof language, it might change the topic, tense or degree of verbosity.\n\u2022 E\ufb00icient likelihood computation: If the model is probabilistic, we would like\nto be able to calculate the probability of new examples e\ufb00iciently and accurately\n21",
  "Do we have good models?\nHow to measure performance within or between categories?  \n\u2022\nOpen research area.  \n22",
  "\u201cGenerative adversarial networks\u201d, Goodfellow et al\nIan Goodfellow\nPhD ML, U de Montr\u00e9al 2014\n\u2022\nGoogle (TensorFlow, Google \nBrain)\n\u2022\nOpenAI\n\u2022\nGoogle Staff/Sr. Staff Research \nScientist\n\u2022\nApple Director of ML\n\u2022\nGoogle Deep Mind, Research \nScientist\n23\nThe GAN Zoo (Github)",
  "General Idea of GANs\n\u2022 Don\u2019t try to build a probability model directly\n\u2022 Learn a transformation from a sample of noise \nto look similar to training data distribution\nGenerator\n\ud835\udc67\n)\ud835\udc65\nnoise\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\n24",
  "Generative Adversarial Networks\nDiscriminator\nGenerator\n\ud835\udc66\n\ud835\udc67\n\ud835\udc65\n)\ud835\udc65\nTrain a generative model to try to fool a \u201cdiscriminator\u201d model.\nfake\nreal\nThe generator turns noise into \nan imitation of the data to try \nto trick the discriminator.\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\nThe discriminator tries to \nidentify real data from fakes \ncreated by the generator.\nfake\nreal\n25",
  "GAN example \n\u2022 We take examples from a real \ndistribution (e.g. shifted \nstandard gaussian)\n\u2022 We generate synthesized \nsamples, \ud835\udc67!, from a standard \ngaussian and shift by \ud835\udf03.\n\u2022 Train a classifier on the data\n<latexit sha1_ba\nse64=\"k1AyaOJtEUWbdMJnVPdiOSDFh\nY=\">AWp3iclZhbU9w2FICd9JamN9JO\nemLp0xm2oQy0EkvL5lJIOQGKRBYIGHJ\njuyVvQJZNrYMC579Ef01fW1/Rv9Nj2zv\nKj5HPJSZLXn+6zLkWRrHWRSFHp5+d8\nbNz/48KOP7n16e3Pv/iy6/m7ny9X6R",
  "zv\nKj5HPJSZLXn+6zLkWRrHWRSFHp5+d8\nbNz/48KOP7n16e3Pv/iy6/m7ny9X6R\nlHvJemMo0PwxYwaVQvKeFlvwyzlLAsk\nPgtM1w/OeV6IVO3py4wfJyxWIhIh0xA\nazN3x4OTd9W9yUO/nwTpuIonR1eDk8\nW+HnHNjv2HPny73wbzC0sLy3Xfz4trL\nSFBa/92x7c+XbYH6ZhmXClQ8mK4mhlOd\nPHFcu1CWf3O6XBc9YeMpifgRFxRJeHF",
  "SFBa/92x7c+XbYH6ZhmXClQ8mK4mhlOd\nPHFcu1CWf3O6XBc9YeMpifgRFxRJeHF\nf1qCb+XYgM/SjN4Z/Sfh19/4qKJUVxmQ\nRgJkyPCsxM0MWOSh39flwJlZWaq7BpK\nCqlr1PfpMgfipyHWl5CgYW5gL764YjlL\nNSQyNt9xS/CNEmYGlb91fWdSdUPeCxUx\nc/KOqmTSdZrx0OxeuM1Rd7s1qE5om4\nqSWjGVXCPweFJVfClewkBwAGKJE5A",
  "xUx\nc/KOqmTSdZrx0OxeuM1Rd7s1qE5om4\nqSWjGVXCPweFJVfClewkBwAGKJE5AqX\nkCdJj9B5K8gCotIAq6aVdEH4/WEVK0\njyEnHe0t0aCQST7uWGvEgqlMOsouKL5/\n1zeA6xmAboKHxzNwW7G1GR6neZjnSdV\nYWK4hZypmNdNwJBDJs2IuoYqpYRLw471\nB7ZeM3XaJi7N6q7mJoKsvbzr6JzmRQ2\n7Th1BFizCuGvVEWRJ2PJDljDIcls",
  "471\nB7ZeM3XaJi7N6q7mJoKsvbzr6JzmRQ2\n7Th1BFizCuGvVEWRJ2PJDljDIclsewIA\nT30TcqlBYFWRhbudp0G07MxG8NscZ7Je\nut16R9J8zlBETgN1nPgVTIe/qa+nM9qf\nJOa9U+BjfwST1b2E5XEzrGkjMKo2NqF\nmnStk0mxBKE8vuqbpjUPlmegO0ATwpi\ntzoaL3tMW6BEvWhPuLMNS8lPzop6Vf+P\ni4WjbxvxHsgkVFWXmqsiE/0d",
  "0ATwpi\ntzoaL3tMW6BEvWhPuLMNS8lPzop6Vf+P\ni4WjbxvxHsgkVFWXmqsiE/0dFQ3jI4P\nUFETx5qUSTB4F68lIJ93c0dSzHC9tE6r\nmDglBMCn2Jtr+IVfeaOoI7myaorxAw9\ncInEwpNchR1ZRMwMnzC49KxgEI0yLAZY\nyjTosw5ufmh9QyRWje3xVyYh1X3hiqN0\nL1vcDm7CsrwcDjn1weoIwGT6DtFRDl\nqNkjs2Ujt/1Cw1bzLX76yl",
  "Yh1X3hiqN0\nL1vcDm7CsrwcDjn1weoIwGT6DtFRDl\nqNkjs2Ujt/1Cw1bzLX76ylvik4r5mcb\nXvQL5idMgz52WADz0dMLOpIVBecT5x1\nSWI52oO6Zsv1/Z5VG+/ukaUdO1y3KUm9\nbS/dtsO9pgf8bNPR203iEYs6EtXV9pB6\nxHK0B3W587jpGoXDdZuS1DvNo9N2uDMT\nLf9ozxBzTEplUNz7EtlvwlhUVNRO8U0\n4TESmxAWk7JrwXes7Ap",
  "1DvNo9N2uDMT\nLf9ozxBzTEplUNz7EtlvwlhUVNRO8U0\n4TESmxAWk7JrwXes7Ap4eHStJoTF7UJ\n0NRPA0pBLPIQmhMVmC3fNobVTYe6Va\nZzEbIbEJYfMYSPOomhMWYirFTPGVZhsQ\nmRPI4wnkc0TxmWMpcEp6RzDEjZEm5FlQ\n+SruSCWBpjFobOxqDHshUoQbIJYLuv\nIK58pTaBUruop7roZ71zSsGarQBLC0Rf\naY39ybrIApxiOWa4kZ",
  "shUoQbIJYLuv\nIK58pTaBUruop7roZ71zSsGarQBLC0Rf\naY39ybrIApxiOWa4kZwJZGU3gNna2qT\nM9/QVRU5yQXRp6SWlF5ZeUHpg6QGlua\nXkF0EQvbaU/DoJonNLzyndt3Sf0tLSkt\nKepT1KI0sjSp9a+pTS0NKQ0jVL1yjVl\npITKTwRLN2jdGTpiNJDSw8pfWPpG0qfW\n/qc0reWvqX0ytIrSh9b+phSZimjdN3Sd\nUq5peTVQRCtWrpK",
  "JDSw8pfWPpG0qfW\n/qc0reWvqX0ytIrSh9b+phSZimjdN3Sd\nUq5peTVQRCtWrpKaWAp+e0He83SbUozS\nzNKn1j6hNKhpeRXMTzPLCXHG3gwWiop\nfWHpC0qFpeT3WxC9svQVpYmlCaUvLX1J\n6YmlJ5Q+s/QZpbGl5N0AnE4s3aXUvgWq\nCkp3LN2h9MzSM/d7AT6bxsC1MLdsBVuU\npamlG5YSn4pwFHC0lNynoxUe1ebvm0i\n97VIzbiDtRm",
  "M/d7AT6bxsC1MLdsBVuU\npamlG5YSn4pwFHC0lNynoxUe1ebvm0i\n97VIzbiDtRmfXk1yHqkZd7D27jS9mty\nfIjXjI9L19f3ZixRIKdzpB3MLK/gtLC3\ns/7y08uvSg50HC49W2ze0t7zvO+9H7w\nV7zfvkfc2/Z6Xuj96f3l/e39M/j/Nb\n8/vxho9680V7zjdf5m2f/AQTJ36E=</l\natexit>x\u21e4\nj = g[zj, \u2713] = zj + \u2713\n26",
  "GAN example \n<latexit sha1_ba\nse64=\"k1AyaOJtEUWbdMJnVPdiOSDFh\nY=\">AWp3iclZhbU9w2FICd9JamN9JO\nemLp0xm2oQy0EkvL5lJIOQGKRBYIGHJ\njuyVvQJZNrYMC579Ef01fW1/Rv9Nj2zv\nKj5HPJSZLXn+6zLkWRrHWRSFHp5+d8\nbNz/48KOP7n16e3Pv/iy6/m7ny9X6R\nlHvJemMo0PwxYwaVQvKeFlvwyzlLAsk\nPgtM1w/OeV6IVO3py4w",
  "iy6/m7ny9X6R\nlHvJemMo0PwxYwaVQvKeFlvwyzlLAsk\nPgtM1w/OeV6IVO3py4wfJyxWIhIh0xA\nazN3x4OTd9W9yUO/nwTpuIonR1eDk8\nW+HnHNjv2HPny73wbzC0sLy3Xfz4trL\nSFBa/92x7c+XbYH6ZhmXClQ8mK4mhlOd\nPHFcu1CWf3O6XBc9YeMpifgRFxRJeHF\nf1qCb+XYgM/SjN4Z/Sfh19/4qKJUVxmQ\nRgJkyPCsxM0MWOSh39",
  "pifgRFxRJeHF\nf1qCb+XYgM/SjN4Z/Sfh19/4qKJUVxmQ\nRgJkyPCsxM0MWOSh39flwJlZWaq7BpK\nCqlr1PfpMgfipyHWl5CgYW5gL764YjlL\nNSQyNt9xS/CNEmYGlb91fWdSdUPeCxUx\nc/KOqmTSdZrx0OxeuM1Rd7s1qE5om4\nqSWjGVXCPweFJVfClewkBwAGKJE5AqX\nkCdJj9B5K8gCotIAq6aVdEH4/WEVK0\njyEnHe0t0aCQST7uWG",
  "ewkBwAGKJE5AqX\nkCdJj9B5K8gCotIAq6aVdEH4/WEVK0\njyEnHe0t0aCQST7uWGvEgqlMOsouKL5/\n1zeA6xmAboKHxzNwW7G1GR6neZjnSdV\nYWK4hZypmNdNwJBDJs2IuoYqpYRLw471\nB7ZeM3XaJi7N6q7mJoKsvbzr6JzmRQ2\n7Th1BFizCuGvVEWRJ2PJDljDIclsewIA\nT30TcqlBYFWRhbudp0G07MxG8NscZ7Je\nut16R9J8zlBETg",
  "J2PJDljDIclsewIA\nT30TcqlBYFWRhbudp0G07MxG8NscZ7Je\nut16R9J8zlBETgN1nPgVTIe/qa+nM9qf\nJOa9U+BjfwST1b2E5XEzrGkjMKo2NqF\nmnStk0mxBKE8vuqbpjUPlmegO0ATwpi\ntzoaL3tMW6BEvWhPuLMNS8lPzop6Vf+P\ni4WjbxvxHsgkVFWXmqsiE/0dFQ3jI4P\nUFETx5qUSTB4F68lIJ93c0dSzHC9tE6r\nmDglBMCn2Jt",
  "VFWXmqsiE/0dFQ3jI4P\nUFETx5qUSTB4F68lIJ93c0dSzHC9tE6r\nmDglBMCn2Jtr+IVfeaOoI7myaorxAw9\ncInEwpNchR1ZRMwMnzC49KxgEI0yLAZY\nyjTosw5ufmh9QyRWje3xVyYh1X3hiqN0\nL1vcDm7CsrwcDjn1weoIwGT6DtFRDl\nqNkjs2Ujt/1Cw1bzLX76ylvik4r5mcb\nXvQL5idMgz52WADz0dMLOpIVBecT5x1\nSWI52oO6Zs",
  "1Cw1bzLX76ylvik4r5mcb\nXvQL5idMgz52WADz0dMLOpIVBecT5x1\nSWI52oO6Zsv1/Z5VG+/ukaUdO1y3KUm9\nbS/dtsO9pgf8bNPR203iEYs6EtXV9pB6\nxHK0B3W587jpGoXDdZuS1DvNo9N2uDMT\nLf9ozxBzTEplUNz7EtlvwlhUVNRO8U0\n4TESmxAWk7JrwXes7Ap4eHStJoTF7UJ\n0NRPA0pBLPIQmhMVmC3fNobVTYe6Va\nZzEbIbEJ",
  "Wk7JrwXes7Ap4eHStJoTF7UJ\n0NRPA0pBLPIQmhMVmC3fNobVTYe6Va\nZzEbIbEJYfMYSPOomhMWYirFTPGVZhsQ\nmRPI4wnkc0TxmWMpcEp6RzDEjZEm5FlQ\n+SruSCWBpjFobOxqDHshUoQbIJYLuv\nIK58pTaBUruop7roZ71zSsGarQBLC0Rf\naY39ybrIApxiOWa4kZwJZGU3gNna2qT\nM9/QVRU5yQXRp6SWlF5ZeUHpg6QGlua\nXkF0EQ",
  "rIApxiOWa4kZwJZGU3gNna2qT\nM9/QVRU5yQXRp6SWlF5ZeUHpg6QGlua\nXkF0EQvbaU/DoJonNLzyndt3Sf0tLSkt\nKepT1KI0sjSp9a+pTS0NKQ0jVL1yjVl\npITKTwRLN2jdGTpiNJDSw8pfWPpG0qfW\n/qc0reWvqX0ytIrSh9b+phSZimjdN3Sd\nUq5peTVQRCtWrpKaWAp+e0He83SbUozS\nzNKn1j6hNKhpeRXMTzPLCXHG3gwWiop\nfW",
  "peTVQRCtWrpKaWAp+e0He83SbUozS\nzNKn1j6hNKhpeRXMTzPLCXHG3gwWiop\nfWHpC0qFpeT3WxC9svQVpYmlCaUvLX1J\n6YmlJ5Q+s/QZpbGl5N0AnE4s3aXUvgWq\nCkp3LN2h9MzSM/d7AT6bxsC1MLdsBVuU\npamlG5YSn4pwFHC0lNynoxUe1ebvm0i\n97VIzbiDtRmfXk1yHqkZd7D27jS9mty\nfIjXjI9L19f3ZixRIKdzpB3MLK/gtLC",
  "97VIzbiDtRmfXk1yHqkZd7D27jS9mty\nfIjXjI9L19f3ZixRIKdzpB3MLK/gtLC3\ns/7y08uvSg50HC49W2ze0t7zvO+9H7w\nV7zfvkfc2/Z6Xuj96f3l/e39M/j/Nb\n8/vxho9680V7zjdf5m2f/AQTJ36E=</l\natexit>x\u21e4\nj = g[zj, \u2713] = zj + \u2713\n<latexit sha1_base64=\"ZyGyGVmwJ\nyNoaezx5HwDyrJnSU=\">AWjXiclZhbU9w2FICd9Ja",
  "sha1_base64=\"ZyGyGVmwJ\nyNoaezx5HwDyrJnSU=\">AWjXiclZhbU9w2FICd9Jaml5B2yktfPGUy0+mkO5BJ\n2j50OgmEkARSlsACgSWM7JW9CrJsbBmWePan9LX9Tf03PbK9q/gc8dCdIauc7M\nuR5KtdZBJUejl5X9v3Pzo408+/ezW57e/+PKr+8s3P1mv0jLPOSDMJVpfhiwgku\nh+EALflhlnOWBJIfBGdrh9c8LwQqdrTVxk/SVisRCRC",
  "mv0jLPOSDMJVpfhiwgku\nh+EALflhlnOWBJIfBGdrh9c8LwQqdrTVxk/SVisRCRCpiF0unA3Oh4GpZRc3x8\nGUTYWJ6cLS8u95frj08JKW1jy2k/9O53o+EoDcuEKx1KVhTHK8uZPqlYrkUo+f\nT2sCx4xsIzFvNjKCqW8OKkqvs+9e9BZORHaQ5/Svt19MrKpYUxVUSgJkwPS4wM0\nEXOy519NtJVRWaq7CpqGolL5OfZMIfyRyHmp5BQUW5gL6",
  "KpYUxVUSgJkwPS4wM0\nEXOy519NtJVRWaq7CpqGolL5OfZMIfyRyHmp5BQUW5gL6odjlrNQ7puDxW/D\nNMkYWpUDVfXd6bVMOCxUBU/L+vUTadZ712OBSvM1Zf7M1rEZon4j0nldSKqeQag\ncfTquK9uIeB4ABEjxOQKl5AnSY/QeSvIApLRQIGHqQT6Fzkv56SqpXmMeSkox0R\nDQqZ5JOtUYsmMqko+yC4v3fAO4zmEWoKvwxdEc7GZMTWfXaT",
  "56SqpXmMeSkox0R\nDQqZ5JOtUYsmMqko+yC4v3fAO4zmEWoKvwxdEc7GZMTWfXaT7ReVIVJoZbyJmK\ned0EDlk0oyoayhY73Bp2LH+xNZrps7axKVZ3dXcRJC1l3cdndO8qFHXqSPIgkUY\nd606giwJG3vEgZbsunMODENxG3KhRWBVmY/TwNum1nJoLX5iSD/dL1iuS/gu\nGMmICsPvMt2Aq5F19LZ3b/iw5F7VvCnzij2GyupewPG6GNWsERtX",
  "D/dL1iuS/gu\nGMmICsPvMt2Aq5F19LZ3b/iw5F7VvCnzij2GyupewPG6GNWsERtXGptSsc4VMmi0\nI5el1zS9cag8E90BmgDedGUuVPSBdr8uwZI14eF9GpeSn78c+8Rn5xUy2bmH\n9INqGiosxcFZnw/6hoBI8SvL4gicvlWjyIFBPXirh/o6mjuV4YZtIPXdQEIpJoa\n/Q9hex6l5TR3Bn0wT1FQKmXvhmQqFJjqKubAJGhm94KDoWUIgGTZjD",
  "XdQEIpJoa\n/Q9hex6l5TR3Bn0wT1FQKmXvhmQqFJjqKubAJGhm94KDoWUIgGTZjDGValDknN\nz+0niFS6+a2mAvzsOreUKURuvcNLudXQRkeDhf8msDlNGgyWeQlmrEcpTMiZnSy\ndthoWGLuXZ/PeVN0WnF/HyzbQ/6BbNThiE/P93E8xETizoS1QWnEGdkliO9qCu\n+XL9sGfV5tufyNKOHa7blKTetpdu2+Fe0wN+vuXo7RbxiEUdiepqe0g9",
  "liO9qCu\n+XL9sGfV5tufyNKOHa7blKTetpdu2+Fe0wN+vuXo7RbxiEUdiepqe0g9Yjnag7rc\nedxyjcLhuk1J6p3l0Wk73LmJln+0N+amWNSKkfm2JfKYRPCoqaidopwmMkNiEs\nJmXgv9jZVfAw6NrNSEs9gvR1UwASyMu8RCaEBabLdw12xhWtxzqltlMhsjswl\nhcYMleNRNCIsxFWOneMayDIlNiORxjPM4pnMsJS5JDwjmWNGyJyLah8nHY",
  "sjswl\nhcYMleNRNCIsxFWOneMayDIlNiORxjPM4pnMsJS5JDwjmWNGyJyLah8nHYlE8D\nSBLU2cTQGPZCpQg2QSwXdOUVzpWn0CpWdBUPXA0PrmlYM1ShCWBpm+wxf7jt3G\nQBTjEcs1xJzgSyMprAPnb61Jmd/oKoIie5ILqy9IrS0svKT2w9IDS3FLyiyCIXl\ntKfp0E0YWlF5TuW7pPaWlpSenA0gGlkaURpc8sfUZpaGlI6Zqla5RqS8mJFJ",
  "IXl\ntKfp0E0YWlF5TuW7pPaWlpSenA0gGlkaURpc8sfUZpaGlI6Zqla5RqS8mJFJ4Il\nu5ROrZ0TOmhpYeUvrH0DaXPLX1O6ZGlR5S+t/Q9pU8sfUIps5Rum7pOqXcUvLqI\nIhWLV2lNLCU/PaDvWZpn9LM0ozSp5Y+pXRkKflVDM8zS8nxBh6MlkpKX1j6glJhK\nfn9FkSvLH1FaWJpQulLS19S+s7Sd5RuWLpBaWwpeTcApxNLdym1b4GqgtId",
  "lJhK\nfn9FkSvLH1FaWJpQulLS19S+s7Sd5RuWLpBaWwpeTcApxNLdym1b4GqgtIdS3co\nPbf03P1egM+nMXAtzG1bwTalqaUpZuWkl8KcJSw9IycJyPV3tVmb5vIfS1Sc+5g\nbcZnV5OcR2rOHay9O82uJvenSM35mHR9fX/+IgVSCnf604WlFfwWlhb2H/RWfuk\n93Hm49Hi1fUN7y/ve+8H70VvxfvUe8+9vjfwQu/S+8v72/tn8c7io8XfF/9",
  "Wfuk\n93Hm49Hi1fUN7y/ve+8H70VvxfvUe8+9vjfwQu/S+8v72/tn8c7io8XfF/9o1Js\n32mu+9TqfxY3/AKo81Ys=</latexit>f[\u2022, \u03c6]\nDiscriminator\n\u2022 Train the discriminator \n\u2022 using logistic regression \nparameterized by \ud835\udf19 \n\u2022 as a binary classifier on \nthe data\n\u2022 e.g. $ real if \ud835\udc53[\u22c5] \u2265 .5\nfake if \ud835\udc53[\u22c5] <  .5\n27",
  "GAN example \n<latexit sha1_ba\nse64=\"k1AyaOJtEUWbdMJnVPdiOSDFh\nY=\">AWp3iclZhbU9w2FICd9JamN9JO\nemLp0xm2oQy0EkvL5lJIOQGKRBYIGHJ\njuyVvQJZNrYMC579Ef01fW1/Rv9Nj2zv\nKj5HPJSZLXn+6zLkWRrHWRSFHp5+d8\nbNz/48KOP7n16e3Pv/iy6/m7ny9X6R\nlHvJemMo0PwxYwaVQvKeFlvwyzlLAsk\nPgtM1w/OeV6IVO3py4w",
  "iy6/m7ny9X6R\nlHvJemMo0PwxYwaVQvKeFlvwyzlLAsk\nPgtM1w/OeV6IVO3py4wfJyxWIhIh0xA\nazN3x4OTd9W9yUO/nwTpuIonR1eDk8\nW+HnHNjv2HPny73wbzC0sLy3Xfz4trL\nSFBa/92x7c+XbYH6ZhmXClQ8mK4mhlOd\nPHFcu1CWf3O6XBc9YeMpifgRFxRJeHF\nf1qCb+XYgM/SjN4Z/Sfh19/4qKJUVxmQ\nRgJkyPCsxM0MWOSh39",
  "pifgRFxRJeHF\nf1qCb+XYgM/SjN4Z/Sfh19/4qKJUVxmQ\nRgJkyPCsxM0MWOSh39flwJlZWaq7BpK\nCqlr1PfpMgfipyHWl5CgYW5gL764YjlL\nNSQyNt9xS/CNEmYGlb91fWdSdUPeCxUx\nc/KOqmTSdZrx0OxeuM1Rd7s1qE5om4\nqSWjGVXCPweFJVfClewkBwAGKJE5AqX\nkCdJj9B5K8gCotIAq6aVdEH4/WEVK0\njyEnHe0t0aCQST7uWG",
  "ewkBwAGKJE5AqX\nkCdJj9B5K8gCotIAq6aVdEH4/WEVK0\njyEnHe0t0aCQST7uWGvEgqlMOsouKL5/\n1zeA6xmAboKHxzNwW7G1GR6neZjnSdV\nYWK4hZypmNdNwJBDJs2IuoYqpYRLw471\nB7ZeM3XaJi7N6q7mJoKsvbzr6JzmRQ2\n7Th1BFizCuGvVEWRJ2PJDljDIclsewIA\nT30TcqlBYFWRhbudp0G07MxG8NscZ7Je\nut16R9J8zlBETg",
  "J2PJDljDIclsewIA\nT30TcqlBYFWRhbudp0G07MxG8NscZ7Je\nut16R9J8zlBETgN1nPgVTIe/qa+nM9qf\nJOa9U+BjfwST1b2E5XEzrGkjMKo2NqF\nmnStk0mxBKE8vuqbpjUPlmegO0ATwpi\ntzoaL3tMW6BEvWhPuLMNS8lPzop6Vf+P\ni4WjbxvxHsgkVFWXmqsiE/0dFQ3jI4P\nUFETx5qUSTB4F68lIJ93c0dSzHC9tE6r\nmDglBMCn2Jt",
  "VFWXmqsiE/0dFQ3jI4P\nUFETx5qUSTB4F68lIJ93c0dSzHC9tE6r\nmDglBMCn2Jtr+IVfeaOoI7myaorxAw9\ncInEwpNchR1ZRMwMnzC49KxgEI0yLAZY\nyjTosw5ufmh9QyRWje3xVyYh1X3hiqN0\nL1vcDm7CsrwcDjn1weoIwGT6DtFRDl\nqNkjs2Ujt/1Cw1bzLX76ylvik4r5mcb\nXvQL5idMgz52WADz0dMLOpIVBecT5x1\nSWI52oO6Zs",
  "1Cw1bzLX76ylvik4r5mcb\nXvQL5idMgz52WADz0dMLOpIVBecT5x1\nSWI52oO6Zsv1/Z5VG+/ukaUdO1y3KUm9\nbS/dtsO9pgf8bNPR203iEYs6EtXV9pB6\nxHK0B3W587jpGoXDdZuS1DvNo9N2uDMT\nLf9ozxBzTEplUNz7EtlvwlhUVNRO8U0\n4TESmxAWk7JrwXes7Ap4eHStJoTF7UJ\n0NRPA0pBLPIQmhMVmC3fNobVTYe6Va\nZzEbIbEJ",
  "Wk7JrwXes7Ap4eHStJoTF7UJ\n0NRPA0pBLPIQmhMVmC3fNobVTYe6Va\nZzEbIbEJYfMYSPOomhMWYirFTPGVZhsQ\nmRPI4wnkc0TxmWMpcEp6RzDEjZEm5FlQ\n+SruSCWBpjFobOxqDHshUoQbIJYLuv\nIK58pTaBUruop7roZ71zSsGarQBLC0Rf\naY39ybrIApxiOWa4kZwJZGU3gNna2qT\nM9/QVRU5yQXRp6SWlF5ZeUHpg6QGlua\nXkF0EQ",
  "rIApxiOWa4kZwJZGU3gNna2qT\nM9/QVRU5yQXRp6SWlF5ZeUHpg6QGlua\nXkF0EQvbaU/DoJonNLzyndt3Sf0tLSkt\nKepT1KI0sjSp9a+pTS0NKQ0jVL1yjVl\npITKTwRLN2jdGTpiNJDSw8pfWPpG0qfW\n/qc0reWvqX0ytIrSh9b+phSZimjdN3Sd\nUq5peTVQRCtWrpKaWAp+e0He83SbUozS\nzNKn1j6hNKhpeRXMTzPLCXHG3gwWiop\nfW",
  "peTVQRCtWrpKaWAp+e0He83SbUozS\nzNKn1j6hNKhpeRXMTzPLCXHG3gwWiop\nfWHpC0qFpeT3WxC9svQVpYmlCaUvLX1J\n6YmlJ5Q+s/QZpbGl5N0AnE4s3aXUvgWq\nCkp3LN2h9MzSM/d7AT6bxsC1MLdsBVuU\npamlG5YSn4pwFHC0lNynoxUe1ebvm0i\n97VIzbiDtRmfXk1yHqkZd7D27jS9mty\nfIjXjI9L19f3ZixRIKdzpB3MLK/gtLC",
  "97VIzbiDtRmfXk1yHqkZd7D27jS9mty\nfIjXjI9L19f3ZixRIKdzpB3MLK/gtLC3\ns/7y08uvSg50HC49W2ze0t7zvO+9H7w\nV7zfvkfc2/Z6Xuj96f3l/e39M/j/Nb\n8/vxho9680V7zjdf5m2f/AQTJ36E=</l\natexit>x\u21e4\nj = g[zj, \u2713] = zj + \u2713\n\u2022 Train the generator \nto update \ud835\udf03 in order \nto increase the loss \non the discriminator\n\u2022 Then train the \ndiscriminator to \ndecrease the loss\n28",
  "GAN example \n<latexit\n sha1_base64=\"k1\nAyaOJtEUWbdMJnVP\ndiOSDFhY=\">AW\np3iclZhbU9w2FICd\n9JamN9JOemLp0xm\n2oQy0EkvL5lJIOQG\nKRBYIGHJjuyVvQJZ\nNrYMC579Ef01fW1/\nRv9Nj2zvKj5HPJS\nZLXn+6zLkWRrHWR\nSFHp5+d8bNz/48KO\nP7n16e3Pv/iy6/\nm7ny9X6RlHvJemMo\n0PwxYwaVQvKeFlvw\nwyzlLAskPgtM1w/\nOeV",
  "6e3Pv/iy6/\nm7ny9X6RlHvJemMo\n0PwxYwaVQvKeFlvw\nwyzlLAskPgtM1w/\nOeV6IVO3py4wfJyx\nWIhIh0xAazN3x4O\nTd9W9yUO/nwTpuI\nonR1eDk8W+HnHNjv\n2HPny73wbzC0sLy\n3Xfz4trLSFBa/92x\n7c+XbYH6ZhmXClQ8\nmK4mhlOdPHFcu1C\nWf3O6XBc9YeMpifg\nRFxRJeHFf1qCb+XY\ngM/SjN4Z/Sfh19/4\nqKJUVxmQRgJk",
  "Wf3O6XBc9YeMpifg\nRFxRJeHFf1qCb+XY\ngM/SjN4Z/Sfh19/4\nqKJUVxmQRgJkyPCs\nxM0MWOSh39flwJl\nZWaq7BpKCqlr1Pfp\nMgfipyHWl5CgYW5g\nL764YjlLNSQyNt9x\nS/CNEmYGlb91fWdS\ndUPeCxUxc/KOqmTS\ndZrx0OxeuM1Rd7s\n1qE5om4qSWjGVX\nCPweFJVfClewkBwA\nGKJE5AqXkCdJj9B\n5K8gCotIAq6aVdEH\n4/WEV",
  "4qSWjGVX\nCPweFJVfClewkBwA\nGKJE5AqXkCdJj9B\n5K8gCotIAq6aVdEH\n4/WEVK0jyEnHe0t\n0aCQST7uWGvEgqlM\nOsouKL5/1zeA6xm\nAboKHxzNwW7G1GR6\nneZjnSdVYWK4hZyp\nmNdNwJBDJs2IuoYq\npYRLw471B7ZeM3Xa\nJi7N6q7mJoKsvbz\nr6JzmRQ27Th1BFiz\nCuGvVEWRJ2PJDljD\nIclsewIAT30TcqlB\nYFWRhbudp0G07",
  "r6JzmRQ27Th1BFiz\nCuGvVEWRJ2PJDljD\nIclsewIAT30TcqlB\nYFWRhbudp0G07MxG\n8NscZ7Jeut16R9J8\nzlBETgN1nPgVTIe/\nqa+nM9qfJOa9U+B\njfwST1b2E5XEzrGk\njMKo2NqFmnStk0mx\nBKE8vuqbpjUPlme\ngO0ATwpitzoaL3tM\nW6BEvWhPuLMNS8lP\nzop6Vf+Pi4Wjbxv\nxHsgkVFWXmqsiE/0\ndFQ3jI4PUFETx5qU\nSTB4",
  "PuLMNS8lP\nzop6Vf+Pi4Wjbxv\nxHsgkVFWXmqsiE/0\ndFQ3jI4PUFETx5qU\nSTB4F68lIJ93c0dS\nzHC9tE6rmDglBMCn\n2Jtr+IVfeaOoI7my\naorxAw9cInEwpNc\nhR1ZRMwMnzC49Kxg\nEI0yLAZYyjTosw5u\nfmh9QyRWje3xVyYh\n1X3hiqN0L1vcDm7C\nsrwcDjn1weoIwGT\nT6DtFRDlqNkjs2Uj\nt/1Cw1bzLX76ylvi\nk4r5mcbXvQL",
  "7C\nsrwcDjn1weoIwGT\nT6DtFRDlqNkjs2Uj\nt/1Cw1bzLX76ylvi\nk4r5mcbXvQL5idM\ngz52WADz0dMLOpI\nVBecT5x1SWI52oO6\nZsv1/Z5VG+/ukaUd\nO1y3KUm9bS/dtsO9\npgf8bNPR203iEYs6\nEtXV9pB6xHK0B3W5\n87jpGoXDdZuS1DvN\no9N2uDMTLf9ozxB\nzTEplUNz7Etlvwlh\nUVNRO8U04TESmxA\nWk7JrwXes7Ap4eHS\ntJo",
  "DMTLf9ozxB\nzTEplUNz7Etlvwlh\nUVNRO8U04TESmxA\nWk7JrwXes7Ap4eHS\ntJoTF7UJ0NRPA0pB\nLPIQmhMVmC3fNob\nVTYe6VaZzEbIbEJ\nYfMYSPOomhMWYirF\nTPGVZhsQmRPI4wnk\nc0TxmWMpcEp6RzDE\njZEm5FlQ+SruSCWB\npjFobOxqDHshUoQ\nbIJYLuvIK58pTaB\nUruop7roZ71zSsGa\nrQBLC0RfaY39ybr\nIApxiOWa4kZwJ",
  "Q\nbIJYLuvIK58pTaB\nUruop7roZ71zSsGa\nrQBLC0RfaY39ybr\nIApxiOWa4kZwJZGU\n3gNna2qTM9/QVRU\n5yQXRp6SWlF5ZeUH\npg6QGluaXkF0EQvb\naU/DoJonNLzyndt3\nSf0tLSktKepT1KI0\nsjSp9a+pTS0NKQ0\njVL1yjVlpITKTwRL\nN2jdGTpiNJDSw8pf\nWPpG0qfW/qc0reWv\nqX0ytIrSh9b+phSZ\nimjdN3SdUq5peTVQ\nRCt",
  "iNJDSw8pf\nWPpG0qfW/qc0reWv\nqX0ytIrSh9b+phSZ\nimjdN3SdUq5peTVQ\nRCtWrpKaWAp+e0He\n83SbUozSzNKn1j6h\nNKhpeRXMTzPLCXHG\n3gwWiopfWHpC0qF\npeT3WxC9svQVpYml\nCaUvLX1J6YmlJ5Q+\ns/QZpbGl5N0AnE4s\n3aXUvgWqCkp3LN2h\n9MzSM/d7AT6bxsC1\nMLdsBVuUpamlG5Y\nSn4pwFHC0lNynoxU\ne1ebvm0i97",
  "N2h\n9MzSM/d7AT6bxsC1\nMLdsBVuUpamlG5Y\nSn4pwFHC0lNynoxU\ne1ebvm0i97VIzbiD\ntRmfXk1yHqkZd7D\n27jS9mtyfIjXjI9L\n19f3ZixRIKdzpB3M\nLK/gtLC3s/7y08uv\nSg50HC49W2ze0t7z\nvO+9H7wV7zfvkf\nc2/Z6Xuj96f3l/e3\n9M/j/Nb8/vxho96\n80V7zjdf5m2f/AQT\nJ36E=</latexit>x\u21e4\nj = g[zj,",
  "l/e3\n9M/j/Nb8/vxho96\n80V7zjdf5m2f/AQT\nJ36E=</latexit>x\u21e4\nj = g[zj, \u2713] = zj + \u2713\n\u2022 Keep repeating till the \ndiscriminator does no better \nthan random chance\n29",
  "Trained to completion\n30\no z: uniform latent variable\no x: samples according to a (green solid) \ngenerative distribution\no black dotted curve: real data distribution\no blue dashed curve: discriminator\nI. Goodfellow et al., \u201cGenerative Adversarial Nets,\u201d 2014",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n31",
  "GAN cost function\n<latexit sha1_base64=\"2ZbhRl\ndbQw5MobBsa/t9X5j2kK0=\">AXK3iclZhbT9xGFICXtP0RlqFl75Y\nRZHSKiC2Si9SVSmBkBukQMItwZvV2Dv2ThiPjT2GJdb+pKo/pk+t+tr\n/0TO2cHnDFK7EtnJ+b65+MzFXgeZFIVeWflz7p13v/gw+vfXT9408\n+/ez+Rtf7BdpmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOM",
  "+/ez+Rtf7BdpmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOMDxIWKxGJkGkIDed/98dMV34QZWMx9X7xfJbHiVDWciXPN\nJHflEmw0pMl273l85N4RtfprG/KmJ51F/ykyCdVIWIp0dNMYJCE2G4\nk7TzGBg1HzgLXl1bVv5P1f1cxGP9WA4v7iyvFJ/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4",
  "9WA4v7iyvFJ/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4xsJjFvMjKCqW8GJQ1Zmd\nercgMvKiNIc/pb06erlGxZKiOE8CMBOmxwVmJuhiR6WOfhpUQmWl5ips\nOopK6enUM9PkjUTOQy3PocDCXMBYvXDMchZqmMzrvuJnYZokTI0qf3V\n9ZwrTxWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nr",
  "0qf3V\n9ZwrTxWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nri/HyxgIDkAscwJSxQto0+QniLw+orCQJeCqWRmwHrznU9K0jyGnH\nS0V0SDQib5pGOtEQumMukoL0DxvFueAVznMAswVPjiaA5eZExNL+pPt\nF5UhUmhnvImYp53QVcsikuaKuoUopoWrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLA",
  "ikuaKuoUopoWrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLAnHzoglDLcluEYyRPRNyqUFgVZ\nGFu52nQ7TszEbw2Jxnsl63XpH0nzKUEROA3We+BVMh7+pr6cz2LpJzW\nvumwCfeGCarW6U5Py91AlfVxqbUrHOFTJotCOXpWdc0o3GoPBPdCzQB\nvOnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM",
  "oPBPdCzQB\nvOnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM\n+H80NIbHV5fEMGTl0o0eRCoJy+VcL6jqWM5XtgmUs8dFIRiUuhztP1\nFrLp16gebJqgsULAtAvfTCg0yVHUlU3AyPANt2zHAgrRYbNYyLcq\nck8MPrWeI1Lo5FnNhblbdA1UaoXtucDmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF",
  "bdA1UaoXtucDmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF/GSj7Q/GBbNThiE/GW7g+YiJR\n2J2oJnJGdbkliO/qCt2XK9PLJq4/W3ZGnHDtdtStJuO0q37XCvGAE/2\nXSMdpN4xKORG21I6QesRz9QVvuPG6rsLhuk1J2r3Io9N2uDMTLf9od\n8w1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh",
  "DMTLf9od\n8w1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh\ncbsQXc0EsDTiEl9CE8Jis4W7ZhvD6qZD3XSrTGZjZDYhLD5iCb7qJoTF\nmIqxUzxmWYbEJkTyOMZ5HNM8ZljKXBKekcwxI2RJuRZUPk67kglgaYJ\n6mzg6gxHIVKEO2yCWC7ryCufKU2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0g",
  "CWC7ryCufKU2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0gdvY2abOxdNfEFXkS6Izi09p/TM0jNKDy\nw9oDS3lPwiCKLnlpJfJ0F0aukpfuW7lNaWlpSumfpHqWRpRGlDy19SG\nloaUjpmqVrlGpLyRMp3BEs3aV0bOmY0kNLDyl9aelLSh9b+pjSV5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sN",
  "elLSh9b+pjSV5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sNUu3Kc0sz\nSh9YOkDSkeWkl/FcD+zlDzewI3RUknpE0ufUCosJb/fguiZpc8oTSxN\nKH1q6VNK31j6htJHlj6iNLaUvBuApxNLX1Bq3wJVBaU7lu5QemLpifu9\nAJ9NY+BamFu2gS1KU0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd",
  "BamFu2gS1KU0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd7D2dLqoTc6nSM34mAx9fX/2IgVSCif9cH6xj9/C0sL\n+d8v9H5bv7txdvLfavqG91vuq93Xvdq/f+7F3r/e4t93b64VzN+d+n\nswt7w28IfC38t/N2o78y1db7sdT4L/wLeTwUhw=</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi,",
  "78y1db7sdT4L/wLeTwUhw=</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi, \u03c6]]\ni\n\u2212 yi log\nh\nsig[f[xi, \u03c6]]\ni#\nDiscriminator uses standard cross entropy loss (see Section 5.4 \u2013 binary classification loss): :\n32",
  "GAN cost function\n<latexit sha1_base64=\"2ZbhRl\ndbQw5MobBsa/t9X5j2kK0=\">AXK3iclZhbT9xGFICXtP0RlqFl75Y\nRZHSKiC2Si9SVSmBkBukQMItwZvV2Dv2ThiPjT2GJdb+pKo/pk+t+tr\n/0TO2cHnDFK7EtnJ+b65+MzFXgeZFIVeWflz7p13v/gw+vfXT9408\n+/ez+Rtf7BdpmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOM",
  "+/ez+Rtf7BdpmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOMDxIWKxGJkGkIDed/98dMV34QZWMx9X7xfJbHiVDWciXPN\nJHflEmw0pMl273l85N4RtfprG/KmJ51F/ykyCdVIWIp0dNMYJCE2G4\nk7TzGBg1HzgLXl1bVv5P1f1cxGP9WA4v7iyvFJ/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4",
  "9WA4v7iyvFJ/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4xsJjFvMjKCqW8GJQ1Zmd\nercgMvKiNIc/pb06erlGxZKiOE8CMBOmxwVmJuhiR6WOfhpUQmWl5ips\nOopK6enUM9PkjUTOQy3PocDCXMBYvXDMchZqmMzrvuJnYZokTI0qf3V\n9ZwrTxWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nr",
  "0qf3V\n9ZwrTxWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nri/HyxgIDkAscwJSxQto0+QniLw+orCQJeCqWRmwHrznU9K0jyGnH\nS0V0SDQib5pGOtEQumMukoL0DxvFueAVznMAswVPjiaA5eZExNL+pPt\nF5UhUmhnvImYp53QVcsikuaKuoUopoWrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLA",
  "ikuaKuoUopoWrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLAnHzoglDLcluEYyRPRNyqUFgVZ\nGFu52nQ7TszEbw2Jxnsl63XpH0nzKUEROA3We+BVMh7+pr6cz2LpJzW\nvumwCfeGCarW6U5Py91AlfVxqbUrHOFTJotCOXpWdc0o3GoPBPdCzQB\nvOnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM",
  "oPBPdCzQB\nvOnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM\n+H80NIbHV5fEMGTl0o0eRCoJy+VcL6jqWM5XtgmUs8dFIRiUuhztP1\nFrLp16gebJqgsULAtAvfTCg0yVHUlU3AyPANt2zHAgrRYbNYyLcq\nck8MPrWeI1Lo5FnNhblbdA1UaoXtucDmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF",
  "bdA1UaoXtucDmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF/GSj7Q/GBbNThiE/GW7g+YiJR\n2J2oJnJGdbkliO/qCt2XK9PLJq4/W3ZGnHDtdtStJuO0q37XCvGAE/2\nXSMdpN4xKORG21I6QesRz9QVvuPG6rsLhuk1J2r3Io9N2uDMTLf9od\n8w1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh",
  "DMTLf9od\n8w1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh\ncbsQXc0EsDTiEl9CE8Jis4W7ZhvD6qZD3XSrTGZjZDYhLD5iCb7qJoTF\nmIqxUzxmWYbEJkTyOMZ5HNM8ZljKXBKekcwxI2RJuRZUPk67kglgaYJ\n6mzg6gxHIVKEO2yCWC7ryCufKU2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0g",
  "CWC7ryCufKU2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0gdvY2abOxdNfEFXkS6Izi09p/TM0jNKDy\nw9oDS3lPwiCKLnlpJfJ0F0aukpfuW7lNaWlpSumfpHqWRpRGlDy19SG\nloaUjpmqVrlGpLyRMp3BEs3aV0bOmY0kNLDyl9aelLSh9b+pjSV5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sN",
  "elLSh9b+pjSV5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sNUu3Kc0sz\nSh9YOkDSkeWkl/FcD+zlDzewI3RUknpE0ufUCosJb/fguiZpc8oTSxN\nKH1q6VNK31j6htJHlj6iNLaUvBuApxNLX1Bq3wJVBaU7lu5QemLpifu9\nAJ9NY+BamFu2gS1KU0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd",
  "BamFu2gS1KU0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd7D2dLqoTc6nSM34mAx9fX/2IgVSCif9cH6xj9/C0sL\n+d8v9H5bv7txdvLfavqG91vuq93Xvdq/f+7F3r/e4t93b64VzN+d+n\nswt7w28IfC38t/N2o78y1db7sdT4L/wLeTwUhw=</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi,",
  "78y1db7sdT4L/wLeTwUhw=</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi, \u03c6]]\ni\n\u2212 yi log\nh\nsig[f[xi, \u03c6]]\ni#\n<latexit sha1_base64=\"bDQB\nfdAPXU69Wg1suhD84icKLts=\">AXJ3iclZhbT9xGFICX9JamN\n9KqvPTFKopURQFBlTZ9qZRAyA1SIFwD3qzG3rF3YDw29hiWPuD\nqv6YvlXtY/9Jz9hmB58zPHQlspPzfXPxmYu9DjIpCr209M/MrQ",
  "F3YDw29hiWPuD\nqv6YvlXtY/9Jz9hmB58zPHQlspPzfXPxmYu9DjIpCr209M/MrQ8\n+/OjT25/euez7/48qvZu1/vF2mZh3wvTGWaHwas4FIovqeFlv\nwyzlLAskPgtNVw/OeV6IVO3qy4z3ExYrEYmQaQgNZv/wR0xXf\nhBlIzHxfvV8lseJUINpyJc80sd+USaD6mSy4Ms09ldELI+XF/wk\nSMdVIeLJcVOMoBE43f3BycPmvr9vpHzvrfgNU2",
  "+USaD6mSy4Ms09ldELI+XF/wk\nSMdVIeLJcVOMoBE43f3BycPmvr9vpHzvrfgNU2IiW3hpvoDgWr\n7uYhHuj+YnV9aXKo/Hi0st4X5XvZGtz9dugP07BMuNKhZEVxvL\nyU6X7Fci1CySd3/LgGQtPWcyPoahYwot+VWd14t2DyNCL0hz+l\nPbq6PUaFUuK4jIJwEyYHhWYmaCLHZc6+qVfCZWVmquw6Sgqpad\nTz0yRNxQ5D7W8hAILcwFj9cIRy",
  "IJwEyYHhWYmaCLHZc6+qVfCZWVmquw6Sgqpad\nTz0yRNxQ5D7W8hAILcwFj9cIRy1moYSLv+IpfhGmSMDWs/JW17Q\nlMFY+FqvhZWU/qZNJ1mqHQ/EmY+Xl7rQVoXki3nPSK2YRm4Qe\nDypKr4YL2IgOACxyAlIFS+gTZOfIPKWEYVFLAFXzcqA9eC9mZCm\nleYx5KSjHRENCpnk461SiyYyqSj7IDiefc8A7jOYRZgqPDF0Rz\nsZExNrupPtZ5U",
  "leYx5KSjHRENCpnk461SiyYyqSj7IDiefc8A7jOYRZgqPDF0Rz\nsZExNrupPtZ5UhUmhnvImYp53QVcsikuaKuoUopoWrYsX7D1h\numTtvEpVk91NxEkLWbdx2d07yoYdepI8iCRh3rTqCLAlHzpAlD\nLcluEIyRPRNyqUFgVZGFu5WnQ7TszEbw2xnsl63VpH0nzOU\nEROA3We+BVMh7+qr6dT2rpJzXvumwMfeCarW6U5O691AlfVxib\nUrHOF",
  "pH0nzOU\nEROA3We+BVMh7+qr6dT2rpJzXvumwMfeCarW6U5O691AlfVxib\nUrHOFTJotCOXpRdc0o3GoPBPdCzQBvOnKXKjomvagLsGSNWH/AV\nxqXkp+vLD4Ex/3qyWzbcw/JvQUFmroZM+H80NISbHF5fEMGTl\n0o0eRCoJy+VcL6jqWM5XtgmUs8dFIRiUuhLtP1FrLp16gebJqg\nsULAtAvfTCg0yVHUlU3AyPANt2vHAgrRYbNYyLcqck8M",
  "UuhLtP1FrLp16gebJqg\nsULAtAvfTCg0yVHUlU3AyPANt2vHAgrRYbNYyLcqck8MPrWe\nI1Lo5FnNhblbdA1UaoXtucDmtBW4OZzG6oHKNBk8gLdWQ5S\niZYzOl43d+oWGLuXZ/PeVN0WnF/Gy97Q/GBbNThiE/G6zj+YiJR\nR2J2oLnI2dbkliO/qCt6XK9PrJq/d19srRjh+s2JWm3HaXbdrg3\njICfbThGu0E8YlFHorbaEVKPWI7+oC13Hjd",
  "PrJq/d19srRjh+s2JWm3HaXbdrg3\njICfbThGu0E8YlFHorbaEVKPWI7+oC13HjdcV+Fw3aYk7V7l0Wk\n73KmJln+0O+KamcekVA7NY18q/SaERU1F7RThMdIbEJYTMquBf\n/Hyo6Am0fXakJY3CpEVzMBLA25xJfQhLDYbOGu2cawuFQN9wq\nk9kImU0Ii89Zgq+6CWExpmLsFE9ZliGxCZE8jnAeRzSPGZYyl4R\nnJHPMCFlSrgWVj9KuZAJYG",
  "9Zgq+6CWExpmLsFE9ZliGxCZE8jnAeRzSPGZYyl4R\nnJHPMCFlSrgWVj9KuZAJYGqPexo7OYAQyVajDNojlgq68wrnyF\nrFiq7iPVfHezd0rBlq0ASwtEn2mOdvOjdZgFMj1muJGcCWRlN4\nBZ2tqhz9fQXRBV5kguiS0svKb2w9ILSA0sPKM0tJb8IguiNpeTX\nSRCdW3pO6b6l+5SWlpaU7lm6R2lkaUTpM0ufURpaGlK6aukqpdp\nS8kQKdwR",
  "eTX\nSRCdW3pO6b6l+5SWlpaU7lm6R2lkaUTpM0ufURpaGlK6aukqpdp\nS8kQKdwRLdykdWTqi9NDSQ0rfWvqW0heWvqD0yNIjSt9b+p7SJ5\nY+oZRZyihds3SNUm4peXUQRCuWrlAaWEp+8Fes3SL0szSjNKnl\nj6ldGgp+VUM9zNLyeMN3BgtlZS+tPQlpcJS8vstiF5b+prSxNKE\n0leWvqL0xNITSp9b+pzS2FLybgCeTizdodS+BaoKSrct3",
  "JS8vstiF5b+prSxNKE\n0leWvqL0xNITSp9b+pzS2FLybgCeTizdodS+BaoKSrct3ab0zNI\nz93sBPp3GwLUwN20Dm5SmlqaUrltKfinAo4Slp+R5MlLtqXb1to\nmca5GacgdrM35Vm+Q8UlPuYO3pdFWbnE+RmvIRGfra/vRFCqQUT\nvrB7PwyfgtLC/s/Li7/vPhw+H845X2De3t3ne973s/9JZ7j3qP\ney96W729Xjhzd+bRzOZJ3O/z/059fc",
  "7/vPhw+H845X2De3t3ne973s/9JZ7j3qP\ney96W729Xjhzd+bRzOZJ3O/z/059fc3416a6at802v85n79z9\nZJRM5</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n2\n4X\nj\n\u2212 log\nh\n1 \u2212 sig[f[x\u21e4\nj, \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\nDiscriminator uses standard cross entropy loss (see Section 5.4 \u2013 binary classification loss):\nGenerated samples, \ud835\udc31!\n\u2217,  \ud835\udc66! = 0, and for real examples, \ud835\udc31!,  \ud835\udc66! = 1 :\nThese are generated \nsamples so \ud835\udc66# = 0\nThese are real samples \nso \ud835\udc66!",
  "\ud835\udc31!\n\u2217,  \ud835\udc66! = 0, and for real examples, \ud835\udc31!,  \ud835\udc66! = 1 :\nThese are generated \nsamples so \ud835\udc66# = 0\nThese are real samples \nso \ud835\udc66! = 1\nWe can separate into two summations that separately index over the generated \nsamples and the real samples.\n33",
  "GAN loss function\n34",
  "GAN loss function\n35",
  "GAN loss function\n36",
  "GAN cost function\n<latexit sha1\n_base64=\"pC+2wogeCSs95gd\nEtDYkU9OAPo=\">AXeHicl\nZhJU9xGFICHrI6z4aQSDrnIoV\nxZClPgcpZLqmw3sABDAPYaD\nzV0rQ0bVotIbVgQDX3XJN/l7\n+SyiGvpZlp9F5zyFQN07zv0+t\nWb1qCTIpCr6z8PfOu+9/8G\nHNz6+fEn372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R",
  "372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R\n8V7CYiUiETINof78v/6Q6coP\nomwoxkvTf/SQazb2fvN8lseJU\nP2p4d/2JY/0cR1no76VZ6Qok\n371dnzXl2nsr4lYHq/6t+8CT\noJ0VBUiHh83xWhagByeiQbRpT\nlyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwu",
  "lyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwuqksNiZfHb\n6t74a+IM0LBOudChZURyvrmS\n6V7Fci1Dy8U2/LHjGwhMW82Mo\nKpbwolfVwzT27kBk4EVpDl+l\nvTp69YiKJUVxkQRgJkwPC8xM\n0MWOSx392quEykrNVdhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQ",
  "dhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQwIj4Wq+GlZz5LxuO1s1A\n6H4nXG2rP9WRaheSIuOUlSKy\nbJNQKPx1XFl+NlDAQHIJY5Aa\nniBeQ0/QMzahVRWBUScGXn3Ms\nxSa0j6FPWtprokEhk3zUsta\nJBUOZtJQ9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZU",
  "Q9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZUKSUc\nGras37H1kqmTScelWd3U3ESQ\ntZ+3HZ3TflGDtlNHkAWTMG5bd\nQRZEvawAYM9Zjwtw2aTJ56Ju\nFWhsCrIxNzJ06Bd2YieG6OM\nlgvbW+jIt1/xlCPmACsPvMrmA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1",
  "mA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1jhUnon2CZoAXnR\nlLlR0RVuqSzBlTdhfglPNS8m\nP7y7/xEe9asUsG/OH9CYkKsr\nMlciE/0eiAVw18fyCB68VKLB\ng0A9eKmE/R0NHcvxDaReuyg\nIBSTQl+g5S9i1T6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkO",
  "6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkOk1s2mAtzsWpvqNI7X2Dy\n9lRUIaLwxm/5vA9WjQ9GeQlm\nrActSZIzOkozd+oWGJuVZ/Pe\nRN0WnF/HRzUh+0C0anDEN+2t/\nE4xETizoS5YIbLmcuSxHfZB\nrNl2vtqzafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us",
  "zafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us3C4\nblOSvN+dNoOd2ai6R/t1/fN\nUErlwNz2pdJvQljUVNROMU14j\nMQmhMWkbFvwP1b2BFw82lYTw\nuJOIdqaCWBpwCU+hSaExWYJt\n81JDKtbDnXLrTKZDZHZhLD4hC\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASy",
  "C\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASyNUG0jR2XQApk\nqVOEkiOWCzrzCOfMUmsWKzuK\nuq+LuNRVrhKaAJa2yRrz/G3\nnIgtwF8NtlquTM4GsjHbgDnZ2\nqDO9+wuitzJBdGFpReUnlt6\nTumhpYeU5paSJ4IgemkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7U",
  "emkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7UrgiWLpP6dDSIaVHlh5R+\nsrSV5Q+tfQpa8tfU3paWXlD\n609CGlzFJG6YalG5RyS8mrgy\nBas3SN0sBS8uwHa83SHUozSz\nNKH1n6iNKBpeSpGK5nlpLbG7g\nwWiopfWbpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xao",
  "bpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xaoKijd\ntXSX0lNLT93vBfhsGAPXxNy2\nCbYpTS1NKd20lDwpwK2EpSfk\nfjJSk1t+raJ7GuRmnEHm/S4f\nR9LjBl3sMnuND2a7E+RmvEha\nfrGwexFCnQp7PT9+cV/BaWF\ng7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+",
  "g7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+Z8Fb+G7h0\nZ9Z25yzJed1mfh3n8tXjE3</l\natexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<latexit sha1_base64=\"2ZbhRl\ndbQw5MobBsa/t9X5j2kK0=\">AXK3iclZhbT9xGFICXtP0Rl",
  "1_base64=\"2ZbhRl\ndbQw5MobBsa/t9X5j2kK0=\">AXK3iclZhbT9xGFICXtP0RlqFl75Y\nRZHSKiC2Si9SVSmBkBukQMItwZvV2Dv2ThiPjT2GJdb+pKo/pk+t+tr\n/0TO2cHnDFK7EtnJ+b65+MzFXgeZFIVeWflz7p13v/gw+vfXT9408\n+/ez+Rtf7BdpmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOMDxIWKxGJkGk",
  "pmYd8L0xlmh8GrOBSKL6nhZb8Ms5SwLJD4LjNcMPTn\nleiFTt6vOMDxIWKxGJkGkIDed/98dMV34QZWMx9X7xfJbHiVDWciXPN\nJHflEmw0pMl273l85N4RtfprG/KmJ51F/ykyCdVIWIp0dNMYJCE2G4\nk7TzGBg1HzgLXl1bVv5P1f1cxGP9WA4v7iyvFJ/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4xsJjFvMjKCq",
  "/PFrot4XFXvZHt64O\nfJHaVgmXOlQsqI46q9kelCxXItQ8ul1vyx4xsJjFvMjKCqW8GJQ1Zmd\nercgMvKiNIc/pb06erlGxZKiOE8CMBOmxwVmJuhiR6WOfhpUQmWl5ips\nOopK6enUM9PkjUTOQy3PocDCXMBYvXDMchZqmMzrvuJnYZokTI0qf3V\n9ZwrTxWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nri/HyxgIDkAs",
  "xWOhKn5S1hM7nXad9drhULzKWH2yO2tFaJ6It5w0UiumkSsEHk+\nri/HyxgIDkAscwJSxQto0+QniLw+orCQJeCqWRmwHrznU9K0jyGnH\nS0V0SDQib5pGOtEQumMukoL0DxvFueAVznMAswVPjiaA5eZExNL+pPt\nF5UhUmhnvImYp53QVcsikuaKuoUopoWrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLAnHzoglDLclu",
  "WrYsX7F1nOmjtvEpVk91NxEkL\nWbdx2d07yoUdepI8iCRh3rTqCLAnHzoglDLcluEYyRPRNyqUFgVZ\nGFu52nQ7TszEbw2Jxnsl63XpH0nzKUEROA3We+BVMh7+pr6cz2LpJzW\nvumwCfeGCarW6U5Py91AlfVxqbUrHOFTJotCOXpWdc0o3GoPBPdCzQB\nvOnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM\n+H80NIbHV5",
  "OnKXKjoknanLsGSNWH/DlxqXkp+tLT8PZ8MqhWzbcw/JvQUFmroZM\n+H80NIbHV5fEMGTl0o0eRCoJy+VcL6jqWM5XtgmUs8dFIRiUuhztP1\nFrLp16gebJqgsULAtAvfTCg0yVHUlU3AyPANt2zHAgrRYbNYyLcq\nck8MPrWeI1Lo5FnNhblbdA1UaoXtucDmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF/GSj7Q/GBbN",
  "DmrBW4OZzyK6oHKNBk8gLd\nWI5SiZEzOlk9d+oWGLuXZ/PeVN0WnF/GSj7Q/GBbNThiE/GW7g+YiJR\n2J2oJnJGdbkliO/qCt2XK9PLJq4/W3ZGnHDtdtStJuO0q37XCvGAE/2\nXSMdpN4xKORG21I6QesRz9QVvuPG6rsLhuk1J2r3Io9N2uDMTLf9od\n8w1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh\ncbsQXc0EsDT",
  "1M49JqRyZx75U+k0Ii5qK2imCY+R2ISwmJRdC/6PlRcCbh5dqwlh\ncbsQXc0EsDTiEl9CE8Jis4W7ZhvD6qZD3XSrTGZjZDYhLD5iCb7qJoTF\nmIqxUzxmWYbEJkTyOMZ5HNM8ZljKXBKekcwxI2RJuRZUPk67kglgaYJ\n6mzg6gxHIVKEO2yCWC7ryCufKU2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0gdvY2abOxdNf",
  "2gVK7qK91wd713RsWaoQRPA0hbZY56\n/5dxkAU4xPGa5kpwJZGU0gdvY2abOxdNfEFXkS6Izi09p/TM0jNKDy\nw9oDS3lPwiCKLnlpJfJ0F0aukpfuW7lNaWlpSumfpHqWRpRGlDy19SG\nloaUjpmqVrlGpLyRMp3BEs3aV0bOmY0kNLDyl9aelLSh9b+pjSV5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sNUu3Kc0sz\nSh",
  "V5a+o\nvStpW8pvW/pfUqZpYzSdUvXKeWklcHQbRq6SqlgaXktx/sNUu3Kc0sz\nSh9YOkDSkeWkl/FcD+zlDzewI3RUknpE0ufUCosJb/fguiZpc8oTSxN\nKH1q6VNK31j6htJHlj6iNLaUvBuApxNLX1Bq3wJVBaU7lu5QemLpifu9\nAJ9NY+BamFu2gS1KU0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd7D2dLqoTc6n",
  "0tTSjcsJb8U4FHC0mPyPBmp9lS7eNtEzrVIzbi\nDtRm/qE1yHqkZd7D2dLqoTc6nSM34mAx9fX/2IgVSCif9cH6xj9/C0sL\n+d8v9H5bv7txdvLfavqG91vuq93Xvdq/f+7F3r/e4t93b64VzN+d+n\nswt7w28IfC38t/N2o78y1db7sdT4L/wLeTwUhw=</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi, \u03c6]]\ni\n\u2212 yi log\nh\nsig[f[xi, \u03c6]]\ni#\n<latexit s",
  "latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n\"X\ni\n\u2212(1 \u2212 yi) log\nh\n1 \u2212 sig[f[xi, \u03c6]]\ni\n\u2212 yi log\nh\nsig[f[xi, \u03c6]]\ni#\n<latexit sha1_base64=\"bDQB\nfdAPXU69Wg1suhD84icKLts=\">AXJ3iclZhbT9xGFICX9JamN\n9KqvPTFKopURQFBlTZ9qZRAyA1SIFwD3qzG3rF3YDw29hiWPuD\nqv6YvlXtY/9Jz9hmB58zPHQlspPzfXPxmYu9DjIpCr209M/MrQ8\n+/OjT25/euez7/48qvZu1/",
  "9Jz9hmB58zPHQlspPzfXPxmYu9DjIpCr209M/MrQ8\n+/OjT25/euez7/48qvZu1/vF2mZh3wvTGWaHwas4FIovqeFlv\nwyzlLAskPgtNVw/OeV6IVO3qy4z3ExYrEYmQaQgNZv/wR0xXf\nhBlIzHxfvV8lseJUINpyJc80sd+USaD6mSy4Ms09ldELI+XF/wk\nSMdVIeLJcVOMoBE43f3BycPmvr9vpHzvrfgNU2IiW3hpvoDgWr\n7uYhHuj+YnV",
  "k\nSMdVIeLJcVOMoBE43f3BycPmvr9vpHzvrfgNU2IiW3hpvoDgWr\n7uYhHuj+YnV9aXKo/Hi0st4X5XvZGtz9dugP07BMuNKhZEVxvL\nyU6X7Fci1CySd3/LgGQtPWcyPoahYwot+VWd14t2DyNCL0hz+l\nPbq6PUaFUuK4jIJwEyYHhWYmaCLHZc6+qVfCZWVmquw6Sgqpad\nTz0yRNxQ5D7W8hAILcwFj9cIRy1moYSLv+IpfhGmSMDWs/JW17",
  "WVmquw6Sgqpad\nTz0yRNxQ5D7W8hAILcwFj9cIRy1moYSLv+IpfhGmSMDWs/JW17Q\nlMFY+FqvhZWU/qZNJ1mqHQ/EmY+Xl7rQVoXki3nPSK2YRm4Qe\nDypKr4YL2IgOACxyAlIFS+gTZOfIPKWEYVFLAFXzcqA9eC9mZCm\nleYx5KSjHRENCpnk461SiyYyqSj7IDiefc8A7jOYRZgqPDF0Rz\nsZExNrupPtZ5UhUmhnvImYp53QVcsikuaKuoU",
  "qSj7IDiefc8A7jOYRZgqPDF0Rz\nsZExNrupPtZ5UhUmhnvImYp53QVcsikuaKuoUopoWrYsX7D1h\numTtvEpVk91NxEkLWbdx2d07yoYdepI8iCRh3rTqCLAlHzpAlD\nLcluEIyRPRNyqUFgVZGFu5WnQ7TszEbw2xnsl63VpH0nzOU\nEROA3We+BVMh7+qr6dT2rpJzXvumwMfeCarW6U5O691AlfVxib\nUrHOFTJotCOXpRdc0o3GoPBPdCzQB",
  "6dT2rpJzXvumwMfeCarW6U5O691AlfVxib\nUrHOFTJotCOXpRdc0o3GoPBPdCzQBvOnKXKjomvagLsGSNWH/AV\nxqXkp+vLD4Ex/3qyWzbcw/JvQUFmroZM+H80NISbHF5fEMGTl\n0o0eRCoJy+VcL6jqWM5XtgmUs8dFIRiUuhLtP1FrLp16gebJqg\nsULAtAvfTCg0yVHUlU3AyPANt2vHAgrRYbNYyLcqck8MPrWe\nI1Lo5FnNhblbdA1UaoX",
  "tAvfTCg0yVHUlU3AyPANt2vHAgrRYbNYyLcqck8MPrWe\nI1Lo5FnNhblbdA1UaoXtucDmtBW4OZzG6oHKNBk8gLdWQ5S\niZYzOl43d+oWGLuXZ/PeVN0WnF/Gy97Q/GBbNThiE/G6zj+YiJR\nR2J2oLnI2dbkliO/qCt6XK9PrJq/d19srRjh+s2JWm3HaXbdrg3\njICfbThGu0E8YlFHorbaEVKPWI7+oC13HjdcV+Fw3aYk7V7l0Wk\n73KmJln",
  "drg3\njICfbThGu0E8YlFHorbaEVKPWI7+oC13HjdcV+Fw3aYk7V7l0Wk\n73KmJln+0O+KamcekVA7NY18q/SaERU1F7RThMdIbEJYTMquBf\n/Hyo6Am0fXakJY3CpEVzMBLA25xJfQhLDYbOGu2cawuFQN9wq\nk9kImU0Ii89Zgq+6CWExpmLsFE9ZliGxCZE8jnAeRzSPGZYyl4R\nnJHPMCFlSrgWVj9KuZAJYGqPexo7OYAQyVajDNojlgq68w",
  "E8jnAeRzSPGZYyl4R\nnJHPMCFlSrgWVj9KuZAJYGqPexo7OYAQyVajDNojlgq68wrnyF\nrFiq7iPVfHezd0rBlq0ASwtEn2mOdvOjdZgFMj1muJGcCWRlN4\nBZ2tqhz9fQXRBV5kguiS0svKb2w9ILSA0sPKM0tJb8IguiNpeTX\nSRCdW3pO6b6l+5SWlpaU7lm6R2lkaUTpM0ufURpaGlK6aukqpdp\nS8kQKdwRLdykdWTqi9NDSQ0rfWvqW0he",
  "7lm6R2lkaUTpM0ufURpaGlK6aukqpdp\nS8kQKdwRLdykdWTqi9NDSQ0rfWvqW0heWvqD0yNIjSt9b+p7SJ5\nY+oZRZyihds3SNUm4peXUQRCuWrlAaWEp+8Fes3SL0szSjNKnl\nj6ldGgp+VUM9zNLyeMN3BgtlZS+tPQlpcJS8vstiF5b+prSxNKE\n0leWvqL0xNITSp9b+pzS2FLybgCeTizdodS+BaoKSrct3ab0zNI\nz93sBPp3GwLUwN20D",
  "qL0xNITSp9b+pzS2FLybgCeTizdodS+BaoKSrct3ab0zNI\nz93sBPp3GwLUwN20Dm5SmlqaUrltKfinAo4Slp+R5MlLtqXb1to\nmca5GacgdrM35Vm+Q8UlPuYO3pdFWbnE+RmvIRGfra/vRFCqQUT\nvrB7PwyfgtLC/s/Li7/vPhw+H845X2De3t3ne973s/9JZ7j3qP\ney96W729Xjhzd+bRzOZJ3O/z/059fc3416a6at802v85n79z9\nZJRM",
  "/9JZ7j3qP\ney96W729Xjhzd+bRzOZJ3O/z/059fc3416a6at802v85n79z9\nZJRM5</latexit>\n\u02c6\u03c6 = argmin\n\u03c6\n2\n4X\nj\n\u2212 log\nh\n1 \u2212 sig[f[x\u21e4\nj, \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\nDiscriminator uses standard cross entropy loss:\nDiscriminator:  generated samples, y = 0,   real examples, y = 1:\nGenerator loss:  make generated samples more likely under discriminator (i.e. make discriminator loss larger)\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z",
  "a1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z4aQSDrnIoVxZClPgcpZLqmw3sABDAPYaDzV0rQ0bVotIb\nVgQDX3XJN/l7+SyiGvpZlp9F5zyFQN07zv0+tWb1qCTIpCr6z\n8PfOu+9/8GHNz6+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5GqfX2R8V7CYiUiETIN\nof",
  "D3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5GqfX2R8V7CYiUiETIN\nof78v/6Q6coPomwoxkvTf/SQazb2fvN8lseJUP2p4d/2JY/0c\nR1no76VZ6Qok371dnzXl2nsr4lYHq/6t+8CToJ0VBUiHh83xW\nhagByeiQbRpTlyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuqksNiZfHb6t74a+IM0L",
  "yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuqksNiZfHb6t74a+IM0LBO\nudChZURyvrmS6V7Fci1Dy8U2/LHjGwhMW82MoKpbwolfVwzT2\n7kBk4EVpDl+lvTp69YiKJUVxkQRgJkwPC8xM0MWOSx392quE\nykrNVdhUFJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4",
  "5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4nXG2rP9WR\naheSIuOUlSKybJNQKPx1XFl+NlDAQHIJY5AaniBeQ0/QMzah\nVRWBUScGXn3MsxSa0j6FPWtprokEhk3zUstaJBUOZtJQ9UDz\nvjmcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras37H1kqmTScelWd3U3E",
  "MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras37H1kqmTScelWd3U3ESQtZ+3HZ3TflGDt\nlNHkAWTMG5bdQRZEvawAYM9Zjwtw2aTJ56JuFWhsCrIxNzJ0\n6Bd2YieG6OMlgvbW+jIt1/xlCPmACsPvMrmAp5W19PZ7Y37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdh",
  "Q5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdhfglPNS8mP7y7/xEe9asU\nsG/OH9CYkKsrMlciE/0eiAVw18fyCB68VKLBg0A9eKmE/R0N\nHcvxDaReuygIBSTQl+g5S9i1T6mjuDGpglqKwRMXvhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA",
  "+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA9WjQ9GeQlmrActSZIzOkoz\nd+oWGJuVZ/PeRN0WnF/HRzUh+0C0anDEN+2t/E4xETizoS5YI\nbLmcuSxHfZBrNl2vtqzafPMjmdqxw3WbkuSdtNJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJ",
  "IesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJvQljUVNROMU14jMQmhMWkbFvwP\n1b2BFw82lYTwuJOIdqaCWBpwCU+hSaExWYJt81JDKtbDnXLr\nTKZDZHZhLD4hCX4rJsQFmMqxk7xhGUZEpsQ6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuN",
  "OSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuNRVrhKaAJa2yRrz/G3nIgtwF8Ntlqu\nTM4GsjHbgDnZ2qDO9+wuitzJBdGFpReUnlt6TumhpYeU5paS\nJ4IgemkpeToJojNLzyg9sPSA0tLSktKupV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tfQpa\n8tfU3paWXlD609",
  "NKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tfQpa\n8tfU3paWXlD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8u\nwHa83SHUozSzNKH1n6iNKBpeSpGK5nlpLbG7gwWiopfWbpM0q\nFpeT5LYheWPqC0sTShNLnlj6n9K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYpTS1NKd20\nlDwpwK2Ep",
  "BuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYpTS1NKd20\nlDwpwK2EpSfkfjJSk1t+raJ7GuRmnEHm/S4fR9LjBl3sMnuN\nD2a7E+RmvEhafrGwexFCnQp7PT9+cV/BaWFg7uLa/+vHx/9/\n7ig7XJG9obnW863a+76x2fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6,",
  "6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<l ate xit  sha 1_b ase 64= \"pC +2wo geC Ss9 5gd EtD YYk U9OA Po= \">A AAX eHi clZ hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN",
  "z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN 07z v0+ tWb 1qC TIp Cr6z 8Pf fOu ++9 /8G HNz 66+f Enn 372 +fy tLw 6KtM xD3 g1T meZ HAS u4F Ip3t dCS H2U 5Z0 kg+ WFws m74 4Rn PC5 Gqf X2R 8V7C YiU iET INo f78 v/6Q 6co Pom wox kvT f/S Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly",
  "UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly aZqz 5y0 11f Z6J lHe 8yA RfJs qoD GzG py5 g2j UFyi Dn4 t4q Kc/ /fn Fle WV+u PRw uqk sNi ZfH b6t7 4a+ IM0 LBO udC hZU Ryvr mS6 V7F ci1 Dy8 U2/L HjG whM W82 MoK pbw olfV wzT 27k Bk4 EVp Dl+ lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha",
  "UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha ZIw Naj 8tY 3dM QwI j4Wq +Gl Zz5 Lxu O1s 1A6 H4nX G2r P9W Rah eSI uOUl SKy bJN QKP x1X Fl+ NlDA QHI JY5 Aan iBe Q0/Q Mza hVR WBU ScG Xn3 MsxS a00 j6F PWt pro kEhk 3zU sta JBU OZt JQ9 UDzv jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl",
  "ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl NHk AWTM G5b dQR ZEv awA YM9 Zjwt w2a TJ5 6Ju FWh sCr IxNz J06 Bdd 2Yi eG6 OMlg vbW +jI t1/ xlC PmA CsPv Mrm Ap5 W19 PZ7 Y37Z yz2 jcF PvK GMF jtQ 5rd9 0ol cFa T2J iad V8h k/YW hPL 0vG 2a1 jhU non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/",
  "lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/ R0N Hcv xxD aRe uyg IBST Ql+ g5S 9i1 T6m juDG pgl qKw RMX vhl QqF BjqK 2bA JGh l+4 /js mUI hOMm zOM ZRp Uea cbH 5oPk Ok1 s22 mAt zsW pvq NII7 X2D y9l RUI aLw xm/5 vAA 9Wj Q9G eQl mrA ctSZ IzO koz d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd",
  "/PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd tNJ tO9 xrW sBPt xyt 3SI esa gjU a5J C6lH LEd 9kM vdj 1uu s3C 4blO SvN N+d NoO d2a i6R/ t1/ fNU Erl wNz 2pd JvQl jUV NRO MU1 4jM QmhM Wkb Fvw P1b 2BF w82 lYTw uJO Idq aCW Bpw CU+ hSaE xWY Jt8 1JD Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2",
  "ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2 XQA pkqV OEk iOW Czr zCO fMU msWK zuK uq+ LuN RVr hhKa AJa 2yR rz/ G3n Igt wF8N tlq uTM 4Gs jHb gDn Z2qD O9+ wui itz JBd GFpR eUn lt6 Tum hpY eU5 paSJ 4Ig emk peT oJo jNLz yg9 sPS A0t LSk tKu pV1K I0s jSh 9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG",
  "0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG 5Ry S8m rgy Bas 3SN0 sBS 8uw Ha8 3SH Uoz SzNK H1n 6iN KBp eSp GK5 nlpL bG7 gwW iop fWb pM0q Fpe T5L Yhe WPq C0s TShN Lnl j6n 9K2 lby l9Yu kTS mNL ybs BuD uxd I9S+ xao Kij dtX SX0 lNL T93v Bfh sGA PXx Ny2 CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ",
  "NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ p7P T9+ cVV/ BaW Fg7 uLa /+v Hx/ 9/7i g7X JG9 obn W86 33a+ 76x 2fu k86 Dzt 7HS 6nXA umP tj7 s+5 v77 +Z8 Fb+G 7hh 0Z9 Z25 yzJ ed1m fh3 n8t XjE 3</ lat exit >\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi,",
  "\u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\nsubstituted the generator function \nfor the generated sample\n37",
  "GAN Cost function\n<latexit sha1\n_base64=\"pC+2wogeCSs95gd\nEtDYkU9OAPo=\">AXeHicl\nZhJU9xGFICHrI6z4aQSDrnIoV\nxZClPgcpZLqmw3sABDAPYaD\nzV0rQ0bVotIbVgQDX3XJN/l7\n+SyiGvpZlp9F5zyFQN07zv0+t\nWb1qCTIpCr6z8PfOu+9/8G\nHNz6+fEn372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R",
  "372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R\n8V7CYiUiETINof78v/6Q6coP\nomwoxkvTf/SQazb2fvN8lseJU\nP2p4d/2JY/0cR1no76VZ6Qok\n371dnzXl2nsr4lYHq/6t+8CT\noJ0VBUiHh83xWhagByeiQbRpT\nlyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwu",
  "lyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwuqksNiZfHb\n6t74a+IM0LBOudChZURyvrmS\n6V7Fci1Dy8U2/LHjGwhMW82Mo\nKpbwolfVwzT27kBk4EVpDl+l\nvTp69YiKJUVxkQRgJkwPC8xM\n0MWOSx392quEykrNVdhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQ",
  "dhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQwIj4Wq+GlZz5LxuO1s1A\n6H4nXG2rP9WRaheSIuOUlSKy\nbJNQKPx1XFl+NlDAQHIJY5Aa\nniBeQ0/QMzahVRWBUScGXn3Ms\nxSa0j6FPWtprokEhk3zUsta\nJBUOZtJQ9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZU",
  "Q9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZUKSUc\nGras37H1kqmTScelWd3U3ESQ\ntZ+3HZ3TflGDtlNHkAWTMG5bd\nQRZEvawAYM9Zjwtw2aTJ56Ju\nFWhsCrIxNzJ06Bd2YieG6OM\nlgvbW+jIt1/xlCPmACsPvMrmA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1",
  "mA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1jhUnon2CZoAXnR\nlLlR0RVuqSzBlTdhfglPNS8m\nP7y7/xEe9asUsG/OH9CYkKsr\nMlciE/0eiAVw18fyCB68VKLB\ng0A9eKmE/R0NHcvxDaReuyg\nIBSTQl+g5S9i1T6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkO",
  "6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkOk1s2mAtzsWpvqNI7X2Dy\n9lRUIaLwxm/5vA9WjQ9GeQlm\nrActSZIzOkozd+oWGJuVZ/Pe\nRN0WnF/HRzUh+0C0anDEN+2t/\nE4xETizoS5YIbLmcuSxHfZB\nrNl2vtqzafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us",
  "zafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us3C4\nblOSvN+dNoOd2ai6R/t1/fN\nUErlwNz2pdJvQljUVNROMU14j\nMQmhMWkbFvwP1b2BFw82lYTw\nuJOIdqaCWBpwCU+hSaExWYJt\n81JDKtbDnXLrTKZDZHZhLD4hC\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASy",
  "C\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASyNUG0jR2XQApk\nqVOEkiOWCzrzCOfMUmsWKzuK\nuq+LuNRVrhKaAJa2yRrz/G3\nnIgtwF8NtlquTM4GsjHbgDnZ2\nqDO9+wuitzJBdGFpReUnlt6\nTumhpYeU5paSJ4IgemkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7U",
  "emkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7UrgiWLpP6dDSIaVHlh5R+\nsrSV5Q+tfQpa8tfU3paWXlD\n609CGlzFJG6YalG5RyS8mrgy\nBas3SN0sBS8uwHa83SHUozSz\nNKH1n6iNKBpeSpGK5nlpLbG7g\nwWiopfWbpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xao",
  "bpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xaoKijd\ntXSX0lNLT93vBfhsGAPXxNy2\nCbYpTS1NKd20lDwpwK2EpSfk\nfjJSk1t+raJ7GuRmnEHm/S4f\nR9LjBl3sMnuND2a7E+RmvEha\nfrGwexFCnQp7PT9+cV/BaWF\ng7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+",
  "g7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+Z8Fb+G7h0\nZ9Z25yzJed1mfh3n8tXjE3</l\natexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\nThe discriminator parameters, \ud835\udf19, are manipulated to minimize the loss function\nThe generator parameters, \ud835\udf03, are manipulated to maximize the loss function.\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYk",
  "are manipulated to minimize the loss function\nThe generator parameters, \ud835\udf03, are manipulated to maximize the loss function.\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z4aQSDrnIoVxZClPgcpZLqmw3sABDAPYaDzV0rQ0bVotIb\nVgQDX3XJN/l7+SyiGvpZlp9F5zyFQN07zv0+tWb1qCTIpCr6z\n8PfOu+9/8GHNz6+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WF",
  "PfOu+9/8GHNz6+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5GqfX2R8V7CYiUiETIN\nof78v/6Q6coPomwoxkvTf/SQazb2fvN8lseJUP2p4d/2JY/0c\nR1no76VZ6Qok371dnzXl2nsr4lYHq/6t+8CToJ0VBUiHh83xW\nhagByeiQbRpTlyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/f",
  "hagByeiQbRpTlyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuqksNiZfHb6t74a+IM0LBO\nudChZURyvrmS6V7Fci1Dy8U2/LHjGwhMW82MoKpbwolfVwzT2\n7kBk4EVpDl+lvTp69YiKJUVxkQRgJkwPC8xM0MWOSx392quE\nykrNVdhUFJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNa",
  "2quE\nykrNVdhUFJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4nXG2rP9WR\naheSIuOUlSKybJNQKPx1XFl+NlDAQHIJY5AaniBeQ0/QMzah\nVRWBUScGXn3MsxSa0j6FPWtprokEhk3zUstaJBUOZtJQ9UDz\nvjmcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTR",
  "UOZtJQ9UDz\nvjmcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras37H1kqmTScelWd3U3ESQtZ+3HZ3TflGDt\nlNHkAWTMG5bdQRZEvawAYM9Zjwtw2aTJ56JuFWhsCrIxNzJ0\n6Bd2YieG6OMlgvbW+jIt1/xlCPmACsPvMrmAp5W19PZ7Y37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1",
  "MrmAp5W19PZ7Y37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdhfglPNS8mP7y7/xEe9asU\nsG/OH9CYkKsrMlciE/0eiAVw18fyCB68VKLBg0A9eKmE/R0N\nHcvxDaReuygIBSTQl+g5S9i1T6mjuDGpglqKwRMXvhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2",
  "juDGpglqKwRMXvhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA9WjQ9GeQlmrActSZIzOkoz\nd+oWGJuVZ/PeRN0WnF/HRzUh+0C0anDEN+2t/E4xETizoS5YI\nbLmcuSxHfZBrNl2vtqzafPMjmdqxw3WbkuSdtNJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOS",
  "mdqxw3WbkuSdtNJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJvQljUVNROMU14jMQmhMWkbFvwP\n1b2BFw82lYTwuJOIdqaCWBpwCU+hSaExWYJt81JDKtbDnXLr\nTKZDZHZhLD4hCX4rJsQFmMqxk7xhGUZEpsQ6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApk",
  "Mqxk7xhGUZEpsQ6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuNRVrhKaAJa2yRrz/G3nIgtwF8Ntlqu\nTM4GsjHbgDnZ2qDO9+wuitzJBdGFpReUnlt6TumhpYeU5paS\nJ4IgemkpeToJojNLzyg9sPSA0tLSktKupV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHl",
  "g9sPSA0tLSktKupV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tfQpa\n8tfU3paWXlD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8u\nwHa83SHUozSzNKH1n6iNKBpeSpGK5nlpLbG7gwWiopfWbpM0q\nFpeT5LYheWPqC0sTShNLnlj6n9K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfh",
  "0sTShNLnlj6n9K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYpTS1NKd20\nlDwpwK2EpSfkfjJSk1t+raJ7GuRmnEHm/S4fR9LjBl3sMnuN\nD2a7E+RmvEhafrGwexFCnQp7PT9+cV/BaWFg7uLa/+vHx/9/\n7ig7XJG9obnW863a+76x2fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mf",
  "obnW863a+76x2fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<l ate xit  sha 1_b ase 64= \"pC +2wo geC Ss9 5gd EtD YYk U9OA Po= \">A AAX eHi clZ hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV",
  "YYk U9OA Po= \">A AAX eHi clZ hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN 07z v0+ tWb 1qC TIp Cr6z 8Pf fOu ++9 /8G HNz 66+f Enn 372 +fy tLw 6KtM xD3 g1T meZ HAS u4F Ip3t dCS H2U 5Z0 kg+ WFws m74 4Rn PC5 Gqf X2R 8V7C YiU iET INo f78 v/6Q 6co Pom wox kvT f/S Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH",
  "f78 v/6Q 6co Pom wox kvT f/S Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly aZqz 5y0 11f Z6J lHe 8yA RfJs qoD GzG py5 g2j UFyi Dn4 t4q Kc/ /fn Fle WV+u PRw uqk sNi ZfH b6t7 4a+ IM0 LBO udC hZU Ryvr mS6 V7F ci1 Dy8 U2/L HjG whM W82 MoK pbw olfV wzT 27k Bk4 EVp Dl+ lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi",
  "MoK pbw olfV wzT 27k Bk4 EVp Dl+ lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha ZIw Naj 8tY 3dM QwI j4Wq +Gl Zz5 Lxu O1s 1A6 H4nX G2r P9W Rah eSI uOUl SKy bJN QKP x1X Fl+ NlDA QHI JY5 Aan iBe Q0/Q Mza hVR WBU ScG Xn3 MsxS a00 j6F PWt pro kEhk 3zU sta JBU OZt JQ9 UDzv jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra",
  "3zU sta JBU OZt JQ9 UDzv jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl NHk AWTM G5b dQR ZEv awA YM9 Zjwt w2a TJ5 6Ju FWh sCr IxNz J06 Bdd 2Yi eG6 OMlg vbW +jI t1/ xlC PmA CsPv Mrm Ap5 W19 PZ7 Y37Z yz2 jcF PvK GMF jtQ 5rd9 0ol cFa T2J iad V8h k/YW hPL 0vG 2a1 jhU non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs",
  "k/YW hPL 0vG 2a1 jhU non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/ R0N Hcv xxD aRe uyg IBST Ql+ g5S 9i1 T6m juDG pgl qKw RMX vhl QqF BjqK 2bA JGh l+4 /js mUI hOMm zOM ZRp Uea cbH 5oPk Ok1 s22 mAt zsW pvq NII7 X2D y9l RUI aLw xm/5 vAA 9Wj Q9G eQl mrA ctSZ IzO koz d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx",
  "Q9G eQl mrA ctSZ IzO koz d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd tNJ tO9 xrW sBPt xyt 3SI esa gjU a5J C6lH LEd 9kM vdj 1uu s3C 4blO SvN N+d NoO d2a i6R/ t1/ fNU Erl wNz 2pd JvQl jUV NRO MU1 4jM QmhM Wkb Fvw P1b 2BF w82 lYTw uJO Idq aCW Bpw CU+ hSaE xWY Jt8 1JD Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi",
  "Bpw CU+ hSaE xWY Jt8 1JD Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2 XQA pkqV OEk iOW Czr zCO fMU msWK zuK uq+ LuN RVr hhKa AJa 2yR rz/ G3n Igt wF8N tlq uTM 4Gs jHb gDn Z2qD O9+ wui itz JBd GFpR eUn lt6 Tum hpY eU5 paSJ 4Ig emk peT oJo jNLz yg9 sPS A0t LSk tKu pV1K I0s jSh 9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8",
  "LSk tKu pV1K I0s jSh 9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG 5Ry S8m rgy Bas 3SN0 sBS 8uw Ha8 3SH Uoz SzNK H1n 6iN KBp eSp GK5 nlpL bG7 gwW iop fWb pM0q Fpe T5L Yhe WPq C0s TShN Lnl j6n 9K2 lby l9Yu kTS mNL ybs BuD uxd I9S+ xao Kij dtX SX0 lNL T93v Bfh sGA PXx Ny2 CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN",
  "lNL T93v Bfh sGA PXx Ny2 CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ p7P T9+ cVV/ BaW Fg7 uLa /+v Hx/ 9/7i g7X JG9 obn W86 33a+ 76x 2fu k86 Dzt 7HS 6nXA umP tj7 s+5 v77 +Z8 Fb+G 7hh 0Z9 Z25 yzJ ed1m fh3 n8t XjE 3</ lat exit >\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi,",
  "\u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n38\n\ud835\udf19\n\ud835\udf03",
  "GAN Cost function\n<latexit sha1\n_base64=\"pC+2wogeCSs95gd\nEtDYkU9OAPo=\">AXeHicl\nZhJU9xGFICHrI6z4aQSDrnIoV\nxZClPgcpZLqmw3sABDAPYaD\nzV0rQ0bVotIbVgQDX3XJN/l7\n+SyiGvpZlp9F5zyFQN07zv0+t\nWb1qCTIpCr6z8PfOu+9/8G\nHNz6+fEn372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R",
  "372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R\n8V7CYiUiETINof78v/6Q6coP\nomwoxkvTf/SQazb2fvN8lseJU\nP2p4d/2JY/0cR1no76VZ6Qok\n371dnzXl2nsr4lYHq/6t+8CT\noJ0VBUiHh83xWhagByeiQbRpT\nlyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwu",
  "lyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwuqksNiZfHb\n6t74a+IM0LBOudChZURyvrmS\n6V7Fci1Dy8U2/LHjGwhMW82Mo\nKpbwolfVwzT27kBk4EVpDl+l\nvTp69YiKJUVxkQRgJkwPC8xM\n0MWOSx392quEykrNVdhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQ",
  "dhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQwIj4Wq+GlZz5LxuO1s1A\n6H4nXG2rP9WRaheSIuOUlSKy\nbJNQKPx1XFl+NlDAQHIJY5Aa\nniBeQ0/QMzahVRWBUScGXn3Ms\nxSa0j6FPWtprokEhk3zUsta\nJBUOZtJQ9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZU",
  "Q9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZUKSUc\nGras37H1kqmTScelWd3U3ESQ\ntZ+3HZ3TflGDtlNHkAWTMG5bd\nQRZEvawAYM9Zjwtw2aTJ56Ju\nFWhsCrIxNzJ06Bd2YieG6OM\nlgvbW+jIt1/xlCPmACsPvMrmA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1",
  "mA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1jhUnon2CZoAXnR\nlLlR0RVuqSzBlTdhfglPNS8m\nP7y7/xEe9asUsG/OH9CYkKsr\nMlciE/0eiAVw18fyCB68VKLB\ng0A9eKmE/R0NHcvxDaReuyg\nIBSTQl+g5S9i1T6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkO",
  "6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkOk1s2mAtzsWpvqNI7X2Dy\n9lRUIaLwxm/5vA9WjQ9GeQlm\nrActSZIzOkozd+oWGJuVZ/Pe\nRN0WnF/HRzUh+0C0anDEN+2t/\nE4xETizoS5YIbLmcuSxHfZB\nrNl2vtqzafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us",
  "zafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us3C4\nblOSvN+dNoOd2ai6R/t1/fN\nUErlwNz2pdJvQljUVNROMU14j\nMQmhMWkbFvwP1b2BFw82lYTw\nuJOIdqaCWBpwCU+hSaExWYJt\n81JDKtbDnXLrTKZDZHZhLD4hC\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASy",
  "C\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASyNUG0jR2XQApk\nqVOEkiOWCzrzCOfMUmsWKzuK\nuq+LuNRVrhKaAJa2yRrz/G3\nnIgtwF8NtlquTM4GsjHbgDnZ2\nqDO9+wuitzJBdGFpReUnlt6\nTumhpYeU5paSJ4IgemkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7U",
  "emkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7UrgiWLpP6dDSIaVHlh5R+\nsrSV5Q+tfQpa8tfU3paWXlD\n609CGlzFJG6YalG5RyS8mrgy\nBas3SN0sBS8uwHa83SHUozSz\nNKH1n6iNKBpeSpGK5nlpLbG7g\nwWiopfWbpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xao",
  "bpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xaoKijd\ntXSX0lNLT93vBfhsGAPXxNy2\nCbYpTS1NKd20lDwpwK2EpSfk\nfjJSk1t+raJ7GuRmnEHm/S4f\nR9LjBl3sMnuND2a7E+RmvEha\nfrGwexFCnQp7PT9+cV/BaWF\ng7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+",
  "g7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+Z8Fb+G7h0\nZ9Z25yzJed1mfh3n8tXjE3</l\natexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<latexit sha1_base64=\"IV8qJV\nqOXtrIamF5uo3eYnirTsA=\">AXk3icrZjbtw2EIbX6Sl1T0",
  "1_base64=\"IV8qJV\nqOXtrIamF5uo3eYnirTsA=\">AXk3icrZjbtw2EIbX6Sl1T0mL1Be\n9EWqkKArHsIv0ABQFEjvOyU7txMfE2iwoLaVlTFGyRNlrC/s0vW0fq\nG/ToaRdWjP0RYEu4IiZ7+eQHA4pikEmRaFXVv6Zu/He+x98+NHNj+c/\n+fSz7+4dfvLgyIt85Dvh6lM86OAFVwKxfe10JIfZTlnSD5YXCybvj\nhGc8Lkao9fZHxfsJiJSIR",
  "gyIt85Dvh6lM86OAFVwKxfe10JIfZTlnSD5YXCybvj\nhGc8Lkao9fZHxfsJiJSIRMg2mwe25r7eO/SDKRqLvfe75xdlMqjeTb\nx7vkxjf03E8nj1np8E6bgqRDw5borRtABVPWMNoktTbQkKesQ163tL\nrde+cZL3wWHtWkysY6fbIBoPBKrsq1SVScBz35+vu9u2cbXD/3N/B7c\nWV5ZX6p9HC6tYbHX/nYGt+8M/WEalglXOpSsKI5XVzLdr1i",
  "vu9u2cbXD/3N/B7c\nWV5ZX6p9HC6tYbHX/nYGt+8M/WEalglXOpSsKI5XVzLdr1iuRSj5ZN\n4vC56x8ITF/BiKiW86Ff1DE68u2AZelGaw5/SXm29WqNiSVFcJAEoE\n6ZHBWbG6GLHpY5+7VdCZaXmKmwaikrp6dQz6eANRc5DLS+gwMJcQF+\n9cMRyFmpImnlf8fMwTRKmhpW/tvFyAgHksVAVPy3rBJpMupqNWsOheJ\n1i7dnezIvQPBGXnDip",
  "nlf8fMwTRKmhpW/tvFyAgHksVAVPy3rBJpMupqNWsOheJ\n1i7dnezIvQPBGXnDipJcbJNQIeT6qKL8fLGAgOQCxzAlLFC/Bp4gMZs\nIoLBgJuLI58mpCXCvNY4hJR/aGyKCQST7uqNaJCqYy6Uh2QeJ5dz0D\nuM5hFqCr8OBoDnYzpibTepqPdZ5UhbHhFnKmYl43AUMOmTQj6ipUKS\nVUDTuqP7DqFVMnbeDSrO5qbixItZd3NTqncVHDrqa2IB",
  "mYl43AUMOmTQj6ipUKS\nVUDTuqP7DqFVMnbeDSrO5qbixItZd3NTqncVHDrqa2IBUkYdxV1Rak\nrC9DVnCIMpteQADTjxjcUuFwlJBEnMnT4Nu25mx4NwcZ7BeurqNioT/\njKGIGAOsPvMUTIW8K19PZ2pvGpyzWm8KfOyNYLK6VgeN8OaNgKjam0\nTqxjhZQ0WmDK0/Ou0vTGIeWZ6A7QGPCiK3OhoiuypboEKWvM/hIMN\nS8lP763/BMf96sVs",
  "Q0WmDK0/Ou0vTGIeWZ6A7QGPCiK3OhoiuypboEKWvM/hIMN\nS8lP763/BMf96sVs2zMPySa4KgoM5cjY/4PjobwQsX5BRY8ealEkweG\nevJSCfs7mjqW48Q2lnruoCAUk0JfoOUvYtWtU1twZ9ME9RUMxi8mVB\nokqOoKzYGI4YnHA0cCRSiQYbNGEOZFmXOyeaH8hkstdxsi7kwL6vuhi\nqNoLtvcDmrBWV4OZzxa6oHKJBE8gLdWQ5SiYzOl47",
  "yeaH8hkstdxsi7kwL6vuhi\nqNoLtvcDmrBWV4OZzxa6oHKJBE8gLdWQ5SiYzOl47d+oWGJuVZ/\nPeVN0amK+elm2x70C2anDEN+OtjE8xETFdVI5AvOYk5fkqgc7YGvWbp\ne7Vm1+fYHktqxQ+tWSuK37aVb7dBe0wN+uXo7RbRERXVSOSr7SHVEZ\nWjPfDljuOWaxQOrVspid9pHJ1qh3amROkf7ZkzqTkmpXJojn2p9BsTF\nmoq1E5hmvAYCRs",
  "juOWaxQOrVspid9pHJ1qh3amROkf7ZkzqTkmpXJojn2p9BsTF\nmoq1E5hmvAYCRsTFiZlVwX/x5JdAS+PrqoxYeFOIboyY8CiIZd4CI0\nJC5sl3FW2Nizdcki3FImsxFSNiYsfMISPOrGhIUxFcZO4QnLMiRsTC\nSOIxzHEY1jhkWZS4RnJHPMCEkpV0Llo7QrMgYsGqPWxo7GoAcyVajB1\nojFBc28wpl5CmWxolm872p4/5qGNUMOjQGLtska8",
  "QrMgYsGqPWxo7GoAcyVajB1\nojFBc28wpl5CmWxolm872p4/5qGNUMOjQGLtska8/xt5yILcIjhmOUK\nciaQKqMB3MGaHaqZnv6CqCInuSC6sPSC0nNLzyk9tPSQ0txS8kUQRK\n8sJV8nQXRm6RmlB5YeUFpaWlK6b+k+pZGlEaWPLX1MaWhpSOm6peuUa\nkvJiRTeCJbuUTqydETpkaVHlL629DWlTy19SukbS9QemnpJaUPLX1I\nKbOUbph6Qa",
  "vJiRTeCJbuUTqydETpkaVHlL629DWlTy19SukbS9QemnpJaUPLX1I\nKbOUbph6Qal3FJydRBEa5auURpYSr79YK1ZukNpZmlG6SNLH1E6tJ\nR8FcP7zFJyvIEXo6WS0meWPqNUWEq+34LohaUvKE0sTSh9bulzSt9Z+\no7SJ5Y+oTS2lNwNwOnE0l1K7S1QVD60tKXlJ5aeuq+F+CzaQxciblt\nHWxTmlqaUrpKflSgKOEpSfkPBmpdleb3jaRfS",
  "D60tKXlJ5aeuq+F+CzaQxciblt\nHWxTmlqaUrpKflSgKOEpSfkPBmpdleb3jaRfS1SM+5gbcTt/SlRzLi\nDtbvTtDbZnyI14yPS9Y2D2UKhBR2+sGtxV8C0sLBz8ur/68fP/l/\nbeFRI70x19b5qtf5Lbz4F+C4OPs=</latexit>cUHa+0N7c3eN71ve9/3Vnu/9B70nvZ2evu9cG4y9+fcX3N/L9xZ+G1h\nL[\u03c6] =\nX\nj\n\u2212 log\nh\n1 \u2212 sig[f[g[zj, \u2713],",
  "9/3Vnu/9B70nvZ2evu9cG4y9+fcX3N/L9xZ+G1h\nL[\u03c6] =\nX\nj\n\u2212 log\nh\n1 \u2212 sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\nL[\u2713] =\nX\nj\nlog\nh\n1 \u2212 sig[f[g[zj, \u2713], \u03c6]]\ni\nThe discriminator parameters, \ud835\udf19, are manipulated to minimize the loss function\nThe generator parameters, \ud835\udf03, are manipulated to maximize the loss function.\n \nCan divide into two parts:\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z4aQSDrnIoVxZClPgcpZLqmw3sABDAPYa",
  "U9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z4aQSDrnIoVxZClPgcpZLqmw3sABDAPYaDzV0rQ0bVotIb\nVgQDX3XJN/l7+SyiGvpZlp9F5zyFQN07zv0+tWb1qCTIpCr6z\n8PfOu+9/8GHNz6+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5GqfX2R8V7CYiUiETIN\nof78v/6Q6coPomwoxkvTf/SQazb2fvN8ls",
  "Fwsm74RnPC5GqfX2R8V7CYiUiETIN\nof78v/6Q6coPomwoxkvTf/SQazb2fvN8lseJUP2p4d/2JY/0c\nR1no76VZ6Qok371dnzXl2nsr4lYHq/6t+8CToJ0VBUiHh83xW\nhagByeiQbRpTlyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuqksNiZfHb6t74a+IM0LBO\nudChZURyvrmS6V7Fci1Dy8U2/LHjG",
  "fnFleWV+uPRwuqksNiZfHb6t74a+IM0LBO\nudChZURyvrmS6V7Fci1Dy8U2/LHjGwhMW82MoKpbwolfVwzT2\n7kBk4EVpDl+lvTp69YiKJUVxkQRgJkwPC8xM0MWOSx392quE\nykrNVdhUFJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4nXG2rP9WR\naheSIuOUlSKybJNQKPx1XF",
  "aj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4nXG2rP9WR\naheSIuOUlSKybJNQKPx1XFl+NlDAQHIJY5AaniBeQ0/QMzah\nVRWBUScGXn3MsxSa0j6FPWtprokEhk3zUstaJBUOZtJQ9UDz\nvjmcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras37H1kqmTScelWd3U3ESQtZ+3HZ3TflGDt\nlNHkAWTMG5bdQRZE",
  "Rn1DZUKSUcGras37H1kqmTScelWd3U3ESQtZ+3HZ3TflGDt\nlNHkAWTMG5bdQRZEvawAYM9Zjwtw2aTJ56JuFWhsCrIxNzJ0\n6Bd2YieG6OMlgvbW+jIt1/xlCPmACsPvMrmAp5W19PZ7Y37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdhfglPNS8mP7y7/xEe9asU\nsG/OH9CYkKs",
  "1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdhfglPNS8mP7y7/xEe9asU\nsG/OH9CYkKsrMlciE/0eiAVw18fyCB68VKLBg0A9eKmE/R0N\nHcvxDaReuygIBSTQl+g5S9i1T6mjuDGpglqKwRMXvhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA9WjQ9GeQlmrActSZIzOkoz\nd+oWGJuVZ",
  "2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA9WjQ9GeQlmrActSZIzOkoz\nd+oWGJuVZ/PeRN0WnF/HRzUh+0C0anDEN+2t/E4xETizoS5YI\nbLmcuSxHfZBrNl2vtqzafPMjmdqxw3WbkuSdtNJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJvQljUVNROMU14jMQmhMWkbFvwP\n1b2BF",
  "SvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJvQljUVNROMU14jMQmhMWkbFvwP\n1b2BFw82lYTwuJOIdqaCWBpwCU+hSaExWYJt81JDKtbDnXLr\nTKZDZHZhLD4hCX4rJsQFmMqxk7xhGUZEpsQ6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuNRVrhKaAJa2yRrz/G3nIgtwF8Ntlqu\nTM",
  "kqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuNRVrhKaAJa2yRrz/G3nIgtwF8Ntlqu\nTM4GsjHbgDnZ2qDO9+wuitzJBdGFpReUnlt6TumhpYeU5paS\nJ4IgemkpeToJojNLzyg9sPSA0tLSktKupV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tfQpa\n8tfU3paWXlD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8",
  "lh5R+srSV5Q+tfQpa\n8tfU3paWXlD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8u\nwHa83SHUozSzNKH1n6iNKBpeSpGK5nlpLbG7gwWiopfWbpM0q\nFpeT5LYheWPqC0sTShNLnlj6n9K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYpTS1NKd20\nlDwpwK2EpSfkfjJSk1t+raJ7GuRmnEHm/S4fR9LjB",
  "hsGAPXxNy2CbYpTS1NKd20\nlDwpwK2EpSfkfjJSk1t+raJ7GuRmnEHm/S4fR9LjBl3sMnuN\nD2a7E+RmvEhafrGwexFCnQp7PT9+cV/BaWFg7uLa/+vHx/9/\n7ig7XJG9obnW863a+76x2fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6,",
  "6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<l ate xit  sha 1_b ase 64= \"pC +2wo geC Ss9 5gd EtD YYk U9OA Po= \">A AAX eHi clZ hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN",
  "z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN 07z v0+ tWb 1qC TIp Cr6z 8Pf fOu ++9 /8G HNz 66+f Enn 372 +fy tLw 6KtM xD3 g1T meZ HAS u4F Ip3t dCS H2U 5Z0 kg+ WFws m74 4Rn PC5 Gqf X2R 8V7C YiU iET INo f78 v/6Q 6co Pom wox kvT f/S Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly",
  "UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly aZqz 5y0 11f Z6J lHe 8yA RfJs qoD GzG py5 g2j UFyi Dn4 t4q Kc/ /fn Fle WV+u PRw uqk sNi ZfH b6t7 4a+ IM0 LBO udC hZU Ryvr mS6 V7F ci1 Dy8 U2/L HjG whM W82 MoK pbw olfV wzT 27k Bk4 EVp Dl+ lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha",
  "UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha ZIw Naj 8tY 3dM QwI j4Wq +Gl Zz5 Lxu O1s 1A6 H4nX G2r P9W Rah eSI uOUl SKy bJN QKP x1X Fl+ NlDA QHI JY5 Aan iBe Q0/Q Mza hVR WBU ScG Xn3 MsxS a00 j6F PWt pro kEhk 3zU sta JBU OZt JQ9 UDzv jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl",
  "ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl NHk AWTM G5b dQR ZEv awA YM9 Zjwt w2a TJ5 6Ju FWh sCr IxNz J06 Bdd 2Yi eG6 OMlg vbW +jI t1/ xlC PmA CsPv Mrm Ap5 W19 PZ7 Y37Z yz2 jcF PvK GMF jtQ 5rd9 0ol cFa T2J iad V8h k/YW hPL 0vG 2a1 jhU non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/",
  "lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/ R0N Hcv xxD aRe uyg IBST Ql+ g5S 9i1 T6m juDG pgl qKw RMX vhl QqF BjqK 2bA JGh l+4 /js mUI hOMm zOM ZRp Uea cbH 5oPk Ok1 s22 mAt zsW pvq NII7 X2D y9l RUI aLw xm/5 vAA 9Wj Q9G eQl mrA ctSZ IzO koz d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd",
  "/PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd tNJ tO9 xrW sBPt xyt 3SI esa gjU a5J C6lH LEd 9kM vdj 1uu s3C 4blO SvN N+d NoO d2a i6R/ t1/ fNU Erl wNz 2pd JvQl jUV NRO MU1 4jM QmhM Wkb Fvw P1b 2BF w82 lYTw uJO Idq aCW Bpw CU+ hSaE xWY Jt8 1JD Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2",
  "ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2 XQA pkqV OEk iOW Czr zCO fMU msWK zuK uq+ LuN RVr hhKa AJa 2yR rz/ G3n Igt wF8N tlq uTM 4Gs jHb gDn Z2qD O9+ wui itz JBd GFpR eUn lt6 Tum hpY eU5 paSJ 4Ig emk peT oJo jNLz yg9 sPS A0t LSk tKu pV1K I0s jSh 9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG",
  "0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG 5Ry S8m rgy Bas 3SN0 sBS 8uw Ha8 3SH Uoz SzNK H1n 6iN KBp eSp GK5 nlpL bG7 gwW iop fWb pM0q Fpe T5L Yhe WPq C0s TShN Lnl j6n 9K2 lby l9Yu kTS mNL ybs BuD uxd I9S+ xao Kij dtX SX0 lNL T93v Bfh sGA PXx Ny2 CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ",
  "NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ p7P T9+ cVV/ BaW Fg7 uLa /+v Hx/ 9/7i g7X JG9 obn W86 33a+ 76x 2fu k86 Dzt 7HS 6nXA umP tj7 s+5 v77 +Z8 Fb+G 7hh 0Z9 Z25 yzJ ed1m fh3 n8t XjE 3</ lat exit >\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi,",
  "\u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\ndiscriminator loss:\nnegated generator loss:\nThe 2nd term is constant w.r.t. \ud835\udf03 \n(gradient \n\u2044\n(\u2112\n(* = 0) so we can drop it)\n39",
  "GAN Solution\noThe solution is the Nash equilibrium\noIt lays at a saddle point\noIs inherently unstable\n40\n<latexit sha1\n_base64=\"pC+2wogeCSs95gd\nEtDYkU9OAPo=\">AXeHicl\nZhJU9xGFICHrI6z4aQSDrnIoV\nxZClPgcpZLqmw3sABDAPYaD\nzV0rQ0bVotIbVgQDX3XJN/l7\n+SyiGvpZlp9F5zyFQN07zv0+t\nWb1qCTIpCr6z8PfOu+9/8G\nHNz6+fEn372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ",
  "r6z8PfOu+9/8G\nHNz6+fEn372+fytLw6KtMx\nD3g1TmeZHASu4FIp3tdCSH2U5\nZ0kg+WFwsm74RnPC5GqfX2R\n8V7CYiUiETINof78v/6Q6coP\nomwoxkvTf/SQazb2fvN8lseJU\nP2p4d/2JY/0cR1no76VZ6Qok\n371dnzXl2nsr4lYHq/6t+8CT\noJ0VBUiHh83xWhagByeiQbRpT\nlyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiD",
  "BUiHh83xWhagByeiQbRpT\nlyaZqz5y01fZ6JlHe8yARfJ\nsqoDGzGpy5g2jUFyiDn4t4qKc\n/fnFleWV+uPRwuqksNiZfHb\n6t74a+IM0LBOudChZURyvrmS\n6V7Fci1Dy8U2/LHjGwhMW82Mo\nKpbwolfVwzT27kBk4EVpDl+l\nvTp69YiKJUVxkQRgJkwPC8xM\n0MWOSx392quEykrNVdhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM",
  "C8xM\n0MWOSx392quEykrNVdhUFJXS0\n6lnxtwbiJyHWl5AgYW5gLZ64\nZDlLNQwM276ip+HaZIwNaj8t\nY3dMQwIj4Wq+GlZz5LxuO1s1A\n6H4nXG2rP9WRaheSIuOUlSKy\nbJNQKPx1XFl+NlDAQHIJY5Aa\nniBeQ0/QMzahVRWBUScGXn3Ms\nxSa0j6FPWtprokEhk3zUsta\nJBUOZtJQ9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsR",
  "tprokEhk3zUsta\nJBUOZtJQ9UDzvjmcA1zmMAjQ\nVfjgag72MqfH0OM1HOk+qwsRw\nDTlTMa+rgFMOmTRn1DZUKSUc\nGras37H1kqmTScelWd3U3ESQ\ntZ+3HZ3TflGDtlNHkAWTMG5bd\nQRZEvawAYM9Zjwtw2aTJ56Ju\nFWhsCrIxNzJ06Bd2YieG6OM\nlgvbW+jIt1/xlCPmACsPvMrmA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT",
  "gvbW+jIt1/xlCPmACsPvMrmA\np5W19PZ7Y37Zyz2jcFPvKGMF\njtQ5rd90olcFaT2JiadV8hk/\nYWhPL0vG2a1jhUnon2CZoAXnR\nlLlR0RVuqSzBlTdhfglPNS8m\nP7y7/xEe9asUsG/OH9CYkKsr\nMlciE/0eiAVw18fyCB68VKLB\ng0A9eKmE/R0NHcvxDaReuyg\nIBSTQl+g5S9i1T6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUI",
  "DaReuyg\nIBSTQl+g5S9i1T6mjuDGpglq\nKwRMXvhlQqFBjqK2bAJGhl+4/\njsmUIhOMmzOMZRpUeacbH5oP\nkOk1s2mAtzsWpvqNI7X2Dy\n9lRUIaLwxm/5vA9WjQ9GeQlm\nrActSZIzOkozd+oWGJuVZ/Pe\nRN0WnF/HRzUh+0C0anDEN+2t/\nE4xETizoS5YIbLmcuSxHfZB\nrNl2vtqzafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesag",
  "5YIbLmcuSxHfZB\nrNl2vtqzafPMjmdqxw3WbkuS\ndtNJtO9xrWsBPtxyt3SIesagj\nUa5JC6lHLEd9kMvdj1us3C4\nblOSvN+dNoOd2ai6R/t1/fN\nUErlwNz2pdJvQljUVNROMU14j\nMQmhMWkbFvwP1b2BFw82lYTw\nuJOIdqaCWBpwCU+hSaExWYJt\n81JDKtbDnXLrTKZDZHZhLD4hC\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwi",
  "JDKtbDnXLrTKZDZHZhLD4hC\nX4rJsQFmMqxk7xhGUZEpsQ6c\nch7sch7cMS5lLwiOSOUaETC\nnXhMqHaVsyASyNUG0jR2XQApk\nqVOEkiOWCzrzCOfMUmsWKzuK\nuq+LuNRVrhKaAJa2yRrz/G3\nnIgtwF8NtlquTM4GsjHbgDnZ2\nqDO9+wuitzJBdGFpReUnlt6\nTumhpYeU5paSJ4IgemkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b",
  "Unlt6\nTumhpYeU5paSJ4IgemkpeToJ\nojNLzyg9sPSA0tLSktKupV1KI\n0sjSh9b+pjS0NKQ0nVL1ynVl\npI7UrgiWLpP6dDSIaVHlh5R+\nsrSV5Q+tfQpa8tfU3paWXlD\n609CGlzFJG6YalG5RyS8mrgy\nBas3SN0sBS8uwHa83SHUozSz\nNKH1n6iNKBpeSpGK5nlpLbG7g\nwWiopfWbpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk",
  "eSpGK5nlpLbG7g\nwWiopfWbpM0qFpeT5LYheWPq\nC0sTShNLnlj6n9K2lbyl9Yuk\nTSmNLybsBuDuxdI9S+xaoKijd\ntXSX0lNLT93vBfhsGAPXxNy2\nCbYpTS1NKd20lDwpwK2EpSfk\nfjJSk1t+raJ7GuRmnEHm/S4f\nR9LjBl3sMnuND2a7E+RmvEha\nfrGwexFCnQp7PT9+cV/BaWF\ng7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86D",
  "rGwexFCnQp7PT9+cV/BaWF\ng7uLa/+vHx/9/7ig7XJG9obnW\n863a+76x2fuk86Dzt7HS6nX\nAumPtj7s+5v7+Z8Fb+G7h0\nZ9Z25yzJed1mfh3n8tXjE3</l\natexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AX",
  "\u03c6]]\ni\n3\n5\n3\n5\n<latexit sha1_base64=\"pC+\n2wogeCSs95gdEtDYkU9OAPo=\">AXeHiclZhJU9xGFICHrI\n6z4aQSDrnIoVxZClPgcpZLqmw3sABDAPYaDzV0rQ0bVotIb\nVgQDX3XJN/l7+SyiGvpZlp9F5zyFQN07zv0+tWb1qCTIpCr6z\n8PfOu+9/8GHNz6+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5Gqf",
  "+fEn372+fytLw6KtMxD3g1TmeZHASu4\nFIp3tdCSH2U5Z0kg+WFwsm74RnPC5GqfX2R8V7CYiUiETIN\nof78v/6Q6coPomwoxkvTf/SQazb2fvN8lseJUP2p4d/2JY/0c\nR1no76VZ6Qok371dnzXl2nsr4lYHq/6t+8CToJ0VBUiHh83xW\nhagByeiQbRpTlyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuq",
  "lyaZqz5y01fZ6JlHe8yARfJsqoDGzGpy5g2\njUFyiDn4t4qKc/fnFleWV+uPRwuqksNiZfHb6t74a+IM0LBO\nudChZURyvrmS6V7Fci1Dy8U2/LHjGwhMW82MoKpbwolfVwzT2\n7kBk4EVpDl+lvTp69YiKJUVxkQRgJkwPC8xM0MWOSx392quE\nykrNVdhUFJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4W",
  "FJXS06lnxtwbiJyHWl5AgYW5gLZ64ZDlLNQwM276i\np+HaZIwNaj8tY3dMQwIj4Wq+GlZz5LxuO1s1A6H4nXG2rP9WR\naheSIuOUlSKybJNQKPx1XFl+NlDAQHIJY5AaniBeQ0/QMzah\nVRWBUScGXn3MsxSa0j6FPWtprokEhk3zUstaJBUOZtJQ9UDz\nvjmcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras",
  "mcA1zmMAjQVfjgag72MqfH0OM1HOk+qwsRwDTlTMa+rgFMO\nmTRn1DZUKSUcGras37H1kqmTScelWd3U3ESQtZ+3HZ3TflGDt\nlNHkAWTMG5bdQRZEvawAYM9Zjwtw2aTJ56JuFWhsCrIxNzJ0\n6Bd2YieG6OMlgvbW+jIt1/xlCPmACsPvMrmAp5W19PZ7Y37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAX",
  "37Z\nyz2jcFPvKGMFjtQ5rd90olcFaT2JiadV8hk/YWhPL0vG2a1jh\nUnon2CZoAXnRlLlR0RVuqSzBlTdhfglPNS8mP7y7/xEe9asU\nsG/OH9CYkKsrMlciE/0eiAVw18fyCB68VKLBg0A9eKmE/R0N\nHcvxDaReuygIBSTQl+g5S9i1T6mjuDGpglqKwRMXvhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7",
  "vhlQqFBj\nqK2bAJGhl+4/jsmUIhOMmzOMZRpUeacbH5oPkOk1s2mAtzs\nWpvqNI7X2Dy9lRUIaLwxm/5vA9WjQ9GeQlmrActSZIzOkoz\nd+oWGJuVZ/PeRN0WnF/HRzUh+0C0anDEN+2t/E4xETizoS5YI\nbLmcuSxHfZBrNl2vtqzafPMjmdqxw3WbkuSdtNJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6",
  "NJtO9xrWsB\nPtxyt3SIesagjUa5JC6lHLEd9kMvdj1us3C4blOSvN+dNoO\nd2ai6R/t1/fNUErlwNz2pdJvQljUVNROMU14jMQmhMWkbFvwP\n1b2BFw82lYTwuJOIdqaCWBpwCU+hSaExWYJt81JDKtbDnXLr\nTKZDZHZhLD4hCX4rJsQFmMqxk7xhGUZEpsQ6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz",
  "Q6ch7sch7cMS5\nlLwiOSOUaETCnXhMqHaVsyASyNUG0jR2XQApkqVOEkiOWCzrz\nCOfMUmsWKzuKuq+LuNRVrhKaAJa2yRrz/G3nIgtwF8Ntlqu\nTM4GsjHbgDnZ2qDO9+wuitzJBdGFpReUnlt6TumhpYeU5paS\nJ4IgemkpeToJojNLzyg9sPSA0tLSktKupV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tf",
  "upV1KI0sjSh9b+pjS0\nNKQ0nVL1ynVlpI7UrgiWLpP6dDSIaVHlh5R+srSV5Q+tfQpa\n8tfU3paWXlD609CGlzFJG6YalG5RyS8mrgyBas3SN0sBS8u\nwHa83SHUozSzNKH1n6iNKBpeSpGK5nlpLbG7gwWiopfWbpM0q\nFpeT5LYheWPqC0sTShNLnlj6n9K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYp",
  "K2lbyl9YukTSmNLybsBuDux\ndI9S+xaoKijdtXSX0lNLT93vBfhsGAPXxNy2CbYpTS1NKd20\nlDwpwK2EpSfkfjJSk1t+raJ7GuRmnEHm/S4fR9LjBl3sMnuN\nD2a7E+RmvEhafrGwexFCnQp7PT9+cV/BaWFg7uLa/+vHx/9/\n7ig7XJG9obnW863a+76x2fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</la",
  "fuk86Dzt7HS6nXAumPtj7s+5v7\n7+Z8Fb+G7h0Z9Z25yzJed1mfh3n8tXjE3</latexit>\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\n<l ate xit  sha 1_b ase 64= \"pC +2wo geC Ss9 5gd EtD YYk U9OA Po= \">A AAX eHi clZ hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG",
  "hJU9 xGF ICH rI6 z4a QSDr nIo VxZ ClP gcp ZLq mww3 sAB DAP YaD zV0 rQ0b Vot IbV gQD X3X JN/ l7+S yiG vpZ lp9 F5z yFQN 07z v0+ tWb 1qC TIp Cr6z 8Pf fOu ++9 /8G HNz 66+f Enn 372 +fy tLw 6KtM xD3 g1T meZ HAS u4F Ip3t dCS H2U 5Z0 kg+ WFws m74 4Rn PC5 Gqf X2R 8V7C YiU iET INo f78 v/6Q 6co Pom wox kvT f/S Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x",
  "Qazb 2fv N8l seJ UP2 p4d /2JY /0c R1n o76 VZ6 Qok3 71d nzX l2n sr4 lYH q/6t +8C ToJ 0VB UiH h83x Wha gBy eiQ bRp Tly aZqz 5y0 11f Z6J lHe 8yA RfJs qoD GzG py5 g2j UFyi Dn4 t4q Kc/ /fn Fle WV+u PRw uqk sNi ZfH b6t7 4a+ IM0 LBO udC hZU Ryvr mS6 V7F ci1 Dy8 U2/L HjG whM W82 MoK pbw olfV wzT 27k Bk4 EVp Dl+ lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ",
  "lvTp 69Y iKJ UVx kQR gJkw PC8 xM0 MWO Sx3 92q uEyk rNV dhU FJX S06 lnxt wbi JyH Wl5 AgY W5g LZ64 ZDl LNQ wM2 76i p+Ha ZIw Naj 8tY 3dM QwI j4Wq +Gl Zz5 Lxu O1s 1A6 H4nX G2r P9W Rah eSI uOUl SKy bJN QKP x1X Fl+ NlDA QHI JY5 Aan iBe Q0/Q Mza hVR WBU ScG Xn3 MsxS a00 j6F PWt pro kEhk 3zU sta JBU OZt JQ9 UDzv jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt",
  "jmc A1z mMA jQV fjg ag72 Mqf H0O M1H Ok+ qwsR wDT lTM a+r gFM OmT Rn1D ZUK SUc Gra s37 H1kq mTS cel Wd3 U3E SQt Z+3H Z3T flG Dtl NHk AWTM G5b dQR ZEv awA YM9 Zjwt w2a TJ5 6Ju FWh sCr IxNz J06 Bdd 2Yi eG6 OMlg vbW +jI t1/ xlC PmA CsPv Mrm Ap5 W19 PZ7 Y37Z yz2 jcF PvK GMF jtQ 5rd9 0ol cFa T2J iad V8h k/YW hPL 0vG 2a1 jhU non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68",
  "non2 CZo AXn RlL lR0 RVu qSzB lTd hfg lPN S8m P7y7 /xE e9a sUs G/O H9C YkKs rMl ciE /0e iAV w18 fyCC B68 VKL Bg0 A9e KmE/ R0N Hcv xxD aRe uyg IBST Ql+ g5S 9i1 T6m juDG pgl qKw RMX vhl QqF BjqK 2bA JGh l+4 /js mUI hOMm zOM ZRp Uea cbH 5oPk Ok1 s22 mAt zsW pvq NII7 X2D y9l RUI aLw xm/5 vAA 9Wj Q9G eQl mrA ctSZ IzO koz d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd",
  "d+o WGJ uVZ /PeR N0W nF/ HRz Uh+ 0C0a nDE N+2 t/E 4xE Tiz oS5Y IbL mcu SSx HfZ BrNl 2vt qza fPM jmd qxw 3Wbk uSd tNJ tO9 xrW sBPt xyt 3SI esa gjU a5J C6lH LEd 9kM vdj 1uu s3C 4blO SvN N+d NoO d2a i6R/ t1/ fNU Erl wNz 2pd JvQl jUV NRO MU1 4jM QmhM Wkb Fvw P1b 2BF w82 lYTw uJO Idq aCW Bpw CU+ hSaE xWY Jt8 1JD Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA",
  "Ktb DnXL rTK ZDZ HZh LD4 hCX 4rJs QFm Mqx k7x hGU ZEps Q6c ch7 sch 7cc MS5 lLwi OSO UaE TCn XhM qHa VsyA SyN UG0 jR2 XQA pkqV OEk iOW Czr zCO fMU msWK zuK uq+ LuN RVr hhKa AJa 2yR rz/ G3n Igt wF8N tlq uTM 4Gs jHb gDn Z2qD O9+ wui itz JBd GFpR eUn lt6 Tum hpY eU5 paSJ 4Ig emk peT oJo jNLz yg9 sPS A0t LSk tKu pV1K I0s jSh 9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF",
  "9b+ pjS 0NK Q0nV L1y nVl pI7 Urg iWLp P6d DSI aVH lh5 R+s rSV5 Q+t fQp pa8 tfU 3ppa WXl D60 9CG lzF JG6 YalG 5Ry S8m rgy Bas 3SN0 sBS 8uw Ha8 3SH Uoz SzNK H1n 6iN KBp eSp GK5 nlpL bG7 gwW iop fWb pM0q Fpe T5L Yhe WPq C0s TShN Lnl j6n 9K2 lby l9Yu kTS mNL ybs BuD uxd I9S+ xao Kij dtX SX0 lNL T93v Bfh sGA PXx Ny2 CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw",
  "CbYp TS1 NKd 20l Dwp wK2 EpSf kfj JSk 11t +ra J7Gu Rmn EHm /S4 fR9 LjB l3sM nuN D2a 7E+ Rmv Eha frGw exF CnQ p7P T9+ cVV/ BaW Fg7 uLa /+v Hx/ 9/7i g7X JG9 obn W86 33a+ 76x 2fu k86 Dzt 7HS 6nXA umP tj7 s+5 v77 +Z8 Fb+G 7hh 0Z9 Z25 yzJ ed1m fh3 n8t XjE 3</ lat exit >\n\u02c6\u03c6, \u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi,",
  "\u02c6\u2713 = argmin\n\u03c6\n2\n4argmax\n\u2713\n2\n4X\nj\n\u2212 log\nh\n1\u2212sig[f[g[zj, \u2713], \u03c6]]\ni\n\u2212\nX\ni\nlog\nh\nsig[f[xi, \u03c6]]\ni\n3\n5\n3\n5\nNash equilibrium\nIn game theory, the Nash equilibrium, named after the \nmathematician John Nash, is the most common way to define \nthe solution of a non-cooperative game involving two or more \nplayers.\n\u2026each player is assumed to know the equilibrium strategies of \nthe other players, and no one has anything to gain by changing \nonly one's own strategy.  Wikipedia",
  "GAN Training Flow Pseudo Python\nfor c_gan_iter in range(n_gan_iters):  # GAN Iterations\n# Run generator to produce synthesized data\nx_syn = generator(z, theta)\n# Update/train the discriminator\nphi = update_discriminator(x_real, x_syn, n_iter_discrim, phi)\n# Update/train the generator\ntheta = update_generator(z, theta, n_iter_gen, phi)\n41\nSee Jupyter Notebook assignment (to be released shortly)",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n42",
  "Deep Convolutional (DC) GAN\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\n\u2022 Early GAN specialized in image generation\n...\n43",
  "DCGAN -- Generator\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\n\u2022 Input is 100D latent variable, z, drawn \nfrom a uniform distribution\n...\n44",
  "DCGAN -- Generator\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\n\u2022 Input is 100D latent variable, z, drawn \nfrom a uniform distribution\n\u2022 Maps to 4x4x1024 via a linear \ntransformation\n...\n45",
  "DCGAN -- Generator\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\n\u2022 Input is 100D latent variable, z, \ndrawn from a uniform distribution\n\u2022 Maps to 4x4x1024 via a linear \ntransformation\n\u2022 Fractionally strided (stride = 0.5) \nconvolutions to double resolution \nin each dimension\n\u2022 Final tanh to limit to [-1,1]\n\u2022 Rescaled to [0,255]\n...\n46",
  "DCGAN -- Discriminator\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\n\u2022 Standard convolution network\n\u2022 Reduces to 1x1\n\u2022 Final sigmoid to create output \nprobability\n...\n47",
  "Deep Convolutional (DC) GAN\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\nTrained as in the earlier example.\n...\n48",
  "Deep Convolutional (DC) GAN\nRadford et al, \u201cUnsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.\u201d,  2015\nWhen training is complete\nDiscard discriminator\nDraw new latent variable\nPass through generator\n...\n49",
  "DC GAN Results\nTrained on a faces dataset.\nTrained on ImageNet dataset.\nTrained on LSUN dataset.\nThe LSUN classification dataset \ncontains 10 scene categories, such \nas dining room, bedroom, chicken, \noutdoor church, and so on.\n50",
  "Common Failures with GANs\nMode Dropping: Only represent a \nsubset of the training distribution.\nMode Collapse: Extreme case \nwhere the generator mostly \nignores the latent variable and \ncollapses all samples to a few \npoints\n51",
  "GAN Performance and Distribution Distance\nSummary of lengthy analysis in \u00a715.2.1 \u201cAnalysis of GAN loss function\u201d\nCan be rewritten in terms of dissimilarities between generated and real \nprobability distributions.\nTwo important takeaways:\nQuality:  Generated samples need to occur where real samples are\nCoverage: Where there is concentrations of real samples, there should be \ngood representation from generated samples\n52",
  "We can conclude that:\n(i) the GAN loss can be interpreted in terms of distances between \nprobability distributions and that \n(ii) the gradient of this distance becomes zero when the generated \nsamples are too easy to distinguish from the real examples.\nWe need a distance metric with better properties.\n53",
  "Wassertein Distance (for continuous distributions)\nEarth Mover\u2019s Distance (for discrete probabilities)\n\u2022 The quantity of work required to transport the probability mass from \none distribution to create the other.\n\u2022 Use linear programming to find an optimal \u201ctransport plan\u201d that \nminimizes \u03a3 \ud835\udc0f \u22c5 |\ud835\udc56 \u2212 \ud835\udc57|\n54\nSee 15.2.4 Wasserstein distance for discrete distributions",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n55",
  "Trick 1:  Progressive growing\nTrain GAN to generate and discriminate 4x4 images\nKeep weights from step (a), add layers to get \nto/from 8x8 images and continue training GAN\nAdd layers to get to 16x16 and continue to train.\nRepeat above steps to get to high resolution.\nWolf (2021), Kerras (2018)\n56",
  "Trick 2: Minibatch discrimination\n\u2022 Add in statistics across minibatches of synthesized and real data\n\u2022 Provided to the discriminator as an additional feature map\n\u2022 Sends signal back to generator to try to better match real batch \nstatistics\n57",
  "Trick 3:  Truncation\n\u2022 Only choose random values of latent \nvariables that are less than a threshold \n\ud835\udf0f distance from the mean of the latent \nvariables.\n\u2022 Reduces variation but improves quality\n58",
  "Interpolation\n59\nWell-behaved latent space: Every latent variable z \nshould correspond to a plausible data example x and \nsmooth changes in z should correspond to smooth\nchanges in x.",
  "Interpolation\n60\nWell-behaved latent space: Every latent variable z \nshould correspond to a plausible data example x and \nsmooth changes in z should correspond to smooth\nchanges in x.",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n61",
  "Lack of control\n62\n\u2022 Cannot specify attributes of generated images from vanilla GANs\n\u2022 E.g. can\u2019t choose ethnicity, age, etc., for a GAN trained on faces.\n\u2022 Conditional generation models provide this control",
  "Conditional GAN models\n63\n\u2022 Passes a vector c of attributes to both the generator and discriminator\n\u2022 Generator learns to generate sample with correct attribute\n\u2022 Discriminator learns to distinguish between generated sample with \ntarget attribute and real sample with real attribute",
  "Auxiliary classifier GAN\n64\n\u2022 Similar to Conditional GAN, but use class label instead of attribute \nvector\n\u2022 Discriminator produces:\n\u2022 Binary real/fake classifier\n\u2022 Multi-class classifier",
  "Auxiliary Classifier GAN results\n65\nTrained on ImageNet images and classes.\nmonarch butterfly\ngoldfinch\ndaisies\nredshanks\ngray whales",
  "InfoGAN\n66\n\u2022 Add random attribute variables c to generator\n\u2022 Discriminator learns to predict discrete and continuous \nvalues of the attributes",
  "InfoGAN results\n67\nLearns classes\nLearns orientation\nLearns stroke thickness",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n68",
  "Image \ntranslation: \nPix2Pix\n69\n\u2022 Maps one image to a different style image using a U-Net type model\n\u2022 Adds a content loss (\u2113*norm) to make the input similar to ground \ntruth\n\u2022 Discriminator fed input/prediction and real/modified pairs to predict \nreal or fake\nIsola et al, \u201cImage-to-Image Translation with Conditional Adversarial Networks.\u201d 2016.",
  "Image translation: Pix2Pix\n70",
  "Image translation: SRGAN\n71",
  "Image translation: SRGAN\n72",
  "Image translation: CycleGAN\n73\nDoesn\u2019t need labeled or matched training \npairs.\nHave two sets of data with distinct styles \nbut no matching pairs.\nE.g. Horses and zebras, or photos and \nMonet paintings\nThree losses\n1. Content loss based on (\u2113$norm)\n2. Discriminator loss (real vs fake)\n3. Cycle-consistency loss\n2nd model is also trained.\nEncourages the generator to be \nreversible",
  "Image translation: CycleGAN\n74",
  "75",
  "GANs\n\u2022 GAN loss function\n\u2022 DCGAN results and problems\n\u2022 Tricks for improving performance\n\u2022 Conditional GANs\n\u2022 Image translation models\n\u2022 StyleGAN\n76",
  "Style GAN\n77\nSeparates style from \nnoise at different scales\nFace Examples\nLarge style changes: face \nshape, head pose\nMedium-scale changes: \nfacial features\nFine-scale: hair and skin \ncolor\nNoise: hair placement, \nfreckles\nlearned \nconstant",
  "This work is subject to a Creative Commons CC-BY-NC-ND license. (C) MIT Press.\n78",
  "Upcoming Topics                           Feedback\n\u2022 VAEs\n\u2022 Diffusion Models\n\u2022 Graph Neural Networks\n\u2022 Reinforcement Learning\nLink\n79",
  "Variational Autoencoders (VAEs)\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos\n Prince, Understanding Deep Learning,\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n Other Content Cited\n1",
  "April Dates\nSunday\nMonday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\nApril 1\n2\n3\n4\nGANs\n5\n6\n7\n8\n9\nVAEs\n10\nDiscussion\n11\nDiffusion Models\n12\n13\n14\n15\n16\nGraph Neural Nets\n(VizWiz Leaders \nShare)\n17\nDiscussion\n18\nReinforcement \nLearning\n19\n20\n21\n22\n23\nTBD/Overflow\n(JEPA Models)\n24\nDiscussion\n25\n\u2605 Project \nPresentations 1 \u2605 \n26\n27\n28\n29\n30\n\u2605 Project \nPresentations 2 \u2605 \nMay 1\nDiscussion??\n2\nStudy Period\n3\nStudy Period\n4\n5\n6\nFinal Exams\n7\n8\nFinal report\n& Repo **\n9\n10\n11\n2\n** Might be earlier. Depends on when grades are due.",
  "Diederik P. Kingma\n2016 OpenAI, founding member\n2017 PhD U. of Amsterdam\n2018\u2013 Google DeepMind\n3",
  "Variational Autoencoder\nVariational Inference:  A method from machine \nlearning that approximates probability densities \nthrough optimization.\nAutoencoder:  A type of artificial neural network \nused to learn efficient codings of unlabeled data in \nan unsupervised manner.\nVAE is an autoencoder whose encodings distribution is \nregularized during the training to ensure that its latent \nspace has good properties allowing us to generate new \ndata.\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n4",
  "Variational Inference:  A method from \nmachine learning that approximates \nprobability densities through optimization.\nAutoencoder:  A type of artificial neural \nnetwork used to learn efficient codings of \nunlabeled data in an unsupervised manner.\nBayesian since joint density is decomposed into \nprior and posterior density distributions using \nBayes Rule:\n\ud835\udc5d \ud835\udc33, \ud835\udc31 = \ud835\udc5d \ud835\udc31 \ud835\udc33  \ud835\udc5d(\ud835\udc33)\n5",
  "Outline\n\u2022 Autoencoder and its limitations\n\u2022 Intuition behind VAEs\n\u2022 Derivation of VAE\n\u2022 Example applications\n6",
  "Dimensionality reduction with an \nautoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n7",
  "Dimensionality reduction with an autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nWe want to find the best encoder, e, and \ndecoder, d, to minimize the error between x \nand d(e(x)).\n\ud835\udc52\u2217, \ud835\udc51\u2217 = argmin\n\",$ \u2208&\u00d7(\n\ud835\udf16(\ud835\udc65, \ud835\udc51 \ud835\udc52 \ud835\udc65\n)\nwhere\n\ud835\udf16(\ud835\udc65, \ud835\udc51 \ud835\udc52 \ud835\udc65\n)\nis the reconstruction error.\n8",
  "Dimensionality reduction with \nPrincipal Component Analysis (PCA)\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nProject the \ud835\udc5b$-dimensional features onto an \northogonal \ud835\udc5b\"-dimensional subspace that minimizes \nEuclidean distance.\n\ud835\udc5b$ = 2  \ud835\udc5b\" = 1\nLinear Transformation!!\n9",
  "Neural Network Autoencoder \u2013 1 Linear Layer\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nWe could define encoder and decoder to each \nhave one linear layer (no activation function), \nbut it wouldn\u2019t necessarily converge during \ntraining to PCA solution.\nbest linear subspace\nAE can end up with any basis\n10",
  "Neural Network Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n11",
  "Autoencoder Reconstruction\nKana, \"Variational Autoencoders (VAEs) for Dummies -- Step by Step Tutorial\", 2020 \nTrained on CelebA dataset.\n12",
  "Can we generate new samples with \nautoencoder?\nTrain encoder and decoder as autoencoder.\nRandomly select a different point in the latent space.\nProvide as input to the decoder to generate an \noutput.\nWill this produce a good quality output? \nWhy?\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n13",
  "Extreme case: Memorization\nEncoder and decoder are so powerful that they can fully \nmemorize the data. \nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n\u201ctraining\u201d data for \nthe autoencoder\npoint sampled from the \nlatent space for new \ncontent generation\nPerfectly decoded \nsamples.\nwithout regularization \nthe decoded output can \nbe meaningless\n14",
  "Outline\n\u2022 Autoencoder and its limitations\n\u2022 Intuition behind VAEs\n\u2022 Derivation of VAE\n\u2022 Example applications\n15",
  "Variational Autoencoder\nIs an autoencoder whose training is regularized to avoid overfitting and \nensure that the latent space has good properties that enable generative \nprocess.\nInstead of encoding as a single point, encode it as a distribution over the \nlatent space.\nAssume distributions are normal.\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n16",
  "Variational Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \n17",
  "Variational Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nEncoder is emitting \ud835\udf07) vector \nand \ud835\udf0e) diagonal vector for \nindependent gaussians \ndensities.\n18",
  "Variational Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nWe then sample z from the \nmultivariate Normal.\n19",
  "Variational Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nThen input z to the decoder \nnetwork to produce output.\n20",
  "Variational Autoencoder\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nKulback-Leibler divergence\nL2 Loss\nThe loss is now the L2 loss as \nwith the autoencoder, but with \nan additional KL-divergence \nterm as regularizer.\n21",
  "Intuitions about Regularization\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nirregular latent space\nregular latent space\npoints that are \nclose in latent \nspace but produce \ndissimilar decoded \noutputs\npoint in the latent \nspace that produce \nmeaningless \ndecoded output\npoints that are close in latent \nspace produce similar decoded \noutputs\n22",
  "Encoding to Normal distributions is not enough\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nwithout regularization\nwith regularization\nWe have to regularize the means and the covariances too!\nRegularize to a standard normal.\n23",
  "Benefit of regularization\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nThe continuity and completeness \nobtained from regularization \ntends to create a \u201cgradient\u201d over \nthe information encoded in latent \nspace.\n24",
  "Dall-E 3\n25",
  "Outline\n\u2022 Autoencoder and its limitations\n\u2022 Intuition behind VAEs\n\u2022 Derivation of VAE\n\u2022 Example applications\n26",
  "Preliminaries: Bayesian Models\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nN\nz\nx\n\ud835\udc5d(\ud835\udc67)\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nprior \u2013 prior knowledge or \nbelief about z\nlikelihood \u2013 probability \nof a sample given z\n27",
  "Bayesian Inference\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n28",
  "Bayesian Inference\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n\ud835\udc5d \ud835\udc67|\ud835\udc65 = \ud835\udc5d \ud835\udc65 \ud835\udc67 \ud835\udc5d(\ud835\udc67)\n\ud835\udc5d(\ud835\udc65)\nlikelihood \u2013 probability of \na sample given z\nposterior \u2013 update our \nknowledge of z given a \nnew sample\nWe can relate the posterior \nto the likelihood via\nBayes Theorem.\n29",
  "Bayesian Inference\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n\ud835\udc5d \ud835\udc67|\ud835\udc65 = \ud835\udc5d \ud835\udc65 \ud835\udc67 \ud835\udc5d(\ud835\udc67)\n\ud835\udc5d(\ud835\udc65)\nlikelihood \u2013 probability of \na sample given z\nprior \u2013 prior knowledge \nor belief about z\nposterior \u2013 update our \nknowledge of z given a \nnew sample\nevidence \u2013 probability distribution \nof our observed data\nlikelihood\nposterior\n30",
  "Bayesian Inference\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n\ud835\udc5d \ud835\udc67|\ud835\udc65 = \ud835\udc5d \ud835\udc65 \ud835\udc67 \ud835\udc5d(\ud835\udc67)\n\ud835\udc5d(\ud835\udc65)\nlikelihood \u2013 probability of \na sample given z\nprior \u2013 prior knowledge \nor belief about z\nposterior \u2013 update our \nknowledge of z given a \nnew sample\n=\n\ud835\udc5d \ud835\udc65 \ud835\udc67 \ud835\udc5d(\ud835\udc67)\n\u222b \ud835\udc5d \ud835\udc65 \ud835\udc67 \ud835\udc5d \ud835\udc67 \ud835\udc51\ud835\udc67\nlikelihood\nposterior\nWe can\u2019t calculate the \nintegral directly, but we can \napproximate it using \nvariational inference\n31",
  "Simplifying Assumptions\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n32\nAssume that the prior is a standard \nGaussian\n\ud835\udc5d(\ud835\udc67) \u2261 \ud835\udca9(0, \ud835\udc3c)\nAnd likelihood is a Gaussian\n\ud835\udc5d(\ud835\udc65|\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc53 \ud835\udc67 , \ud835\udc50\ud835\udc3c)\nwhere \ud835\udc53 \u2208 \ud835\udc39 is a family of functions we \nwill specify later and \ud835\udc50 > 0.\nlikelihood\nposterior",
  "Variational Inference Formulation\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nz\nProbabilistic Decoder\n\ud835\udc5d(\ud835\udc65|\ud835\udc67)\nProbabilistic Encoder\n\ud835\udc5d(\ud835\udc67|\ud835\udc65)\n33\nWe are going to approximate posterior \nto parameterized set of Gaussians.\nApproximate \ud835\udc5d(\ud835\udc67|\ud835\udc65) by a Gaussian \n\ud835\udc5e!(\ud835\udc67).\n\ud835\udc5e!(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\nwhere \ud835\udc54 \u2208 \ud835\udc3a and \u210e \u2208 \ud835\udc3b are a family of \nfunctions we will define shortly.\nlikelihood\nposterior",
  "Variational Inference\n34\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\nWe want to find the best functions, g and h, to minimize the KL-\ndivergence from the posterior \ud835\udc5d(\ud835\udc67|\ud835\udc65).",
  "Variational Inference\n35\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\n\u00d8 Rewriting KL divergence as Expectation,\n\u00d8 log of division is difference of the logs\n\u00d8 substituting for the posterior using Bayes Theorem",
  "Variational Inference\n36\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\n\u00d8 log of product becomes sum of logs\n\u00d8 log of division becomes difference of logs",
  "Variational Inference\n37\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\n\u00d8 negating and converting from argmin to argmax\n\u00d8 collecting terms to form KL divergence",
  "Variational Inference\n38\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\nMaximize the expected log \nlikelihood.\nMinimize the difference \nbetween the approximate \nposterior and the prior.",
  "Variational Inference\n39\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\nLog of the Gaussian likelihood \ud835\udc5d \ud835\udc65 \ud835\udc67 \u2261 \ud835\udca9 \ud835\udc53 \ud835\udc67 , \ud835\udc50\ud835\udc3c .\nThis brings our function, \ud835\udc53, into the equation, so\u2026",
  "Variational Inference\n40\n\ud835\udc5e)(\ud835\udc67) \u2261 \ud835\udca9(\ud835\udc54 \ud835\udc65 , \u210e \ud835\udc65 )\nWe are looking for optimal f*, g* and h* such that\nNote that the constant, c, determines the balance between \nreconstruction error and the regularization term given by KL \ndivergence.",
  "Enter the Neural Networks\n41\nEncoder produces the mean and \nvariance.\nDecoder reconstructs the input \n(during training)",
  "But one more problem to solve\n42\nWe can\u2019t backpropagate through the sampling step.",
  "Use the reparameterization trick\n43",
  "Putting it all together\n44\nWe have as trainable neural network!\nWe use a Monte-Carlo \napproximation to the \nexpectation of \nreconstruction loss\nConvert C = 1/(2c).",
  "Probability Distribution Divergence Measures\nPrince, Understanding Deep Learning\n45",
  "Dall-E 3\n46",
  "Outline\n\u2022 Autoencoder and its limitations\n\u2022 Intuition behind VAEs\n\u2022 Derivation of VAE\n\u2022 Example applications\n47",
  "Generating high quality images\n48\nVahdat & Kautz (2020) \u201cNVAE: A deep hierarchical variational autoencoder\u201d",
  "Resynthesizing real data with changes\n49",
  "Disentanglement of the latent space\n50\nChen et al (2021) \u201cCross-layer distillation with semantic calibration.\u201d",
  "Kingma and Welling, \u201cAuto-Encoding Variational Bayes.\u201d, 2013.\n51",
  "Conditional VAEs\nExample from https://towardsdatascience.com/variational-\nautoencoders-vaes-for-dummies-step-by-step-tutorial-69e6d1c9d8e9 \n52",
  "Amini et al, \u201cUncovering and Mitigating Algorithmic Bias through Learned Latent Structure,\u201d 2019\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\nDebiasing\n53",
  "A. Amini et al, \u201cVariational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing,\u201d 2018\n\u00a9 Alexander Amini and Ava Amini, MIT 6.S191: Introduction to Deep Learning, IntroToDeepLearning.com\nOutlier Detection\n54",
  "Upcoming Topics                           Feedback\n\u2022 Diffusion Models\n\u2022 Graph Neural Networks\n\u2022 Reinforcement Learning\nLink\n55",
  "Diffusion Models\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos\n Prince, Understanding Deep Learning,\nOther Content Cited\n1\nBased on Rocca, 2022, \"Understanding Diffusion Probabilistic Models (DPMs)\", Towards Data Science",
  "Outline\n\u2022 Contextualizing Diffusion Models\n\u2022 Theory behind diffusion models\n\u2022 Architectures and Training\n\u2022 Applications\n2",
  "Given a probability distribution only \ndescribed by some available samples, \nhow can one generate a new \nsample?\n3",
  "Introduction\nGenerative models aims at learning a function that takes data from a simple distribution and transform it into \ndata from a complex distribution.\n4\nRocca, 2022",
  "Retrospective\n\u2022 2013: Kingma and Welling introduce Variational AutoEncoder. Train an \nautoencoder with regularized latent space. Encoder is regularized towards \na gaussian distribution. Decoder is like the G() function. Takes a sample \nfrom the gaussian like distribution and produces new sample close the the \noriginal distribution.\n\u2022 2014: Goodfellow et al introduced Generative Adversarial Networks \n(GANs). Train a generative network, G(), to produce a sample from a \nrandom input such as a Gaussian distribution and output a sample \nindistinguishable from the target distribution. D tries to guess, G tries to \nfool D.\n\u2022 2015: Sohl-Dickstein \u201cDeep Unsupervised Learning using Nonequilibrium \nThermodynamics\u201d\n5",
  "Commercial Diffusion Solutions\n\u2022 Dall-E 2 and 3 from OpenAI\n\u2022 Imagen from Google\n\u2022 Make-A-Scene from Meta\n\u2022 Imagen Video from Google\n\u2022 Make-A-Video from Meta\n\u2022 Stable Diffusion 3 from stability.ai\n\u2022 Model Version 6 from midjourney.com\n6",
  "Examples\nExamples above have been generated by Meta Make-A-Scene model, that generates images from \nboth a text prompt and a basic sketch for greater level of creative control.\n7",
  "Basic idea of Diffusion Probabilistic Models\n\u2022 learn the reverse process of \n\u2022 a well defined stochastic forward process that progressively destroys \ninformation, taking data from our complex target distribution and \nbringing them to a simple gaussian distribution. \n\u2022 reverse process is then expected to take the path in the opposite \ndirection, taking gaussian noise as an input and generating data from \nthe distribution of interest.\n8\nRocca, 2022",
  "Outline\nFirst:\n\u2022 Stochastic Process\n\u2022 Diffusion Process\nThen intuition behind DPMs\nThen some math basis\nThen how trained in practice\n9",
  "Markov stochastic process\nStochastic Processes\n\u2022 Discrete:   \ud835\udc4b!,     \u2200 \ud835\udc5b \u2208  \u2115\n\u2022 Continuous:   \ud835\udc4b\",     \u2200\ud835\udc61 \u2265  0\nRealization of a random variable \u00e0 sample\nRealization of a stochastic process \u00e0 sample path or trajectory\n10",
  "Different types of stochastic processes\nDiscrete Time\nContinuous Time\nDiscrete \nValue\nContinuous \nValue\nCoin Flipping\nQueue Length Over Time\nDaily Average Temperature\nStock Price\n11\nRocca, 2022",
  "Different types of stochastic processes\nDiscrete Time\nContinuous Time\nDiscrete \nValue\nContinuous \nValue\nCoin Flipping\nQueue Length Over Time\nDaily Average Temperature\nStock Price\n12\nRocca, 2022",
  "Markov (Stochastic) Process\nA Markov process is a stochastic process with no memory.\nFuture behavior only depends on the present.\n\ud835\udc43 \ud835\udc4b\"! \ud835\udc4b\"!\"#, \u2026 , \ud835\udc4b\"$\n= \ud835\udc43 \ud835\udc4b\"! \ud835\udc4b\"!\"# \n\u2200\ud835\udc61# < \ud835\udc61$ <  \u2026 < \ud835\udc61!%$ < \ud835\udc61!\n13",
  "Diffusion Process\nAny diffusion process can be described by a stochastic differential \nequation (SDE)\n\ud835\udc51\ud835\udc4b\" = \ud835\udc4e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc4a\"\nwhere: \n\ud835\udc4e(\u22c5) is called the drift coefficient\n\ud835\udf0e(\u22c5) is called the diffusion coefficient\n\ud835\udc4a is the Wiener process\nBoth \ud835\udc4e and \ud835\udf0e are a function of the value and time\n14",
  "Stochastic part\nDiffusion Process\nAny diffusion process can be described by a stochastic differential \nequation (SDE)\n\ud835\udc51\ud835\udc4b\" = \ud835\udc4e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc4a\"\nwhere: \n\ud835\udc4e(\u22c5) is called the drift coefficient\n\ud835\udf0e(\u22c5) is called the diffusion coefficient\n\ud835\udc4a is the Wiener process\nSimple differential equation\n15",
  "Wiener Process (Brownian Motion)\nContinuous time stochastic process\nhttps://en.wikipedia.org/wiki/Wiener_process \nFive sampled \nprocesses\nExpected standard \ndeviation\n16",
  "Norbert Weiner\nNorbert Wiener (November 26, 1894 \u2013 March \n18, 1964) was an American computer scientist, \nmathematician and philosopher. He became a \nprofessor of mathematics at the Massachusetts \nInstitute of Technology (MIT). \nA child prodigy, Wiener later became an early \nresearcher in stochastic and mathematical noise \nprocesses, contributing work relevant to \nelectronic engineering, electronic \ncommunication, and control systems.\nWiener is considered the originator of \ncybernetics, the science of communication as it \nrelates to living things and machines.\nHeavily influenced John von Neumann, Claude \nShannon, etc\u2026\nWrote \u201cThe Machine Age\u201d in 1949 anticipating \nrobots, etc.\ngreat, great \ngrand advisor J\nhttps://en.wikipedia.org/wiki/Norbert_Wiener\nhttps://www.nytimes.com/2013/05/21/science/mit-scholars-1949-essay-on-machine-age-is-found.html  \n17",
  "Discretizing\nSo \n\ud835\udc51\ud835\udc4a\" \u2248 \ud835\udc4a\"()\" \u2212 \ud835\udc4a\" ~ \ud835\udca9(0, \ud835\udc51\ud835\udc61)\nDiscretizing the SDE\n\ud835\udc4b\"()\" \u2212 \ud835\udc4b\"  \u2248 \ud835\udc4e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc4b\", \ud835\udc61 \ud835\udc48     where   \ud835\udc48 ~ \ud835\udca9(0, \ud835\udc51\ud835\udc61)\nWhich can also be rewritten\n\ud835\udc4b\"()\"  \u2248 \ud835\udc4b\" + \ud835\udc4e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udc48\u2032   where    \ud835\udc48*~ \ud835\udca9(0, \ud835\udf0e \ud835\udc4b\", \ud835\udc61 \ud835\udc51\ud835\udc61) \nProperty of Weiner Process: \nThe std is equal to the time \nstep.\nDeterministic drift term\nNormal RV with std proportional to diffusion term\n18\nRocca, 2022",
  "Diffusion process samples\n19\nRocca, 2022",
  "Reversed time process\nIf \ud835\udc4b% is a diffusion process such that\n\ud835\udc51\ud835\udc4b% = \ud835\udc4e \ud835\udc4b%, \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc61 \ud835\udc51\ud835\udc4a%\nthen the reversed-time process, *\ud835\udc4b% = \ud835\udc4b&'% is also a diffusion process\n\ud835\udc51 *\ud835\udc4b% = \ud835\udc4e *\ud835\udc4b%, \ud835\udc61 \u2212 \ud835\udf0e( \ud835\udc61 \u2207)! log \ud835\udc5d(\ud835\udc4b%) \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc61 \ud835\udc51\ud835\udc4a%\n= *\ud835\udc4e *\ud835\udc4b%, \ud835\udc61 \ud835\udc51\ud835\udc61 + \ud835\udf0e \ud835\udc61 \ud835\udc51\ud835\udc4a%\nwhere \u2207)! log \ud835\udc5d(\ud835\udc4b%) is called the score function and \ud835\udc5d(\ud835\udc4b%) is the marginal probability of \ud835\udc4b%\n20\nRocca, 2022",
  "Intuition behind diffusion processes\nProgressively destroys relevant information\nE.g. with shrinking ( \ud835\udc4e < 1) drift coefficient and non-zero diffusion coefficient will \nturn complex distribution into isotropic gaussian\n21\nRocca, 2022",
  "Intuition behind diffusion processes\nTowards Gaussian\nFor the diffusion process\nAfter a given number of steps \ud835\udc47 we can write\nThis is a sum of independent gaussians, \nso can express as single gaussian with \nvariance the sum of the variances.\n22\nRocca, 2022",
  "Intuition behind diffusion processes\nTowards Gaussian\nAfter a given number of steps \ud835\udc47 we can write\nFor number of steps \ud835\udc47 large enough, we have\nSo for any starting point, we tend to a normal gaussian.\n23\nRocca, 2022\nVariance of the gaussian\nGeometric series",
  "Same idea but for images\nBut in \ud835\udc3b\u00d7\ud835\udc4a\u00d7\ud835\udc36 dimensions, e.g. 100\u00d7100\u00d73 for 100\u00d7100 resolution \nRGB images\n24\nRocca, 2022",
  "Why use diffusion?\nAnswer: \nGives us a progressive and structured way to go from a complex \ndistribution to an isotropic gaussian noise \nthat will enable the learning of the reverse process\n25",
  "Intuition behind learning the reverse process\nReversing process in one step is extremely difficult\n26\nRocca, 2022",
  "Doing it in steps gives us some clues\n27\nRocca, 2022",
  "One step versus multi-step\n28\nRocca, 2022",
  "Advantage of multi-step reverse process\n1. Don\u2019t have to learn a unique transform \nGi for each step, but rather a single \ntransform that is a function of the \nindex step. Drastically reduces size of \nthe model.\n2. Gradient descent is much more difficult \nin one step and can exploit coarse to \nfine adjustments in multiple steps,.\n29\nRocca, 2022",
  "Iterative versus one step\n30\nRocca, 2022",
  "Similarities and differences to VAEs\nSimilarities:\n\u2022 An encoder transforms a complex \ndistribution into a simple \ndistribution in a structured way to \nlearn a decoder that produces a \nsimilar sample\nDifferences:\n\u2022 DPM is multi-step, versus one step \nfor VAE\n\u2022 DPM encoder is fixed and does not \nget trained\n\u2022 DPM will be trained based on the \nstructure of the diffusion process\n\u2022 DPM latent space is exactly same \nas input, as opposed to VAE which \nreduces dimensionality\n31\nRocca, 2022",
  "Dall-E 3\n32",
  "Mathematics of Diffusion Models\nAssume the forward and reverse process operate in \ud835\udc47 steps.\nBoth forward and reverse process are discrete so becomes a Markov \nchain with gaussian transition probability.\n33",
  "Mathematics of Diffusion Models\nDenote \ud835\udc65# as a sample from a distribution \ud835\udc5e(\ud835\udc65#).\nForward process: gaussian transition probability\n\ud835\udc5e \ud835\udc65\" \ud835\udc65\"%$) = \ud835\udca9(\ud835\udc65\";\n1 \u2212 \ud835\udefd\"  \ud835\udc65\"%$, \ud835\udefd\"\ud835\udc3c)    where    \ud835\udc61 \u2208 \u2115\nand where \ud835\udefd\" indicates trade-off between info to be kept from previous \nstep and new noise added.\n\ud835\udca9(\ud835\udf07, \ud835\udf0e+)\n34\nRocca, 2022",
  "Mathematics of Diffusion Models\nDenote \ud835\udc65# as a sample from a distribution \ud835\udc5e(\ud835\udc65#).\nForward process: gaussian transition probability\n\ud835\udc5e \ud835\udc65\" \ud835\udc65\"%$) = \ud835\udca9(\ud835\udc65\";\n1 \u2212 \ud835\udefd\"  \ud835\udc65\"%$, \ud835\udefd\"\ud835\udc3c)    where    \ud835\udc61 \u2208 \u2115\nand where \ud835\udefd\" indicates trade-off between info to be kept from previous \nstep and new noise added.\nWe can equivalently write\n\ud835\udc65\" =\n(1 \u2212 \ud835\udefd\") \ud835\udc65\"%$ +\n\ud835\udefd\" \ud835\udf16\"       \ud835\udf16\"~\ud835\udca9(0, \ud835\udc3c)\n\ud835\udca9(\ud835\udf07, \ud835\udf0e+)\n35\nRocca, 2022\nDiscretized diffusion process",
  "Mathematics of Diffusion Models\nThrough recurrence, we can represent any step in the chain as directly \nrepresented from \ud835\udc65#:\n\ud835\udc5e \ud835\udc65\" \ud835\udc65#) = \ud835\udca9(\ud835\udc65\";\n\ud835\udefc\" \ud835\udc65#, (1 \u2212 J\ud835\udefc\")\ud835\udc3c)    \nwhere\n\ud835\udefc\" = (1 \u2212 \ud835\udefd\")    and    J\ud835\udefc\" =  \u220f,-$\n\"\n\ud835\udefc, = \u220f,-$\n\"\n(1 \u2212 \ud835\udefd,)\nand from the Markov property, the entire forward trajectory is\n\ud835\udc5e \ud835\udc65#:/ = \ud835\udc5e \ud835\udc65# L\n\"-$\n/\n\ud835\udc5e \ud835\udc65\" \ud835\udc65\"%$)\n\ud835\udca9(\ud835\udf07, \ud835\udf0e+)\n36\nRocca, 2022",
  "The reverse process\nWith the assumption on the drift and diffusion coefficients, the reverse \nof the diffusion process takes the same form.\nReverse gaussian transition probability\n\ud835\udc5e(\ud835\udc65\"%$|\ud835\udc65\")\ncan then be approximated by\n\ud835\udc5d0 \ud835\udc65\"%$ \ud835\udc65\") = \ud835\udca9(\ud835\udc65\"%$; \ud835\udf070 \ud835\udc65\", \ud835\udc61 , \u03a30 \ud835\udc65\", \ud835\udc61 )\nwhere \ud835\udf070 and  \u03a30 are two functions parameterized by \ud835\udf03 and learned.\n37\nRocca, 2022",
  "The reverse process\nUsing the Markov property, the probability of a given backward \ntrajectory can be approximated by\np0 \ud835\udc65#:/ = \ud835\udc5d \ud835\udc65/ L\n\"-$\n/\n\ud835\udc5d0 \ud835\udc65\"%$ \ud835\udc65\")\nwhere \ud835\udc5d \ud835\udc65/  is an isotropic gaussian distribution that does not depend \non \ud835\udf03\n\ud835\udc5d \ud835\udc65/ = \ud835\udca9(\ud835\udc65/; 0, \ud835\udc3c)\n38\nRocca, 2022",
  "39\nRocca, 2022",
  "Questions\nHow do we learn the parameters \ud835\udf03 for \ud835\udf070 and \u03a30? \nWhat is the loss to be optimized?\n\u2022 We hope that \ud835\udc5d0(\ud835\udc65#), the distribution of the last step of the reverse \nprocess, will be close to \ud835\udc5e(\ud835\udc65#)\n40",
  "Optimization Objective\n41",
  "Skipping a lot more math\n\u2022 Expand p-theta as marginalization integral\n\u2022 Use Jensen\u2019s inequality to define a slightly simpler upper bound to \nthe loss\n\u2022 Some manipulations with Bayes\u2019 Theorem\n\u2022 Properties of KL divergence of two gaussian distributions\n\u2022 An additional simplification suggested by [Ho et al 2020]\n42",
  "Diffusion models in practice\nWe have the forward process\nand our reverse process\nand we want to train to minimize this simplified upper bound\n43",
  "Dall-E 3 44",
  "Training Process\n45\nRocca, 2022",
  "To sample/generate\n46\nRocca, 2022",
  "To sample/generate\n47\nRocca, 2022",
  "U-Net (2016) as the Model\nOutput is same size as the input.\n48",
  "U-Net for reverse diffusion\n49",
  "Conditional generation\nClassifier Guidance\n\u2022 Modify the denoising update from \ud835\udc67\" to \ud835\udc67\"%$ to incorporate class \ninformation, \ud835\udc50.\nGuidance from text\n\u2022 Condition on a sentence embedding computed from a language \nmodel\n50",
  "Conditional generation using classifier \nguidance\nDhariwal and Nichol, \u201cDiffusion Models Beat GANs on Image Synthesis.\u201d 2021.\n51",
  "Cascaded \nconditional \ngeneration \nbased on a \ntext prompt\n52",
  "Conditional generation using text prompts\nC. Saharia et al., \u201cPhotorealistic Text-to-Image Diffusion Models with Deep Language Understanding,\u201d 2022.\n53",
  "Stable Diffusion \u2013 Latent Diffusion Models\nProject the original data to a smaller latent space using a conventional \nautoencoder and then run the diffusion process in the smaller space.\nRombach et al, \u201cHigh-Resolution Image Synthesis with Latent Diffusion Models,\u201d 2022\n54",
  "A Victorian-dressed butcher is \noperating a large, antique meat \ngrinder. Instead of meat, a stack \nof math textbooks is being fed \ninto the top of the grinder. As \nthese textbooks enter the \ngrinder, they are transformed, \nand out of the bottom, small, \nintricate photographs emerge. \nThese photographs represent the \ntransformation of complex \nmathematics into visual beauty. \nThe butcher, wearing a traditional \nVictorian outfit complete with an \napron, has a focused expression, \nemphasizing the craftsmanship \ninvolved in this unique process. \nThe setting is reminiscent of an \nold-fashioned butcher's shop, \nadding to the Victorian theme. \nThe image is in a 16:9 aspect \nratio, focusing on the \ntransformation where math \ntextbooks turn into small \nphotographs falling out of the \nmachine.\nshow a picture of a \nvictorian dressed \nbutcher cranking a \nmeat grinder, except \nmath symbols are \nfalling in the top and \npretty pictures are \ndropping out the \nbottom. Make the \npicture 16:9 aspect \nratio.\n55\nDall-E 3",
  "Resources\n\u2022 https://towardsdatascience.com/understan\nding-diffusion-probabilistic-models-dpms-\n1940329d6048 \n\u2022 CVPR 2023 Tutorial: Denoising Diffusion \nModels: A Generative Learning Big Bang, \nhttps://cvpr.thecvf.com/virtual/2023/tutori\nal/18546 \n56\nLink",
  "Graph Neural Networks\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos\n Prince, Understanding Deep Learning, Creative Commons CC-BY-NC-ND license. (C) MIT Press\nOther Content Cited\n1",
  "April Dates\nSunday\nMonday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\nApril 1\n2\n3\n4\nGANs\n5\n6\n7\n8\n9\nVAEs\n10\nDiscussion\n11\nDiffusion Models\n12\n13\n14\n15\n16\nGraph Neural Nets\n(VizWiz Leaders \nShare)\n17\nDiscussion\n18\nReinforcement \nLearning\n19\n20\n21\n22\n23\nTBD/Overflow\n(JEPA Models)\n24\nDiscussion\n25\n\u2605 Project \nPresentations 1 \u2605 \n26\n27\n28\n29\n30\n\u2605 Project \nPresentations 2 \u2605 \nMay 1\nDiscussion??\n2\nStudy Period\n3\nStudy Period\n4\n5\n6\nFinal Exams\n7\nFinal report\n& Repo **\n8\n9\n10\n11\n2\n** Might be earlier. Depends on when grades are due.",
  "Project Presentations\nApril 25 \u2013 75 minutes\n\u2022 Slot 1\n\u2022 Slot 2\n\u2022 Slot 3\n\u2022 Slot 4\n\u2022 Slot 5\n\u2022 Slot 6\n\u2022 Slot 7\n\u2022 Slot 8\nApril 30 \u2013 75 minutes\n\u2022 Slot 9\n\u2022 Slot 10\n\u2022 Slot 11\n\u2022 Slot 12\n\u2022 Slot 13\n\u2022 Slot 14\n\u2022 Slot 15\n\u2022 Slot 16\n\u2022 Slot 17\nFormat:\n\u2264 3 minutes screencast/video\n\u2264 2 minutes additional presentation\n~2 minutes Q&A\nWill post slot assignments tonight!!\nFinal project info updated on Gradescope and website.\n3",
  "Graph Neural Networks\nNeural architectures that process graphs.\nThree challenges:\n1. Variable topology\n2. Size (billions of nodes)\n3. Single monolithic graph\n4",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n5",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n6",
  "Graph (Network)\n\u2022 general structure composed of nodes \n(vertices) and edges (links)\n\u2022 edges can be undirected or directed\n\u2022 a graph with directed edges and no cycles \n(no loops) is called directed acyclic graph \n(DAG)\n7\nnode or vertex\nedge\nor link\nundirected\ndirected",
  "Directed Example \u2013 Feed Forward Network\n8",
  "Directed Example \u2013 Bayesian Graphical Model\n9\nPreliminaries: Bayesian Models\nRocca, \"Understanding Variational Autoencoders (VAEs)\", 2019 \nN\nz\nx\n!(#)\n!(%|#)\nprior \u2013 prior knowledge or \nbelief about z\nlikelihood \u2013 probability \nof a sample given z\n27\nFrom lecture 18 \u2013 Variational Autoencoders",
  "Undirected Examples\n10\nroad networks\nnodes: physical locations or \nlandmarks\nedges: connecting roads\nchemical molecules\nnodes: atoms\nedges: chemical bonds\nelectrical circuits\nnodes: components or junctions\nedges: wires/electrical connections",
  "Examples\n11\nsocial networks\nnodes: people\nedges: friendships\n(undirected)\nscience literature\nnodes: papers\nedges: citations\n(acyclic directed)\nknowledge graph\nnodes: objects\nedges: named relationship\n(cyclic directed)",
  "Example \u2013 Geometric Point Cloud\n12\nnodes: positions in 3D space (vertex in 3D graphics)\nedges: connections to nearby points\n(undirected)\nhttps://en.wikipedia.org/wiki/Vertex_(computer_graphics)",
  "Example \u2013 Scene Graph \n13\nhierarchical graph showing relationship between objects in a 3D scene\n \nnodes: composite graphs or objects in 3D space\nedges: connections to nearby points\n(undirected)\nFernandez-Madrigal and Gonzalez, \"Multi-hierarchical graph search,\" 2002\nArmeni et al, \u201c3D Scene Graph: A structure for unified semantics, 3D space and camera,\u201d 2020? \nWald et al,  \u201cLearning 3D Semantic Scene Graphs with Instance Embeddings,\u201d 2022",
  "Other examples\n\u2022 Wikipedia \u2013 nodes are articles, edges are hyperlinks between articles\n\u2022 Computer programs \u2013 nodes are syntax tokens, edges are \ncomputation between tokens (tensor graph from Gradients lecture)\n\u2022 Protein interactions \u2013 nodes are proteins, edges exist where two \nproteins interface\n\u2022 Set or list \u2013 every element is connected to every other element\n\u2022 image \u2013 each pixel is a node with edges to the eight adjacent pixels\n14",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n15",
  "Graph representation\n16\nExample undirected graph with 6 nodes",
  "Graph representation \u2013 node embedding\n17\nExample undirected graph with 6 nodes\nInformation about a node is stored in a node \nembedding",
  "Graph representation \u2013 edge embedding\n18\nExample undirected graph with 6 nodes\nInformation about a node is stored in a node \nembedding\nInformation about an edge is stored in an edge \nembedding",
  "Graph representation \u2013 adjacency matrix\n19\nAssume we have \ud835\udc41 nodes\nThe graph connections can be \nrepresented by an adjacency matrix\nWhere a value of 1 at (\ud835\udc5a, \ud835\udc5b) represents \na connection between nodes \ud835\udc5a and \ud835\udc5b.\nFor undirected graphs the matrix is \nalways symmetric about the diagonal\nDiagonal is zero \u2013 no edge to itself\nCan be very sparse",
  "Graph representation \u2013 node data matrix\n20\nAll the node data in the form of \nnode embeddings can represented \nby a Node data matrix\nWhere \ud835\udc37 is the dimension of the \nnote embedding and\n\ud835\udc41 is the number of nodes",
  "Graph representation \u2013 edge data matrix\n21\nSimilarly, all the edge embedding information can be \nstored in an Edge data matrix, where:\n\ud835\udc37! is the dimension of the edge embedding vector and  \n\ud835\udc38 is the number of edges",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n22",
  "Adjacency Matrix\n23\nAssume we have an 8-node undirected graph",
  "Adjacency Matrix\n24\nAdjacency matrix for this graph.",
  "Adjacency Matrix\n25\nadjacency matrix\nWe can one hot encode \nrepresentation of node 6",
  "Adjacency Matrix\n26\nIf we pre-multiply the one-hot encoded \ndata node vector x by adjacency matrix A \nwe get the 6th column of A indicating direct \nconnections to other nodes\nOne-hot encoding vector of all \nnodes directly connected node 6",
  "Adjacency Matrix\n27\nGraph showing all nodes that can \nbe reached in exactly 2 steps.\nIf we pre-multiply again by A, we get a \nvector showing the number of times we can \nget to each node in 2 steps.",
  "Adjacency Matrix\n28\nPre-multiplying x by A twice is equivalent to the matrix A2\nShows how many times you can get from node m to node \nn in 2 steps",
  "Adjacency Matrix\n29\nWhen you raise the adjacency matrix to the power of \nL,\nthe entry at position (m, n) of AL contains the number \nof unique walks of length \ud835\udc3f from node \ud835\udc5b to node \ud835\udc5a\nNote: this is not the same as the number of unique \npaths since it includes routes that visit the same node \nmore than once. \na non-zero entry at position (\ud835\udc5a, \ud835\udc5b) indicates that the \ndistance from \ud835\udc5a to \ud835\udc5b must be less than or equal to \ud835\udc3f.\nExample for \ud835\udc3f = 2",
  "Permutation of node indices\nSince node indexing is arbitrary, we can permute the node indices \n30\n\ud835\udc17 =\n1\n2\n3\n4\n5\n6\n7\n8\n 9\n10\n11\n12\n\ud835\udc00 =\n0\n1\n0\n0\n1\n0\n1\n1\n0\n1\n0\n1\n0\n1\n1\n0\n1\n2\n3\n4\n1\n5\n9\n2\n6\n10\n3\n7\n11\n4\n8\n12\n(1      2       3       4)\nnode data\nadjacency matrix",
  "Permutation of node indices\nSince node indexing is arbitrary, we can permute the node indices \n31\nPermute the columns of the \nNode data matrix\nPermute both the rows and \ncolumn of the Adjacency matrix\nWe can express this  \nmathematically with a \npermutation matrix, P\n\ud835\udc0f =\n0\n0\n0\n1\n0\n0\n1\n0\n1\n0\n0\n0\n0\n1\n0\n0\n\ud835\udc17 =\n1\n2\n3\n4\n5\n6\n7\n8\n 9\n10\n11\n12\n\ud835\udc17- = \ud835\udc17\ud835\udc0f =\n3\n4\n2\n1\n7\n8\n6\n5\n 11\n12\n10\n9\n\ud835\udc00 =\n0\n1\n0\n0\n1\n0\n1\n1\n0\n1\n0\n1\n0\n1\n1\n0\n1\n2\n3\n4\n1\n5\n9\n2\n6\n10\n3\n7\n11\n4\n8\n12\n(3    4    2     1)\n(1",
  "4    2     1)\n(1      2       3       4)\nNew: (1       2       3      4)\nOld:   (3       4       2      1)\n\ud835\udc00- = \ud835\udc0f\ud835\udc13\ud835\udc00\ud835\udc0f =\n0\n1\n1\n0\n1\n0\n1\n0\n1\n1\n0\n1\n0\n0\n1\n0\nnode data\nadjacency matrix\n4\n3\n1\n2\n1\n5\n9\n2\n6\n10\n3\n7\n11\n4\n8\n12",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n32",
  "Graph Neural Network\n\u2022 A graph neural network is a model that takes the node embeddings X and \nthe adjacency matrix A as inputs and passes them through a series of \ud835\udc3e \nlayers. \n\u2022 The node embeddings are updated at each layer to create intermediate \n\u201chidden\u201d representations \ud835\udc07\ud835\udc3e before finally computing output embeddings \n\ud835\udc07\ud835\udc3e. \n\u2022 At the start of this network, each column of the input node embeddings \ud835\udc17 \njust contains information about the node itself. \n\u2022 At the end, each column of the model output \ud835\udc07\ud835\udc3e includes information \nabout the node and its context within the graph. \n\u2022 This is like word embeddings passing through a transformer network. These \nrepresent words at the start but represent the word meanings in the \ncontext of the sentence at the end.\n33",
  "Graph Level Tasks\nDetermine \n\u2022 class categories, e.g. molecule is poisonous \n\u2022 regression values, e.g. molecure boiling and freezing point\nbased on graph structure and node embeddings\nFor graph-level tasks, the output node embeddings are combined (e.g., \nby averaging), and the resulting vector is mapped via a linear \ntransformation or neural network to a fixed-size vector\n34",
  "Graph level classification\n35\nBinary Classification: Pr \ud835\udc66 = 1 \ud835\udc17, \ud835\udc00 = sigmoid[\ud835\udefd@ + \ud835\udf14@\ud835\udc07@\ud835\udfcf /\ud835\udc41] \n\ud835\udc07@\n\ud835\udf14!is 1\u00d7\ud835\udc37 row vector\n\ud835\udefd/ is  scalar\n\ud835\udc07/ is the output embedding matrix \n\ud835\udfcf is the output embedding matrix \nMean pooling\nmulti-class \nclassification",
  "Node level binary classification\n36\nPr \ud835\udc66(D) = 1 \ud835\udc17, \ud835\udc00 = sigmoid[\ud835\udefd@ + \ud835\udf14@\ud835\udc21@\n(D)] \n\ud835\udc21/\n(1)is the output embedding vector node for \ud835\udc5b",
  "Edge prediction\nPredict whether edge should exist or not.\n37\nPr \ud835\udc66(FD) = 1 \ud835\udc17, \ud835\udc00 = sigmoid[\ud835\udc21@\nF G\ud835\udc21@\n(D)]",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n38",
  "Graph convolutional network\nThese models are convolutional in that they update each node by \naggregating information from nearby nodes. \nAs such, they induce a relational inductive bias (i.e., a bias toward \nprioritizing information from neighbors).\n39\nA function \ud835\udc39[\u22c5]  with parameters \ud835\udf19\" that \ntakes the node embeddings and \nadjacency matrix and outputs new node \nembeddings",
  "Equivariance and Invariance\nEvery layer should be equivariant to index permutations\nAnd for node classification and edge prediction the output should be \ninvariant to index permutations\n40\n\ud835\udc07HIJ\ud835\udc0f = \ud835\udc05[\ud835\udc07H\ud835\udc0f, \ud835\udc0fG\ud835\udc00\ud835\udc0f, \ud835\udf19H]\n\ud835\udc66 = sigmoid \ud835\udefd@ + \ud835\udf14@\ud835\udc07@\ud835\udfcf/\ud835\udc41 = sigmoid \ud835\udefd@ + \ud835\udf14@\ud835\udc07@\ud835\udc0f\ud835\udfcf/\ud835\udc41",
  "Example Graph Convolution Network (GCN) layer\nAggregate information from neighboring nodes\nagg \ud835\udc5b, \ud835\udc58 =\n'\n*\u2208,-[.]\n\ud835\udc21/\n(*)\nwhere ne[\ud835\udc5b] returns the set of indices of the neighbors of node \ud835\udc5b.\n41",
  "Example Graph Convolution Network (GCN) layer\nAggregate information from neighboring nodes\nagg \ud835\udc5b, \ud835\udc58 =\n'\n*\u2208,-[.]\n\ud835\udc21/\n(*)\nwhere ne[\ud835\udc5b] returns the set of indices of the neighbors of node \ud835\udc5b.\nThen a linear transform to the current node vector and the aggregate \nfor the current node and add a bias.\n\ud835\udc21/23\n(.) = \ud835\udc1a \ud835\udefd/ + \u03a9/ \u22c5 \ud835\udc21/\n. + \u03a9/ \u22c5 agg[\ud835\udc5b, \ud835\udc58]\n42",
  "Graph convolution layers\n43\nInput\n1st Layer\nk+1st Layer",
  "Example Graph Convolution Network (GCN) layer\nWe apply the following equation\n\ud835\udc21/23\n(.) = \ud835\udc1a \ud835\udefd/ + \u03a9/ \u22c5 \ud835\udc21/\n. + \u03a9/ \u22c5 agg[\ud835\udc5b, \ud835\udc58]\nto the entire node hidden layers matrix, \ud835\udc07/, by noting that \ud835\udc07/\ud835\udc00 \nproduces a matrix where the nth column is agg[\ud835\udc5b, \ud835\udc58].\n44",
  "Example Graph Convolution Network (GCN) layer\nWe apply the following equation\n\ud835\udc21/23\n(.) = \ud835\udc1a \ud835\udefd/ + \u03a9/ \u22c5 \ud835\udc21/\n. + \u03a9/ \u22c5 agg[\ud835\udc5b, \ud835\udc58]\nto the entire node hidden layers matrix, \ud835\udc07/, by noting that \ud835\udc07/\ud835\udc00 \nproduces a matrix where the nth column is agg[\ud835\udc5b, \ud835\udc58].\n45\nNote that this is (1) equivariant to permutations, (2) handles arbitrary number of \nneighbors, (3) exploits graph structure and (4) share parameters",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n46",
  "Graph classification example\nWe can put it all together and add a sigmoid layer\n47\nYou\u2019ll be implementing this in notebook 13.2.\nFor classification on molecules,\n\ud835\udc4b \u2208 \u211d334\u00d76: one hot encoding of 118 \nelements\n\u03a97 \u2208 \u211d8\u00d7334: convert to \ud835\udc37-dimensional \nembeddings\n\ud835\udefd/: is a scalar\n\ud835\udf14/: a 1\u00d7\ud835\udc37 parameters row vector\nMean pooling",
  "Inductive             vs.   Transductive\n48\nsemi-supervised learning: train with \nthe labeled nodes, then run \ninference to determine label for \nunlabeled nodes\nsupervised learning: train with the \nlabeled graphs and then run \ninference on the unlabeled (test) \ngraphs",
  "Node classification example\nAssume transductive binary node classification with \nmillions of nodes, partially labeled.\nSame network body as graph classification, but \ndifferent head:\n\ud835\udc1f \ud835\udc17, \ud835\udc00, \ud835\udebd = sigmoid[\ud835\udefd4\ud835\udfcf5 + \ud835\udf4e4\ud835\udc074]\nNo mean pooling. Output is 1\u00d7\ud835\udc41.\nTrain with binary cross-entropy loss on nodes with \nlabels.\n49\nsource",
  "Node classification example\nAssume transductive binary node classification with \nmillions of nodes, partially labeled.\nChallenges:\n1. memory limitations: need to store every node \nand hidden layer embedding during training\n2. how to perform SGD with basically one batch!\n50\nsource",
  "Solutions: Choosing batches for graphs\n1. Choose random subset of nodes\n2. Neighborhood sampling\n3. Graph partitioning\n51",
  "Batches: Random subset\n52\nYou can pick a random batch of labeled nodes at each \ntraining step.",
  "Batches: Random subset\n53\n\ud835\udc219:3\n(1) = \ud835\udc1a \ud835\udefd9 + \u03a99 \u22c5 \ud835\udc219\n1 + \u03a99 \u22c5 agg[\ud835\udc5b, \ud835\udc58]\nEach node is dependent on the same node in the previous layer and its neighbors because of agg[]\nReceptive Field",
  "Batches: Random subset\n54\n\ud835\udc219:3\n(1) = \ud835\udc1a \ud835\udefd9 + \u03a99 \u22c5 \ud835\udc219\n1 + \u03a99 \u22c5 agg[\ud835\udc5b, \ud835\udc58]\nEach node is dependent on the same node in the previous layer and its neighbors because of agg[].\nWith many layers and dense connection, it can quickly expand to encompass every node.\nReceptive Field",
  "Neighborhood Sampling\n55\nRandom Sampling:\nUse all the neighbors\nNeighborhood Sampling:\nUse max \ud835\udc5b of the neighbors.\nHere \ud835\udc5b = 3.\nNotebook 13.3",
  "Graph Partitioning\n56\nDisconnect edges of the original to create maximally \nconnected disjoint subsets\nSplit into train, test and validation sets and train just like in the \ninductive setting.",
  "Alternatives to Mean Pooling for Node Combinations\n\u2022 Diagonal enhancement: current node is multiplied by (1 + \ud835\udf16#), where \ud835\udf16# is a learned scalar for \neach layer \n\ud835\udc07#$% = \ud835\udc1a \ud835\udefd#\ud835\udfcf& + \ud835\udec0#\ud835\udc07#(\ud835\udc00 + (1 + \ud835\udf16#)\ud835\udc08)\n\u2022 Residual connections: Include the current node in the sum\n\ud835\udc07#$% = \ud835\udc1a \ud835\udefd#\ud835\udfcf& + \ud835\udec0#\ud835\udc07#\ud835\udc00) + \ud835\udc07#\n\u2022 Mean aggregation: take average instead of sum of neighbors\nagg \ud835\udc5b =\n1\nne[\ud835\udc5b]\nE\n'\u2208)*[,]\n\ud835\udc21'\n\u2022 Kipf normalization: downweight neighboring nodes with a lot of neighbors\nagg \ud835\udc5b =\nE\n'\u2208)*[,]\n\u210e'\nne[\ud835\udc5b] ne[\ud835\udc5a]\n\u2022 Max pool aggregation: element-wise max of all neighbors to current node\nagg \ud835\udc5b =\nmax\n'\u2208)*[,][\ud835\udc21']\n57\nUDL book sections 13.",
  "]\n\u210e'\nne[\ud835\udc5b] ne[\ud835\udc5a]\n\u2022 Max pool aggregation: element-wise max of all neighbors to current node\nagg \ud835\udc5b =\nmax\n'\u2208)*[,][\ud835\udc21']\n57\nUDL book sections 13.8.1 \u2013 13.8.5",
  "Aggregation by Attention\nWeights depend on data at the nodes.\nApply linear transform to current node:\n\ud835\udc07\u2032! = \ud835\udefd!\ud835\udfcf\" + \ud835\udec0!\ud835\udc07\nThen the similarity \ud835\udc60\ud835\udc5a\ud835\udc5b of each transformed node embedding \ud835\udc21\u2032\ud835\udc5a to the \ntransformed node embedding \ud835\udc21\u2032\ud835\udc5b is computed by concatenating the pairs, \ntaking a dot product with a column vector \ud835\udf19! of learned parameters, and \napplying an activation function:\n\ud835\udc60#$ = a \ud835\udf19!\n\" \ud835\udc21\u2032#\n\ud835\udc21\u2032$\n\ud835\udc07!%& = \ud835\udc1a[\ud835\udc07'\n! \u22c5 Softmask \ud835\udc12, \ud835\udc00 + \ud835\udc08 ]\n58",
  "Graph Attention\n59\nRegular graph convolution\nGraph attention",
  "Graph Attention\n60\nRegular graph convolution\nGraph attention\nTransformer se",
  "Topics\n\u2022 Basic definition and examples\n\u2022 Graph representation\n\u2022 Properties of Adjacency Matrix\n\u2022 Graph neural network, tasks and loss functions\n\u2022 Graph convolutional network\n\u2022 Graph & Node classification\n\u2022 Edge graphs\n61",
  "Edge Graphs\n62\nHandled by simple transformation from node graphs.",
  "Next                                                 Feedback?\n\u2022 Reinforcement Learning\n\u2022 Joint Embedding Predictive Architecture\n\u2022 Project Presentations\n63\nLink",
  "Reinforcement Learning &\nRL with Human Feedback (RLHF)\nDL4DS \u2013 Spring 2024\nDS598 B1 Gardos\n Prince, Understanding Deep Learning, Creative Commons CC-BY-NC-ND license. (C) MIT Press\nOther Content Cited\n1",
  "April Dates\nSunday\nMonday\nTuesday\nWednesday\nThursday\nFriday\nSaturday\nApril 1\n2\n3\n4\nGANs\n5\n6\n7\n8\n9\nVAEs\n10\nDiscussion\n11\nDiffusion Models\n12\n13\n14\n15\n16\nGraph Neural Nets\n(VizWiz Leaders \nShare)\n17\nDiscussion\n18\nOffice Hours\n19\n20\n21\n22\n23\nRL/RLHF\n24\nDiscussion\n25\n\u2605 Project \nPresentations 1 \u2605 \n26\n27\n28\n29\n30\n\u2605 Project \nPresentations 2 \u2605 \nMay 1\nDiscussion??\n2\nStudy Period\n3\nStudy Period\n4\n5\n6\nFinal Exams\n7\nFinal report\n& Repo **\n8\n9\n10\n11\n2\n** Might be earlier. Depends on when grades are due.",
  "Project Presentations\nApril 25 \u2013 75 minutes\n1.\nOsama Dabbousi\n2.\nCarmen Pelayo Fernandez\n3.\nAnush Veeranala, Lilin Jin, Xinyu \nZhang\n4.\nBowen Li\n5.\nYuta Tsukumo\n6.\nZhengxiong Zouxu\n7.\nHang Yu, Yinzhou Lu\n8.\nZhandong Jiao\nApril 30 \u2013 75 minutes\n1.\nSeung Hee Lee, Xinyi Hu, Yuke \nZhang\n2.\nJessica Cannon\n3.\nSungjoon Park\n4.\nAnh Pham, Farid Karimli\n5.\nNikhita Mantravadi\n6.\nIshan Ranjan, Jack Campbell, Rani \nShah\n7.\nAndy Yang, Weining Mai\n8.\nRuozhu Wang, Yi Liu, Zhuoyan Ma\n9.\nKevin Quinn\nFormat:\n\u2264 3 minutes screencast/video\n\u2264 2 minutes additional presentation\n~2 minutes Q&A\nFinal project info updated on Gradescope and website.\n3",
  "Outline\n\u2022 RL basic concepts\n\u2022 Deep reinforcement learning from human preferences (2017)\n\u2022 Training language models to follow instructions with human feedback \n(2022)\n4",
  "Outline\n\u2022 RL basic concepts\n\u2022 Deep reinforcement learning from human preferences (2017)\n\u2022 Training language models to follow instructions with human feedback \n(2022)\n5",
  "In a nutshell\u2026\nRL is the study of agents and how they learn by trial and error.\nIt formalizes the idea that rewarding or penalizing an agent for \nits behavior makes it more or less likely, respectively, to repeat \nthat behavior in the future.\n6",
  "Markov Process\nWorld is described by a set of states \ud835\udc60\nChanges between states are represented by \ntransition probabilities Pr(\ud835\udc60!\"#|\ud835\udc60!)\nMarkov process produces a sequence of \nstates \ud835\udc60#, \ud835\udc60$, \ud835\udc60%, \u2026\nA trajectory is the sequence of states\n \ud835\udf0f = [\ud835\udc60#, \ud835\udc60$, \ud835\udc60%, \u2026 ]\nCan only go",
  "Markov Process \u2013 Transition Probabilities\nTransition probability from \nsquare 1 to square n\nEqually likely to go in any allowable direction.",
  "Markov reward process\nDistribution of rewards at next time step given current state: Pr(\ud835\udc5f$%&|\ud835\udc60$)\nThe return \ud835\udc3a$ is the sum of discounted future rewards\n \n \n\ud835\udc3a$ = \u2211'()\n*\n\ud835\udefe'\ud835\udc5f$%'%&      where     \ud835\udefe \u2208 (0,1]\nTrajectory now comprised of  state and the next reward",
  "Markov decision process (MDP)\nAdds a set of of possible actions at each time step that changes \ntransition and reward probabilities\n \n \nPr(\ud835\udc60!\"#|\ud835\udc60!, \ud835\udc4e!)     and    Pr(\ud835\udc5f!\"#|\ud835\udc60!, \ud835\udc4e!)\nTrajectory now consists of states, actions and rewards\nAction is not deterministic",
  "Partially observable Markov decision process (POMDP)\nThe state is not directly visible, but \ninstead receives an observation \ud835\udc5c! \ndrawn from Pr(\ud835\udc5c!|\ud835\udc60!)\nPenguin can only see what is in the \ndashed box.\nIndistinguishable from what it would \nsee from box 9.",
  "Policy\nThe rules that determine the agent\u2019s (e.g. penguin\u2019s) action for each \nstate:  \ud835\udf0b[\ud835\udc4e|\ud835\udc60]\nCan be deterministic or stochastic\nCan be stationary or non-stationary (time dependent)\nBetter deterministic policy\nPoorer deterministic policy\nStochastic policy",
  "Full reinforcement learning loop\nAgent receives the state (or observation) \nand reward.\nThen (optionally modifies the policy and) \nchoose next action.\nEnvironment then assigns next state and \nreward according to transition \nprobabilities.",
  "RLHF as RL\n\u201cWe can think of the main model as an agent that takes sequential \nactions (choose tokens) and receives a delayed reward from the reward \nmodel when the last token is chosen\u201d\n14\nPrince, \u201cTraining and fine-tuning large language models,\u201d 2023, blog",
  "Outline\n\u2022 RL basic concepts\n\u2022 Deep reinforcement learning from human preferences (2017)\n\u2022 Training language models to follow instructions with human feedback \n(2022)\n15",
  "Aligning to Human Preferences\n\u201cOne step towards building safe AI systems is to remove the need for \nhumans to write goal functions, since using a simple proxy for a \ncomplex goal, or getting the complex goal a bit wrong, can lead to \nundesirable and even dangerous behavior. \n\u2026, we\u2019ve developed an algorithm which can infer what humans want by \nbeing told which of two proposed behaviors is better.\u201d\n16\nhttps://openai.com/research/learning-from-human-preferences, 2017",
  "Originally developed to improve RL systems\n\u2022 Starts by acting randomly\n\u2022 Gives 2 examples to a \nhuman who votes on which \nis closer to achieving goal\n\u2022 AI builds reward model to \nmatch human votes\n\u2022 Continues to seek feedback \non trajectory pairs that are \nmost uncertain\n17\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017",
  "Originally developed to improve RL systems\n\u2022 Trained with ~1 hour \nof evaluator time\n\u2022 Background policy \naccumulated ~70 \nhours of experience\n18\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nhttps://openai.com/research/learning-from-human-preferences, 2017",
  "Originally developed to improve RL systems\n\u2022 Learned Atari with only human reward model (right vertical bar)\n\u2022 Without access to game score as reward\n\u2022 Human feedback sometimes does better than normal reward function\n19\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nhttps://openai.com/research/learning-from-human-preferences, 2017",
  "Deep RL from Human Preferences\n20\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nAssume human overseer who can express preferences between \ntrajectory segments\n\ud835\udf0e =\n\ud835\udc5c&, \ud835\udc4e& , \ud835\udc5c#, \ud835\udc4e# , \u2026 , \ud835\udc5c'(#, \ud835\udc4e'(#\n\u2208 \ud835\udcaa\u00d7\ud835\udc9c '\nWe write \n\ud835\udf0e# \u227b \ud835\udf0e$\nto indicate that the human preferred trajectory \ud835\udf0e# to \ud835\udf0e$.\nGoal of Agent: Produce trajectories preferred by human, while making as few queries as \npossible to the human.",
  "Deep RL from Human \nPreferences\n21\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nPreferences are generated by a reward function \ud835\udc5f \u2236  \ud835\udcaa \u00d7 \ud835\udc9c \u2192  \u211d if\n\ud835\udc5c&\n#, \ud835\udc4e&\n# , \u2026 , \ud835\udc5c'(#\n#\n, \ud835\udc4e'(#\n#\n\u227b ( \ud835\udc5c&\n$,\n\ud835\udc4e&\n$ , \u2026 , \ud835\udc5c'(#\n$\n,\n\ud835\udc4e'(#\n$\n)\nwhenever\n\ud835\udc5f \ud835\udc5c&\n#, \ud835\udc4e&\n# + \u22ef + \ud835\udc5f \ud835\udc5c'\n#, \ud835\udc4e'\n#\n> \ud835\udc5f \ud835\udc5c&\n$, \ud835\udc4e&\n$ + \u22ef + \ud835\udc5f(\ud835\udc5c'(#\n$\n, \ud835\udc5c'(#\n$\n)",
  "Deep RL from Human \nPreferences -- Method\n22\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nAt each point in time, \n\u2022 maintain a policy  \ud835\udf0b \u2236  \ud835\udcaa \u2192  \ud835\udc9c\n\u2022 and a reward function estimate   \u0302\ud835\udc5f \u2236  \ud835\udcaa \u00d7 \ud835\udc9c \u2192  \u211d \neach parameterized by deep neural networks.",
  "Deep RL from Human \nPreferences -- Method\n23\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nThe policy and reward estimate networks are updated by three \nprocesses\n1. Policy \ud835\udf0b interacts with the environment to produce a set of \ntrajectories \ud835\udf0f#, \u2026 , \ud835\udf0f) , where \ud835\udf0b is updated with traditional RL to \nmaximize sum of predicted rewards\n2. Select pairs of segments (\ud835\udf0e#, \ud835\udf0e$)from trajectories \ud835\udf0f#, \u2026 , \ud835\udf0f)  and \nquery human for comparison.\n3. Parameters of reward estimate \u0302\ud835\udc5f are optimized via supervised \nlearning to fit human comparisons",
  "Deep RL from Human \nPreferences -- Method\n24\nChristiano et al (OpenAI), \u201cDeep Reinforcement Learning from Human Preferences,\u201d 2017\nThe policy and reward estimate networks are updated by three processes\n1.\nLearn policy \ud835\udf0b and produce trajectories \ud835\udf0f!, \u2026 , \ud835\udf0f\"\n2.\nSelect pairs of segments (\ud835\udf0e!, \ud835\udf0e#) and query human\n3.\nUpdate reward estimate \u0302\ud835\udc5f to match human comparisons\n(1)\n(2)\n(3)\nTrajectories: \n\ud835\udf0f!, \u2026 , \ud835\udf0f\"\nHuman comparisons:\n(\ud835\udf0e! <> \ud835\udf0e#) \nUpdated reward: \u0302\ud835\udc5f\nProcesses run \nasynchronously",
  "Outline\n\u2022 RL basic concepts\n\u2022 Deep reinforcement learning from human preferences (2017)\n\u2022 Training language models to follow instructions with human feedback \n(2022)\n25",
  "3 Steps\n26\nL. Ouyang et al., \u201cTraining language models to follow instructions with human feedback,\u201d 2022\nSupervised Fine-\nTuning\nReward Model (RM) \nTraining\nReinforcement Learning via \nProximal Policy Optimization \non this Reward Model",
  "Supervised (Instruction) Fine Tuning\nCreate a dataset of ~10,000 prompts and fine-tuned responses.\n27\nPrince, \u201cTraining fine-tuning LLMs\u201d, 2023, blog",
  "Supervised (Instruction) Fine Tuning\n28\nPrince, \u201cTraining fine-tuning LLMs\u201d, 2023, blog\nPrompt\nGround \nTruth \nResponse \n(Teacher \nForcing)\nResponse",
  "Reward Modeling\nTrain a neural network, e.g. start with the SFT model with \nthe last layers replaced.\nPresent labelers between \ud835\udc3e = 4 and \ud835\udc3e = 9 responses to \nrank.\nThis produces \ud835\udc3e\n2  comparisons for each prompt.\nThe loss function for the reward model is:\nwhere \ud835\udc5f!(\ud835\udc65, \ud835\udc66) is the scalar output of the reward model for \nprompt \ud835\udc65 and completion \ud835\udc66 with parameters \ud835\udf03, \ud835\udc66\" is the \npreferred completion (winner) versus \ud835\udc66#, and \ud835\udc37 is the data \nset of human comparisons.\n29\nL. Ouyang et al., \u201cTraining language models to follow instructions with human feedback,\u201d 2022",
  "Reinforcement Learning (RL)\nContinue training the SFT model to maximize the \nfollowing objective:\n30\nL. Ouyang et al., \u201cTraining language models to follow instructions with human feedback,\u201d 2022\nN. Lambert, \u201cReinforcement Learning from Human Feedback,\u201d 2023",
  "Next                              \n\u2022 Project Presentations\n\u2022 Fill out course evaluations\n31\nLast lecture!!\nDALL-E 3",
  "Deep Learning for Data Science (DL4DS) / Spring 2024 - Boston University Faculty for Computing and Data Science\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\nDeep Learning for Data Science (DL4DS) / Spring 2024 \n\nAnnouncements\n\n\n        \n            Apr 22, 2024: To make your screen recordings, one free and easy option is to use Kaltura\nCapture, which is integrated with BU\u2019s MyMedia streaming media solution.\nThe instruction video, including how to download the app, is\n here.\nThe project page is also updated with this information.\n\n\n        \n            Apr 15, 2024: Final project instructions have been updated on the \nproject page. \nA revised report\ntemplate is posted along with updated instructions on the project\nvideo and the project code repository.\n\n\n        \n            Apr 11, 2024: Future lecture topics and schedule have been updated and posted.",
  "Apr 11, 2024: Future lecture topics and schedule have been updated and posted.\n\n\n        \n            Mar 17, 2024: Additional instructions have been added for the project\nmid-point check-in.\nA gradescope assignment has also been posted for it.\n\n\n\n\nCourse Abstract\nIn this course we will dive into Deep Learning. We'll balance important theoretical concepts with hands on network training and applications using modern deep learning python frameworks. We'll explore numerous network architectures like CNNs, transformers, and the rapidly developing state-of-the-art of large pre-trained foundation models. You'll have the chance to apply what you've learned in a final project.",
  "We'll explore numerous network architectures like CNNs, transformers, and the rapidly developing state-of-the-art of large pre-trained foundation models. You'll have the chance to apply what you've learned in a final project.\nLectures: Tuesdays and Thursdays, 3:30pm \u2013 4:45pm\nLocation: CAS 208\nDiscussion Session I: Wednesdays, 11:15am \u2013 12:05pm\nLocation: CDS 164\nDiscussion Session II: Wednesdays, 3:35pm \u2013 4:25pm\nLocation: CDS 1526\nInstructor: Thomas Gardos\n\nOffice: CDS 1623\nOffice Hours: Fridays, 1:30 \u2013 3:30pm on Zoom (see Piazza) and on demand. E-mail for appointment.",
  "E-mail for appointment.\nemail: tgardos <at> bu <dot> edu\n\nTeaching Assistant: Xavier Thomas\n\nOffice Hours: Mondays, 3:00pm \u2013 5:00pm\nOffice Hours Location: CDS, 16th Floor\nemail: xthomas <at> bu <dot> edu\n\nCourse Description\nIn this course students will gain an understanding of the fundamentals in deep learning and then apply those concepts in exercises and applications in python. We\u2019ll start with the origins of artificial neural networks, learn about loss functions, understand gradient descent, back propagation and various training optimization techniques. Students will be familiar with canonical network architecture such as multi-layer perceptions, convolutional neural networks, recursive neural networks, LSTMs and GRU, attention and transformers. Through explanations, examples and exercises students will build intuition on how deep learning algorithms work and how they are implemented in popular deep learning frameworks such as PyTorch. Students will be able to define, train and evaluate deep learning models as well as adapt deep learning frameworks to new functionality.  Students will gain exposure to pre-trained large language models and other foundation models and the concepts of few-shot learning and reasoning.",
  "Students will be able to define, train and evaluate deep learning models as well as adapt deep learning frameworks to new functionality.  Students will gain exposure to pre-trained large language models and other foundation models and the concepts of few-shot learning and reasoning. Finally, students will be able to apply many of the techniques they learned in a final class project.\nLearning Outcomes\nUpon successful completion of this course, students will be able to:\n\nGrasp foundational theories and practices in the deep learning arena.\nDesign and implement various neural network architectures using Python and PyTorch.\nEmploy regularization, optimization, and advanced training techniques to enhance model performance.\nAnalyze real-world datasets, applying suitable deep learning techniques to derive actionable insights.\nUnderstand the benefits and drawbacks of various neural architectures in specific contexts.\nUnderstand the pros and cons of pre-trained large language and other foundation models and how best to employ them\nComplete a data-centric project, showcasing end-to-end deep learning implementation.\n\nPrerequisites\nPython Programming \u2013 Should be proficient in python and associated data science packages,\nor studiously working towards proficiency. See for example\nScientific Python Lectures for lessons on python language and relevant packages, or\nThe Python Tutorial for a tutorial on the core\nlanguage.",
  "Prerequisites\nPython Programming \u2013 Should be proficient in python and associated data science packages,\nor studiously working towards proficiency. See for example\nScientific Python Lectures for lessons on python language and relevant packages, or\nThe Python Tutorial for a tutorial on the core\nlanguage.\nPackages such as NumPy NumPy - Learn  and \nSciPy (SciPy User Guide\nhave tutorials and documentation as well.\nThe more proficient you are, the more effective you will be at the assignments and projects. We will dedicate some discussion sessions to ensure your environment is setup correctly and review some of the basics as well as answer any questions.\nMath Proficiency \u2013 In order to understand the foundational concepts, it is important to have proficiency in a number of areas of mathematics. These include linear algebra, first year calculus and trigonometry as well as some concepts from Real Analysis. We will cover these concepts in the class and some recitation sessions, but refreshing or building your foundation will help.\nReference Material\nThe primary textbook for this course will be \nUnderstanding Deep Learning, by Simon Prince . \nThe book is available online as a preprint and should be available in print from \nMIT Press in early 2024.",
  "Reference Material\nThe primary textbook for this course will be \nUnderstanding Deep Learning, by Simon Prince . \nThe book is available online as a preprint and should be available in print from \nMIT Press in early 2024.\nWe\u2019ll also reference \nIntroduction to Linear Algebra, Sixth Edition (2023), by Gilbert Strang.\nLecture notes, consisting primarily of Jupyter notebooks will be posted online as well.\nGiven the fast moving nature of this area, we\u2019ll also be citing many articles available online as well as other online reference materials in each lecture. As part of the class, we will guide the students in constructing their own bibliography and give tips on how to efficiently and effectively read research papers.\nComputing Environment\nStudents are of course free to use their own personal computer, but you will also have access to Boston University\u2019s Shared Computing Cluster and GPUs. Access instructions will be provided. For more information, see:\n\nShared Computing Cluster : TechWeb : Boston University\nGPU Computing : TechWeb : Boston University\n\nLearning Management Software\nTo be added.",
  "Access instructions will be provided. For more information, see:\n\nShared Computing Cluster : TechWeb : Boston University\nGPU Computing : TechWeb : Boston University\n\nLearning Management Software\nTo be added.\n\nPiazza\nGradescope\nBlackboard\n\nCourse Requirements\n\nHomework: For the 1st half of the class we\u2019ll assign\nJupyter Notebook coding assignments and a few homework questions\napproximately every week to help anchor the key concepts and python/pytorch\ncoding patterns. These will become less frequent in the 2nd half of the course\nto allow you to concentrate and make progress on your projects.\nA mid-term deep learning training contest\nA final project where you will apply deep learning methods to a problem or\napplication of interest to you.\n\nCourse Assessment\n\nFinal Project: 45%\nMid-term Project/Competition: 25%\nJupyter Notebooks: 15%\nHomeworks: 10%\nClass Participation/Attendance: 5%\n\nFinal Project\nSee the Project page for more\ninformation.",
  "Course Assessment\n\nFinal Project: 45%\nMid-term Project/Competition: 25%\nJupyter Notebooks: 15%\nHomeworks: 10%\nClass Participation/Attendance: 5%\n\nFinal Project\nSee the Project page for more\ninformation.\nStudent Code of Conduct\nAll students are expected to abide by University conduct policies as detailed in the following links:\n\nBoston University Student Codes of Conduct\nCollege of Arts & Sciences Codes of Conduct\nBoston University Student Responsibilities\n\nAcademic Honesty\nYou may discuss homework assignments with classmates, but you are solely responsible for what you turn in. Collaboration\nin the form of discussion is allowed, but all forms of cheating (copying parts of a classmate\u2019s assignment, plagiarism\nfrom books or old posted solutions) are NOT allowed. We \u2013 both teaching staff and students \u2013 are expected to abide by the\nguidelines and rules of the Academic Code of Conduct.\nGraduate students must also be aware of and abide by the GRS Academic Conduct code.\nYou can probably, if you try hard enough, find solutions for homework problems online. Given the nature of the Internet,\nthis is inevitable.",
  "Graduate students must also be aware of and abide by the GRS Academic Conduct code.\nYou can probably, if you try hard enough, find solutions for homework problems online. Given the nature of the Internet,\nthis is inevitable. Let me make a couple of comments about that:\n\nIf you are looking online for an answer because you don\u2019t know how to start thinking about a problem, talk to the TA\nor instructor, who may be able to give you pointers to get you started. Piazza is great for this \u2013 you can usually\nget an answer in an hour if not a few minutes.\nIf you are looking online for an answer because you want to see if your solution is correct, ask yourself if there is\nsome way to verify the solution yourself. Usually, there is. You will understand what you have done much better if\nyou do that. So \u2026 it would be better to simply submit what you have at the deadline (without going online to cheat)\nand plan to allocate more time for homeworks in the future.\n\nGenerative AI Assistance (GAIA) Policy\nIn general, we follow the policy outlined in the\nCDS GAIA Policy.\nExtracting and paraphrasing from the student responsibilities of that policy.",
  "Generative AI Assistance (GAIA) Policy\nIn general, we follow the policy outlined in the\nCDS GAIA Policy.\nExtracting and paraphrasing from the student responsibilities of that policy. \nWhere there is conflicting information between the CDS policy and below, the\npolicy below should take precedence.\nStudents shall:\n\nGive credit to AI tools whenever used, even if only to generate ideas rather than usable text, illustrations or code.\nWhen using AI tools on written assignments, unless prohibited, add an appendix showing\n    \nthe entire exchange, highlighting the most relevant sections;\na description of precisely which AI tools were used (e.g. ChatGPT private subscription version or DALL-E free\n version),\nan explanation of how the AI tools were used (e.g. to generate ideas, turns of phrase, elements of text, long\n stretches of text, lines of argument, pieces of evidence, maps of conceptual territory, illustrations of key\n concepts, etc.);\nan account of why AI tools were used (e.g. to save time, to surmount writer\u2019s block, to stimulate thinking, to\nhandle mounting stress, to clarify prose, to translate text, to experiment for fun, etc.).",
  ");\nan account of why AI tools were used (e.g. to save time, to surmount writer\u2019s block, to stimulate thinking, to\nhandle mounting stress, to clarify prose, to translate text, to experiment for fun, etc.).\nOptional but recommended: Employ AI detection tools and originality checks prior to submission, ensuring that\ntheir submitted work is not mistakenly flagged.\n\n\nWhen using AI tools on coding assignments, unless prohibited\n    \nAdd the prompt text and tool used as comments before the generated code.\nClarify whether the code was used as is, or modified somewhat, moderately\nor significantly.\n\n\nNot use AI tools during in-class examinations, or assignments, unless explicitly permitted and instructed.\nUse AI tools wisely and intelligently, aiming to deepen understanding of subject matter and to support learning.\n\nAs these generative assistive tools become widely deployed and pervasive, we believe they will become integral\nto most people\u2019s workflow. However, for foundational concepts, as are taught in this course, it is in your\nbest interest and worth it to struggle some in creating your answers and solutions. It is just as important\nto learn what doesn\u2019t work, and which paths are dead ends, as it is to learn what does work.",
  "However, for foundational concepts, as are taught in this course, it is in your\nbest interest and worth it to struggle some in creating your answers and solutions. It is just as important\nto learn what doesn\u2019t work, and which paths are dead ends, as it is to learn what does work. When you are posed\nwith new and unique problems, that intuition you develop will be vital in choosing directions. More pragmatically,\nsome of the most coveted jobs at the most selective companies require technical interviews where they expect you\nto know these foundational concepts without assistance.\nAnd finally, to reiterate, it is vitally important, and a core part of academic integrity, to cite when you are\nusing Generative AI Assistive technologies. Arguably, not citing and risking plagiarism is worse than taking\nshort cuts and using and then citing GAIA.\nAccommodations for Students with Disabilities\nIf you have a disability and have an accommodations letter from the Disability & Access Services office, I encourage you to discuss your accommodations and needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate.",
  "Accommodations for Students with Disabilities\nIf you have a disability and have an accommodations letter from the Disability & Access Services office, I encourage you to discuss your accommodations and needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with BU Disability & Access Services, I encourage you to find more information at https://www.bu.edu/disability/.\nThis syllabus provides a general plan for the course; deviations may be necessary depending on the progress of the class.\n\n\n\n\n\n\nInstructor\n\n\nThomas Gardos\n\n\n\n\n\nTeaching Assistants\n\n\nXavier Thomas\n\n\n\nThe AI Terrier Tutor (ChatGPT Subscription Required)\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Schedule - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\nSchedule\n\nThe schedule is subject to change. Check back often. Notes/codes/slides links\nare placeholders for future lectures. They will be updated no later than the \nlecture date.\n\n\n\nEvent\nDate\nDescription\nCourse Material\n\n\nLecture\n\n    01/18/2024\n    Thursday\n\n\n    01 - Intro to Deep Learning and Course Logistics\n    \n    \n    \n    \n    [slides]\n    \n    [Jupyter Notebook]\n    \n    \n\n    \n\n\n\nUnfortunately the recording failed for this lecture. Will re-record it at some\npoint.\nSuggested Readings:\n\nUDL Chapter 1\n\n\n\n\nLecture\n\n    01/23/2024\n    Tuesday\n\n\n    02 - Supervised Learning\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [Jupyter Notebook]",
  "Lecture\n\n    01/23/2024\n    Tuesday\n\n\n    02 - Supervised Learning\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [Jupyter Notebook]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 2\n\n\n\n\nAssignment\n\n    01/23/2024\n    Tuesday\n\n\n    PS2 -- Supervised Learning released!\n\n\n    [PS2 -- Supervised Learning]\n    \n\n\n\nDiscussion\n\n    01/24/2024\n    Wednesday\n\n\n    Discussion_01 - Environment Setup and an Intro to Pytorch, Tensors, and Tensor Operations\n    \n    \n    \n    \n    [slides]\n    \n    \n\n    \n\n\n\nNotebook: 00_fundamentals.ipynb \nSuggested Readings:\n\nSCC Resources\nIntroduction to Pytorch\nWhat\u2019s a Tensor? - Dan Fleisch\nPytorch Internals - Edward Z. Yang \n\n\n\n\nLecture\n\n    01/25/2024\n    Thursday\n\n\n    03 - Shallow Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 3\n\n\n\n\nAssignment\n\n    01/25/2024\n    Thursday\n\n\n    PS3 -- Shallow Networks released!",
  "03 - Shallow Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 3\n\n\n\n\nAssignment\n\n    01/25/2024\n    Thursday\n\n\n    PS3 -- Shallow Networks released!\n\n\n    [PS3 -- Shallow Networks]\n    \n\n\n\nAssignment\n\n    01/29/2024\n    Monday\n\n\n    Project Proposal released!\n\n\n    [Project Proposal]\n    \n\n\n\nLecture\n\n    01/30/2024\n    Tuesday\n\n\n    04 - Deep Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 4\n\n\n\n\nAssignment\n\n    01/30/2024\n    Tuesday\n\n\n    PS4 -- Deep Networks released!\n\n\n    [PS4 -- Deep Networks]\n    \n\n\n\nDue\n\n    01/30/2024\n    23:59\n    Tuesday\n\n\n    PS2 due\n\n\n\n\n\nDiscussion\n\n    01/31/2024\n    Wednesday\n\n\n    Discussion_02 - Autograd, and Computational Graphs in Pytorch. Intro to Model Building in Pytorch.",
  "PS2 due\n\n\n\n\n\nDiscussion\n\n    01/31/2024\n    Wednesday\n\n\n    Discussion_02 - Autograd, and Computational Graphs in Pytorch. Intro to Model Building in Pytorch.\n    \n\n\n\n\nNotebook: 01_autograd.ipynb \nSuggested Readings:\n\nFundamentals of Autograd\nOverview of PyTorch Autograd Engine\nPytorch Internals - Edward Z. Yang \nBuilders\u2019 guide (d2l.ai)\n\n\n\n\nLecture\n\n    02/01/2024\n    Thursday\n\n\n    05 - Loss Functions\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 5\n\n\n\n\nAssignment\n\n    02/01/2024\n    Thursday\n\n\n    KC5 -- Loss Functions released!\n\n\n    [KC5 -- Loss Functions]\n    \n\n\n\nDue\n\n    02/01/2024\n    23:59\n    Thursday\n\n\n    PS3 due\n\n\n\n\n\nLecture\n\n    02/06/2024\n    Tuesday\n\n\n    06 - Fitting Models\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [lecture recording part 2]",
  "PS3 due\n\n\n\n\n\nLecture\n\n    02/06/2024\n    Tuesday\n\n\n    06 - Fitting Models\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [lecture recording part 2]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 6\n\n\n\n\nDue\n\n    02/06/2024\n    23:59\n    Tuesday\n\n\n    PS4 due\n\n\n\n\n\nDiscussion\n\n    02/07/2024\n    Wednesday\n\n\n    Discussion_03 - Intro to Model Training in Pytorch. (Image Classification, Text Classifcation)\n    \n\n\n\n\nNotebook: 02_intro_nn_training.ipynb\n\n\n\nLecture\n\n    02/08/2024\n    Thursday\n\n\n    07a - Gradients and Backpropagation\n    \n    \n    \n    \n    [slides]\n    \n    [jupyter notebook]\n    \n    [Lecture Part 1 - Scalar Gradient Descent]\n    \n    [Lecture Part 2 - Review Jupyter Notebook]\n    \n    [Lecture Part 3 - Matrix Gradient Descent]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Sections 7.1 - 7.4",
  "Suggested Readings:\n\nUDL Sections 7.1 - 7.4\n\n\n\n\nDue\n\n    02/08/2024\n    23:59\n    Thursday\n\n\n    KC5 due\n\n\n\n\n\nDiscussion\n\n    02/14/2024\n    Wednesday\n\n\n    Discussion_04 - Deep Dive 1: How to read, load, and process data. (Examples on Object Detection, Deep Learning on Tabular Data). Using GPUs on SCC.\n    \n\n\n\n\nGithub Link: disc4 \nGoogle Drive: disc4\n\n\n\nLecture\n\n    02/15/2024\n    Thursday\n\n\n    07b - Initialization\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [how to read research papers]\n    \n    [recording - how to read research paper]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Sections 7.5 - 7.6\n\n\n\n\nDue\n\n    02/16/2024\n    23:59\n    Friday\n\n\n    Project Proposal Due\n\n\n\n\n\nAssignment\n\n    02/20/2024\n    Tuesday\n\n\n    KC6-9 -- Chapters 6-9 Knowledge Checks released!",
  "Due\n\n    02/16/2024\n    23:59\n    Friday\n\n\n    Project Proposal Due\n\n\n\n\n\nAssignment\n\n    02/20/2024\n    Tuesday\n\n\n    KC6-9 -- Chapters 6-9 Knowledge Checks released!\n\n\n    [KC6-9 -- Chapters 6-9 Knowledge Checks]\n    \n\n\n\nLecture\n\n    02/20/2024\n    Tuesday\n\n\n    08 - Measuring Performance\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 8\n\n\n\n\nNo Discussion Session\n\n    02/21/2024\n    \n    20:30\n    \n    Wednesday\n\nSubstitute Monday schedule\n\n\n\n\nLecture\n\n    02/22/2024\n    Thursday\n\n\n    09 - Regularization\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 9\n\n\n\n\nDue\n\n    02/26/2024\n    23:59\n    Monday\n\n\n    KC6-9 due\n\n\n\n\n\nLecture\n\n    02/27/2024\n    Tuesday",
  "Suggested Readings:\n\nUDL Chapter 9\n\n\n\n\nDue\n\n    02/26/2024\n    23:59\n    Monday\n\n\n    KC6-9 due\n\n\n\n\n\nLecture\n\n    02/27/2024\n    Tuesday\n\n\n    10 - Convolutional Neural Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 10\n\n\n\n\nLecture\n\n    02/29/2024\n    Thursday\n\n\n    11 - Residual Networks\n    \n    \n    \n    \n    [slides]\n    \n    \n\n    \n\n\n\nUnfortunately the lecture recording cut off after 1 minute. I will try to re-record it\nat some point.\nSuggested Readings:\n\nUDL Chapter 11\n\n\n\n\nLecture\n\n    02/29/2024\n    Thursday\n\n\n    11a - Recurrent Neural Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 11\n\n\n\n\nLecture\n\n    03/05/2024\n    Tuesday\n\n\n    12 - Transformers\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]",
  "Suggested Readings:\n\nUDL Chapter 11\n\n\n\n\nLecture\n\n    03/05/2024\n    Tuesday\n\n\n    12 - Transformers\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 12\nOptional The Illustrated Transformer\n\n\n\n\nDiscussion\n\n    03/06/2024\n    Wednesday\n\n\n    Discussion_05 - Deep Dive 2: Deep Learning Modules in Pytorch (CNN, RNN/LSTM, Transformer)\n    \n\n\n\n\nGithub Link: disc5\n\n\n\nLecture\n\n    03/07/2024\n    Thursday\n\n\n    13 - Transformers Part 2\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 12\nOptional The Illustrated Transformer\n\n\n\n\nSession Ends\n\n    03/08/2024\n    \n    20:30\n    \n    Friday\n\nFirst 7 week session ends\n\n\n\n\nStart of Spring Recess\n\n    03/09/2024\n    \n    20:30\n    \n    Saturday\n\nSpring recess begins -- Have a great break!\n\n\n\n\nEnd of Spring Recess\n\n    03/17/2024\n    \n    20:30\n    \n    Sunday\n\nSpring recess ends",
  "Start of Spring Recess\n\n    03/09/2024\n    \n    20:30\n    \n    Saturday\n\nSpring recess begins -- Have a great break!\n\n\n\n\nEnd of Spring Recess\n\n    03/17/2024\n    \n    20:30\n    \n    Sunday\n\nSpring recess ends\n\n\n\n\nSession Begins\n\n    03/18/2024\n    \n    20:30\n    \n    Monday\n\n2nd 7 week session begins\n\n\n\n\nLecture\n\n    03/19/2024\n    Tuesday\n\n\n    14 -- Vision & Multimodal Transformers\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nSee slides for references\n\n\n\n\nDiscussion\n\n    03/20/2024\n    Wednesday\n\n\n    Discussion_06 - Deep Dive 3: Logging, Model Checkpointing, Tracking Experiments, etc. Hyperparameter Tuning/Search (Optuna). Walkthrough of the VizWiz (Midterm Project) codebase.\n    \n\n\n\n\nGithub Link: disc6\n\n\n\nLecture\n\n    03/21/2024\n    Thursday\n\n\n    15 -- Improving LLM Perf\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]",
  "Github Link: disc6\n\n\n\nLecture\n\n    03/21/2024\n    Thursday\n\n\n    15 -- Improving LLM Perf\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nSee slides for references\n\n\n\n\nLecture\n\n    03/26/2024\n    Tuesday\n\n\n    16 - Parameter Efficient Fine Tuning\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\nReferences are in the lecture slides.\n\n\n\nDiscussion\n\n    03/27/2024\n    Wednesday\n\n\n    Discussion_07 - Midterm Project Lab Session\n    \n\n\n\n\n\n\n\nDiscussion\n\n    04/03/2024\n    Wednesday\n\n\n    Discussion_08 - Huggingface, ViT, CLIP, Kosmos2\n    \n\n\n\n\nGithub Link: disc8\n\n\n\nLecture\n\n    04/04/2024\n    Thursday\n\n\n    17 -- Unsupervised Learning and GANs\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapters 14 and 15\n\n\n\n\nLecture\n\n    04/09/2024\n    Tuesday",
  "17 -- Unsupervised Learning and GANs\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapters 14 and 15\n\n\n\n\nLecture\n\n    04/09/2024\n    Tuesday\n\n\n    18 - Variational Autoencoders (VAEs)\n    \n    \n    \n    \n    [slides]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUnderstanding Variational Autoencoders\nUDL Chapter 17 (optional)\n\nUnfortunately the lecture recorded with no sound, so there is no lecture recording.\n\n\n\nDiscussion\n\n    04/10/2024\n    Wednesday\n\n\n    Discussion_09 - VAEs, and GANs\n    \n\n\n\n\nGithub Link: disc9\n\n\n\nLecture\n\n    04/11/2024\n    Thursday\n\n\n    19 -- Diffusion Models\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nRocca, Understanding Diffusion Probabilistic Models\nUDL Chapter 18\n\n\n\n\nLecture\n\n    04/16/2024\n    Tuesday\n\n\n    20 -- Graph Neural Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]",
  "Suggested Readings:\n\nRocca, Understanding Diffusion Probabilistic Models\nUDL Chapter 18\n\n\n\n\nLecture\n\n    04/16/2024\n    Tuesday\n\n\n    20 -- Graph Neural Networks\n    \n    \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 13\n\n\n\n\nLecture\n\n    04/23/2024\n    Tuesday\n\n\n    21 - Reinforcement Learning\n    \n    \n    \n    \n    [slides]\n    \n    \n\n    \n\n\n\nSuggested Readings:\n\nUDL Chapter 19\n\n\n\n\nDue\n\n    04/25/2024\n    20:30\n    Thursday\n\n\n    Project Presentations Round 1\n\n\n\n\n\nDue\n\n    04/30/2024\n    20:30\n    Tuesday\n\n\n    Project Presentations Round 2\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Lectures - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nLectures\n\n\nYou can download the lectures here. We will try to upload lectures prior to\ntheir corresponding classes. Future lectures listed below likely have\nbroken links.\n\n\n\n\n\n\n\n\n\n01 - Intro to Deep Learning and Course Logistics\ntl;dr: We will introduce the topic of deep learning, a bit about it's history, and what impact it has had. Then we'll go over the course logistics, the lecture topics, problem sets and the mid-term and final projects.\n        \n\n        \n    \n    \n    [slides]\n    \n    [Jupyter Notebook]\n    \n    \n\n        \n\n\nUnfortunately the recording failed for this lecture. Will re-record it at some\npoint.\nSuggested Readings:\n\nUDL Chapter 1",
  "Unfortunately the recording failed for this lecture. Will re-record it at some\npoint.\nSuggested Readings:\n\nUDL Chapter 1\n\n\n\n\n\n\n\n\n\n\n\n\n\n02 - Supervised Learning\ntl;dr: We go a little deeper into supervised learning, introducing terminology and illustrating with a simple example of a linear model.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [Jupyter Notebook]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 2\n\n\n\n\n\n\n\n\n\n\n\n\n\n03 - Shallow Networks\ntl;dr: In this lecture we consider networks with one layer of hidden units and explore their representational power.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 3\n\n\n\n\n\n\n\n\n\n\n\n\n\n04 - Deep Networks\ntl;dr: We dive into deep networks by composing two shallow networks and visualizing their representational capabilities. We then generalize fully connected networks with two and more layers of hidden units. We'll compare the modeling efficiency between deep and shallow networks.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 4",
  "Suggested Readings:\n\nUDL Chapter 4\n\n\n\n\n\n\n\n\n\n\n\n\n\n05 - Loss Functions\ntl;dr: We reconsider loss functions as a measure of how well the data fits to parametric probability distribution. We show that for univariate gaussian distributions we arrive back at least squares loss. We then introduce the notion of maximum likelihood and see how we can use that to define loss functions for many types data distributions. We cover some examples and then show how to generalize. This is a key topic to aid you in applying deep learning models to new types of data.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 5\n\n\n\n\n\n\n\n\n\n\n\n\n\n06 - Fitting Models\ntl;dr: In this lecture we look at different ways minimizing the loss function for models given a training dataset. We'll formally define gradient descent, then show the advantages of stochastic gradient descent and then finally see how momentum and normalized gradients (ADAM) can improve model training farther.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [lecture recording part 2]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 6",
  "Suggested Readings:\n\nUDL Chapter 6\n\n\n\n\n\n\n\n\n\n\n\n\n\n07a - Gradients and Backpropagation\ntl;dr: In this lecture we show how to efficienctly calculate gradients over more complex functions like deep neural networks using backpropagation. We also show an example simple implementation in the accompanying Jupyter notebook.\n        \n\n        \n    \n    \n    [slides]\n    \n    [jupyter notebook]\n    \n    [Lecture Part 1 - Scalar Gradient Descent]\n    \n    [Lecture Part 2 - Review Jupyter Notebook]\n    \n    [Lecture Part 3 - Matrix Gradient Descent]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Sections 7.1 - 7.4\n\n\n\n\n\n\n\n\n\n\n\n\n\n07b - Initialization\ntl;dr: In this lecture we talk about weight initialization and how it can impact the training results. We'll also go back and finish model fitting with the Adam optimizer. We'll also give some tips and tricks on how to efficiently scan and read research papers.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    [how to read research papers]\n    \n    [recording - how to read research paper]",
  "Suggested Readings:\n\nUDL Sections 7.5 - 7.6\n\n\n\n\n\n\n\n\n\n\n\n\n\n08 - Measuring Performance\ntl;dr: We look at measuring model training performance, the importance of test sets as well as how noise, bias and variance play a role in training outcomes.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 8\n\n\n\n\n\n\n\n\n\n\n\n\n\n09 - Regularization\ntl;dr: We explain explicit and implicit regularization techniques and how they help generalize models.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 9\n\n\n\n\n\n\n\n\n\n\n\n\n\n10 - Convolutional Neural Networks\ntl;dr: We cover 1D and 2D convolutional neural networks along with subsampling and upsampling operations.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 10",
  "Suggested Readings:\n\nUDL Chapter 10\n\n\n\n\n\n\n\n\n\n\n\n\n\n11 - Residual Networks\ntl;dr: In this lecture we introduce residual networks, the types of problems they solve, why we need batch normalization and then review some example residual network architectures.\n        \n\n        \n    \n    \n    [slides]\n    \n    \n\n        \n\n\nUnfortunately the lecture recording cut off after 1 minute. I will try to re-record it\nat some point.\nSuggested Readings:\n\nUDL Chapter 11\n\n\n\n\n\n\n\n\n\n\n\n\n\n11a - Recurrent Neural Networks\ntl;dr: In this lecture we introduce recurrent neural networks, starting the plain vanilla RNN, the problem of vanishing gradients, LSTM and GRU and batch normalization.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 11\n\n\n\n\n\n\n\n\n\n\n\n\n\n12 - Transformers\ntl;dr: In this lecture we cover the transformer architecture, starting with the motivation that required a new type of model, the concept and implementation of self-attention and then the full transformer architecture for encoder, decoder and encoder-decoder type models.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]",
  "Suggested Readings:\n\nUDL Chapter 12\nOptional The Illustrated Transformer\n\n\n\n\n\n\n\n\n\n\n\n\n\n13 - Transformers Part 2\ntl;dr: In this lecture we continue to review the transformer architecture. We continue the discussion of decoders and encoder-decoder architectures, then discuss scaling to large contexts and then tokenization and embedding.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 12\nOptional The Illustrated Transformer\n\n\n\n\n\n\n\n\n\n\n\n\n\n14 -- Vision & Multimodal Transformers\ntl;dr: In this lecture we'll cover vision and multimodal transformers as a survey of three papers.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nSee slides for references\n\n\n\n\n\n\n\n\n\n\n\n\n\n15 -- Improving LLM Perf\ntl;dr: In this lecture we talk about ways to improve LLM performance short of retraining or finetuning. We cover more sophisticated prompt strategies, retrieval augmentation and cognitive architectures building systems and agents based on LLMs.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nSee slides for references",
  "Suggested Readings:\n\nSee slides for references\n\n\n\n\n\n\n\n\n\n\n\n\n\n16 - Parameter Efficient Fine Tuning\ntl;dr: In this lecture we'll do a quick review of full model fine tuning then review the parameter efficient finetuning techniques Low Rank Adaptation and Prompt Tuning.,\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\nReferences are in the lecture slides.\n\n\n\n\n\n\n\n\n\n\n\n\n17 -- Unsupervised Learning and GANs\ntl;dr: In this lecture we revisit the concept of unsupervised learning in the context of generative models. We will then dive into Generative Adversarial Networks (GANs) and their applications. We will also discuss the challenges and limitations of GANs and some of the recent advances in the field.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapters 14 and 15",
  "Suggested Readings:\n\nUDL Chapters 14 and 15\n\n\n\n\n\n\n\n\n\n\n\n\n\n18 - Variational Autoencoders (VAEs)\ntl;dr: In this lecture we dive into Variational Autoencoders or VAEs. We start by looking at autoencoders and their ability to reduce dimensions of inputs into a latent space. We'll see why they don't make good generative models and then generalize to VAEs. We'll finish with some examples of generative output of VAEs.\n        \n\n        \n    \n    \n    [slides]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUnderstanding Variational Autoencoders\nUDL Chapter 17 (optional)\n\nUnfortunately the lecture recorded with no sound, so there is no lecture recording.\n\n\n\n\n\n\n\n\n\n\n\n\n19 -- Diffusion Models\ntl;dr: Short text to discribe what this lecture is about.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nRocca, Understanding Diffusion Probabilistic Models\nUDL Chapter 18",
  "19 -- Diffusion Models\ntl;dr: Short text to discribe what this lecture is about.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nRocca, Understanding Diffusion Probabilistic Models\nUDL Chapter 18\n\n\n\n\n\n\n\n\n\n\n\n\n\n20 -- Graph Neural Networks\ntl;dr: In this lecture we introduce graph neural networks, define matrix representations, how to do graph level classification and regression, and how to define graph convolutional network layers.\n        \n\n        \n    \n    \n    [slides]\n    \n    [lecture recording]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 13\n\n\n\n\n\n\n\n\n\n\n\n\n\n21 - Reinforcement Learning\ntl;dr: We cover the basic concepts of reinforcement learning then review reinforcement learning from human feedback via the two seminal papers on the topic.\n        \n\n        \n    \n    \n    [slides]\n    \n    \n\n        \n\n\nSuggested Readings:\n\nUDL Chapter 19\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Discussions - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nDiscussions\n\n\nYou can download the discussions here. We will try to upload discussion material prior to\ntheir corresponding sessions. Future discussions listed below likely have\nbroken links.\n\n\n\n\n\n\n\n\n\nDiscussion_01 - Environment Setup and an Intro to Pytorch, Tensors, and Tensor Operations\ntl;dr: Setting up SCC, virtual environments, and an intro to Pytorch, Tensors, and Tensor Operations. Will also go over on how to run the dl4ds_tutor on SCC.\n        \n\n        \n    \n    \n    [slides]\n    \n    \n\n        \n\n\nNotebook: 00_fundamentals.ipynb \nSuggested Readings:\n\nSCC Resources\nIntroduction to Pytorch\nWhat\u2019s a Tensor? - Dan Fleisch\nPytorch Internals - Edward Z. Yang",
  "Notebook: 00_fundamentals.ipynb \nSuggested Readings:\n\nSCC Resources\nIntroduction to Pytorch\nWhat\u2019s a Tensor? - Dan Fleisch\nPytorch Internals - Edward Z. Yang \n\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_02 - Autograd, and Computational Graphs in Pytorch. Intro to Model Building in Pytorch.\ntl;dr: Fundamentals of Autograd and Computational Graphs in Pytorch. Introduction to Defining a Neural Network in Pytorch (Basics)\n        \n\n\n\n\nNotebook: 01_autograd.ipynb \nSuggested Readings:\n\nFundamentals of Autograd\nOverview of PyTorch Autograd Engine\nPytorch Internals - Edward Z. Yang \nBuilders\u2019 guide (d2l.ai)\n\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_03 - Intro to Model Training in Pytorch. (Image Classification, Text Classifcation)\ntl;dr: Model Training Code Walkthrough. Intro on how to handle Text for Deep Learning Networks.\n        \n\n\n\n\nNotebook: 02_intro_nn_training.ipynb",
  "Discussion_03 - Intro to Model Training in Pytorch. (Image Classification, Text Classifcation)\ntl;dr: Model Training Code Walkthrough. Intro on how to handle Text for Deep Learning Networks.\n        \n\n\n\n\nNotebook: 02_intro_nn_training.ipynb\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_04 - Deep Dive 1: How to read, load, and process data. (Examples on Object Detection, Deep Learning on Tabular Data). Using GPUs on SCC.\ntl;dr: Closer look into how to handle data in Pytorch. With examples on how to create and use custom datasets for Object Detection and Tabular Data. Using and monitoring GPUs on SCC to speed up your models.\n        \n\n\n\n\nGithub Link: disc4 \nGoogle Drive: disc4\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_05 - Deep Dive 2: Deep Learning Modules in Pytorch (CNN, RNN/LSTM, Transformer)\ntl;dr: Fine-tuning Pre-trained Models, Text Generation (LSTM), Text Summarization (Custom Transformer).\n        \n\n\n\n\nGithub Link: disc5",
  "Discussion_05 - Deep Dive 2: Deep Learning Modules in Pytorch (CNN, RNN/LSTM, Transformer)\ntl;dr: Fine-tuning Pre-trained Models, Text Generation (LSTM), Text Summarization (Custom Transformer).\n        \n\n\n\n\nGithub Link: disc5\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_06 - Deep Dive 3: Logging, Model Checkpointing, Tracking Experiments, etc. Hyperparameter Tuning/Search (Optuna). Walkthrough of the VizWiz (Midterm Project) codebase.\ntl;dr: Creating a robust training pipeline. How to log, checkpoint, and track experiments. Hyperparameter Tuning/Search using Optuna. Walkthrough of the VizWiz (Midterm Project) codebase.\n        \n\n\n\n\nGithub Link: disc6\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_07 - Midterm Project Lab Session\ntl;dr: Midterm Project Lab Session for the VizWiz Captioning Challenge.\n        \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_08 - Huggingface, ViT, CLIP, Kosmos2\ntl;dr: Huggingface Transformers, Vision Transformers (ViT), CLIP, Kosmos2.\n        \n\n\n\n\nGithub Link: disc8",
  "Discussion_08 - Huggingface, ViT, CLIP, Kosmos2\ntl;dr: Huggingface Transformers, Vision Transformers (ViT), CLIP, Kosmos2.\n        \n\n\n\n\nGithub Link: disc8\n\n\n\n\n\n\n\n\n\n\n\n\nDiscussion_09 - VAEs, and GANs\ntl;dr: Code examples for VAEs, and GANs.\n        \n\n\n\n\nGithub Link: disc9\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Assignments - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nAssignments\n\n\nAssignments are posted on Gradescope.\nSee the welcome message for the Entry Code if you don\u00e2\u20ac\u2122t have access.\n\n\nPS2 -- Supervised Learning\n\n \n\n\u00a0 \n\n\n\nPS3 -- Shallow Networks\n\n \n\n\u00a0 \n\n\n\nProject Proposal\n\n \n\n\u00a0 \n\n\n\nPS4 -- Deep Networks\n\n \n\n\u00a0 \n\n\n\nKC5 -- Loss Functions\n\n \n\n\u00a0 \n\n\n\nKC6-9 -- Chapters 6-9 Knowledge Checks\n\n \n\n\u00a0 \n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Project - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nProject\n\n\nTable of Contents:\n\nProject Categories\nProject Topics\n\nProject Ideas\n\n\nProject Deliverables\n\nProposal \u2013 Feb 16\nMidpoint Checkin \u2013 Mar 29\nFinal Report \u2013 Apr 30\nVideo \u2013 Apr 30\nProject Code Repository\n\n\n\nHere\u2019s your chance to apply the deep learning techniques you learn in this \nclass to real world applications.\n\nNote: we\u2019re modeling our project definition on Prof. Andrew Ng\u2019s \nCS230 Deep Learning class.\n\nProject Categories\nChoose a project that aligns with your interests and utilizes deep learning\nas part of the solution.\nYou may pick one of the three following categoies of projects.\n\nApplication Project: We expect most students will pick this category. Pick a\nproblem or application that interests you.",
  "Project Categories\nChoose a project that aligns with your interests and utilizes deep learning\nas part of the solution.\nYou may pick one of the three following categoies of projects.\n\nApplication Project: We expect most students will pick this category. Pick a\nproblem or application that interests you. Consider whether there are suitable\ndatasets available already or whether you will have to augment or create a \ndataset. Outcomes are expected to be implementation with an accompanying\ngithub repo and a report.\nAlgorithmic Project: In this category, you will develop a new deep learning\nalgorithm or substantively improve an existing one. One would typically\nbenchmark against some well known dataset and show non-trivial improvement\nover prior work. Outcomes would typically be a short conference style \narticle and an implementation with a github repo.\nTheoretical Project: Prove an interesting property of a new or existing \nlearning algorithm. For a purely theoretical project, the output may only be\na conference style report, but an implementation (and accompanying GitHub repo)\nmay be appropriate as well.\n\nIt\u2019s possible that your project may blend more than one category.\nProject Topics\nDesign a project that piques your interest.",
  "For a purely theoretical project, the output may only be\na conference style report, but an implementation (and accompanying GitHub repo)\nmay be appropriate as well.\n\nIt\u2019s possible that your project may blend more than one category.\nProject Topics\nDesign a project that piques your interest. The more the project aligns with\nyour interest, the more invested you will be in it and the more effort you will\nlikely put into it. Maybe you want to work on something that brings social \nbenefit, or perhaps you want to prototpye a potential future commercial venture!\nIt\u2019s ok to be ambitious and aim high.\nHaving said that, the project is time bound so do your best to scope it to fit\nin the semester timeline. Some factors that may impact the scope:\n\nIs there an existing, labeled dataset that you can use? Or will you have to\ncreate one from scratch? Creating and labeling a dataset is a great experience\nbut can be quite time consuming. On the other hand, one way to get attention\nto a problem or application is to provide an interesting dataset and invite\nthe community to propose solutions.",
  "Or will you have to\ncreate one from scratch? Creating and labeling a dataset is a great experience\nbut can be quite time consuming. On the other hand, one way to get attention\nto a problem or application is to provide an interesting dataset and invite\nthe community to propose solutions.\nWill you be simply prompting an existing model, or fine-tuning a pre-trained\nmodel, or training a new model from scratch. The three approaches are\nincreasingly time consuming.\nIs the problem or application amenable to deep learning solutions? This can \nsometimes be hard to assess without experimenting, but one trick is to reflect\non how hard it would be for a person to solve the problem. Generally, if a\nperson can do the task very quickly (think classify dog versus cat), then there\u2019s\na good chance a deep learning model can be applied. Of course LLMs can be\ncounterintuitive in this regard.\n\nWe encourage you to bounce ideas off the instructor and the TA. We can help you\nbrainstorm your project ideas and help estimate the scope.\nThere are places you can look to help give you ideas:\n\nApplication workshops at major conferences can be good sources of ideas.",
  "We encourage you to bounce ideas off the instructor and the TA. We can help you\nbrainstorm your project ideas and help estimate the scope.\nThere are places you can look to help give you ideas:\n\nApplication workshops at major conferences can be good sources of ideas. Often\ntimes they are associated with new and interesting datasets. Some potential\nconferences include:\n    \nNeurIps,\nCVPR,\nICML,\nICMLA,\nSPIE",
  "There are places you can look to help give you ideas:\n\nApplication workshops at major conferences can be good sources of ideas. Often\ntimes they are associated with new and interesting datasets. Some potential\nconferences include:\n    \nNeurIps,\nCVPR,\nICML,\nICMLA,\nSPIE\n\n\nKaggle and other competition websites can be a\n source of ideas.\nYou might find some interesting datasets at \nPapers with Code\nLot of applications are posted on X/Twitter, Reddit, LinkedIn, etc.\n\nProject Ideas\nIn case it is helpful, here are some project ideas you can also consider:\n\nClass AI Tutor (LLM-based assistant)\n\nThis topic can incorporate multiple student projects.\nEnhance the current primitive class AI tutor with a customized tutor built on a \u201ccognitive architecture\u201d\nframework using langchain or llamaindex, to incorporate things like retrieval augmentation based on course\nmaterials. Have the tutor be socratic in style (e.g. not directly give answers but guide the user on how to\narrive at the answer) and reference course material and lectures in responses.\nExperiment and evaluate with different foundation models.\nIn addition to the core functionality, it would be helpful to have a data collection/model improvement mode\nwhere at a minimum the user can provide feedback on how helpful the response is. This can take multiple forms.\n        \nSimple thumbs up or thumbs down.\nA more sophisticated data collection mode where the user is presented with two responses side by side and they\ncan then pick which one they find more helpful.",
  "Use the feedback data to fine-tune the model.\nAn initial draft at a customized AI tutor is this GitHub repo, currently\nas a Pull Request.\nIdeally, we provide this AI tutor as a template for other instructors at BU and elsewhere.\n\n\nTeacher\u2019s AI Assistant (LLM-based assistant)\n\nThis would complement the student-facing AI tutor with an instructor-facing assistant that would give feedback\nto the instructor on what topics the student are asking about and which ones the students might be struggling with.\nFor both this and the above assistant, build in privacy-preserving features as necessary, so students have control\nover privacy should they choose. The feedback to the instructor would primarily be aggregrated with no personal\nidentifying information. Work out privacy policy so that students could opt to share identity and instructors could\nbe better prepared to work with individual students.\n\n\nCDS Curriculum AI Assistant (LLM-based Assistant)\n\nBuild an LLM-based assistant that could help students navigate the CDS curriculum with tasks such as helping to\nchoose electives based on students\u2019 interests and priorities.\nPossibly provide feedbac to CDS administration in a privacy preserving way.",
  "CDS Curriculum AI Assistant (LLM-based Assistant)\n\nBuild an LLM-based assistant that could help students navigate the CDS curriculum with tasks such as helping to\nchoose electives based on students\u2019 interests and priorities.\nPossibly provide feedbac to CDS administration in a privacy preserving way.\n\n\nCDS Building Recycling Advisor (Computer Vision)\n\nA computer vision based system that directs a person as to which bin an item should be placed.\nEstablish baselines on waste/recycle streams, contamination rates so that if/when prototypes are deployed one\ncan gauge any changes/improvements.",
  "CDS Building Recycling Advisor (Computer Vision)\n\nA computer vision based system that directs a person as to which bin an item should be placed.\nEstablish baselines on waste/recycle streams, contamination rates so that if/when prototypes are deployed one\ncan gauge any changes/improvements.\n\n\nNAACP/WGBH bias detection (Computer Vision, NLP)\n\nBU Spark has a project underway with NAACP and WGBH to understand if there is bias in media reporting of minorities\nand primarily minority neighborhoods. The work to date is using explicit mention of minority status and geographic\nlocations in the text and then applying sentiment analysis. There are two possible extensions to this work:\n        \nExtend the bias analyis to any accompanying photographs to the news stories.\nUse LLMs and other foundation models to infer minority status and geographic location when not explicitly\nmentioned while carefully considering the ethical implications of doing so.\nUse LLMs to infer more nuanced bias in the text than classical NLP techniques may uncover.\n\n\n\n\nHerbaria Foundation Model (Computer Vision, OCR, Multimodal)\n\nAn Herbarium is an institution, usually affiliated with a university or museum, that collects and catalogs plant\nsamples.",
  "Herbaria Foundation Model (Computer Vision, OCR, Multimodal)\n\nAn Herbarium is an institution, usually affiliated with a university or museum, that collects and catalogs plant\nsamples. The plant samples are often dried, pressed and mounted on paper with accompanying descriptive labels,\neither handwritten or typed. Many collections, such as that of the Harvard University Herbaria go back more than\n100 years. There has been a concerted effort over recent years to digitize the images of these plant samples to make\nthem available online. There are now millions of these digitized records online, and tens of millions of records\nyet to be scanned but likely online in the future.\nBU Spark has a project underway to streamline the capture and digitization process of these herbaria plant samples.\nPart of that effort is to implement OCR methods to speed transcription of the sample labels in english, cyrillic and\nchinese characters.\nFor this project you will go beyond OCR to analyze the plant samples themselves, as well as better understand all\ncontent of the digitized herbaria sample.",
  "Part of that effort is to implement OCR methods to speed transcription of the sample labels in english, cyrillic and\nchinese characters.\nFor this project you will go beyond OCR to analyze the plant samples themselves, as well as better understand all\ncontent of the digitized herbaria sample.\nTasks to consider in this project could include: building a plant classifier based on the labeled plant species; \nfrom the plant classification, identify possible misclassification candidates \u2013 propose correct classification or\npossibly identify new species; determine phenological features of the plants \u2013 i.e. the state of any fruit or\nflowers;\nGiven the complexity of the plant samples themselves, would the problem warrant finetuning some kind of foundation\nmodel to eventually make available to the broader scientific community?",
  "the state of any fruit or\nflowers;\nGiven the complexity of the plant samples themselves, would the problem warrant finetuning some kind of foundation\nmodel to eventually make available to the broader scientific community?\n\n\nModern Implementation of Classic Papers\n\nWhat can we learn from some of the earliest papers on neural networks? For this project you will reimplement\nsome of the seminal neural networks from these papers and write an accompanying report that recasts the early \nwork in modern nomemclature, compares and contrasts to modern networks and then perform evaluations of the network\ntraining and performance. Ideally, you provide all these networks in a public GitHub repo.\nEarly works to consider are: Perceptron by Rosenblatt, 1957; ADALINE by Widrow and Hoff, 1960; Neocognitron, by\nFukushima, 1980; Hopfield Networks, by John Hopfield, 1982; Boltzmann Machines by Hinton & Sejnowski, 1983, etc.\n\n\nmore to come\n\nOf course you can pursue any other ideas you have as well!",
  "more to come\n\nOf course you can pursue any other ideas you have as well!\nProject Deliverables\nProposal\nDeadline: February 16, Friday 11:59 PM\nHere\u2019s a proposal template pdf and the source\n\\(\\LaTeX\\).\nThe proposal format is:\n\nProject Title\nAbstract\nTeam Members (From one to three people.)\nIntroduction: Give the motivation for the problem you are solving or application\nyou are developing and why it is worthwhile.\nRelated Work: Results from initial literature search.\nProposed Work: What are you going to do and how are you going to do it?\nDatasets: What dataset will you be using? Does it exist already? What dataset preparation will be needed?\nEvaluation: How are you going to evaluate your results?\nTimeline: Approximate time line for the project over the course of the semester.\nConclusion: You can recap your proposal.\nReferences: References for your citations.\n\nMore explanation of each section is in the template.\nSubmission will be on GradeScope, but feel free to share early draft with the instructor\nto get early feedback.",
  "Timeline: Approximate time line for the project over the course of the semester.\nConclusion: You can recap your proposal.\nReferences: References for your citations.\n\nMore explanation of each section is in the template.\nSubmission will be on GradeScope, but feel free to share early draft with the instructor\nto get early feedback.\nIf you don\u2019t have a \\(\\LaTeX\\) authoring environment set up, we recommend using\nOverleaf or\nthe LaTeX Workshop\nextension for Visual Studio Code.\nMid-Point Check-In\nDeadline: March 31, Sunday 11:59 PM\nPrepare an update on your project status.\nFor the format, update your project proposal with the additional information you have learned since making the proposal.\nIf you didn\u2019t previously use the LaTeX template,\nwe highly recommend you do, and make sure you have content for each of the sections. Feel free to revise any content\nfrom your original proposal to make the update more coherent.\nIdeally, at this point you have:\n\nUpdated or refined your problem statement based on any learnings so that it is more aligned with your interests or objectives and perhaps more feasible.\nUpdated your dataset choices and performed some initial exploratory data analysis to better understand your dataset(s).",
  "Ideally, at this point you have:\n\nUpdated or refined your problem statement based on any learnings so that it is more aligned with your interests or objectives and perhaps more feasible.\nUpdated your dataset choices and performed some initial exploratory data analysis to better understand your dataset(s).\nUpdated your literature and repo survey to indicate the most relevant references and source repos.\nDefined and trained some initial, perhaps greatly simplified, models to start getting a sense of how they may perform on your dataset towards your problem.\nCreated a github repo where you are collecting your work so far. The repo doesn\u2019t have to be clean and well-documented at this point, but it\u2019s not a bad idea to start filling in the top-level README with some description and any learnings or experiments you have done so far.\n\nYou don\u2019t necessarily have to cover all these items completely (hence the word \u201cIdeally\u201d), but you are highly encouraged to show some progress on each item.\nFinal Report and Presentation\nDeadline: April 30, Tuesday 11:59 PM, with a late deadline (with no penalty)\nof May 7, 11:59 PM.\nHere\u2019s the project report template pdf and the source\n\\(\\LaTeX\\).",
  "Final Report and Presentation\nDeadline: April 30, Tuesday 11:59 PM, with a late deadline (with no penalty)\nof May 7, 11:59 PM.\nHere\u2019s the project report template pdf and the source\n\\(\\LaTeX\\).\nThe report should include:\n\nProject Title\nTeam Members (From one to three people.)\nAbstract\nIntroduction\nRelated Work\nApproach (or Methodology)\nDatasets\nEvaluation Results\nConclusion\nReferences\n\nVideo\nDeadline: April 30, Tuesday 11:59 PM\nCreate a 3-4 minute video (no more than 4 minutes) that describe your\nproject. Generally, the video should include:\n\nIntroduce the team\nState the problem or application and why it is important\nProvide the approach taken, models and methods used\nShow the results and how evaluated\n\nWhen complete, upload your video to the BU MyMedia channel for this semester\u2019s\nprojects.\nTo make your screen recordings, one free and easy option is to use Kaltura\nCapture, which is integrated with BU\u2019s MyMedia streaming media solution.\nInstruction video is here,\nincluding how to download the app.",
  "To make your screen recordings, one free and easy option is to use Kaltura\nCapture, which is integrated with BU\u2019s MyMedia streaming media solution.\nInstruction video is here,\nincluding how to download the app.\nProject Code Repository\nAs part of your final project, you should have your project code checked into \na github repo and include the link to your project repo in your report.\nThe repo should be documented enough so that someone can reproduce your work.\n\n\n\n\n\n\n\n\n\n\nBoston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo",
  "Materials - Deep Learning for Data Science (DL4DS) / Spring 2024 \n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nBoston University Faculty for Computing and Data Science\n\nDeep Learning for Data Science (DL4DS)\n\nSpring 2024\n\n\n\n\n\n\n\n\n\n\nMain Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nMaterials",
  "Main Navigation\n\n\n\n Home\n        \n\n\n\n Schedule\n        \n\n\n\n Lectures\n        \n\n\n\n Discussions\n        \n\n\n\n Assignments\n        \n\n\n\n Project\n        \n\n\n\n Materials\n        \n\n\n\n\n\n\n\n\n\n\n\nMaterials\n\n\n\n\n\n\nBook\nUnderstanding Deep Learning\nby Simon J.D. Prince\nMIT Press, 2023\nFree PDF and other book material available here.\nAdditional Course Materials\n\nScientific Python Lectures for lessons on python language and relevant packages, or\nThe Python Tutorial for a tutorial on the core language.\nNumPy - Learn\nSciPy (SciPy User Guide) have tutorials and documentation  as well.\nIntroduction to Linear Algebra, Sixth Edition (2023), by Gilbert Strang.\n\nSCC Resources\n\nRCS website: https://www.bu.edu/tech/support/research/\nCheatsheets:  \n  Linux: https://scv.bu.edu/documents/Linux_SCC_CheatSheet.pdf \n  SCC: https://scv.bu.edu/documents/SCC_CheatSheet.pdf\nVideo Tutorials: https://www.bu.edu/tech/support/research/training-consulting/rcs-tutorial-videos-and-third-party-tutorials/\nSCC Python webpages: https://www.bu.edu/tech/support/research/software-and-programming/common-languages/python/",
  "Boston, MA\nUSA\n\n\n\n\n\n trgardos\n    \n\n\n\n thomas-gardos\n  \n\n\n\n GitHub Repo"
]