AI/機械学習/深層学習入門
2019年8月2日
大阪大学データビリティフロンティア機構
中島悠太

自己紹介
• 中島 悠太
大阪大学データビリティフロンティア機構
准教授• 略歴-2012: Ph.D Course, Osaka University•
2012: Visiting Scholar, UNC Charlotte•• 2012 -2016: Assistant Prof., Nara Institute for Science and Technology• 2015 -2016: Visiting Scholar, CMU• 2017 -2017: Current Position• 研究分野• コンピュータビジョン; CV• パターン認識; PR• （自然言語処理; NLP）

今日の内容• AI? 機械学習? 深層学習?
• なぜ今深層学習か?
• ニューラルネットワークの訓練• ニューラルネットワークの問題• 最近の取り組み• 深層学習利用の例

AI? 機械学習? 深層学習?

AI関連の分野の関係
人工知能; Artificial Intelligence, AI•ルールベース•エキスパートシステム

機械学習; Machine Learning, ML•サポートベクトルマシン•ロジスティック回帰•リッジ回帰•(浅い) ニューラルネットワーク

表現学習; Representation Learning, RL•(浅い) オートエンコーダ•辞書学習

深層学習; Deep Learning, DL•ディープニューラルネットワーク•制限ボルツマンマシン

機械学習のカテゴリ
機械学習…
教師なし学習教師あり学習• 次元削減• 識別• クラスタリング

• 回帰

強化学習
…
“cat” 2 L<latexit sha1_base64="OEPoQbY0S0B+cXpR2czlJ3/rFQU=">AAACeXichVG7SgNBFD1ZXzG+4qMQbKLBBxZhkgiKVdDGwiJR84BE4u466uC+2N0EY/AH/AELGxVEop9h4w9Y+AliqWChhTebBVFR7zAzZ87cc+fMjGJpwnEZewhIbe0dnV3B7lBPb1//QHhwKOeYFVvlWdXUTLugyA7XhMGzrnA1XrBsLuuKxvPK/nJzP1/ltiNMY8OtWXxTl3cNsSNU2SWqHB4p6Yp5UN/aImJ6+ihSEkZktRyOshjzIvITxH0QhR9pM3yFErZhQkUFOjgMuIQ1yHCoFREHg0XcJurE2YSEt89xhBBpK5TFKUMmdp/GXVoVfdagdbOm46lVOkWjbpMygkl2zxrsmd2xG/bI3n+tVfdqNL3UaFZaWm6VB45H11//Vek0u9j7VP3p2cUOFjyvgrxbHtO8hdrSVw9PntcX1ybrU+yCPZH/c/bAbukGRvVFvczwtVOE6APi35/7J8glYvFkLJGZi6aW/K8IYgwTmKH3nkcKK0gjS+ce4gwNXAfepHFpRpptpUoBXzOMLyElPwC+EpGC</latexit>• 検出• セグメンテーション

35.4 2 R+<latexit sha1_base64="gGKFT9Kl3bwUC7c+gcWIKDFkhTc=">AAACfHichVHNLkNBGD29/qr+igWJTaOIRDRTrZ9YCRtLSpGoNPdeg4n7l3unTeqmL+AFLKwQEcFT2HgBC48gliQ2Ir7e3kQQfJOZOXPmO9+cmdEcQ3iSsYeI0tDY1NwSbY21tXd0dsW7e9Y8u+TqPK/bhu1uaKrHDWHxvBTS4BuOy1VTM/i6tr9Q218vc9cTtrUqKw7fMtVdS+wIXZVEFeN9mclUNlEQVqJgqnJP0/xcteiPVYvxJEuxIBI/QToESYSxZMcvUMA2bOgowQSHBUnYgAqP2ibSYHCI24JPnEtIBPscVcRIW6IsThkqsfs07tJqM2QtWtdqeoFap1MM6i4pExhm9+ySPbM7dsUe2duvtfygRs1LhWatruVOseuwf+X1X5VJs8Tep+pPzxI7mAm8CvLuBEztFnpdXz44el6ZzQ37I+yUPZH/E/bAbukGVvlFP1/muWPE6APS35/7J1ibSKUzqYnlbHJuPvyKKAYwiFF672nMYRFLyAfnnuEaN5F3ZUgZU8brqUok1PTiSyhTH7pSkmQ=</latexit>なぜ今深層学習か?

いままでの機械学習と深層学習の違い• 今までの機械学習
特徴抽出器

v

Class label

識別器

“cat”

<latexit sha1_base64="wQd+3GVxZAnAisDyqgzhx+LUl+s=">AAACY3ichVHLSsNAFD2N71ptrS4EEcRScVVuRfCxEt24bKuxhVpKEkcNTZOQpIVa/AHdKi5cKYiIn+HGH3DhDwjiUsGNC2/TgGhR7zAzZ87cc+fMjGobuusRPYakru6e3r7+gfBgZGg4GhuJb7lWzdGErFmG5RRUxRWGbgrZ0z1DFGxHKFXVEHm1stbaz9eF4+qWuek1bFGqKnumvqtrisdUtl6OJShFfkx1gnQAEggiY8WusY0dWNBQQxUCJjzGBhS43IpIg2AzV0KTOYeR7u8LHCLM2hpnCc5QmK3wuMerYsCavG7VdH21xqcY3B1WTiFJD3RDr3RPt/RMH7/Wavo1Wl4aPKttrbDL0aPxjfd/VVWePex/qf707GEXi75Xnb3bPtO6hdbW1w/OXjeWc8nmDF3SC/u/oEe64xuY9TftKity5wjzB6R/PncnkOdSSynKzidWVoOf6McEpjHLz72AFawjA5mPFTjGCU5DT1JEiktj7VQpFGhG8S2kyU9h1onJ</latexit><latexit

h(x)

x

g(v)

y

<latexit sha1_base64="+8N3sbrz5ixVcaxM5DnZRTz1qIk=">AAACZnichVHLSsNAFD2N7/poVUTBTbEodVNuRfCxEt247MOqUEWSOG1D0yQkaaEWf8CFWwVXCiLiZ7jxB1z4CcVlBTcuvEkDokW9w8ycOXPPnTMziqVrjkv0EpJ6evv6BwaHwsMjo2OR6PjErmPWbFXkVVM37X1FdoSuGSLvaq4u9i1byFVFF3tKZcvb36sL29FMY8dtWOKwKpcMraipsutRpUR98SgapyT5EesGqQDEEUTajN7hAMcwoaKGKgQMuIx1yHC4FZACwWLuEE3mbEaavy9wijBra5wlOENmtsJjiVeFgDV47dV0fLXKp+jcbVbGME/PdE9teqIHatHHr7Wafg3PS4NnpaMV1lHkbCb3/q+qyrOL8pfqT88uilj1vWrs3fIZ7xZqR18/uWjn1rPzzQW6oVf2f00v9Mg3MOpv6m1GZK8Q5g9I/XzubpBfSq4lKbMc39gMfmIQs5hDgp97BRvYRhp5PraMc1zgMtSSItKUNN1JlUKBZhLfQop9AkNaip8=</latexit><latexit sha1_base64="zuIlM0RAQY0lng67ZdduYdhsEQ4=">AAACZnichVFNS8NAEH2N3/WjVREFL8Gi6KVMa2nVU9GLR61WBRVJ4rYNTZOQpEUt/gEPXit4UhARf4YX/4CH/gTxqODFg5O0Ih7UWXZn9u28mbe7qm3orkfUDEkdnV3dPb194f6BwaFIdHhky7WqjibymmVYzo6quMLQTZH3dM8QO7YjlIpqiG21vOKfb9eE4+qWuekd22K/ohRNvaBriudDpdmjuYNojOKLC+lkKi1TnCiTSCb8IJlJzafkBCO+xdC2NSt6iz0cwoKGKioQMOFxbECBy2MXCRBsxvZRZ8zhSA/OBU4RZm6VswRnKIyWeS3ybreNmrz3a7oBW+MuBk+HmTKm6Ynu6JUe6Z6e6ePXWvWghq/lmL3a4gr7IHI2sfH+L6vC3kPpm/WnZg8FLARaddZuB4h/C63Fr500XjeWctP1GbqmF9Z/RU164BuYtTftZl3kLhHmD/h6Zfn3IJ+ML8ZpPRXLLrd/oheTmMIsP3cGWaxiDXluW8I5GrgIPUsRaUwab6VKoTZnFD9Mkj8BySGK5A==</latexit><latexit sha1_base64="hD7/ktgyxzdhBNl3aapAK2PlYps=">AAACY3ichVHLSsNAFD2Nr1qrrdWFIEKxKK7KrQg+VkU3Lq0aW1ApSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/oAgLiu4ceFtGhAV9Q4zc+bMPXfOzKi2obse0WNI6uru6e0L90cGooNDsfhwYsu1ao4mZM0yLKegKq4wdFPInu4ZomA7QqmqhsirlZX2fr4uHFe3zE2vYYvdqlI29ZKuKR5TuUYxnqI0+ZH8CTIBSCGINSt+jR3swYKGGqoQMOExNqDA5baNDAg2c7toMucw0v19gUNEWFvjLMEZCrMVHsu82g5Yk9ftmq6v1vgUg7vDyiSm6IFuqEX3dEvP9P5rraZfo+2lwbPa0Qq7GDsa23j7V1Xl2cP+p+pPzx5KWPC96uzd9pn2LbSOvn5w1tpYWp9qTtMlvbD/C3qkO76BWX/VrnJi/RwR/oDM9+f+CeTZ9GKacnOp7HLwE2GMYxIz/NzzyGIVa5D5WIFjnOA09CRFpYQ02kmVQoFmBF9CmvgAZ9OJzA==</latexit>
sha1_base64="VulHcwe1WqcoysuvXhIjEJ/vgDg=">AAACVHichVG7TgJBFD2sL1wfrFYmNkSCsSIXGx+ViY0lDxESJGR3GXDDsrvZXUiQ+AO0FhZWmhhj/Awbf8CC3sZYYmJj4WUhMUrUO5mZM2fm3DlzR3NMw/OJeiFpYnJqeiY8K8/NywuLEWX+yLObri5yum3abkFTPWEalsj5hm+KguMKtaGZIq/V9wf7+ZZwPcO2Dv22I0oNtWYZVUNXfaZSZSVGCQoiOg6SIxDDKGzlFseowIaOJhoQsOAzNqHC41ZEEgSHuRI6zLmMjGBf4Awya5t8SvAJldk6jzVeFUesxetBTi9Q63yLyd1lZRRxeqI76tMj3dMLffyaqxPkGHhp86wNtcIpR7or2fd/VQ2efZx8qf707KOK7cCrwd6dgBm8Qh/qW6cX/exuJt5Zp2t6Zf9X1KMHfoHVetNv0iJzCZnrn/xZ7XGQ20zsJChNCGMVa9jgKm9hDwdIIce3VdDFeehZkiVl+E1SaPRfS/gW0vIneJ2Hiw==</latexit>
sha1_base64="PVbi+JADnEehGzwly0XRFpuzUxA=">AAACWHichVG7SgNBFD1ZXzFGE4OFYBMMEatwY+OjEmws8zAmEEPYXSdxcV/sbgIx+APaKhZWCiLiZ9j4Axb5AUEsI9hYeLMJiAb1DjNz5sycO2fmKrauuR5RJyCNjI6NTwQnQ1Ph6ZlIdDa861oNRxUF1dItp6TIrtA1UxQ8zdNFyXaEbCi6KCqHW739YlM4rmaZO17LFhVDrptaTVNlj6lsqxpNUIr8iA+D9AAkMIiMFb3FHvZhQUUDBgRMeIx1yHC5lZEGwWaugjZzDiPN3xc4Roi1DT4l+ITM7CGPdV6VB6zJ615O11erfIvO3WFlHEl6ojvq0iPd0wt9/Jqr7efoeWnxrPS1wq5GTubz7/+qDJ49HHyp/vTsoYY136vG3m2f6b1C7eubRxfd/EYu2V6ia3pl/1fUoQd+gdl8U2+yIneJEBcg/fO7h0FhJbWeoiwhiAUsYpl/eRWb2EYGBb5N4BRnOA88S2Ep1q+UFBiULIZvIc19AnQjiP0=</latexit>
sha1_base64="Lmw5f1V4ZMPkGHLSJDViDUe14tk=">AAACY3ichVG7SgNBFD1ZXzE+EqOFIEIwRKzCjY2PKmhjmYcxgSiyu050yWZ32d0EYvAHtFUsrBRExM+w8Qcs8gOCWEawsfBmsyAq6h1m5syZe+6cmVEsXXNconZA6usfGBwKDodGRsfGw5GJ6JZj1m1VFFRTN+2SIjtC1wxRcDVXFyXLFnJN0UVRqa5394sNYTuaaWy6TUvs1OR9Q6toquwylW3uRuKUJC9iP0HKB3H4kTEjN9jGHkyoqKMGAQMuYx0yHG5lpECwmNtBizmbkebtCxwhxNo6ZwnOkJmt8rjPq7LPGrzu1nQ8tcqn6NxtVsaQoEe6pQ490B090/uvtVpeja6XJs9KTyus3fDxdP7tX1WNZxcHn6o/PbuoYNnzqrF3y2O6t1B7+sbheSe/mku05umKXtj/JbXpnm9gNF7V66zIXSDEH5D6/tw/QWExuZKkLMXTa/5PBDGDOSzwcy8hjQ1kUOBjBU5wirPAkzQqRaWpXqoU8DWT+BLS7Admk4nI</latexit><latexit sha1_base64="Qj4U1j/Mx929t2ieD/QbKvBgPOI=">AAACY3ichVHLSsNAFD2Nr1ofrdWFIEKxVFyVWxF8rIpuXPZhraBSkjhqaJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAV9Q4zc+bMPXfOzCiWrjkuUTMgdXX39PYF+0MDg0PD4chIdN0xq7YqCqqpm/aGIjtC1wxRcDVXFxuWLeSKoouiUl5p7xdrwnY001hz65bYrsh7hrarqbLLVPagFIlTkryI/QQpH8ThR8aM3GALOzChoooKBAy4jHXIcLhtIgWCxdw2GszZjDRvX+AIIdZWOUtwhsxsmcc9Xm36rMHrdk3HU6t8is7dZmUMCXqkW2rRA93RM73/Wqvh1Wh7qfOsdLTCKoWPx/Nv/6oqPLvY/1T96dnFLhY8rxp7tzymfQu1o68dnrfyS7lEY5qu6IX9X1KT7vkGRu1Vvc6K3AVC/AGp78/9ExRmk4tJys7F08v+TwQxgSnM8HPPI41VZFDgYwVOcIqzwJM0KEWlsU6qFPA1o/gS0uQHZdSJyw==</latexit><latexit• 深層学習
マッピング

x<latexit sha1_base64="Qj4U1j/Mx929t2ieD/QbKvBgPOI=">AAACY3ichVHLSsNAFD2Nr1ofrdWFIEKxVFyVWxF8rIpuXPZhraBSkjhqaJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAV9Q4zc+bMPXfOzCiWrjkuUTMgdXX39PYF+0MDg0PD4chIdN0xq7YqCqqpm/aGIjtC1wxRcDVXFxuWLeSKoouiUl5p7xdrwnY001hz65bYrsh7hrarqbLLVPagFIlTkryI/QQpH8ThR8aM3GALOzChoooKBAy4jHXIcLhtIgWCxdw2GszZjDRvX+AIIdZWOUtwhsxsmcc9Xm36rMHrdk3HU6t8is7dZmUMCXqkW2rRA93RM73/Wqvh1Wh7qfOsdLTCKoWPx/Nv/6oqPLvY/1T96dnFLhY8rxp7tzymfQu1o68dnrfyS7lEY5qu6IX9X1KT7vkGRu1Vvc6K3AVC/AGp78/9ExRmk4tJys7F08v+TwQxgSnM8HPPI41VZFDgYwVOcIqzwJM0KEWlsU6qFPA1o/gS0uQHZdSJyw==</latexit><latexit

f (x)<latexit sha1_base64="qZ8OICKBRCWlqVDBdMmzpXsTuD8=">AAACkXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvoVseADYkuSk+KrTbQMzSyMDUz0jHQM7A0tzAHMUyMjMxNTGrTNCo0a+MFlGFKFGBKFGBKFAz1DMBAmQEKAvIFljPEMKQw5DMkM5Qy5DKkMuQxlADZOQyJDMVAGM1gyGDAUAAUi2WoBooVAVmZYPlUhloGLqDeUqCqVKCKRKBoNpBMB/KioaJ5QD7IzGKw7mSgLTlAXATUqcCganDVYKXBZ4MTBqsNXhr8wWlWNdgMkFsqgXQSRG9qQTx/l0Twd4K6coF0CUMGQhdeN5cwpDFYgN2aCXR7AVgE5ItkiP6yqumfg62CVKvVDBYZvAa6f6HBTYPDQB/klX1JXhqYGjSbgQsYAbBQVsDNCDXSs9QzCDRRdnCCxgQHgzSDEoMGMLjNGRwYPBgCGEKB1k5m2MlwiOEwkwiTFZMDE1QtEyNUjzADCmDyBgBr+5c8</latexit>

Class label

“cat”

y
<latexit sha1_base64="hD7/ktgyxzdhBNl3aapAK2PlYps=">AAACY3ichVHLSsNAFD2Nr1qrrdWFIEKxKK7KrQg+VkU3Lq0aW1ApSZzW0DQJSVqoxR/QreLClYKI+Blu/AEX/oAgLiu4ceFtGhAV9Q4zc+bMPXfOzKi2obse0WNI6uru6e0L90cGooNDsfhwYsu1ao4mZM0yLKegKq4wdFPInu4ZomA7QqmqhsirlZX2fr4uHFe3zE2vYYvdqlI29ZKuKR5TuUYxnqI0+ZH8CTIBSCGINSt+jR3swYKGGqoQMOExNqDA5baNDAg2c7toMucw0v19gUNEWFvjLMEZCrMVHsu82g5Yk9ftmq6v1vgUg7vDyiSm6IFuqEX3dEvP9P5rraZfo+2lwbPa0Qq7GDsa23j7V1Xl2cP+p+pPzx5KWPC96uzd9pn2LbSOvn5w1tpYWp9qTtMlvbD/C3qkO76BWX/VrnJi/RwR/oDM9+f+CeTZ9GKacnOp7HLwE2GMYxIz/NzzyGIVa5D5WIFjnOA09CRFpYQ02kmVQoFmBF9CmvgAZ9OJzA==</latexit>
sha1_base64="VulHcwe1WqcoysuvXhIjEJ/vgDg=">AAACVHichVG7TgJBFD2sL1wfrFYmNkSCsSIXGx+ViY0lDxESJGR3GXDDsrvZXUiQ+AO0FhZWmhhj/Awbf8CC3sZYYmJj4WUhMUrUO5mZM2fm3DlzR3NMw/OJeiFpYnJqeiY8K8/NywuLEWX+yLObri5yum3abkFTPWEalsj5hm+KguMKtaGZIq/V9wf7+ZZwPcO2Dv22I0oNtWYZVUNXfaZSZSVGCQoiOg6SIxDDKGzlFseowIaOJhoQsOAzNqHC41ZEEgSHuRI6zLmMjGBf4Awya5t8SvAJldk6jzVeFUesxetBTi9Q63yLyd1lZRRxeqI76tMj3dMLffyaqxPkGHhp86wNtcIpR7or2fd/VQ2efZx8qf707KOK7cCrwd6dgBm8Qh/qW6cX/exuJt5Zp2t6Zf9X1KMHfoHVetNv0iJzCZnrn/xZ7XGQ20zsJChNCGMVa9jgKm9hDwdIIce3VdDFeehZkiVl+E1SaPRfS/gW0vIneJ2Hiw==</latexit>
sha1_base64="PVbi+JADnEehGzwly0XRFpuzUxA=">AAACWHichVG7SgNBFD1ZXzFGE4OFYBMMEatwY+OjEmws8zAmEEPYXSdxcV/sbgIx+APaKhZWCiLiZ9j4Axb5AUEsI9hYeLMJiAb1DjNz5sycO2fmKrauuR5RJyCNjI6NTwQnQ1Ph6ZlIdDa861oNRxUF1dItp6TIrtA1UxQ8zdNFyXaEbCi6KCqHW739YlM4rmaZO17LFhVDrptaTVNlj6lsqxpNUIr8iA+D9AAkMIiMFb3FHvZhQUUDBgRMeIx1yHC5lZEGwWaugjZzDiPN3xc4Roi1DT4l+ITM7CGPdV6VB6zJ615O11erfIvO3WFlHEl6ojvq0iPd0wt9/Jqr7efoeWnxrPS1wq5GTubz7/+qDJ49HHyp/vTsoYY136vG3m2f6b1C7eubRxfd/EYu2V6ia3pl/1fUoQd+gdl8U2+yIneJEBcg/fO7h0FhJbWeoiwhiAUsYpl/eRWb2EYGBb5N4BRnOA88S2Ep1q+UFBiULIZvIc19AnQjiP0=</latexit>
sha1_base64="Lmw5f1V4ZMPkGHLSJDViDUe14tk=">AAACY3ichVG7SgNBFD1ZXzE+EqOFIEIwRKzCjY2PKmhjmYcxgSiyu050yWZ32d0EYvAHtFUsrBRExM+w8Qcs8gOCWEawsfBmsyAq6h1m5syZe+6cmVEsXXNconZA6usfGBwKDodGRsfGw5GJ6JZj1m1VFFRTN+2SIjtC1wxRcDVXFyXLFnJN0UVRqa5394sNYTuaaWy6TUvs1OR9Q6toquwylW3uRuKUJC9iP0HKB3H4kTEjN9jGHkyoqKMGAQMuYx0yHG5lpECwmNtBizmbkebtCxwhxNo6ZwnOkJmt8rjPq7LPGrzu1nQ8tcqn6NxtVsaQoEe6pQ490B090/uvtVpeja6XJs9KTyus3fDxdP7tX1WNZxcHn6o/PbuoYNnzqrF3y2O6t1B7+sbheSe/mku05umKXtj/JbXpnm9gNF7V66zIXSDEH5D6/tw/QWExuZKkLMXTa/5PBDGDOSzwcy8hjQ1kUOBjBU5wirPAkzQqRaWpXqoU8DWT+BLS7Admk4nI</latexit>

ニューラルネットワークの構成要素• 全結合層: h = W x + b

教師信号 tロス関数<latexit sha1_base64="MMgHjpXlkeOHsPydpO4ACxuWX/s=">AAACZHichVHLSsNAFD2Nr1ofrRZBEEQsiiu5kYLiquDGZR9WBRVJ4qihaRKSaaEWf0C3igtXCiLiZ7jxB1z0BwRxqeDGhTdpQFTUO8zMmTP33Dkzo7uW6UuiVkzp6Ozq7on3Jvr6BwaTqaHhVd+peYYoG47leOu65gvLtEVZmtIS664ntKpuiTW9shTsr9WF55uOvSIbrtiqanu2uWsammSqILdTGZqlMCZ+AjUCGUSRd1LX2MQOHBiooQoBG5KxBQ0+tw2oILjMbaHJnMfIDPcFDpFgbY2zBGdozFZ43OPVRsTavA5q+qHa4FMs7h4rJzBFD3RDL3RPt/RE77/WaoY1Ai8NnvW2VrjbyaPR0tu/qirPEvufqj89S+xiIfRqsnc3ZIJbGG19/eDspbRYnGpO0yU9s/8LatEd38CuvxpXBVE8R4I/QP3+3D/B6tysyriQzeSy0VfEMYZJzPB7zyOHZeRR5nMFjnGC09ij0q+klZF2qhKLNGl8CWX8A+oYieY=</latexit><latexit<latexit sha1_base64="kkWpxS4h7ewksvkQbSyWsYp6kWM=">AAACbXichVHLSgMxFD0d3/VVFUFQZLD4AKGkIiiCUHDjsg/7wAcyM0YNnc4MM2mxFn/AteBCFBRExM9w4w+48BPEhQsFNy68nQ6IinpDkpOTe25OEt0xhScZewgpTc0trW3tHeHOru6e3khff86zy67Bs4Zt2m5B1zxuCotnpZAmLzgu10q6yfN6cam+n69w1xO2tSKrDt8oaTuW2BaGJola3VUX1fyeOq3qm5EoizE/1J8gHoAogkjakSusYws2DJRRAocFSdiEBo/aGuJgcIjbQI04l5Dw9zkOECZtmbI4ZWjEFmncodVawFq0rtf0fLVBp5jUXVKqGGf37Jq9sDt2wx7Z+6+1an6NupcqzXpDy53N3sOhzNu/qhLNErufqj89S2xj3vcqyLvjM/VbGA19Zf/4JbOQHq9NsAv2RP7P2QO7pRtYlVfjMsXTJwjTB8S/P/dPkJuJxQmnZqOJ+eAr2jGMMUzRe88hgWUkkaVzLRzhFGehZ2VQGVFGG6lKKNAM4Esokx9hyoxR</latexit>• 画像の場合は畳み込み層

Output y<latexit sha1_base64="JdYEc2e/Wb2b5VTnyuHjrlS9J1M=">AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KRAqKq4Ibl33YB9RSkjitoWkSkrRQiz+gW8WFKwUR8TPc+AMu+gOCuKzgxoW3aUC0qHeYmTNn7rlzZkY2NdV2GOv5hLHxickp/3RgJjg7FwrPLxRso2UpPK8YmmGVZMnmmqrzvKM6Gi+ZFpeassaLcmNnsF9sc8tWDX3P6Zi80pTqulpTFckhKtOphmMswdyIjgLRAzF4kTbCt9jHAQwoaKEJDh0OYQ0SbGpliGAwiaugS5xFSHX3OY4RIG2LsjhlSMQ2aKzTquyxOq0HNW1XrdApGnWLlFHE2RO7Y332yO7ZC/v4tVbXrTHw0qFZHmq5WQ2dLOXe/1U1aXZw+KX607ODGrZcryp5N11mcAtlqG8fXfRz29l4d41ds1fyf8V67IFuoLfflJsMz14iQB8g/nzuUVDYSIiEM8lYKul9hR/LWMU6vfcmUthFGnk6l+MUZzj3PQtBISIsDlMFn6eJ4FsIK5/0GInr</latexit>• 活性化関数(非線形性): y = (h)活性化関数• Hyperbolic tangent (h) = tanh(h)• ReLU (h) = max(0, h)• etc.

全結合層<latexit sha1_base64="LZuXmJY/Hvr0FzepabxkSatbrQI=">AAACcHichVHLSgMxFD0dX7W+qm4EF44WRV1IKoJFEAQ3LvuwtlCLzIyxBuflzLRQiz/gD3Thxgoi4me48Qdc9BPEnQpuXHhnOiAq6g1JTk7uuTlJVFsXrsdYOyJ1dff09kX7YwODQ8Mj8dGxHdeqOhrPa5ZuOUVVcbkuTJ73hKfzou1wxVB1XlCPNv39Qo07rrDMba9u87KhVExxIDTFI6pcl9flXVdUDGX+cGEvnmBLLAj5J0iGIIEw0lb8GrvYhwUNVRjgMOER1qHApVZCEgw2cWU0iHMIiWCf4xQx0lYpi1OGQuwRjRValULWpLVf0w3UGp2iU3dIKWOWPbAb9sLu2S17ZO+/1moENXwvdZrVjpbbeyNnE7m3f1UGzR4OP1V/evZwgFTgVZB3O2D8W2gdfe2k+ZJby8425tgleyL/LdZmd3QDs/aqXWV49hwx+oDk9+f+CXaWl5KEMyuJjVT4FVFMYgbz9N6r2MAW0sjTucdo4gKtyLM0IU1J051UKRJqxvElpMUPFYGOCg==</latexit><latexit sha1_base64="+aYzdaK2XVCSpQ7+k4ZgAIOnSik=">AAACeXichVHLSgMxFD0d3/U1PhaCm2qpVBeSqqAIguDGpVZrC1bKzJi2wXkxkxZs8Qf8ARduVBBRP8ONP+DCTxCXCi504e10QLSoNyQ5Obnn5iTRXVP4krHHiNLW3tHZ1d0T7e3rHxhUh4Z3fKfiGTxjOKbj5XTN56aweUYKafKc63HN0k2e1Q/WGvvZKvd84djb8tDle5ZWskVRGJokqqCO5n1RsrRkeTq2EstLzS4TLKhxNsuCiLWCVAjiCGPDUa+Qxz4cGKjAAocNSdiEBp/aLlJgcInbQ504j5AI9jmOECVthbI4ZWjEHtBYotVuyNq0btT0A7VBp5jUPVLGkGAP7Jq9sHt2y57Yx6+16kGNhpdDmvWmlruFweOxrbd/VRbNEuUv1Z+eJYpYCrwK8u4GTOMWRlNfrZ28bC2nE/UpdsGeyf85e2R3dAO7+mpcbvL0KaL0Aamfz90KduZmU4Q3F+KrC+FXdGMck0jSey9iFevYQIbOreEM17iJvCsTSlKZaaYqkVAzgm+hzH8CV6+QxA==</latexit>• Others:• バッチ正規化• プーリング• etc.

活性化関数<latexit sha1_base64="pNbcj3NpK98tXpPOs/xTlVli50w=">AAACZ3ichVG7SgNBFD1ZXzE+EhUkYKMGRZtwI4JiFbCxzMPEQBTZXSe6ZF/sboIa/AELWwUrBRHxM2z8AYt8QrBUsLHw7mZBVNQ7zMyZM/fcOTOj2LrmekTtiNTT29c/EB2MDQ2PjMYTY+Nl12o4qiiplm45FUV2ha6ZouRpni4qtiNkQ9HFllJf9/e3msJxNcvc9I5ssWPI+6ZW01TZ86nawuHibiJFaQpi+ifIhCCFMHJW4hbb2IMFFQ0YEDDhMdYhw+VWRQYEm7kdtJhzGGnBvsAJYqxtcJbgDJnZOo/7vKqGrMlrv6YbqFU+RefusHIac/REd/RCj3RPHXr/tVYrqOF7OeJZ6WqFvRs/TRbf/lUZPHs4+FT96dlDDauBV4292wHj30Lt6pvH5y/FtcJca56u6Zn9X1GbHvgGZvNVvcmLwiVi/AGZ78/9E5SX0hnG+eVUdjn8iiimMIsFfu8VZLGBHEp87gHOcI6LSEeKS5NSspsqRULNBL6ENPMB0laKvw==</latexit>
sha1_base64="1tqRKu4qqxHI8AQAeehg2wQidig=">AAACVXichVG7TgJBFD2sL1xRIDYmNkSCsSIXG42ViY0lD3kkSMjuMuCGZXezu5Ag8QdoLSysNDHG+Bk2/oAFvY2xxMTGwstCYpSodzIzZ87MuXPmjmobuusRDQLSzOzc/EJwUV4Kycsr4Uio4FptRxN5zTIsp6QqrjB0U+Q93TNEyXaE0lINUVSbB6P9Ykc4rm6ZR17XFpWW0jD1uq4pHlPpaiROSfIjNg1SExDHJKzILY5RgwUNbbQgYMJjbECBy62MFAg2cxX0mHMY6f6+wBlk1rb5lOATCrNNHhu8Kk9Yk9ejnK6v1vgWg7vDyhgS9ER3NKRHuqcX+vg1V8/PMfLS5Vkda4VdDffXcu//qlo8ezj5Uv3p2UMdu75Xnb3bPjN6hTbWd04vhrm9bKK3Sdf0yv6vaEAP/AKz86bdZET2EjLXP/Wz2tOgsJ1MMc4QgljHBra4zDvYxyHSyPN1NfRxHniWZCk6/icpMPmwKL6FtPoJBguHtw==</latexit>
sha1_base64="6eLMJmF33J4+WTuVCsbLWkeP7Xs=">AAACXHichVG7SgNBFD1ZXzE+Ei0kYCOGSGzkro1iJdhY5mESIUrYXSfJ4r7Y3QQ1+AMWtgpWCiLiZ9j4AxZ+QrCMYGPh3U1ANKh3mJkzZ+bcOTNXdQzd84leItLQ8MjoWHQ8NjE5NR1PzEyWPLvpaqKo2Ybt7qqKJwzdEkVf9w2x67hCMVVDlNXDrWC/3BKup9vWjn/siH1TqVt6TdcUP6BqmaPlaiJFKxTGwiCQ+yCFfmTtxB32cAAbGpowIWDBZ2xAgcetAhkEh7l9tJlzGenhvsApYqxt8inBJxRmD3ms86rSZy1eBzm9UK3xLQZ3l5ULSNMz3VOXnuiBOvTxa652mCPwcsyz2tMKpxo/Sxbe/1WZPPtofKn+9OyjhvXQq87enZAJXqH19K2Ti25hI59uL9ENvbL/a3qhR36B1XrTbnMif4UYF0D++d2DoLS6IjPOEaKYxyIy/M1r2MQ2sijydQ2c4wKXkY4Ul+Z6pZIi/ZrN4ltIyU/YU4n6</latexit>
sha1_base64="+z3hxTKa+tlEihVqdkmzg9QvjCI=">AAACZ3ichVHLSsNAFD2Nr1ofrQpScKMWRTflVgTFleDGZavWFmqRJE7b0DQJSVqsxR9w4VbBlYKI+Blu/AEX/YTisoIbF96kAVFR7zAzZ87cc+fMjGLpmuMStUNSX//A4FB4ODIyOjYejU1M7jtm3VZFVjV1084rsiN0zRBZV3N1kbdsIdcUXeSU6pa3n2sI29FMY89tWqJYk8uGVtJU2fWo0tLx8mEsQUnyY/YnSAUggSDSZuwOBziCCRV11CBgwGWsQ4bDrYAUCBZzRbSYsxlp/r7AKSKsrXOW4AyZ2SqPZV4VAtbgtVfT8dUqn6Jzt1k5iwV6pnvq0hM9UIfef63V8mt4Xpo8Kz2tsA6jZ/Hdt39VNZ5dVD5Vf3p2UcK671Vj75bPeLdQe/rGyUV3d2NnobVIN/TC/q+pTY98A6Pxqt5mxM4VIvwBqe/P/RPsryRTjDOU2FwNviKMGcxjid97DZvYRhpZPreCc1zgMtSRotK0FO+lSqFAM4UvIc19ANEWirs=</latexit><latexit sha1_base64="ZWQHyVxiGrfol6tYU9WAWn9TgfA=">AAACanichVFNLwNBGH66vqq+qhxILw0qlUgzi4Q4NXFxcGhRlSCyu6ZsbHc3u9MGjT/g5iTRE4mI+Bku/oBDf4JwI3Fx8O60iSB4JzPzzDPv884zM7prmb5grB5SWlrb2jvCnZGu7p7evmh/bM13yp7B84ZjOd66rvncMm2eF6aw+Lrrca2kW7yg7y8E+4UK93zTsVfFocu3StqubRZNQxNEFZZSB5MJMbEdHWVpJiPxE6hNMJoZLMrIOtFrbGIHDgyUUQKHDUHYggaf2gZUMLjEbaFKnEfIlPscx4iQtkxZnDI0Yvdp3KXVRpO1aR3U9KXaoFMs6h4pE0iyB3bDXtg9u2WP7P3XWlVZI/BySLPe0HJ3u+9kaOXtX1WJZoG9T9WfngWKmJNeTfLuSia4hdHQV47OXlbml5PVcXbJnsj/BauzO7qBXXk1rnJ8uYYIfYD6/bl/grWptDqdnsrRT8ygEWHEMYIUvfcsMlhEFnnp7hTnqIWelZgyrMQbqUqoqRnAl1DGPgCLG447</latexit>f (x)• ロス関数: L(x, t)…

<latexit sha1_base64="OGOx8LVL+sJcFmit/3vNVSDAGyY=">AAACeXichVHLSgMxFD0d3/VVHwvBTbUoVURutaAIguDGpa+qYKXMjGkbnBcz02It/oA/4MKNCiLqZ7jxB1z4CeJSwYUuvJ0OiIp6Q5KTk3tuThLNMaTnEz1ElIbGpuaW1rZoe0dnV3esp3fDs0uuLjK6bdjulqZ6wpCWyPjSN8SW4wrV1Ayxqe0t1vY3y8L1pG2t+xVH7JhqwZJ5qas+U7lYf9aTBVNNFsfms6a6n6SJeHEsF0vQJAUR/wlSIUggjGU7doksdmFDRwkmBCz4jA2o8LhtIwWCw9wOqsy5jGSwL3CIKGtLnCU4Q2V2j8cCr7ZD1uJ1raYXqHU+xeDusjKOEbqnK3qmO7qhR3r/tVY1qFHzUuFZq2uFk+s+Glh7/Vdl8uyj+Kn607OPPGYDr5K9OwFTu4Ve15cPjp/X5lZHqqN0Tk/s/4we6JZvYJVf9IsVsXqCKH9A6vtz/wQbU5MpxivpxEI6/IpWDGIYSX7vGSxgCcvI8LkHOMUVriNvypCSVMbrqUok1PThSyjTHweykJs=</latexit>

全結合層
活性化関数
全結合層
Input x<latexit sha1_base64="MQ2Wn/quWMCC8abSWNLYORWSBvk=">AAACZHichVFNSwJBGH7cvswsLQmCICQxOsUYQtFJ6NLRj/wAE9ndRlvcL3ZXyaQ/UNeiQ6eCiOhndOkPdPAPBNHRoEuHXteFKKneYWaeeeZ93nlmRjJVxXYY6/qEkdGx8Qn/ZGAqOD0TCs/OFWyjack8LxuqYZUk0eaqovO8ozgqL5kWFzVJ5UWpsd3fL7a4ZSuGvuu0TV7RxLqu1BRZdIjKHFbDMbbG3IgOg4QHYvAibYRvsYd9GJDRhAYOHQ5hFSJsamUkwGASV0GHOIuQ4u5zHCNA2iZlccoQiW3QWKdV2WN1Wvdr2q5aplNU6hYpo4izJ3bHeuyR3bMX9vFrrY5bo++lTbM00HKzGjpZyL3/q9JodnDwpfrTs4MaNl2vCnk3XaZ/C3mgbx1d9HJb2XhnhV2zV/J/xbrsgW6gt97kmwzPXiJAH5D4+dzDoLC+liCcScZSSe8r/FjEMlbpvTeQwg7SyNO5HKc4w7nvWQgKEWF+kCr4PE0E30JY+gTyGInq</latexit><latexitニューラルネットワークのいいところ
f (x)<latexit sha1_base64="pNbcj3NpK98tXpPOs/xTlVli50w=">AAACZ3ichVG7SgNBFD1ZXzE+EhUkYKMGRZtwI4JiFbCxzMPEQBTZXSe6ZF/sboIa/AELWwUrBRHxM2z8AYt8QrBUsLHw7mZBVNQ7zMyZM/fcOTOj2LrmekTtiNTT29c/EB2MDQ2PjMYTY+Nl12o4qiiplm45FUV2ha6ZouRpni4qtiNkQ9HFllJf9/e3msJxNcvc9I5ssWPI+6ZW01TZ86nawuHibiJFaQpi+ifIhCCFMHJW4hbb2IMFFQ0YEDDhMdYhw+VWRQYEm7kdtJhzGGnBvsAJYqxtcJbgDJnZOo/7vKqGrMlrv6YbqFU+RefusHIac/REd/RCj3RPHXr/tVYrqOF7OeJZ6WqFvRs/TRbf/lUZPHs4+FT96dlDDauBV4292wHj30Lt6pvH5y/FtcJca56u6Zn9X1GbHvgGZvNVvcmLwiVi/AGZ78/9E5SX0hnG+eVUdjn8iiimMIsFfu8VZLGBHEp87gHOcI6LSEeKS5NSspsqRULNBL6ENPMB0laKvw==</latexit>
sha1_base64="1tqRKu4qqxHI8AQAeehg2wQidig=">AAACVXichVG7TgJBFD2sL1xRIDYmNkSCsSIXG42ViY0lD3kkSMjuMuCGZXezu5Ag8QdoLSysNDHG+Bk2/oAFvY2xxMTGwstCYpSodzIzZ87MuXPmjmobuusRDQLSzOzc/EJwUV4Kycsr4Uio4FptRxN5zTIsp6QqrjB0U+Q93TNEyXaE0lINUVSbB6P9Ykc4rm6ZR17XFpWW0jD1uq4pHlPpaiROSfIjNg1SExDHJKzILY5RgwUNbbQgYMJjbECBy62MFAg2cxX0mHMY6f6+wBlk1rb5lOATCrNNHhu8Kk9Yk9ejnK6v1vgWg7vDyhgS9ER3NKRHuqcX+vg1V8/PMfLS5Vkda4VdDffXcu//qlo8ezj5Uv3p2UMdu75Xnb3bPjN6hTbWd04vhrm9bKK3Sdf0yv6vaEAP/AKz86bdZET2EjLXP/Wz2tOgsJ1MMc4QgljHBra4zDvYxyHSyPN1NfRxHniWZCk6/icpMPmwKL6FtPoJBguHtw==</latexit>
sha1_base64="6eLMJmF33J4+WTuVCsbLWkeP7Xs=">AAACXHichVG7SgNBFD1ZXzE+Ei0kYCOGSGzkro1iJdhY5mESIUrYXSfJ4r7Y3QQ1+AMWtgpWCiLiZ9j4AxZ+QrCMYGPh3U1ANKh3mJkzZ+bcOTNXdQzd84leItLQ8MjoWHQ8NjE5NR1PzEyWPLvpaqKo2Ybt7qqKJwzdEkVf9w2x67hCMVVDlNXDrWC/3BKup9vWjn/siH1TqVt6TdcUP6BqmaPlaiJFKxTGwiCQ+yCFfmTtxB32cAAbGpowIWDBZ2xAgcetAhkEh7l9tJlzGenhvsApYqxt8inBJxRmD3ms86rSZy1eBzm9UK3xLQZ3l5ULSNMz3VOXnuiBOvTxa652mCPwcsyz2tMKpxo/Sxbe/1WZPPtofKn+9OyjhvXQq87enZAJXqH19K2Ti25hI59uL9ENvbL/a3qhR36B1XrTbnMif4UYF0D++d2DoLS6IjPOEaKYxyIy/M1r2MQ2sijydQ2c4wKXkY4Ul+Z6pZIi/ZrN4ltIyU/YU4n6</latexit>
sha1_base64="+z3hxTKa+tlEihVqdkmzg9QvjCI=">AAACZ3ichVHLSsNAFD2Nr1ofrQpScKMWRTflVgTFleDGZavWFmqRJE7b0DQJSVqsxR9w4VbBlYKI+Blu/AEX/YTisoIbF96kAVFR7zAzZ87cc+fMjGLpmuMStUNSX//A4FB4ODIyOjYejU1M7jtm3VZFVjV1084rsiN0zRBZV3N1kbdsIdcUXeSU6pa3n2sI29FMY89tWqJYk8uGVtJU2fWo0tLx8mEsQUnyY/YnSAUggSDSZuwOBziCCRV11CBgwGWsQ4bDrYAUCBZzRbSYsxlp/r7AKSKsrXOW4AyZ2SqPZV4VAtbgtVfT8dUqn6Jzt1k5iwV6pnvq0hM9UIfef63V8mt4Xpo8Kz2tsA6jZ/Hdt39VNZ5dVD5Vf3p2UcK671Vj75bPeLdQe/rGyUV3d2NnobVIN/TC/q+pTY98A6Pxqt5mxM4VIvwBqe/P/RPsryRTjDOU2FwNviKMGcxjid97DZvYRhpZPreCc1zgMtSRotK0FO+lSqFAM4UvIc19ANEWirs=</latexit><latexit sha1_base64="JdYEc2e/Wb2b5VTnyuHjrlS9J1M=">AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KRAqKq4Ibl33YB9RSkjitoWkSkrRQiz+gW8WFKwUR8TPc+AMu+gOCuKzgxoW3aUC0qHeYmTNn7rlzZkY2NdV2GOv5hLHxickp/3RgJjg7FwrPLxRso2UpPK8YmmGVZMnmmqrzvKM6Gi+ZFpeassaLcmNnsF9sc8tWDX3P6Zi80pTqulpTFckhKtOphmMswdyIjgLRAzF4kTbCt9jHAQwoaKEJDh0OYQ0SbGpliGAwiaugS5xFSHX3OY4RIG2LsjhlSMQ2aKzTquyxOq0HNW1XrdApGnWLlFHE2RO7Y332yO7ZC/v4tVbXrTHw0qFZHmq5WQ2dLOXe/1U1aXZw+KX607ODGrZcryp5N11mcAtlqG8fXfRz29l4d41ds1fyf8V67IFuoLfflJsMz14iQB8g/nzuUVDYSIiEM8lYKul9hR/LWMU6vfcmUthFGnk6l+MUZzj3PQtBISIsDlMFn6eJ4FsIK5/0GInr</latexit>

Output y

活性化関数

全結合層

活性化関数

全結合層

活性化関数

全結合層

<latexit sha1_base64="Qj4U1j/Mx929t2ieD/QbKvBgPOI=">AAACY3ichVHLSsNAFD2Nr1ofrdWFIEKxVFyVWxF8rIpuXPZhraBSkjhqaJqEJC3W4g/oVnHhSkFE/Aw3/oCL/oAgLiu4ceFtGhAV9Q4zc+bMPXfOzCiWrjkuUTMgdXX39PYF+0MDg0PD4chIdN0xq7YqCqqpm/aGIjtC1wxRcDVXFxuWLeSKoouiUl5p7xdrwnY001hz65bYrsh7hrarqbLLVPagFIlTkryI/QQpH8ThR8aM3GALOzChoooKBAy4jHXIcLhtIgWCxdw2GszZjDRvX+AIIdZWOUtwhsxsmcc9Xm36rMHrdk3HU6t8is7dZmUMCXqkW2rRA93RM73/Wqvh1Wh7qfOsdLTCKoWPx/Nv/6oqPLvY/1T96dnFLhY8rxp7tzymfQu1o68dnrfyS7lEY5qu6IX9X1KT7vkGRu1Vvc6K3AVC/AGp78/9ExRmk4tJys7F08v+TwQxgSnM8HPPI41VZFDgYwVOcIqzwJM0KEWlsU6qFPA1o/gS0uQHZdSJyw==</latexit><latexit

活性化関数

全結合層

x

…

事前学習とファインチューニング
• 事前学習; Pretraining• やりたいタスクと関係するけど、違うタ
スクで学習• 例) 画像認識タスクで事前学習、画像のキャプショニングで利用• 大量のデータを使える可能性がある！？

活性化関数

活性化関数

全結合層

全結合層
活性化関数

• ファインチューニング; Fine-tuning

…

全結合層

• 事前学習したニューラルネットワークを、
やりたいタスクで再学習

活性化関数• 付け替えた先だけ学習してもOK• 付け替えた先と同時に、事前学習した部
分を再学習してもOK

全結合層• やりたいタスクで使えるデータが少ない
場合でもうまくいく！？

活性化関数
全結合層

ニューラルネットワークの訓練ニューラルネットワークの構成要素(再掲)• 全結合層: h = W x + b

教師信号 tロス関数<latexit sha1_base64="MMgHjpXlkeOHsPydpO4ACxuWX/s=">AAACZHichVHLSsNAFD2Nr1ofrRZBEEQsiiu5kYLiquDGZR9WBRVJ4qihaRKSaaEWf0C3igtXCiLiZ7jxB1z0BwRxqeDGhTdpQFTUO8zMmTP33Dkzo7uW6UuiVkzp6Ozq7on3Jvr6BwaTqaHhVd+peYYoG47leOu65gvLtEVZmtIS664ntKpuiTW9shTsr9WF55uOvSIbrtiqanu2uWsammSqILdTGZqlMCZ+AjUCGUSRd1LX2MQOHBiooQoBG5KxBQ0+tw2oILjMbaHJnMfIDPcFDpFgbY2zBGdozFZ43OPVRsTavA5q+qHa4FMs7h4rJzBFD3RDL3RPt/RE77/WaoY1Ai8NnvW2VrjbyaPR0tu/qirPEvufqj89S+xiIfRqsnc3ZIJbGG19/eDspbRYnGpO0yU9s/8LatEd38CuvxpXBVE8R4I/QP3+3D/B6tysyriQzeSy0VfEMYZJzPB7zyOHZeRR5nMFjnGC09ij0q+klZF2qhKLNGl8CWX8A+oYieY=</latexit><latexit<latexit sha1_base64="kkWpxS4h7ewksvkQbSyWsYp6kWM=">AAACbXichVHLSgMxFD0d3/VVFUFQZLD4AKGkIiiCUHDjsg/7wAcyM0YNnc4MM2mxFn/AteBCFBRExM9w4w+48BPEhQsFNy68nQ6IinpDkpOTe25OEt0xhScZewgpTc0trW3tHeHOru6e3khff86zy67Bs4Zt2m5B1zxuCotnpZAmLzgu10q6yfN6cam+n69w1xO2tSKrDt8oaTuW2BaGJola3VUX1fyeOq3qm5EoizE/1J8gHoAogkjakSusYws2DJRRAocFSdiEBo/aGuJgcIjbQI04l5Dw9zkOECZtmbI4ZWjEFmncodVawFq0rtf0fLVBp5jUXVKqGGf37Jq9sDt2wx7Z+6+1an6NupcqzXpDy53N3sOhzNu/qhLNErufqj89S2xj3vcqyLvjM/VbGA19Zf/4JbOQHq9NsAv2RP7P2QO7pRtYlVfjMsXTJwjTB8S/P/dPkJuJxQmnZqOJ+eAr2jGMMUzRe88hgWUkkaVzLRzhFGehZ2VQGVFGG6lKKNAM4Esokx9hyoxR</latexit>• 画像の場合は畳み込み層

Output y<latexit sha1_base64="JdYEc2e/Wb2b5VTnyuHjrlS9J1M=">AAACZHichVHLSsNAFD2Nr1qrrRZBEKRYKq7KRAqKq4Ibl33YB9RSkjitoWkSkrRQiz+gW8WFKwUR8TPc+AMu+gOCuKzgxoW3aUC0qHeYmTNn7rlzZkY2NdV2GOv5hLHxickp/3RgJjg7FwrPLxRso2UpPK8YmmGVZMnmmqrzvKM6Gi+ZFpeassaLcmNnsF9sc8tWDX3P6Zi80pTqulpTFckhKtOphmMswdyIjgLRAzF4kTbCt9jHAQwoaKEJDh0OYQ0SbGpliGAwiaugS5xFSHX3OY4RIG2LsjhlSMQ2aKzTquyxOq0HNW1XrdApGnWLlFHE2RO7Y332yO7ZC/v4tVbXrTHw0qFZHmq5WQ2dLOXe/1U1aXZw+KX607ODGrZcryp5N11mcAtlqG8fXfRz29l4d41ds1fyf8V67IFuoLfflJsMz14iQB8g/nzuUVDYSIiEM8lYKul9hR/LWMU6vfcmUthFGnk6l+MUZzj3PQtBISIsDlMFn6eJ4FsIK5/0GInr</latexit>• 活性化関数(非線形性): y = (h)活性化関数• Hyperbolic tangent (h) = tanh(h)• ReLU (h) = max(0, h)• etc.

全結合層<latexit sha1_base64="LZuXmJY/Hvr0FzepabxkSatbrQI=">AAACcHichVHLSgMxFD0dX7W+qm4EF44WRV1IKoJFEAQ3LvuwtlCLzIyxBuflzLRQiz/gD3Thxgoi4me48Qdc9BPEnQpuXHhnOiAq6g1JTk7uuTlJVFsXrsdYOyJ1dff09kX7YwODQ8Mj8dGxHdeqOhrPa5ZuOUVVcbkuTJ73hKfzou1wxVB1XlCPNv39Qo07rrDMba9u87KhVExxIDTFI6pcl9flXVdUDGX+cGEvnmBLLAj5J0iGIIEw0lb8GrvYhwUNVRjgMOER1qHApVZCEgw2cWU0iHMIiWCf4xQx0lYpi1OGQuwRjRValULWpLVf0w3UGp2iU3dIKWOWPbAb9sLu2S17ZO+/1moENXwvdZrVjpbbeyNnE7m3f1UGzR4OP1V/evZwgFTgVZB3O2D8W2gdfe2k+ZJby8425tgleyL/LdZmd3QDs/aqXWV49hwx+oDk9+f+CXaWl5KEMyuJjVT4FVFMYgbz9N6r2MAW0sjTucdo4gKtyLM0IU1J051UKRJqxvElpMUPFYGOCg==</latexit><latexit sha1_base64="+aYzdaK2XVCSpQ7+k4ZgAIOnSik=">AAACeXichVHLSgMxFD0d3/U1PhaCm2qpVBeSqqAIguDGpVZrC1bKzJi2wXkxkxZs8Qf8ARduVBBRP8ONP+DCTxCXCi504e10QLSoNyQ5Obnn5iTRXVP4krHHiNLW3tHZ1d0T7e3rHxhUh4Z3fKfiGTxjOKbj5XTN56aweUYKafKc63HN0k2e1Q/WGvvZKvd84djb8tDle5ZWskVRGJokqqCO5n1RsrRkeTq2EstLzS4TLKhxNsuCiLWCVAjiCGPDUa+Qxz4cGKjAAocNSdiEBp/aLlJgcInbQ504j5AI9jmOECVthbI4ZWjEHtBYotVuyNq0btT0A7VBp5jUPVLGkGAP7Jq9sHt2y57Yx6+16kGNhpdDmvWmlruFweOxrbd/VRbNEuUv1Z+eJYpYCrwK8u4GTOMWRlNfrZ28bC2nE/UpdsGeyf85e2R3dAO7+mpcbvL0KaL0Aamfz90KduZmU4Q3F+KrC+FXdGMck0jSey9iFevYQIbOreEM17iJvCsTSlKZaaYqkVAzgm+hzH8CV6+QxA==</latexit>• Others:• バッチ正規化• プーリング• etc.

活性化関数<latexit sha1_base64="pNbcj3NpK98tXpPOs/xTlVli50w=">AAACZ3ichVG7SgNBFD1ZXzE+EhUkYKMGRZtwI4JiFbCxzMPEQBTZXSe6ZF/sboIa/AELWwUrBRHxM2z8AYt8QrBUsLHw7mZBVNQ7zMyZM/fcOTOj2LrmekTtiNTT29c/EB2MDQ2PjMYTY+Nl12o4qiiplm45FUV2ha6ZouRpni4qtiNkQ9HFllJf9/e3msJxNcvc9I5ssWPI+6ZW01TZ86nawuHibiJFaQpi+ifIhCCFMHJW4hbb2IMFFQ0YEDDhMdYhw+VWRQYEm7kdtJhzGGnBvsAJYqxtcJbgDJnZOo/7vKqGrMlrv6YbqFU+RefusHIac/REd/RCj3RPHXr/tVYrqOF7OeJZ6WqFvRs/TRbf/lUZPHs4+FT96dlDDauBV4292wHj30Lt6pvH5y/FtcJca56u6Zn9X1GbHvgGZvNVvcmLwiVi/AGZ78/9E5SX0hnG+eVUdjn8iiimMIsFfu8VZLGBHEp87gHOcI6LSEeKS5NSspsqRULNBL6ENPMB0laKvw==</latexit>
sha1_base64="1tqRKu4qqxHI8AQAeehg2wQidig=">AAACVXichVG7TgJBFD2sL1xRIDYmNkSCsSIXG42ViY0lD3kkSMjuMuCGZXezu5Ag8QdoLSysNDHG+Bk2/oAFvY2xxMTGwstCYpSodzIzZ87MuXPmjmobuusRDQLSzOzc/EJwUV4Kycsr4Uio4FptRxN5zTIsp6QqrjB0U+Q93TNEyXaE0lINUVSbB6P9Ykc4rm6ZR17XFpWW0jD1uq4pHlPpaiROSfIjNg1SExDHJKzILY5RgwUNbbQgYMJjbECBy62MFAg2cxX0mHMY6f6+wBlk1rb5lOATCrNNHhu8Kk9Yk9ejnK6v1vgWg7vDyhgS9ER3NKRHuqcX+vg1V8/PMfLS5Vkda4VdDffXcu//qlo8ezj5Uv3p2UMdu75Xnb3bPjN6hTbWd04vhrm9bKK3Sdf0yv6vaEAP/AKz86bdZET2EjLXP/Wz2tOgsJ1MMc4QgljHBra4zDvYxyHSyPN1NfRxHniWZCk6/icpMPmwKL6FtPoJBguHtw==</latexit>
sha1_base64="6eLMJmF33J4+WTuVCsbLWkeP7Xs=">AAACXHichVG7SgNBFD1ZXzE+Ei0kYCOGSGzkro1iJdhY5mESIUrYXSfJ4r7Y3QQ1+AMWtgpWCiLiZ9j4AxZ+QrCMYGPh3U1ANKh3mJkzZ+bcOTNXdQzd84leItLQ8MjoWHQ8NjE5NR1PzEyWPLvpaqKo2Ybt7qqKJwzdEkVf9w2x67hCMVVDlNXDrWC/3BKup9vWjn/siH1TqVt6TdcUP6BqmaPlaiJFKxTGwiCQ+yCFfmTtxB32cAAbGpowIWDBZ2xAgcetAhkEh7l9tJlzGenhvsApYqxt8inBJxRmD3ms86rSZy1eBzm9UK3xLQZ3l5ULSNMz3VOXnuiBOvTxa652mCPwcsyz2tMKpxo/Sxbe/1WZPPtofKn+9OyjhvXQq87enZAJXqH19K2Ti25hI59uL9ENvbL/a3qhR36B1XrTbnMif4UYF0D++d2DoLS6IjPOEaKYxyIy/M1r2MQ2sijydQ2c4wKXkY4Ul+Z6pZIi/ZrN4ltIyU/YU4n6</latexit>
sha1_base64="+z3hxTKa+tlEihVqdkmzg9QvjCI=">AAACZ3ichVHLSsNAFD2Nr1ofrQpScKMWRTflVgTFleDGZavWFmqRJE7b0DQJSVqsxR9w4VbBlYKI+Blu/AEX/YTisoIbF96kAVFR7zAzZ87cc+fMjGLpmuMStUNSX//A4FB4ODIyOjYejU1M7jtm3VZFVjV1084rsiN0zRBZV3N1kbdsIdcUXeSU6pa3n2sI29FMY89tWqJYk8uGVtJU2fWo0tLx8mEsQUnyY/YnSAUggSDSZuwOBziCCRV11CBgwGWsQ4bDrYAUCBZzRbSYsxlp/r7AKSKsrXOW4AyZ2SqPZV4VAtbgtVfT8dUqn6Jzt1k5iwV6pnvq0hM9UIfef63V8mt4Xpo8Kz2tsA6jZ/Hdt39VNZ5dVD5Vf3p2UcK671Vj75bPeLdQe/rGyUV3d2NnobVIN/TC/q+pTY98A6Pxqt5mxM4VIvwBqe/P/RPsryRTjDOU2FwNviKMGcxjid97DZvYRhpZPreCc1zgMtSRotK0FO+lSqFAM4UvIc19ANEWirs=</latexit><latexit sha1_base64="ZWQHyVxiGrfol6tYU9WAWn9TgfA=">AAACanichVFNLwNBGH66vqq+qhxILw0qlUgzi4Q4NXFxcGhRlSCyu6ZsbHc3u9MGjT/g5iTRE4mI+Bku/oBDf4JwI3Fx8O60iSB4JzPzzDPv884zM7prmb5grB5SWlrb2jvCnZGu7p7evmh/bM13yp7B84ZjOd66rvncMm2eF6aw+Lrrca2kW7yg7y8E+4UK93zTsVfFocu3StqubRZNQxNEFZZSB5MJMbEdHWVpJiPxE6hNMJoZLMrIOtFrbGIHDgyUUQKHDUHYggaf2gZUMLjEbaFKnEfIlPscx4iQtkxZnDI0Yvdp3KXVRpO1aR3U9KXaoFMs6h4pE0iyB3bDXtg9u2WP7P3XWlVZI/BySLPe0HJ3u+9kaOXtX1WJZoG9T9WfngWKmJNeTfLuSia4hdHQV47OXlbml5PVcXbJnsj/BauzO7qBXXk1rnJ8uYYIfYD6/bl/grWptDqdnsrRT8ygEWHEMYIUvfcsMlhEFnnp7hTnqIWelZgyrMQbqUqoqRnAl1DGPgCLG447</latexit>f (x)• ロス関数: L(x, t)…

<latexit sha1_base64="OGOx8LVL+sJcFmit/3vNVSDAGyY=">AAACeXichVHLSgMxFD0d3/VVHwvBTbUoVURutaAIguDGpa+qYKXMjGkbnBcz02It/oA/4MKNCiLqZ7jxB1z4CeJSwYUuvJ0OiIp6Q5KTk3tuThLNMaTnEz1ElIbGpuaW1rZoe0dnV3esp3fDs0uuLjK6bdjulqZ6wpCWyPjSN8SW4wrV1Ayxqe0t1vY3y8L1pG2t+xVH7JhqwZJ5qas+U7lYf9aTBVNNFsfms6a6n6SJeHEsF0vQJAUR/wlSIUggjGU7doksdmFDRwkmBCz4jA2o8LhtIwWCw9wOqsy5jGSwL3CIKGtLnCU4Q2V2j8cCr7ZD1uJ1raYXqHU+xeDusjKOEbqnK3qmO7qhR3r/tVY1qFHzUuFZq2uFk+s+Glh7/Vdl8uyj+Kn607OPPGYDr5K9OwFTu4Ve15cPjp/X5lZHqqN0Tk/s/4we6JZvYJVf9IsVsXqCKH9A6vtz/wQbU5MpxivpxEI6/IpWDGIYSX7vGSxgCcvI8LkHOMUVriNvypCSVMbrqUok1PThSyjTHweykJs=</latexit>

全結合層
活性化関数
全結合層
Input x<latexit sha1_base64="MQ2Wn/quWMCC8abSWNLYORWSBvk=">AAACZHichVFNSwJBGH7cvswsLQmCICQxOsUYQtFJ6NLRj/wAE9ndRlvcL3ZXyaQ/UNeiQ6eCiOhndOkPdPAPBNHRoEuHXteFKKneYWaeeeZ93nlmRjJVxXYY6/qEkdGx8Qn/ZGAqOD0TCs/OFWyjack8LxuqYZUk0eaqovO8ozgqL5kWFzVJ5UWpsd3fL7a4ZSuGvuu0TV7RxLqu1BRZdIjKHFbDMbbG3IgOg4QHYvAibYRvsYd9GJDRhAYOHQ5hFSJsamUkwGASV0GHOIuQ4u5zHCNA2iZlccoQiW3QWKdV2WN1Wvdr2q5aplNU6hYpo4izJ3bHeuyR3bMX9vFrrY5bo++lTbM00HKzGjpZyL3/q9JodnDwpfrTs4MaNl2vCnk3XaZ/C3mgbx1d9HJb2XhnhV2zV/J/xbrsgW6gt97kmwzPXiJAH5D4+dzDoLC+liCcScZSSe8r/FjEMlbpvTeQwg7SyNO5HKc4w7nvWQgKEWF+kCr4PE0E30JY+gTyGInq</latexit><latexitニューラルネット、どうやって学習するの?
• 確率的勾配降下法; Stochastic gradient descent• 勾配降下法（aka 最急降下法）の一種• 誤差逆伝播法; Back-propagation• 連鎖律（chain rule; 合成関数の微分）

勾配降下法• (NNの)パラメータ ✓ をどんどん更新して関数を最小化<latexit sha1_base64="CPMMUXlBzWIqoiwan4WNoM0nWpk=">AAACaXichVG7SgNBFD1ZXzG+ojZBGzEoVuFGBcVGwcYyPhIDiYTddRJXN7vL7iQQgz9gZSfRSkFE/AJrG3/Awk8QSwUbC+9uAqJBvcPMnDlzz50zM5pjGp4kegopHZ1d3T3h3khf/8DgUHR4JOPZFVcXad02bTerqZ4wDUukpSFNkXVcoZY1U2xrB6v+/nZVuJ5hW1uy5oidslqyjKKhq5KpTF7uCakWonFKUBAT7SDZAvHlu4YfZyk7eo08dmFDRwVlCFiQjE2o8LjlkATBYW4HdeZcRkawL3CECGsrnCU4Q2X2gMcSr3It1uK1X9ML1DqfYnJ3WTmBKXqkG3qlB7qlZ/r4tVY9qOF7qfGsNbXCKQwdxzbf/1WVeZbY+1L96VmiiMXAq8HenYDxb6E39dXD09fNpY2p+jRd0gv7v6AnuucbWNU3/WpdbJwjwh+Q/Pnc7SAzm0jOJWbXKb4yj2aEMY5JzPB7L2AFa0ghzefu4wQNnIVelGElpow1U5VQSzOKb6HEPwH+spCl</latexit>✓<latexit sha1_base64="PkpAicVbXB21YaObyPOZ9EIoz7w=">AAACpHichVFBSxtBGH1uq8ZoTaoXoZdFUYRimGihxYuClx48qDFRcDXMrpNkcLK77E4CNuQP5AfowVNbpIg/w4vQqx78AR7Eo4IXD37ZDYiK+i0787433/vmzYztKxlqxi66jA8fu3t6E33J/oFPg6n056FC6NUCR+QdT3nBus1DoaQr8lpqJdb9QPCqrcSavbPQXl+riyCUnruqd32xWeVlV5akwzVRxfScpStCc9MqCx2anWTKtLjyK8SWAu40LJ8HWnJlLjYfcVzaLKbHWIZFYb4E2Q4Ym//6t7B32ZpZ8tL/YGEbHhzUUIWAC01YgSOkbwNZMPjEbaJBXEBIRusCTSRJW6MqQRWc2B0ay5RtdFiX8nbPMFI7tIuiPyCliXF2zo7YDTtlx+yK3b/aqxH1aHvZpdmOtcIvplojubt3VVWaNSqPqjc9a5TwI/IqybsfMe1TOLG+/mv/Jje7Mt6YYH/YNfn/zS7YCZ3Ard86h8ti5QBJeoDs8+t+CQrTmexMZnqZXuIb4kjgC0YxSff9HfP4iSXkad9D/McZzo0JY9HIGfm41OjqaIbxJIytByBepkY=</latexit>

✓

@L
↵
@✓

✓ は W や b のすべての値↵ は学習率<latexit sha1_base64="CPMMUXlBzWIqoiwan4WNoM0nWpk=">AAACaXichVG7SgNBFD1ZXzG+ojZBGzEoVuFGBcVGwcYyPhIDiYTddRJXN7vL7iQQgz9gZSfRSkFE/AJrG3/Awk8QSwUbC+9uAqJBvcPMnDlzz50zM5pjGp4kegopHZ1d3T3h3khf/8DgUHR4JOPZFVcXad02bTerqZ4wDUukpSFNkXVcoZY1U2xrB6v+/nZVuJ5hW1uy5oidslqyjKKhq5KpTF7uCakWonFKUBAT7SDZAvHlu4YfZyk7eo08dmFDRwVlCFiQjE2o8LjlkATBYW4HdeZcRkawL3CECGsrnCU4Q2X2gMcSr3It1uK1X9ML1DqfYnJ3WTmBKXqkG3qlB7qlZ/r4tVY9qOF7qfGsNbXCKQwdxzbf/1WVeZbY+1L96VmiiMXAq8HenYDxb6E39dXD09fNpY2p+jRd0gv7v6AnuucbWNU3/WpdbJwjwh+Q/Pnc7SAzm0jOJWbXKb4yj2aEMY5JzPB7L2AFa0ghzefu4wQNnIVelGElpow1U5VQSzOKb6HEPwH+spCl</latexit><latexit sha1_base64="JDpNcH5uLVTLupPgdE5OEOCtsOM=">AAACZHichVHLSsNAFD2N7/poVQRBkGJRXJUbFRQ3Fty4bNXaQhVJ4lRD0yQkaUGLP6BbRdSVgoj4Ba7d+AMu/AFBXFZw48KbtCBa1DvMzJkz99w5M6Pahu56RE8hqaW1rb2jsyvc3dPbF4n2D6y5VtnRREazDMvJqYorDN0UGU/3DJGzHaGUVENk1eKiv5+tCMfVLXPV27XFRknZNvWCrikeU+nsZjROCQoi1gzkBogv3J36cZayotdYxxYsaCijBAETHmMDClxuecgg2MxtoMqcw0gP9gX2EWZtmbMEZyjMFnnc5lW+wZq89mu6gVrjUwzuDitjGKdHuqEaPdAtvdDHr7WqQQ3fyy7Pal0r7M3IwfDK+7+qEs8edr5Uf3r2UMBc4FVn73bA+LfQ6vrK3kltZX55vDpBl/TK/i/oie75BmblTbtKi+VzhPkD5J/P3QzWphLydGIqTfHkDOrRiRGMYZLfexZJLCGFDJ8rcIgjHIeepR5pUBqqp0qhhmYQ30Ia/QQHcI5Y</latexit><latexit sha1_base64="UMj9xksbZGAk668cvE2cJ6/kBuU=">AAACZHichVHLSsNAFD2Nr1qrrYogCFIsiqtyo4LiRsGNS6v2AbVIEkcNpklI0kIt/oBuFVFXCiLiF7h24w+48AcEcVnBjQtv0oKoqHeYmTNn7rlzZka1Dd31iB5DUktrW3tHuDPSFe3uicV7+7KuVXY0kdEsw3LyquIKQzdFxtM9Q+RtRygl1RA5dWfB389VhOPqlrnqVW1RLClbpr6pa4rHVFpdjycpRUEkfgK5CZJztyd+nC5Z8SusYQMWNJRRgoAJj7EBBS63AmQQbOaKqDHnMNKDfYE9RFhb5izBGQqzOzxu8arQZE1e+zXdQK3xKQZ3h5UJjNIDXVOd7umGnun911q1oIbvpcqz2tAKez22P7jy9q+qxLOH7U/Vn549bGIm8Kqzdztg/FtoDX1l97i+Mrs8WhujC3ph/+f0SHd8A7Pyql2mxfIZIvwB8vfn/gmyEyl5MjWRpuT8FBoRxhBGMM7vPY15LGIJGT5X4ACHOAo9SVGpXxpopEqhpqYfX0Ia/gAdcI5j</latexit><latexit sha1_base64="anJh4W6pvlf+ZSk8kB5iSTLO6Cs=">AAACaXichVHLSsNAFD2N7/pqdSO6EYPiqtxWQXFjwY1LX62FVkoSpzWaJiFJC1r8AVfupHWlICJ+gWs3/oALP0FcKrhx4U1SEC3qHWbmzJl77pyZUW1Ddz2ip4jU0dnV3dPbF+0fGBwajsVHsq5VdTSR0SzDcnKq4gpDN0XG0z1D5GxHKBXVENvqwYq/v10Tjqtb5pZ3aIudilI29ZKuKR5T2YJi2HtKMSZTgoKYbAfJFpCX7xp+NNes2DUK2IUFDVVUIGDCY2xAgcstjyQINnM7qDPnMNKDfYFjRFlb5SzBGQqzBzyWeZVvsSav/ZpuoNb4FIO7w8pJTNMj3dArPdAtPdPHr7XqQQ3fyyHPaqgVdnH4ZGzz/V9VhWcPe1+qPz17KGEx8Kqzdztg/Ftoob52dPa6ubQxXZ+hS3ph/xf0RPd8A7P2pl2ti41zRPkDkj+fux1kU4nkXCK1TnJ6HmH0YgJTmOX3XkAaq1hDhs/dxykaaEZepLg0Jo2HqVKkpRnFt5DkT958kJU=</latexit>• 確率的勾配降下法は、勾配降下法の一種:• データセットをランダムに並び替え• データセットを小さい塊(mini-batch)に分割
into B mini-batches• 勾配降下法をmini-batchに適用

（確率的）勾配降下法のいろいろな亜種• (Vanilla) stochastic gradient descent algorithm• Stochastic gradient descent with momentum• AdaGrad• RMSProp• Adamシンプルなニューラルネットワークの例⇥ = {W1 , b1 , W2 , b2 }<latexit sha1_base64="oWzyno4B/d0+pz0l7GSSUby2dBM=">AAAChXichVFNLwNBGH66vqq+iovEZaMhDjSz1SASIXFxbEtVorLZXYON7e5md9qkGlcHf8DBiUREuPIHXPwBBz9BHElcHLy73UQQvJOZeeaZ93nnmRndtUxfMPYYk1pa29o74p2Jru6e3r5k/8Ca71Q9gxcNx3K8dV3zuWXavChMYfF11+NaRbd4Sd9bCvZLNe75pmOvirrLNyvajm1um4YmiFKTcnl1lwtNnpfLjZKqTMh6MJTUTIAy5YOEmkyxNAtD/gmUCKQQRc5JXqCMLTgwUEUFHDYEYQsafGobUMDgEreJBnEeITPc5zhAgrRVyuKUoRG7R+MOrTYi1qZ1UNMP1QadYlH3SCljlD2wS/bC7tkVe2Lvv9ZqhDUCL3Wa9aaWu2rf0dDK27+qCs0Cu5+qPz0LbGM29GqSdzdkglsYTX1t//hlZa4w2hhjZ+yZ/J+yR3ZHN7Brr8Z5nhdOEHyA8v25f4K1TFqZSmfy2dRiNvqKOIYxgnF67xksYhk5FOncQ1zjBrdShzQpZaXpZqoUizSD+BLSwgfJEpL/</latexit>

Supervision t<latexit sha1_base64="MMgHjpXlkeOHsPydpO4ACxuWX/s=">AAACZHichVHLSsNAFD2Nr1ofrRZBEEQsiiu5kYLiquDGZR9WBRVJ4qihaRKSaaEWf0C3igtXCiLiZ7jxB1z0BwRxqeDGhTdpQFTUO8zMmTP33Dkzo7uW6UuiVkzp6Ozq7on3Jvr6BwaTqaHhVd+peYYoG47leOu65gvLtEVZmtIS664ntKpuiTW9shTsr9WF55uOvSIbrtiqanu2uWsammSqILdTGZqlMCZ+AjUCGUSRd1LX2MQOHBiooQoBG5KxBQ0+tw2oILjMbaHJnMfIDPcFDpFgbY2zBGdozFZ43OPVRsTavA5q+qHa4FMs7h4rJzBFD3RDL3RPt/RE77/WaoY1Ai8NnvW2VrjbyaPR0tu/qirPEvufqj89S+xiIfRqsnc3ZIJbGG19/eDspbRYnGpO0yU9s/8LatEd38CuvxpXBVE8R4I/QP3+3D/B6tysyriQzeSy0VfEMYZJzPB7zyOHZeRR5nMFjnGC09ij0q+klZF2qhKLNGl8CWX8A+oYieY=</latexit><latexit

Loss= XE(y, t|⇥)0hL(x,= gt|⇥)
1 (x|W1 , b1 ) = W1 x + b1
<latexit sha1_base64="2MvwaCjB2IRiiRGaTJcopLiTMsg=">AAACjXichVG7SgNBFD2urxgfiY9CsFkUJYKEiQ8UURFEtLDwFQ0YCbvrJC7Z7C67k2CM/oCFrYWVgojY2mpj4w9Y+AliqWBj4d1NwBfqXXbmzLn33Dkzo9qG7grGHqqk6prauvpAQ7CxqbklFG5tW3OtvKPxuGYZlpNQFZcbusnjQhcGT9gOV3KqwdfV7IyXXy9wx9Utc1UUbb6ZUzKmntY1RRCVCkcWIjsDsthLrm5zofTLk3JS8B1RSszuR4qfEsFUuIdFmR/yTxCrgJ7pjrQfi1b4HElswYKGPHLgMCEIG1Dg0reBGBhs4jZRIs4hpPt5jn0ESZunKk4VCrFZGjO02qiwJq29nq6v1mgXg36HlDJ62T27YM/sjl2yR/b2a6+S38PzUqRZLWu5nQoddK68/qvK0Syw/aH607NAGmO+V5282z7jnUIr6wu7R88r48u9pT52yp7I/wl7YLd0ArPwop0t8eVjeA8Q+37dP8HaYDQ2FB1copcYRjkC6EI3InTfo5jGPBYRp30PcYVr3EghaUSakKbKpVJVRdOOLyHNvQOIG5oa</latexit>

h = (h0 ) = tanh(h0 )Linearity<latexit sha1_base64="ftxYxNo3LE/yKA4i0yfD5P1kQWE=">AAADEHichVFLa9RQFD6JrxofnepGcHNxaJvBdriJhYpQGHHjwkVf0yk0JdzEO5PQvMi9M2Sc9g+4dOPClYKI9A+4E3Ej7l30J4jLCm5ceG4SkFrUE5L7nXO+78u593pZFApJ6ZGmnzl77vyFqYvGpctXrk43Zq5tiXSY+7zrp1Gab3tM8ChMeFeGMuLbWc5Z7EW85+09UP3eiOciTJNNOc74bswGSdgPfSax5DYOg3kyt0IGrmUW+6TnWgvEc60WWVGYFOS2SonjGIGiOSIcxMwM5hXBkSwJEBst1R9XNrYZKBtb2diVjU2C0sZWtEdmsUDkvrMZcMlapafkhZyItC9jViz6eSrEIk9knmbjgxNkw200aZuWQU4DqwZNqGM1bbwBBx5DCj4MIQYOCUjEETAQ+OyABRQyrO3CBGs5orDsczgAA7VDZHFkMKzu4XeA2U5dTTBXnqJU+/iXCN8clQRm6Rf6lh7TT/SQfqU//+o1KT3ULGNcvUrLM3f66Y2NH/9VxbhKCH6r/jmzhD7cLWcNcfasrKhd+JV+9OT58ca99dnJHH1Fv+H8L+kR/Yg7SEbf/ddrfP0FqAuw/jzu02DLblt32vbaUrOzVF/FFNyEW2DieS9DBx7CKnTB12a0Za2j3def6e/09/qHiqprteY6nAj98y+Ygbz6</latexit>

y = g2 (h|W2 , b2 ) = W2 h + b2

L(x, t|⇥)h0 = gsoftmax-cross-entropy(x,1 (x|W1 , b1 ) = W1 x + b1 t|⇥)Activationh = (h0 ) = tanh(h0 )y = g2 (h|W2 , b2 ) = W2 h + b2

LinearityL(x, t|⇥)h0 == softmax-cross-entropy(x,g1 (x|W1 , b1 ) = W1 x + b1 t|⇥)<latexit sha1_base64="ftxYxNo3LE/yKA4i0yfD5P1kQWE=">AAADEHichVFLa9RQFD6JrxofnepGcHNxaJvBdriJhYpQGHHjwkVf0yk0JdzEO5PQvMi9M2Sc9g+4dOPClYKI9A+4E3Ej7l30J4jLCm5ceG4SkFrUE5L7nXO+78u593pZFApJ6ZGmnzl77vyFqYvGpctXrk43Zq5tiXSY+7zrp1Gab3tM8ChMeFeGMuLbWc5Z7EW85+09UP3eiOciTJNNOc74bswGSdgPfSax5DYOg3kyt0IGrmUW+6TnWgvEc60WWVGYFOS2SonjGIGiOSIcxMwM5hXBkSwJEBst1R9XNrYZKBtb2diVjU2C0sZWtEdmsUDkvrMZcMlapafkhZyItC9jViz6eSrEIk9knmbjgxNkw200aZuWQU4DqwZNqGM1bbwBBx5DCj4MIQYOCUjEETAQ+OyABRQyrO3CBGs5orDsczgAA7VDZHFkMKzu4XeA2U5dTTBXnqJU+/iXCN8clQRm6Rf6lh7TT/SQfqU//+o1KT3ULGNcvUrLM3f66Y2NH/9VxbhKCH6r/jmzhD7cLWcNcfasrKhd+JV+9OT58ca99dnJHH1Fv+H8L+kR/Yg7SEbf/ddrfP0FqAuw/jzu02DLblt32vbaUrOzVF/FFNyEW2DieS9DBx7CKnTB12a0Za2j3def6e/09/qHiqprteY6nAj98y+Ygbz6</latexit>

h = (h0 ) = tanh(h0 )input xy = g2 (h|W2 , b2 ) = W2 h + b2

<latexit sha1_base64="MQ2Wn/quWMCC8abSWNLYORWSBvk=">AAACZHichVFNSwJBGH7cvswsLQmCICQxOsUYQtFJ6NLRj/wAE9ndRlvcL3ZXyaQ/UNeiQ6eCiOhndOkPdPAPBNHRoEuHXteFKKneYWaeeeZ93nlmRjJVxXYY6/qEkdGx8Qn/ZGAqOD0TCs/OFWyjack8LxuqYZUk0eaqovO8ozgqL5kWFzVJ5UWpsd3fL7a4ZSuGvuu0TV7RxLqu1BRZdIjKHFbDMbbG3IgOg4QHYvAibYRvsYd9GJDRhAYOHQ5hFSJsamUkwGASV0GHOIuQ4u5zHCNA2iZlccoQiW3QWKdV2WN1Wvdr2q5aplNU6hYpo4izJ3bHeuyR3bMX9vFrrY5bo++lTbM00HKzGjpZyL3/q9JodnDwpfrTs4MaNl2vCnk3XaZ/C3mgbx1d9HJb2XhnhV2zV/J/xbrsgW6gt97kmwzPXiJAH5D4+dzDoLC+liCcScZSSe8r/FjEMlbpvTeQwg7SyNO5HKc4w7nvWQgKEWF+kCr4PE0E30JY+gTyGInq</latexit><latexit

L(x, t|⇥) = softmax-cross-entropy(x, t|⇥)勾配降下法に必要な勾配は・・・<latexit sha1_base64="ftxYxNo3LE/yKA4i0yfD5P1kQWE=">AAADEHichVFLa9RQFD6JrxofnepGcHNxaJvBdriJhYpQGHHjwkVf0yk0JdzEO5PQvMi9M2Sc9g+4dOPClYKI9A+4E3Ej7l30J4jLCm5ceG4SkFrUE5L7nXO+78u593pZFApJ6ZGmnzl77vyFqYvGpctXrk43Zq5tiXSY+7zrp1Gab3tM8ChMeFeGMuLbWc5Z7EW85+09UP3eiOciTJNNOc74bswGSdgPfSax5DYOg3kyt0IGrmUW+6TnWgvEc60WWVGYFOS2SonjGIGiOSIcxMwM5hXBkSwJEBst1R9XNrYZKBtb2diVjU2C0sZWtEdmsUDkvrMZcMlapafkhZyItC9jViz6eSrEIk9knmbjgxNkw200aZuWQU4DqwZNqGM1bbwBBx5DCj4MIQYOCUjEETAQ+OyABRQyrO3CBGs5orDsczgAA7VDZHFkMKzu4XeA2U5dTTBXnqJU+/iXCN8clQRm6Rf6lh7TT/SQfqU//+o1KT3ULGNcvUrLM3f66Y2NH/9VxbhKCH6r/jmzhD7cLWcNcfasrKhd+JV+9OT58ca99dnJHH1Fv+H8L+kR/Yg7SEbf/ddrfP0FqAuw/jzu02DLblt32vbaUrOzVF/FFNyEW2DieS9DBx7CKnTB12a0Za2j3def6e/09/qHiqprteY6nAj98y+Ygbz6</latexit>@L@W1<latexit sha1_base64="1bbEPHLcLRxNdO8/1HOsSoKlXtE=">AAAChXichVFNLwNBGH6sj1JfxUXistEQF80sjeJC4uLg4KsqUWlm15SJ7e5md9qEplcHf8DBiUREuPIHXPwBBz9BHElcHLzdrogD3snMPPPM+7zzzIzp2TJQjD01ac0trW2x9o54Z1d3T2+ir38jcMu+JbKWa7v+pskDYUtHZJVUttj0fMFLpi1y5v5CfT9XEX4gXWddHXhiu8R3HVmUFldEFRJ6vuhzq5r3uK8kt/Wl2jfOFYyaXkgkWWomDL0BMukIzBi6kWJhJBHFspu4RB47cGGhjBIEHCjCNjgCalswwOARt40qcT4hGe4L1BAnbZmyBGVwYvdp3KXVVsQ6tK7XDEK1RafY1H1S6hhhj+yKvbIHds2e2cevtaphjbqXA5rNhlZ4hd7jwbX3f1UlmhX2vlV/elYoYjr0Ksm7FzL1W1gNfeXw5HVtdnWkOsrO2Qv5P2NP7J5u4FTerIsVsXqKOH3A1yvrv4ONiZQxmZpYSSfn09FXtGMIwxij985gHotYRpbOPcINbnGnxbRxLa1NNVK1pkgzgB+hzX0C68GWfg==</latexit>@L@b1<latexit sha1_base64="JvUIuhDr2Jz/iltKAxQdZIu54Eo=">AAAChXichVFNLwNBGH6sr6qv4iJx2WiIi2aWRnEhcXFw8FUkKs3smtbEdnezO21C06uDP+DgRCIiXPkDLv6Ag58gjiQuDt5uV8QB72Rmnnnmfd55Zsb0bBkoxp6atOaW1rb2WEe8s6u7pzfR178RuGXfElnLtV1/y+SBsKUjskoqW2x5vuAl0xab5v5CfX+zIvxAus66OvDETokXHVmQFldE5RN6ruBzq5rzuK8kt/Wl2jc280ZNzyeSLDUTht4AmXQEZgzdSLEwkohi2U1cIodduLBQRgkCDhRhGxwBtW0YYPCI20GVOJ+QDPcFaoiTtkxZgjI4sfs0Fmm1HbEOres1g1Bt0Sk2dZ+UOkbYI7tir+yBXbNn9vFrrWpYo+7lgGazoRVevvd4cO39X1WJZoW9b9WfnhUKmA69SvLuhUz9FlZDXzk8eV2bXR2pjrJz9kL+z9gTu6cbOJU362JFrJ4iTh/w9cr672BjImVMpiZW0sn5dPQVMQxhGGP03hnMYxHLyNK5R7jBLe60dm1cS2tTjVStKdIM4Edoc58B/JaJ</latexit>@L@W2<latexit sha1_base64="Pxru4wmRkRwzMY2u4xPkOqjca5k=">AAAChXichVFNLwNBGH6sj1JfxUXistEQF820GsWFxMXBwVdVotLMrmlNbHc3u9MmNL06+AMOTiQiwpU/4OIPOPgJ4kji4uDtdkUc8E5m5pln3uedZ2YM15K+YuypRWtta++IdHZFu3t6+/pjA4ObvlPxTJE1HcvxtgzuC0vaIqukssSW6wleNiyRM/YXG/u5qvB86dgb6sAVO2VesmVRmlwRVYjp+aLHzVre5Z6S3NKX6984V0jV9UIszhKzQehNkEmHYDapJxMsiDjCWHFil8hjFw5MVFCGgA1F2AKHT20bSTC4xO2gRpxHSAb7AnVESVuhLEEZnNh9Gku02g5Zm9aNmn6gNukUi7pHSh1j7JFdsVf2wK7ZM/v4tVYtqNHwckCz0dQKt9B/PLz+/q+qTLPC3rfqT88KRcwEXiV5dwOmcQuzqa8enryuz62N1cbZOXsh/2fsid3TDezqm3mxKtZOEaUP+Hpl/XewmUokpxKp1XR8IR1+RSdGMIoJeu8MFrCEFWTp3CPc4BZ3WkSb1NLadDNVawk1Q/gR2vwn7cOWfw==</latexit>@L@b2<latexit sha1_base64="miJAPZ9zPsileaOkzs2fN7AE1wk=">AAAChXichVG7SgNBFD2u7/iK2gg2i0GxMUxi8NUo2FhY5GFUMBJm14kObnaX3UlAQ1oLf8DCSkFEtNUfsPEHLPwEsVSwsfBmsyIW6h1m5syZe+6cmTFcS/qKsacWrbWtvaOzqzvS09vXPxAdHFr3nYpnirzpWI63aXBfWNIWeSWVJTZdT/CyYYkNY3+5sb9RFZ4vHXtNHbhiu8x3bVmSJldEFaN6oeRxs1Zwuackt/TV+jc2ism6XozGWHw+CL0JZlMhmE/oiTgLIoYw0k70EgXswIGJCsoQsKEIW+DwqW0hAQaXuG3UiPMIyWBfoI4IaSuUJSiDE7tP4y6ttkLWpnWjph+oTTrFou6RUsc4e2RX7JU9sGv2zD5+rVULajS8HNBsNLXCLQ4cj+Te/1WVaVbY+1b96VmhhLnAqyTvbsA0bmE29dXDk9fcQna8NsHO2Qv5P2NP7J5uYFffzIuMyJ4iQh/w9cr672A9GU9Mx5OZVGwpFX5FF0Yxhkl671ksYQVp5OncI9zgFndapzalpbSZZqrWEmqG8SO0xU8D/paK</latexit>

誤差逆伝播法; Back-propagation, back-prop• 連鎖律:<latexit sha1_base64="40ADY/eZvfOjj7dTNVvzEuIuWOY=">AAACxXichVHLSiNBFD22j3Eyo0ZnM+CmMWSIm1BxBEUQBBfO0ldUSCRT3Vbawn7RXQlqCO7nB1y4ckBE/AxB/AEXfoI4O2d048KbToMmYfQ2XXXq3HtunaoyfFuGirGbLq27p7fvQ//HxKfPA4NDyeGRtdCrBKbIm57tBRsGD4UtXZFXUtliww8EdwxbrBs78438elUEofTcVbXni02HW64sS5MrokrJn8VywM1a0eeBktzWyxkrszs+Xn9hduv6rN5e9Spv1duSVou4lEyxLItC7wS5GKQQx6KXPEURW/BgogIHAi4UYRscIX0F5MDgE7eJGnEBIRnlBepIkLZCVYIqOLE7NFq0KsSsS+tGzzBSm7SLTX9ASh1pds3O2D27Yufslj39t1ct6tHwskez0dQKvzT06+vK47sqh2aF7RfVm54VypiOvEry7kdM4xRmU1/dP7xfmVlO176x3+yO/B+zG3ZBJ3Crf82TJbF8hAQ9QK79ujvB2kQ29z07sTSZmpuMn6IfoxhDhu57CnP4gUXkad9L/ME/PGgLmqMprdos1bpizRe0hHbwDI/UsRQ=</latexit>@f (g(x))
@f @g
=
@x
@g @x

L(x, t|⇥) = XE(y, t|⇥)=)h0 = g1 (x|W1 , b1 ) = W1 x + b1
<latexit sha1_base64="OU8tyLIxY4o+Q1iDduw7+pSE2eQ=">AAACdHichVG7SgNBFD1Z3/GRqI2ghRgiVmFWBcUqYGNh4SuJoBJ210kcsi9mJ5EY/AF/wMJKUUT8DBt/wMJPEEtFGwtvNguiQb3L7Jw5c8+dM3NN3xaBYuwxpnV0dnX39PbF+wcGhxLJ4ZF84FWlxXOWZ3ty2zQCbguX55RQNt/2JTcc0+YFs7Lc3C/UuAyE526pus/3HKPsipKwDEVUMZnYXfXcshTlA2VI6R0WkymWYWFMtgM9AilEseYlr7GLfXiwUIUDDheKsA0DAX070MHgE7eHBnGSkAj3OY4RJ22VsjhlGMRW6F+m1U7EurRu1gxCtUWn2DQkKSeRZg/shr2we3bLntjHr7UaYY2mlzrNZkvL/WLiZGzz/V+VQ7PCwZfqT88KJSyGXgV590OmeQurpa8dnb5sLm2kG9Psgj2T/3P2yO7oBm7t1bpa5xtniFMD9J/P3Q7ysxl9LjO7Pp/Kzket6MU4pjBD772ALFawhlzYkzNc4ir2pk1oKS3dStVikWYU30LLfAIqyJBo</latexit><latexit sha1_base64="2MvwaCjB2IRiiRGaTJcopLiTMsg=">AAACjXichVG7SgNBFD2urxgfiY9CsFkUJYKEiQ8UURFEtLDwFQ0YCbvrJC7Z7C67k2CM/oCFrYWVgojY2mpj4w9Y+AliqWBj4d1NwBfqXXbmzLn33Dkzo9qG7grGHqqk6prauvpAQ7CxqbklFG5tW3OtvKPxuGYZlpNQFZcbusnjQhcGT9gOV3KqwdfV7IyXXy9wx9Utc1UUbb6ZUzKmntY1RRCVCkcWIjsDsthLrm5zofTLk3JS8B1RSszuR4qfEsFUuIdFmR/yTxCrgJ7pjrQfi1b4HElswYKGPHLgMCEIG1Dg0reBGBhs4jZRIs4hpPt5jn0ESZunKk4VCrFZGjO02qiwJq29nq6v1mgXg36HlDJ62T27YM/sjl2yR/b2a6+S38PzUqRZLWu5nQoddK68/qvK0Syw/aH607NAGmO+V5282z7jnUIr6wu7R88r48u9pT52yp7I/wl7YLd0ArPwop0t8eVjeA8Q+37dP8HaYDQ2FB1copcYRjkC6EI3InTfo5jGPBYRp30PcYVr3EghaUSakKbKpVJVRdOOLyHNvQOIG5oa</latexit><latexit sha1_base64="O/sg4TxlzEDzl5zNAdGthnNIE/4=">AAACgnichVG7SgNBFD1Z4yu+ojaCCMGgiEWYVUERC8HGwiKJRgUjYXad6OBmd9mdBGJIZecPWFgpiAQ7/QQbf8DCTxBLBRsLbzYLoqLeYWbOnLnnzpkZw7Wkrxh7jGht0faOzq7uWE9vX/9AfHBo03fKnilypmM53rbBfWFJW+SUVJbYdj3BS4YltozDleb+VkV4vnTsDVV1xW6J79uyKE2uiCrEx/JFj5u1vMs9JbmVWKt/4mq9EE+yFAsi8RPoIUgijLQTv0Iee3BgoowSBGwowhY4fGo70MHgEreLGnEeIRnsC9QRI22ZsgRlcGIPadyn1U7I2rRu1vQDtUmnWNQ9UiYwwR5Yg72we3bNntj7r7VqQY2mlyrNRksr3MLAycj627+qEs0KB5+qPz0rFLEQeJXk3Q2Y5i3Mlr5ydPqyvpidqE2yC/ZM/s/ZI7ujG9iVV/MyI7JniNEH6N+f+yfYnEnps6mZzFxyeS78ii6MYhxT9N7zWMYq0sjRucdo4Aa3WlSb1nRttpWqRULNML6EtvQBH5+VPw==</latexit>@L@yh = (h0 ) = tanh(h0 )

@L
@L @y
y = g2 (h|W2 , b2 ) = W2 h + b2 =)=@W2@y @W2x, t|⇥) = softmax-cross-entropy(x, t|⇥)<latexit sha1_base64="OU8tyLIxY4o+Q1iDduw7+pSE2eQ=">AAACdHichVG7SgNBFD1Z3/GRqI2ghRgiVmFWBcUqYGNh4SuJoBJ210kcsi9mJ5EY/AF/wMJKUUT8DBt/wMJPEEtFGwtvNguiQb3L7Jw5c8+dM3NN3xaBYuwxpnV0dnX39PbF+wcGhxLJ4ZF84FWlxXOWZ3ty2zQCbguX55RQNt/2JTcc0+YFs7Lc3C/UuAyE526pus/3HKPsipKwDEVUMZnYXfXcshTlA2VI6R0WkymWYWFMtgM9AilEseYlr7GLfXiwUIUDDheKsA0DAX070MHgE7eHBnGSkAj3OY4RJ22VsjhlGMRW6F+m1U7EurRu1gxCtUWn2DQkKSeRZg/shr2we3bLntjHr7UaYY2mlzrNZkvL/WLiZGzz/V+VQ7PCwZfqT88KJSyGXgV590OmeQurpa8dnb5sLm2kG9Psgj2T/3P2yO7oBm7t1bpa5xtniFMD9J/P3Q7ysxl9LjO7Pp/Kzket6MU4pjBD772ALFawhlzYkzNc4ir2pk1oKS3dStVikWYU30LLfAIqyJBo</latexit><latexit sha1_base64="twF7fZkk8wo2lJxUwWiYIMUdf+s=">AAACxHichVHLSiNBFD3pGR2Nr8y4Edw0BsVVqETBYUAQBHHhwiTGCEaa6raihZXuprsSiCF+wPzALFwpiIifIYI/4MJPGFyquJnF3HQaxIh6m646de49t05V2b6SoWbsLmF8+drX/21gMDk0PDI6lvr+YzP06oEjSo6nvGDL5qFQ0hUlLbUSW34geM1WomwfLHfy5YYIQum5G7rpi50a33NlVTpcE2WlrEo14E6r4vNAS67MtfYLLlu5trloflDRbPdmmz16K5VmGRaF+RZkY5BGHOte6hwV7MKDgzpqEHChCStwhPRtIwsGn7gdtIgLCMkoL9BGkrR1qhJUwYk9oHGPVtsx69K60zOM1A7tougPSGlimt2yC/bAbtgl+8v+vdurFfXoeGnSbHe1wrfGfk8Unz9V1WjW2H9RfehZo4qfkVdJ3v2I6ZzC6eobh38eir8K060Zdsruyf8Ju2NXdAK38eic5UXhGEl6gGzvdb8Fm7lMdi6Ty8+nl+bjpxjAJKYwS/e9gCWsYh0l2vca93jEk7FiKCM06t1SIxFrxvEqjKP/OcOweQ==</latexit>

h0 = g1 (x|W1 , b1 ) = W1 x + b1
h = (h0 ) = tanh(h0 )=)<latexit sha1_base64="OU8tyLIxY4o+Q1iDduw7+pSE2eQ=">AAACdHichVG7SgNBFD1Z3/GRqI2ghRgiVmFWBcUqYGNh4SuJoBJ210kcsi9mJ5EY/AF/wMJKUUT8DBt/wMJPEEtFGwtvNguiQb3L7Jw5c8+dM3NN3xaBYuwxpnV0dnX39PbF+wcGhxLJ4ZF84FWlxXOWZ3ty2zQCbguX55RQNt/2JTcc0+YFs7Lc3C/UuAyE526pus/3HKPsipKwDEVUMZnYXfXcshTlA2VI6R0WkymWYWFMtgM9AilEseYlr7GLfXiwUIUDDheKsA0DAX070MHgE7eHBnGSkAj3OY4RJ22VsjhlGMRW6F+m1U7EurRu1gxCtUWn2DQkKSeRZg/shr2we3bLntjHr7UaYY2mlzrNZkvL/WLiZGzz/V+VQ7PCwZfqT88KJSyGXgV590OmeQurpa8dnb5sLm2kG9Psgj2T/3P2yO7oBm7t1bpa5xtniFMD9J/P3Q7ysxl9LjO7Pp/Kzket6MU4pjBD772ALFawhlzYkzNc4ir2pk1oKS3dStVikWYU30LLfAIqyJBo</latexit>

y = g2 (h|W2 , b2 ) = W2 h + b2

<latexit sha1_base64="oQLx0Yl0tpX483/EfwO4kvaG8Zw=">AAACwnichVHLSsNAFD3Gd31V3QhugsXHqky1oAiCoAsXLnxVBVvKJE7bsWkSk2lBa3/AH3DhSkFE/Aw3/oALP0HcWcGNC2/TgKioN2TmzLn33DkzY7iW9BVjjy1aa1t7R2dXd6Snt69/IDo4tO07Zc8UKdOxHG/X4L6wpC1SSipL7Lqe4CXDEjtGcamR36kIz5eOvaWOXJEp8bwtc9LkiqhsNJPOedyspl3uKcktfbX2iQuTNX1B/6ug9j1b+CrPRmMszoLQf4JECGIIY82JXiONfTgwUUYJAjYUYQscPn17SIDBJS6DKnEeIRnkBWqIkLZMVYIqOLFFGvO02gtZm9aNnn6gNmkXi36PlDrG2QO7YXV2z27ZE3v/tVc16NHwckSz0dQKNztwOrL59q+qRLNC4VP1p2eFHOYCr5K8uwHTOIXZ1FeOz+qb8xvj1Ql2yZ7J/wV7ZHd0Arvyal6ti41zROgBEt+v+yfYno4nZuLT68nYYjJ8ii6MYgxTdN+zWMQK1pCife/whBfUtWXtQDvU/Gap1hJqhvEltJMP/OyvkQ==</latexit><latexit sha1_base64="7v2mdFKAd4Jibb8o30gR5EGCMxE=">AAACxHichVHLSiNBFD32OE6MrzhuBDeNQXEVKlFwEISAIC5c+IoKRprqtqJFKt1NdyUQQ/wAf8CFKwUR8TNEmB+YhZ8gLlXcuPCm0yBGRm/TVafOvefWqSrbVzLUjN11GT+6f/b8SvQm+/oHBodSw783Q68aOKLgeMoLtm0eCiVdUdBSK7HtB4JXbCW27PJCK79VE0EoPXdD132xW+H7rixJh2uirJRVLAXcaRR9HmjJlbncfMe2lWua8+YXFfVmZ7beobdSaZZhUZifQTYGacSx4qUuUcQePDioogIBF5qwAkdI3w6yYPCJ20WDuICQjPICTSRJW6UqQRWc2DKN+7TaiVmX1q2eYaR2aBdFf0BKExPsH7tij+wvu2b37PW/vRpRj5aXOs12Wyt8a+h4dP3lW1WFZo2Dd9WXnjVK+BN5leTdj5jWKZy2vnZ48rg+tzbRmGTn7IH8n7E7dkMncGtPzsWqWDtFkh4g23ndn8FmLpOdzuRWZ9L5mfgpEhjDOKbovmeRxxJWUKB9b/GAJzwbi4YyQqPaLjW6Ys0IPoRx9AZoxbCP</latexit>

@L
@L @y
=
@b2
@y @b2

@L
@L @h
=
@h0
@h @h0x, t|⇥) = softmax-cross-entropy(x, t|⇥)h0 = g1 (x|W1 , b1 ) = W1 x + b1
h = (h0 ) = tanh(h0 )@L@L @h0==)@W1@h0 @W1<latexit sha1_base64="OU8tyLIxY4o+Q1iDduw7+pSE2eQ=">AAACdHichVG7SgNBFD1Z3/GRqI2ghRgiVmFWBcUqYGNh4SuJoBJ210kcsi9mJ5EY/AF/wMJKUUT8DBt/wMJPEEtFGwtvNguiQb3L7Jw5c8+dM3NN3xaBYuwxpnV0dnX39PbF+wcGhxLJ4ZF84FWlxXOWZ3ty2zQCbguX55RQNt/2JTcc0+YFs7Lc3C/UuAyE526pus/3HKPsipKwDEVUMZnYXfXcshTlA2VI6R0WkymWYWFMtgM9AilEseYlr7GLfXiwUIUDDheKsA0DAX070MHgE7eHBnGSkAj3OY4RJ22VsjhlGMRW6F+m1U7EurRu1gxCtUWn2DQkKSeRZg/shr2we3bLntjHr7UaYY2mlzrNZkvL/WLiZGzz/V+VQ7PCwZfqT88KJSyGXgV590OmeQurpa8dnb5sLm2kG9Psgj2T/3P2yO7oBm7t1bpa5xtniFMD9J/P3Q7ysxl9LjO7Pp/Kzket6MU4pjBD772ALFawhlzYkzNc4ir2pk1oKS3dStVikWYU30LLfAIqyJBo</latexit>

y = g2 (h|W2 , b2 ) = W2 h + b2

x, t|⇥) = softmax-cross-entropy(x, t|⇥)<latexit sha1_base64="dukibumJTZ/HjfZfBUj59BEGBbI=">AAACxXichVHLSitBFCzHd/Rq1I3gZjCodxV6VFAEQXChCxe+YgQjsWduJzbOi5lOQENw7w+4cKUgIn6GIP6ACz9BdOdr48KTyYCo6D3DdFfXOXW6utv0bRkqxm4btMam5pbWtvZER+efru5kT+9a6JUCS2Qsz/aCdZOHwpauyCipbLHuB4I7pi2y5s5sLZ8tiyCUnruqdn2x6fCiKwvS4oqofHIrVwi4Vcn5PFCS2/pC9QNn80ZVn9Z/qdgeqX7JEvO5QT6ZYmkWhf4dGDFIIY5FL3mGHP7Bg4USHAi4UIRtcIT0bcAAg0/cJirEBYRklBeoIkHaElUJquDE7tBYpNVGzLq0rvUMI7VFu9j0B6TUMcRu2Dl7ZNfsgt2xtx97VaIeNS+7NJt1rfDz3Qf9K6//VTk0K2x/qH71rFDAZORVknc/YmqnsOr68t7h48rU8lBlmJ2we/J/zG7ZJZ3ALT9Zp0ti+QgJegDj63V/B2ujaWMsPbo0npoZj5+iDQMYxF+67wnMYB6LyNC+V3jAM160Oc3RlFaul2oNsaYPn0LbfweRm7CN</latexit><latexit sha1_base64="DbcBSZVmjgZPjd1pnWggTR1Cuos=">AAACwHichVHLSsNAFD3Gd31V3QhugkVxVaZVUASh6MaFC19VoZUyiVM7mCYxmRba0h/wB1y4UhARv0Lc+AMu/AIRN4qCGxfepgFRUW/IzJlz77lzZsZwLekrxu5atNa29o7Oru5IT29f/0B0cGjTd0qeKdKmYznetsF9YUlbpJVUlth2PcGLhiW2jP3FRn6rLDxfOvaGqrhip8j3bJmXJldE5aKZbN7jZi3rck9JbunL9U9cqOvz+h/5Sv17tvJFnYvGWJwFof8EiRDEEMaKEz1HFrtwYKKEIgRsKMIWOHz6MkiAwSVuBzXiPEIyyAvUESFtiaoEVXBi92nco1UmZG1aN3r6gdqkXSz6PVLqGGe37IK9sBt2yR7Y+6+9akGPhpcKzUZTK9zcwOHI+tu/qiLNCoVP1Z+eFfKYDbxK8u4GTOMUZlNfrh69rM+tjdcm2Cl7JP8n7I5d0wns8qt5tirWjhGhB0h8v+6fYDMZT0zFk6vTsdR0+BRdGMUYJum+Z5DCElaQpn2vcI8nPGsLWkFztINmqdYSaobxJbTqByQSr1E=</latexit>

@L
@L @y
=
@h
@y @h

<latexit sha1_base64="/r/QGxZaIWxRA1nwoptJ3LOncZk=">AAACxXichVG7SgNBFD2u7/iK2gg2i8FHFWZVUARBsNDCwldUMBJn14kO7ovdSUBDsPcHLKwURMTPEMQfsPATRDtfjYU3mwVRUe+yM2fOvefOmRnTt2WoGLur0Wrr6hsam5oTLa1t7R3Jzq6V0CsElshYnu0FayYPhS1dkVFS2WLNDwR3TFusmrvTlfxqUQSh9NxlteeLDYdvuzIvLa6IyiU3s/mAW6WszwMlua3PlT+xmTPK+qT+R8XOYPlblpivDXLJFEuzKPSfwIhBCnHMe8lzZLEFDxYKcCDgQhG2wRHStw4DDD5xGygRFxCSUV6gjARpC1QlqIITu0vjNq3WY9aldaVnGKkt2sWmPyCljn52yy7YE7thl+yevf/aqxT1qHjZo9msaoWf6zjsWXr7V+XQrLDzqfrTs0Ie45FXSd79iKmcwqrqi/tHT0sTi/2lAXbKHsj/CbtjV3QCt/hsnS2IxWMk6AGM79f9E6wMp42R9PDCaGpqNH6KJvSiD0N032OYwizmkaF9r/GIF7xqM5qjKa1YLdVqYk03voR28AHAqLCj</latexit>@L@L @h0=@b1@h0 @b1✓<latexit sha1_base64="AP5bl8cDjNYPRPHV1F03g6TgOzU=">AAAC0HichVFBSxtBGH27ba1NW031UvCyNFjtJUxUaBEEwYuHHjQ2KrgSZreTZMhkd5mdBG0I4jV/wIMnhSKlP6OXXnvoIT+heIzgpYd+2SxIFO237Myb933vmzczXqRkbBjrWfajx0/Gno4/yzx/8XJiMvtqajsOm9oXJT9Uod71eCyUDETJSKPEbqQFb3hK7Hj1tUF+pyV0LMPgkzmMxH6DVwNZkT43RJWzdVeJipl33IrmftuNuDaSK+dj5wYfdJyVh/K1uc6tLDEjclfLas28K2dzLM+ScO6CQgpySGMjzF7AxWeE8NFEAwIBDGEFjpi+PRTAEBG3jzZxmpBM8gIdZEjbpCpBFZzYOo1VWu2lbEDrQc84Ufu0i6Jfk9LBLPvNvrE++8m+sz/s77292kmPgZdDmr2hVkTlye7rrev/qho0G9RuVA96NqjgQ+JVkvcoYQan8If61peT/tZycbb9lp2zS/J/xnrsB50gaF35XzdF8RQZeoDC7eu+C7YX8oXF/MLmUm51KX2KcczgDebpvt9jFevYQIn2/YVrC5ZlF+0D+8g+HpbaVqqZxkjY3X/lj7Mx</latexit>@L@L @h0=@x@h0 @x◆ニューラルネットワークの問題

Universal approximation theoremによると・・・• ニューラルネットワークは（任意の/ある種の）関数を近似できる（らしい）• パラメータの数が多いとよりよい近似になる（らしい）• 層の数が多い方がいい（らしい）• 一方で、全結合層は大量のパラメータの塊• 入力1,000、出力1,000の全結合層1つのパラメータの数は1,001,000個！

パラメータの数の比較
Performance# layers# Parameters• AlexNet

0.16

8

61M

• VGG-19

0.088?

19

138M• GoogLeNet

0.07

22

11M

• ResNet

0.036

152

25M

How does this happen?

過学習• 識別器が学習に使ったデータに過剰に適応して（学習データ
での性能が高い）、学習に使ってないデータでの性能が低下

どうしたらいい？
• 畳み込み層の利用• 正則化• Dropout• Weight decayその他の問題• 勾配消失• ReLU関数とその変種• バッチ正規化• Skip-connection• Squeeze and excitation• ハイパーパラメータチューニング（層の数、学習率、etc.）• ベイズ最適化• 勘と経験

最近の取り組み「形式知」の組み込み

「形式知」の有効活用• 形式知• テキスト•• 知識グラフ• 数式 ex)で与えられるTripletの集合<latexit sha1_base64="hd9c3W2Gx4hoCRStXiPg0nQL8yc=">AAACenichVHLSsNAFD2N7/qqiiC4CRbFosikCIogFN0Iblq1VdASkjitoWkSkrRQQ3/AH3DhSkFE/Qw3/oALP0FcVnDThTdpQFTUGzL3zJl77pyZUW1Ddz3GnmNCV3dPb1//QHxwaHhkNDE2XnCtmqPxvGYZlnOgKi43dJPnPd0z+IHtcKWqGnxfrWwG6/t17ri6Ze55DZsXq0rZ1Eu6pnhEyYnJbXFdnOeytChy2bebQUqn5ESSLbEwxJ9AikASUWStxA2OcAwLGmqogsOER9iAApe+Q0hgsIkrwifOIaSH6xxNxElboypOFQqxFRrLNDuMWJPmQU83VGu0i0G/Q0oRs+yJ3bIWe2T37IW1f+3lhz0CLw3KakfLbXn0bGr3/V9VlbKHk0/Vn549lLAaetXJux0ywSm0jr5+et7aXduZ9efYFXsl/5fsmT3QCcz6m3ad4zsXiNMDSN+v+ycopJckwrnlZGYjeop+TGMG83TfK8hgC1nkaV8fl7jDfawtzAgpYaFTKsQizQS+hLD8AUJokEE=</latexit><latexit sha1_base64="Fqxstqx4UzG0K5uYMO9n2QPEoWY=">AAACfXichVHLSsNAFD2Nr1pfVTcFN8VScSFlIoIiCKIgLlu1WqhSkji1oXmRTAta6gf4Ay5cKYqI6E+48Qdc9BPEZQU3gt6mEVFRb8jMuWfuuXNmRnUM3ROMNUJSR2dXd0+4N9LXPzA4FB0e2fTsiqvxrGYbtptTFY8busWzQhcGzzkuV0zV4Ftqebm1vlXlrqfb1obYd/iOqexZelHXFEFUIRrbNhVRUou1lXp8IW5+ZEq9EE2wFPMj/hPIAUggiLQdvcQ2dmFDQwUmOCwIwgYUePTlIYPBIW4HNeJcQrq/zlFHhLQVquJUoRBbpnGPsnzAWpS3enq+WqNdDPpdUsaRZA/sijXZPbtmj+z11141v0fLyz7NalvLncLQUWz95V+VSbNA6VP1p2eBIuZ8rzp5d3ymdQqtra8eHDfX59eStQl2xp7I/ylrsDs6gVV91i4yfO0EEXoA+ft1/wSb0ymZcGYmsbgUPEUYYxjHJN33LBaxijSytO8hznGD29CblJSmpFS7VAoFmlF8CWn2HY3lk7Q=</latexit>,<latexit sha1_base64="B3DLg/PzuitHOizhoK8mw+wBgD8=">AAACoHichVHLLgRBFD3ae7wGG4lNx2Q8IpEakRAiETaTWDDDIIZMulsNHf1Kd80Ek/4BP2BhNRIR8RkWLFlY+ASxJLGxcKenEQS301Xnnrrn1qkq1TF0TzB2XyPV1tU3NDY1R1pa29o7op1dK55dcDWe0WzDdtdUxeOGbvGM0IXB1xyXK6Zq8FV1d66yvlrkrqfb1rLYd/imqWxbel7XFEFULjq1YSpiR82X9vwBeVp+z+b9wXeY9j9L5OGPCuEP5aIxNsKCkH+CRAhiCGPRjp5hA1uwoaEAExwWBGEDCjz6skiAwSFuEyXiXEJ6sM7hI0LaAlVxqlCI3aVxm7JsyFqUV3p6gVqjXQz6XVLKiLM7ds6e2DW7YA/s9ddepaBHxcs+zWpVy51cx2HP0su/KpNmgZ1P1Z+eBfKYCLzq5N0JmMoptKq+eHD0tDSZjpf62Ql7JP9lds8u6QRW8Vk7TfH0MSL0AInv1/0TrIyOJAinxmIzs+FTNKEXfRik+x7HDJJYRIb2LeMKN7iV+qSktCClqqVSTajpxpeQ1t8A9DehNQ==</latexit>• 形式知を使う方法が確立されると・・・• 形式知化が難しいところは暗黙知を抽出する学習で、形式知で記述できるものは形式知で• 形式知を追加することで賢くなるエージェント• 形式知の自動獲得

26

絵画の自動分類

In Guernica, Pablo Picasso, by means of his own style built upon many
artistic influences, such as Cubism or African art, expressed his
emotions against war inspired by its historical and political context.

27

絵画の自動分類のための知識（抜粋）

28

モデル

29

結果

30

深層学習利用の例

核物理データの識別に向けた共同研究の紹介• フレーバー識別
日本物理学会全国大会
岸田, 岩崎 他
機械学習を用いたフレーバー識別用ツールの開発• 可変長データの3クラス識別問題• cc / bb / uds イベントに分類• 従来は、低次元の「表現」を手で設計して、浅いNNで識別

深層学習の実力発揮！
• 少しだけ、データを確認

・・・

とりあえず、作ってみよう！

コード（200+行）• 必要なモジュールのロード• データの読み込み• ニューラルネットのモデルの定義• 評価• 各種設定＆学習

学習結果

Takeaway message• 深層学習は機械学習の一種• 非常に簡単、便利• 特徴量抽出の設計が不要（その分野の知識はいらない！？）• 深層学習の中身は簡単な掛け算足し算がほとんど• 忘れてはいけない色々な問題点• 過学習• 勾配消失• ハイパーパラメータチューニング

プログラム担当行事委員より技術研修会をより意義あるものとするため、技術研修会アンケートへの回答をお願いいたします。
QRコードから⇒
URL：https://forms.gle/UKTQsYiJiqCcGEme7