Carnegie Mellon University

More is Less?
Non-parametric Language Models and Eﬃciency
Graham Neubig
Carnegie Mellon University
Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick1

Carnegie Mellon University

Parametric Language Models

Parametric LM
2

Carnegie Mellon University

Parametric Language Models
I

Parametric LM
3

Carnegie Mellon University

Parametric Language Models
I

Parametric LM


 I

4

Carnegie Mellon University

Parametric Language Models
I ordered

Parametric LM


 I

5

Carnegie Mellon University

Parametric Language Models
I ordered a

Parametric LM
 I ordered

6Carnegie Mellon University

Parametric Language Models
I ordered a pizza with sauce . 


Parametric LM
 I ordered a pizza with sauce .

7

Carnegie Mellon University

Non-Parametric Language Models

Non-Parametric LM

Non-parametric datastore (typically training dataset)8Carnegie Mellon University

Non-Parametric Language Models

Non-Parametric LM
Prototype sentence
I ordered a burger with fries

Non-parametric datastore (typically training dataset)9Carnegie Mellon University

Non-Parametric Language Models

Non-Parametric LM
Prototype sentence
I ordered a burger with fries 
Non-parametric datastore (typically training dataset)
10Carnegie Mellon University

Non-Parametric Language Models
I

Non-Parametric LM
Prototype sentence
I ordered a burger with fries 
Non-parametric datastore (typically training dataset)
11Carnegie Mellon University

Non-Parametric Language Models
I ordered a pizza with sauce . 


Non-Parametric LM
Prototype sentence
I ordered a burger with fries 
 I ordered a pizza with sauce .

Non-parametric datastore (typically training dataset)
12Carnegie Mellon University

Non-Parametric Language Models
I ordered a pizza with sauce . 


Non-Parametric LM
Prototype sentence
I ordered a burger with fries 
 I ordered a pizza with sauce .

Non-parametric datastore (typically training dataset)
13

Carnegie Mellon University

A Human Processing Analogy

14

Carnegie Mellon University

A Human Processing Analogy

Our Language Ability/
Knowledge

14

Carnegie Mellon University

A Human Processing Analogy

Our Language Ability/
Knowledge +Wikipedia/Google

Our Language Ability/
Knowledge

14

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP

15

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP•Count-based non-parametric data stores:•Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
•Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
•Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.

15

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP•Count-based non-parametric data stores:•Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
•Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
•Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
•Bias probabili6es based on count-based aggrega6on of retrieved sentences:•Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
•Deep Weighted Averaging Classiﬁers. Card et al. 2019.
•Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.

15

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP•Count-based non-parametric data stores:•Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
•Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
•Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
•Bias probabili6es based on count-based aggrega6on of retrieved sentences:•Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
•Deep Weighted Averaging Classiﬁers. Card et al. 2019.
•Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
•A;end to retrieved sentences:•Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
•Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
•REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.

15

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP•Count-based non-parametric data stores:•Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
•Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
•Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
•Bias probabili6es based on count-based aggrega6on of retrieved sentences:•Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
•Deep Weighted Averaging Classiﬁers. Card et al. 2019.
•Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
•A;end to retrieved sentences:•Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
•Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
•REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.
•Feed retrieved states into model:•Learning to Remember Rare Events. Kaiser et al. 2017.
15

Carnegie Mellon University

A (Very Incomplete) Survey of NeuralNon-parametric Models for NLP•Count-based non-parametric data stores:•Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
•Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
•Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
•Bias probabili6es based on count-based aggrega6on of retrieved sentences:•Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
•Deep Weighted Averaging Classiﬁers. Card et al. 2019.
•Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
•A;end to retrieved sentences:•Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
•Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
•REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.
•Feed retrieved states into model:•Learning to Remember Rare Events. Kaiser et al. 2017.
•Fine-tune models on retrieved sentences:•One Sentence One Model for Neural Machine Transla/on. Li et al. 2016.

15

Carnegie Mellon University

Non-Parametric Language Models:
Pros/Cons
Pros• Micgate pressure on the parametric models• Improved interpretability in the modeling processCarnegie Mellon University

Non-Parametric Language Models:
Pros/Cons
Pros• Micgate pressure on the parametric models• Improved interpretability in the modeling processCons•In almost all cases, large parametric datastore leads to signiﬁcantissues with memory and speed eﬃciency at test cme

Carnegie Mellon University

A Concrete Example:
Prototype-based Language Models(Guu et al. TACL 2018)18

Carnegie Mellon University

Previous ApproachPrototype
distributionp(t)Uniform<latexit sha1_base64="BqEu/64RuYCyn2jfc6RbHK9rVtg=">AAA1pXiclVtbc9vKkVay2d1Eye6ebB7zgorsOs6WrLJ8jmu38hRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/Sc8MBt0DQj61qrKE+b6exly+nm6C8DCPIyHfvPm/n/38H37xj//0z7/81fqvf/Mv//pv3/z2329ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9T/G0JhYiytC/nOXxK/DCNxlHgS4Ru81eDopR//PzNxputN/rHW73Yri421qqf3ufffv8/g1EWTBNIZRD7Qvyw/SaXnxZ+IaMghuX6YCog94N7P4Qf8DL1ExCfFnq8S+8lIiNvnBX4L5WeRnmPhZ8IMU+GaJn4ciKanALbuB+mcvxfnxZRmk8lpIG50XgaezLz1OS9UVRAIOM5XvhBEeFYvWDiF34gcYnW11+qH++sc+ud7vSPvP3OwfHZcf/4/OzK09R620A28a+ahtgcJkv04Z36xb0n8D64wMLLxl7g5+ZazbiAMRRFlIZqUKOojIQ1G0fhtACcUAqPQZYkfjpaDBCMYSyXi8UAEu9VF6//uFyu2AS4D1BYqz3darMronBSO7tUjTYrmeXWpp/lbRbDTMossUa7urViV83bt2b+cxZDazF8ziKwFsFzFiNrMVIWuA1HOLtYzdDzPbRXmw5jjJKRh2uTuD7wWoHLH7Y/oZfh2NvYVk6a054tF4PEL0IUmF8sDo7vmmPBa8cEpdQ06Z/vn+v7DCTMpJb+ogAcvSL+ZG7s+uxol3KS5YtBp8l2HpDtfF5gGD8NRJR4D3hd5pNoqSL76S/4a7ayZJ38qdErV73kBKT/9X51t1nVbfSq1fLhCdeqdSiuXa7snrl5w3L2tGI5U5ZPq5arhis2I200aifVnV61uX54as6qaaEWFuEGOtPorIHONTpvoIlGk6YKNJo2badSiWOKQ5ptenbGK0a5Y6TG3jAZmY54G38Y+7R2TTPVlRm1eUJsxRlifHQYnAf6rDOHIZ7UsOnF2SMUrwNMZVvrA4xUfVrBeGN7Yc7F/x5ga6HDo607ngKR9OMt7wDPWCExD6kjVaiDEHnj8cB6PGh61LR8zOw9N95WdxWeNfJwelXjre3xMPVH1GXju43vV7pt1n3s1Xfc1fd6OlcmWXx1OTChmMFXmcVZjxYHdkFM7yvb+6ql96XtpfPkY1Ynr616YczdhV6ZOrU9szRNh5MCoOmS+dv4btUjrRrz/d2qbz/1ADdBdW5ZMngwc7Ymz0/a8TPNcyg85ce46VRuOm1udrzCf6R1bzh7/fq1X2bRyJsKlfGjsZdnQkRYnBnXeexjRqr8Pz86VaTkmKBa5qgY072y+X9PsnK0Vzva+0lHOOc0BF3aGFthfGi4HhFKxdLW1evXz8oER+fHYYZF2SRpmSdyZnS10VcnylytzHTHutppcWUFb++Hk6h9ff0w6Duddn6y08qiYsEgq5kz9SnUDFddfW1TTP+ment1/57b3860vgGOWl0/O+BKcBDFSqyxusByBQ3UVeVvHGdZoWl9ZXh9WRkgNUwWK0WOLDAQqjon8OPFftOg9ONoxA0+m+siWRhqueIShGzvoJllPSPIhSodcxHFWarLPlxadJElXukXESYxsPrG/LUwhVuaFQl6fTFA6MXSLmfRoH1ihi4zJCZwmYCYkcuMiAGXAWLGLjMmJnSZkJiJy0yIiVwmIuaLy3wh5t5l7omJXSZeahkXiRcJjFj82Dqaq8PO7OCm92UqpDfK0m+lpz4/ohzn6uRxNsZLKt+p6zulu2YukxGTu0xOzIPLPBBTuExBjHAZQYx0GUnM1GWmxJQuUxLz6DKPxMxcZkbM3GXmxDy5zNPSFGg2ADAzZ/XxXlZBsjChNByzsKnHras8ZlFVfTXPOA4PCWaxUQYEs8AoRwSzqCiBYBYS5ZhgFg9lSDALhnJCMIuEckowC4PyC8EsBsp7glkAlDHBMYMTghMGs4XmK5wRzMRc5gQzJZcPBDMZlwXBTMOlIFjwTSVYtq8Jl25JMNNt+UgwE205I5gptpwTzORaPhFstdqJQT2H0g9RihbdghFd67kMRnmtJzMY+bWezWA02Ho6gxFi6/kMRo2tJzQYSbae0WB02XpKI/fsOQ1Goa0nNRiZtp7VYLTaPK0tl7hcwrlnT2Iw0m09i8Hot/U0BiPi1vMYjJJbT2Qwcm49k8FouvVUBiPs1nMZjLpbT2YwEm89m8HovPV0BiP21vMZjOKfP6ExFoooqCuUZIfiY4fCJtkleJfBewTvMXif4H0GdwjuMPiA4AMGHxJ8yOAjgo8YfEzwMYPfE/yewScEnzC4S3CXwacEnzL4jOAzBp8TfM7gHsE9Bl8QfMHgS4IvGXxF8BWD+wT3GXxN8DWDbwi+YfAtwbcMviP4jsEfCP7A4I8Ef3z+eHVFB0Z1TKM7TL9aeozb5dyey+1xbt/l9jnXcbkO5w5c7oBzhy53yLkjlzvi3LHLHXPuvcu959yJy51wrutyXc6dutwp585c7oxz5y53zrmey/U4d+FyF5y7dLlLzl253BXn+i7X59y1y11z7sblbjh363K3nLtzuTvOfXC5D5z76HJW9je8hCifQH+OwM+ub+q+ZZbCwn6etVgyNdAgoaRR18QKd+th9Wy0IvTTVAtX1sxwaBAqT3RxgggVJbokQYRKkbIaIBUguvxAhMoOXXQgQsWGLjUQoRKjrAbJRvjFIFRO6GICESoidAmBSMyWxyBUMOhyAZGULatBMrZIBqGSQBcEiFAhoMsARCj96+SPiGD7YBBK9WW1W2yvSoNQWtdJHRFK5jqVI0IpXCdwRChx67SNSFuR6lanpR/nE7Xf+m8tzHJYaaZ6EG9A+gRGDywqKvaT4Uj1MBdEZAmECtd/CdZKVSq1ADpEBH8TJKIwUV31X4KtnutvCaqJLBZ8/AslVttCsQbUQqGO2KQWSqC2hQIdUwvFGVILhTmhFg6XjRUF+YVaKMZ7tjYLJcJ65gslQNvCxWSriOLL2JIslOhsC0X3QC0UXMFWaqGEVi/QQonMtnCh2TKjwEpqobgeqYXCmlELRTWnFgrqaVl9w4zpd2ZwnXpRZ5RydcJFhBKtTrOIUHrVyRURSqo6pSJCqVQnUkQoger0iQilTZ00EaFkqVMlIpQidYJEhBKjTouIUDrUyRARSoI6BSJCqU8nPkQo4el0hwilOZ3kEKHkplMbIpTSdEJDhBKZTmOIUPrSyQsRSlo6ZSFCqUonKkQoQen0hAilJZ2UEKFkpFMRIpSCdAJC5CPbQUoXQ54tkl6dLXosWyRdG/qK6VbhX0+uimHFXZk41irqQyrUexf7EMR+ASiqyY46gfCOpgYU40g9QYU0yEZRGqIzfxorRIzr62S5EOrh7xXI5xwMs3j0U26Gs+Wi+eWmxPGZb8p1Oq386YfX1dSkKTtTwdQvdy1G+pd7FqMIkPsWoxiQHYtRFMgDi1EcyEOLUSTII4tRLMhji1E0yPcWo3iQJxajiJBdi1FMyFOLUVTIM4tRXMhzi1FkyJ7FKDbkhcUoOuSlxSg+5JXFKEJk32IUI/LaYhQl8sZiFCfy1mIUKfLOYhQr8oPFKFrkR4uZQg2FfFj4+cSwof34GzifQsJdBpMuwj0GkzTCfQaTOsIOg0kg4QGDSSPhIYNJJuERg0kp4TGDSSzhewaTXsITBpNkwi6DSTXhKYNJOOEZg0k74TmDST5hj8GkoPCCwSSi8JLBpKPwisEkpbDPYFJTeM1gElR4w2DSVHjLYJJVeMdgUlb4gcEkrvAjg+0HATzaqlJN1A9XhkxcYpdQ0pbYI5SkJfYJ1cp66e3rLzimAjzfEyA9vHUMI6+z6Q0h8BUuJ5HwHrNpPEIIW+AJ/XUI1pLTwlMvymUxOlJvl8Esx9pSf8drv2nv0B1JtOKAUNKsOCSUJCuOCCXFimNCSbDiPaGkV3FCKMlVdAkltYpTQkms4oxQ0qo4J5SkKnqEklLFBaEkVHFJKOlUXBFKMhV9Qkml4ppQEqm4IZQ0Km4JJYmKO0JJoeIDoSRQ8ZHQ+vlMitUg6A8WvnkyU5WGQHVB1/1IoIrGHWqhgHephcLdoxYKdp9aKKYOtVBEB9RC8RxSC0VzRC0UyzG1UCTvqYXiOKEWiqJLLRTDKbVQBGfUws0/pxZueo9auNkX1MJNvqQWbu4VtXBT+9TCzbymFm7iDbVw826phZt2Ry3crA/Uwk36yO5X1V9V7aW2DPiWSVOH4UGjolq/EYuhbdBN7zGSk2wqPSyCvEdMdDkUbpkEVCc5NVJ1e1lrQBuulIegiyhoVFGgyyho1FGgCyloVFKgSylo1FKgiyloVFOgyylo1FOgCypoVFSgSypo1FSgiypoVFWgyypo1FWgCytoVFagSyto1FagiytoVFegyyto1FegCyxoVFigSyxo1FigiyxoVFmgyyxo1FmgCy1oVFqgSy1o1Fqgiy1oVFugyy1o1FugCy5oVFygSy5o1Fygiy5oVF2gyy5gdRd+fsBEJIspeNN0BEU8V284jXzpeyGkUGAOUu1IoNKHU5WQmm9g+uolQfWiZpEsdEOnQ+UVkjwqIkyETv/6/d3hXCdB/c6IuglmzYZv+zrJxJf4+d29hWPZ45a9ZdtgkmwE8dcmog3qmZjWyn0qo97XjHIZxSOoLAe6UY++7oHHhMyCiS/Uy+v+VGb6cxUUzggbL5HnxqYeY9VldQAjcOxMs8WuQAIPHWtnmqiFQD9Zc41jP4/9AJb16zfdClh6L73q2l3exkvHS16/uFxXsDd8uk32cslze+PUTHK2xg0yLpb2aZxLFBAu6+drTSqQNEfVisYRFE3XIhvLxJ+RpQWadpgsMv3Gk3n0tuolj6dq9k/q+YDLnnSX/HWnk+7KBt74BY1ANZr+Jf7xC9z7ImOWVysbsJeVRKuGEuhtFo8LP1GPqSaPWYFlq/DnwnvR/fHtC/Wqj/6PH9PUvLIqctx/oV81ezGAOGY29jHpS28XEyCGfKp+zTHeIVGvvKna2Dhl1up902wa6pypS+VIwqZ2LzJvlIFy9xjdRzmMIn+r8ZJ1ViSxetK/XHR/fLNsIbMUFLfdxslH3e9tG5crJm9htBa6Pw6idCznzdDJ/UI9MsZjw1fBcgV41go/BC9KvTSrynwJsy1vb5IJtTyZKgCDibePn4hT+FZ4wyy731p3HvKc5+p0zor/QI0XoR4A/h1sqquvGapz0hjiVbtLrVY007+fseijoPrqdcAY5MAfYpzF2eOwAP9+3WCZOgRzOcdjfWAvDFPDpV/g/Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++H7jz7vV/2L65drv1/6w9mpte+0/1/68drTWW7teC9bu1/669re1/3337bvTd/13N8b05z+r+vxuzfl59/nv9mfMZg==</latexit>

0 1 2 3 N

Guu et al. 2018

Carnegie Mellon University

Previous ApproachPrototype
distributionp(t)Uniform<latexit sha1_base64="BqEu/64RuYCyn2jfc6RbHK9rVtg=">AAA1pXiclVtbc9vKkVay2d1Eye6ebB7zgorsOs6WrLJ8jmu38hRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/Sc8MBt0DQj61qrKE+b6exly+nm6C8DCPIyHfvPm/n/38H37xj//0z7/81fqvf/Mv//pv3/z2329ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9T/G0JhYiytC/nOXxK/DCNxlHgS4Ru81eDopR//PzNxputN/rHW73Yri421qqf3ufffv8/g1EWTBNIZRD7Qvyw/SaXnxZ+IaMghuX6YCog94N7P4Qf8DL1ExCfFnq8S+8lIiNvnBX4L5WeRnmPhZ8IMU+GaJn4ciKanALbuB+mcvxfnxZRmk8lpIG50XgaezLz1OS9UVRAIOM5XvhBEeFYvWDiF34gcYnW11+qH++sc+ud7vSPvP3OwfHZcf/4/OzK09R620A28a+ahtgcJkv04Z36xb0n8D64wMLLxl7g5+ZazbiAMRRFlIZqUKOojIQ1G0fhtACcUAqPQZYkfjpaDBCMYSyXi8UAEu9VF6//uFyu2AS4D1BYqz3darMronBSO7tUjTYrmeXWpp/lbRbDTMossUa7urViV83bt2b+cxZDazF8ziKwFsFzFiNrMVIWuA1HOLtYzdDzPbRXmw5jjJKRh2uTuD7wWoHLH7Y/oZfh2NvYVk6a054tF4PEL0IUmF8sDo7vmmPBa8cEpdQ06Z/vn+v7DCTMpJb+ogAcvSL+ZG7s+uxol3KS5YtBp8l2HpDtfF5gGD8NRJR4D3hd5pNoqSL76S/4a7ayZJ38qdErV73kBKT/9X51t1nVbfSq1fLhCdeqdSiuXa7snrl5w3L2tGI5U5ZPq5arhis2I200aifVnV61uX54as6qaaEWFuEGOtPorIHONTpvoIlGk6YKNJo2badSiWOKQ5ptenbGK0a5Y6TG3jAZmY54G38Y+7R2TTPVlRm1eUJsxRlifHQYnAf6rDOHIZ7UsOnF2SMUrwNMZVvrA4xUfVrBeGN7Yc7F/x5ga6HDo607ngKR9OMt7wDPWCExD6kjVaiDEHnj8cB6PGh61LR8zOw9N95WdxWeNfJwelXjre3xMPVH1GXju43vV7pt1n3s1Xfc1fd6OlcmWXx1OTChmMFXmcVZjxYHdkFM7yvb+6ql96XtpfPkY1Ynr616YczdhV6ZOrU9szRNh5MCoOmS+dv4btUjrRrz/d2qbz/1ADdBdW5ZMngwc7Ymz0/a8TPNcyg85ce46VRuOm1udrzCf6R1bzh7/fq1X2bRyJsKlfGjsZdnQkRYnBnXeexjRqr8Pz86VaTkmKBa5qgY072y+X9PsnK0Vzva+0lHOOc0BF3aGFthfGi4HhFKxdLW1evXz8oER+fHYYZF2SRpmSdyZnS10VcnylytzHTHutppcWUFb++Hk6h9ff0w6Duddn6y08qiYsEgq5kz9SnUDFddfW1TTP+ment1/57b3860vgGOWl0/O+BKcBDFSqyxusByBQ3UVeVvHGdZoWl9ZXh9WRkgNUwWK0WOLDAQqjon8OPFftOg9ONoxA0+m+siWRhqueIShGzvoJllPSPIhSodcxHFWarLPlxadJElXukXESYxsPrG/LUwhVuaFQl6fTFA6MXSLmfRoH1ihi4zJCZwmYCYkcuMiAGXAWLGLjMmJnSZkJiJy0yIiVwmIuaLy3wh5t5l7omJXSZeahkXiRcJjFj82Dqaq8PO7OCm92UqpDfK0m+lpz4/ohzn6uRxNsZLKt+p6zulu2YukxGTu0xOzIPLPBBTuExBjHAZQYx0GUnM1GWmxJQuUxLz6DKPxMxcZkbM3GXmxDy5zNPSFGg2ADAzZ/XxXlZBsjChNByzsKnHras8ZlFVfTXPOA4PCWaxUQYEs8AoRwSzqCiBYBYS5ZhgFg9lSDALhnJCMIuEckowC4PyC8EsBsp7glkAlDHBMYMTghMGs4XmK5wRzMRc5gQzJZcPBDMZlwXBTMOlIFjwTSVYtq8Jl25JMNNt+UgwE205I5gptpwTzORaPhFstdqJQT2H0g9RihbdghFd67kMRnmtJzMY+bWezWA02Ho6gxFi6/kMRo2tJzQYSbae0WB02XpKI/fsOQ1Goa0nNRiZtp7VYLTaPK0tl7hcwrlnT2Iw0m09i8Hot/U0BiPi1vMYjJJbT2Qwcm49k8FouvVUBiPs1nMZjLpbT2YwEm89m8HovPV0BiP21vMZjOKfP6ExFoooqCuUZIfiY4fCJtkleJfBewTvMXif4H0GdwjuMPiA4AMGHxJ8yOAjgo8YfEzwMYPfE/yewScEnzC4S3CXwacEnzL4jOAzBp8TfM7gHsE9Bl8QfMHgS4IvGXxF8BWD+wT3GXxN8DWDbwi+YfAtwbcMviP4jsEfCP7A4I8Ef3z+eHVFB0Z1TKM7TL9aeozb5dyey+1xbt/l9jnXcbkO5w5c7oBzhy53yLkjlzvi3LHLHXPuvcu959yJy51wrutyXc6dutwp585c7oxz5y53zrmey/U4d+FyF5y7dLlLzl253BXn+i7X59y1y11z7sblbjh363K3nLtzuTvOfXC5D5z76HJW9je8hCifQH+OwM+ub+q+ZZbCwn6etVgyNdAgoaRR18QKd+th9Wy0IvTTVAtX1sxwaBAqT3RxgggVJbokQYRKkbIaIBUguvxAhMoOXXQgQsWGLjUQoRKjrAbJRvjFIFRO6GICESoidAmBSMyWxyBUMOhyAZGULatBMrZIBqGSQBcEiFAhoMsARCj96+SPiGD7YBBK9WW1W2yvSoNQWtdJHRFK5jqVI0IpXCdwRChx67SNSFuR6lanpR/nE7Xf+m8tzHJYaaZ6EG9A+gRGDywqKvaT4Uj1MBdEZAmECtd/CdZKVSq1ADpEBH8TJKIwUV31X4KtnutvCaqJLBZ8/AslVttCsQbUQqGO2KQWSqC2hQIdUwvFGVILhTmhFg6XjRUF+YVaKMZ7tjYLJcJ65gslQNvCxWSriOLL2JIslOhsC0X3QC0UXMFWaqGEVi/QQonMtnCh2TKjwEpqobgeqYXCmlELRTWnFgrqaVl9w4zpd2ZwnXpRZ5RydcJFhBKtTrOIUHrVyRURSqo6pSJCqVQnUkQoger0iQilTZ00EaFkqVMlIpQidYJEhBKjTouIUDrUyRARSoI6BSJCqU8nPkQo4el0hwilOZ3kEKHkplMbIpTSdEJDhBKZTmOIUPrSyQsRSlo6ZSFCqUonKkQoQen0hAilJZ2UEKFkpFMRIpSCdAJC5CPbQUoXQ54tkl6dLXosWyRdG/qK6VbhX0+uimHFXZk41irqQyrUexf7EMR+ASiqyY46gfCOpgYU40g9QYU0yEZRGqIzfxorRIzr62S5EOrh7xXI5xwMs3j0U26Gs+Wi+eWmxPGZb8p1Oq386YfX1dSkKTtTwdQvdy1G+pd7FqMIkPsWoxiQHYtRFMgDi1EcyEOLUSTII4tRLMhji1E0yPcWo3iQJxajiJBdi1FMyFOLUVTIM4tRXMhzi1FkyJ7FKDbkhcUoOuSlxSg+5JXFKEJk32IUI/LaYhQl8sZiFCfy1mIUKfLOYhQr8oPFKFrkR4uZQg2FfFj4+cSwof34GzifQsJdBpMuwj0GkzTCfQaTOsIOg0kg4QGDSSPhIYNJJuERg0kp4TGDSSzhewaTXsITBpNkwi6DSTXhKYNJOOEZg0k74TmDST5hj8GkoPCCwSSi8JLBpKPwisEkpbDPYFJTeM1gElR4w2DSVHjLYJJVeMdgUlb4gcEkrvAjg+0HATzaqlJN1A9XhkxcYpdQ0pbYI5SkJfYJ1cp66e3rLzimAjzfEyA9vHUMI6+z6Q0h8BUuJ5HwHrNpPEIIW+AJ/XUI1pLTwlMvymUxOlJvl8Esx9pSf8drv2nv0B1JtOKAUNKsOCSUJCuOCCXFimNCSbDiPaGkV3FCKMlVdAkltYpTQkms4oxQ0qo4J5SkKnqEklLFBaEkVHFJKOlUXBFKMhV9Qkml4ppQEqm4IZQ0Km4JJYmKO0JJoeIDoSRQ8ZHQ+vlMitUg6A8WvnkyU5WGQHVB1/1IoIrGHWqhgHephcLdoxYKdp9aKKYOtVBEB9RC8RxSC0VzRC0UyzG1UCTvqYXiOKEWiqJLLRTDKbVQBGfUws0/pxZueo9auNkX1MJNvqQWbu4VtXBT+9TCzbymFm7iDbVw826phZt2Ry3crA/Uwk36yO5X1V9V7aW2DPiWSVOH4UGjolq/EYuhbdBN7zGSk2wqPSyCvEdMdDkUbpkEVCc5NVJ1e1lrQBuulIegiyhoVFGgyyho1FGgCyloVFKgSylo1FKgiyloVFOgyylo1FOgCypoVFSgSypo1FSgiypoVFWgyypo1FWgCytoVFagSyto1FagiytoVFegyyto1FegCyxoVFigSyxo1FigiyxoVFmgyyxo1FmgCy1oVFqgSy1o1Fqgiy1oVFugyy1o1FugCy5oVFygSy5o1Fygiy5oVF2gyy5gdRd+fsBEJIspeNN0BEU8V284jXzpeyGkUGAOUu1IoNKHU5WQmm9g+uolQfWiZpEsdEOnQ+UVkjwqIkyETv/6/d3hXCdB/c6IuglmzYZv+zrJxJf4+d29hWPZ45a9ZdtgkmwE8dcmog3qmZjWyn0qo97XjHIZxSOoLAe6UY++7oHHhMyCiS/Uy+v+VGb6cxUUzggbL5HnxqYeY9VldQAjcOxMs8WuQAIPHWtnmqiFQD9Zc41jP4/9AJb16zfdClh6L73q2l3exkvHS16/uFxXsDd8uk32cslze+PUTHK2xg0yLpb2aZxLFBAu6+drTSqQNEfVisYRFE3XIhvLxJ+RpQWadpgsMv3Gk3n0tuolj6dq9k/q+YDLnnSX/HWnk+7KBt74BY1ANZr+Jf7xC9z7ImOWVysbsJeVRKuGEuhtFo8LP1GPqSaPWYFlq/DnwnvR/fHtC/Wqj/6PH9PUvLIqctx/oV81ezGAOGY29jHpS28XEyCGfKp+zTHeIVGvvKna2Dhl1up902wa6pypS+VIwqZ2LzJvlIFy9xjdRzmMIn+r8ZJ1ViSxetK/XHR/fLNsIbMUFLfdxslH3e9tG5crJm9htBa6Pw6idCznzdDJ/UI9MsZjw1fBcgV41go/BC9KvTSrynwJsy1vb5IJtTyZKgCDibePn4hT+FZ4wyy731p3HvKc5+p0zor/QI0XoR4A/h1sqquvGapz0hjiVbtLrVY007+fseijoPrqdcAY5MAfYpzF2eOwAP9+3WCZOgRzOcdjfWAvDFPDpV/g/Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++H7jz7vV/2L65drv1/6w9mpte+0/1/68drTWW7teC9bu1/669re1/3337bvTd/13N8b05z+r+vxuzfl59/nv9mfMZg==</latexit>

0 1 2 3 N

Prototype sentence
I ordered a burger
with friestnN

Guu et al. 2018

Carnegie Mellon University

Previous ApproachPrototype
distributionp(t)Uniform<latexit sha1_base64="BqEu/64RuYCyn2jfc6RbHK9rVtg=">AAA1pXiclVtbc9vKkVay2d1Eye6ebB7zgorsOs6WrLJ8jmu38hRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/Sc8MBt0DQj61qrKE+b6exly+nm6C8DCPIyHfvPm/n/38H37xj//0z7/81fqvf/Mv//pv3/z2329ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9T/G0JhYiytC/nOXxK/DCNxlHgS4Ru81eDopR//PzNxputN/rHW73Yri421qqf3ufffv8/g1EWTBNIZRD7Qvyw/SaXnxZ+IaMghuX6YCog94N7P4Qf8DL1ExCfFnq8S+8lIiNvnBX4L5WeRnmPhZ8IMU+GaJn4ciKanALbuB+mcvxfnxZRmk8lpIG50XgaezLz1OS9UVRAIOM5XvhBEeFYvWDiF34gcYnW11+qH++sc+ud7vSPvP3OwfHZcf/4/OzK09R620A28a+ahtgcJkv04Z36xb0n8D64wMLLxl7g5+ZazbiAMRRFlIZqUKOojIQ1G0fhtACcUAqPQZYkfjpaDBCMYSyXi8UAEu9VF6//uFyu2AS4D1BYqz3darMronBSO7tUjTYrmeXWpp/lbRbDTMossUa7urViV83bt2b+cxZDazF8ziKwFsFzFiNrMVIWuA1HOLtYzdDzPbRXmw5jjJKRh2uTuD7wWoHLH7Y/oZfh2NvYVk6a054tF4PEL0IUmF8sDo7vmmPBa8cEpdQ06Z/vn+v7DCTMpJb+ogAcvSL+ZG7s+uxol3KS5YtBp8l2HpDtfF5gGD8NRJR4D3hd5pNoqSL76S/4a7ayZJ38qdErV73kBKT/9X51t1nVbfSq1fLhCdeqdSiuXa7snrl5w3L2tGI5U5ZPq5arhis2I200aifVnV61uX54as6qaaEWFuEGOtPorIHONTpvoIlGk6YKNJo2badSiWOKQ5ptenbGK0a5Y6TG3jAZmY54G38Y+7R2TTPVlRm1eUJsxRlifHQYnAf6rDOHIZ7UsOnF2SMUrwNMZVvrA4xUfVrBeGN7Yc7F/x5ga6HDo607ngKR9OMt7wDPWCExD6kjVaiDEHnj8cB6PGh61LR8zOw9N95WdxWeNfJwelXjre3xMPVH1GXju43vV7pt1n3s1Xfc1fd6OlcmWXx1OTChmMFXmcVZjxYHdkFM7yvb+6ql96XtpfPkY1Ynr616YczdhV6ZOrU9szRNh5MCoOmS+dv4btUjrRrz/d2qbz/1ADdBdW5ZMngwc7Ymz0/a8TPNcyg85ce46VRuOm1udrzCf6R1bzh7/fq1X2bRyJsKlfGjsZdnQkRYnBnXeexjRqr8Pz86VaTkmKBa5qgY072y+X9PsnK0Vzva+0lHOOc0BF3aGFthfGi4HhFKxdLW1evXz8oER+fHYYZF2SRpmSdyZnS10VcnylytzHTHutppcWUFb++Hk6h9ff0w6Duddn6y08qiYsEgq5kz9SnUDFddfW1TTP+ment1/57b3860vgGOWl0/O+BKcBDFSqyxusByBQ3UVeVvHGdZoWl9ZXh9WRkgNUwWK0WOLDAQqjon8OPFftOg9ONoxA0+m+siWRhqueIShGzvoJllPSPIhSodcxHFWarLPlxadJElXukXESYxsPrG/LUwhVuaFQl6fTFA6MXSLmfRoH1ihi4zJCZwmYCYkcuMiAGXAWLGLjMmJnSZkJiJy0yIiVwmIuaLy3wh5t5l7omJXSZeahkXiRcJjFj82Dqaq8PO7OCm92UqpDfK0m+lpz4/ohzn6uRxNsZLKt+p6zulu2YukxGTu0xOzIPLPBBTuExBjHAZQYx0GUnM1GWmxJQuUxLz6DKPxMxcZkbM3GXmxDy5zNPSFGg2ADAzZ/XxXlZBsjChNByzsKnHras8ZlFVfTXPOA4PCWaxUQYEs8AoRwSzqCiBYBYS5ZhgFg9lSDALhnJCMIuEckowC4PyC8EsBsp7glkAlDHBMYMTghMGs4XmK5wRzMRc5gQzJZcPBDMZlwXBTMOlIFjwTSVYtq8Jl25JMNNt+UgwE205I5gptpwTzORaPhFstdqJQT2H0g9RihbdghFd67kMRnmtJzMY+bWezWA02Ho6gxFi6/kMRo2tJzQYSbae0WB02XpKI/fsOQ1Goa0nNRiZtp7VYLTaPK0tl7hcwrlnT2Iw0m09i8Hot/U0BiPi1vMYjJJbT2Qwcm49k8FouvVUBiPs1nMZjLpbT2YwEm89m8HovPV0BiP21vMZjOKfP6ExFoooqCuUZIfiY4fCJtkleJfBewTvMXif4H0GdwjuMPiA4AMGHxJ8yOAjgo8YfEzwMYPfE/yewScEnzC4S3CXwacEnzL4jOAzBp8TfM7gHsE9Bl8QfMHgS4IvGXxF8BWD+wT3GXxN8DWDbwi+YfAtwbcMviP4jsEfCP7A4I8Ef3z+eHVFB0Z1TKM7TL9aeozb5dyey+1xbt/l9jnXcbkO5w5c7oBzhy53yLkjlzvi3LHLHXPuvcu959yJy51wrutyXc6dutwp585c7oxz5y53zrmey/U4d+FyF5y7dLlLzl253BXn+i7X59y1y11z7sblbjh363K3nLtzuTvOfXC5D5z76HJW9je8hCifQH+OwM+ub+q+ZZbCwn6etVgyNdAgoaRR18QKd+th9Wy0IvTTVAtX1sxwaBAqT3RxgggVJbokQYRKkbIaIBUguvxAhMoOXXQgQsWGLjUQoRKjrAbJRvjFIFRO6GICESoidAmBSMyWxyBUMOhyAZGULatBMrZIBqGSQBcEiFAhoMsARCj96+SPiGD7YBBK9WW1W2yvSoNQWtdJHRFK5jqVI0IpXCdwRChx67SNSFuR6lanpR/nE7Xf+m8tzHJYaaZ6EG9A+gRGDywqKvaT4Uj1MBdEZAmECtd/CdZKVSq1ADpEBH8TJKIwUV31X4KtnutvCaqJLBZ8/AslVttCsQbUQqGO2KQWSqC2hQIdUwvFGVILhTmhFg6XjRUF+YVaKMZ7tjYLJcJ65gslQNvCxWSriOLL2JIslOhsC0X3QC0UXMFWaqGEVi/QQonMtnCh2TKjwEpqobgeqYXCmlELRTWnFgrqaVl9w4zpd2ZwnXpRZ5RydcJFhBKtTrOIUHrVyRURSqo6pSJCqVQnUkQoger0iQilTZ00EaFkqVMlIpQidYJEhBKjTouIUDrUyRARSoI6BSJCqU8nPkQo4el0hwilOZ3kEKHkplMbIpTSdEJDhBKZTmOIUPrSyQsRSlo6ZSFCqUonKkQoQen0hAilJZ2UEKFkpFMRIpSCdAJC5CPbQUoXQ54tkl6dLXosWyRdG/qK6VbhX0+uimHFXZk41irqQyrUexf7EMR+ASiqyY46gfCOpgYU40g9QYU0yEZRGqIzfxorRIzr62S5EOrh7xXI5xwMs3j0U26Gs+Wi+eWmxPGZb8p1Oq386YfX1dSkKTtTwdQvdy1G+pd7FqMIkPsWoxiQHYtRFMgDi1EcyEOLUSTII4tRLMhji1E0yPcWo3iQJxajiJBdi1FMyFOLUVTIM4tRXMhzi1FkyJ7FKDbkhcUoOuSlxSg+5JXFKEJk32IUI/LaYhQl8sZiFCfy1mIUKfLOYhQr8oPFKFrkR4uZQg2FfFj4+cSwof34GzifQsJdBpMuwj0GkzTCfQaTOsIOg0kg4QGDSSPhIYNJJuERg0kp4TGDSSzhewaTXsITBpNkwi6DSTXhKYNJOOEZg0k74TmDST5hj8GkoPCCwSSi8JLBpKPwisEkpbDPYFJTeM1gElR4w2DSVHjLYJJVeMdgUlb4gcEkrvAjg+0HATzaqlJN1A9XhkxcYpdQ0pbYI5SkJfYJ1cp66e3rLzimAjzfEyA9vHUMI6+z6Q0h8BUuJ5HwHrNpPEIIW+AJ/XUI1pLTwlMvymUxOlJvl8Esx9pSf8drv2nv0B1JtOKAUNKsOCSUJCuOCCXFimNCSbDiPaGkV3FCKMlVdAkltYpTQkms4oxQ0qo4J5SkKnqEklLFBaEkVHFJKOlUXBFKMhV9Qkml4ppQEqm4IZQ0Km4JJYmKO0JJoeIDoSRQ8ZHQ+vlMitUg6A8WvnkyU5WGQHVB1/1IoIrGHWqhgHephcLdoxYKdp9aKKYOtVBEB9RC8RxSC0VzRC0UyzG1UCTvqYXiOKEWiqJLLRTDKbVQBGfUws0/pxZueo9auNkX1MJNvqQWbu4VtXBT+9TCzbymFm7iDbVw826phZt2Ry3crA/Uwk36yO5X1V9V7aW2DPiWSVOH4UGjolq/EYuhbdBN7zGSk2wqPSyCvEdMdDkUbpkEVCc5NVJ1e1lrQBuulIegiyhoVFGgyyho1FGgCyloVFKgSylo1FKgiyloVFOgyylo1FOgCypoVFSgSypo1FSgiypoVFWgyypo1FWgCytoVFagSyto1FagiytoVFegyyto1FegCyxoVFigSyxo1FigiyxoVFmgyyxo1FmgCy1oVFqgSy1o1Fqgiy1oVFugyy1o1FugCy5oVFygSy5o1Fygiy5oVF2gyy5gdRd+fsBEJIspeNN0BEU8V284jXzpeyGkUGAOUu1IoNKHU5WQmm9g+uolQfWiZpEsdEOnQ+UVkjwqIkyETv/6/d3hXCdB/c6IuglmzYZv+zrJxJf4+d29hWPZ45a9ZdtgkmwE8dcmog3qmZjWyn0qo97XjHIZxSOoLAe6UY++7oHHhMyCiS/Uy+v+VGb6cxUUzggbL5HnxqYeY9VldQAjcOxMs8WuQAIPHWtnmqiFQD9Zc41jP4/9AJb16zfdClh6L73q2l3exkvHS16/uFxXsDd8uk32cslze+PUTHK2xg0yLpb2aZxLFBAu6+drTSqQNEfVisYRFE3XIhvLxJ+RpQWadpgsMv3Gk3n0tuolj6dq9k/q+YDLnnSX/HWnk+7KBt74BY1ANZr+Jf7xC9z7ImOWVysbsJeVRKuGEuhtFo8LP1GPqSaPWYFlq/DnwnvR/fHtC/Wqj/6PH9PUvLIqctx/oV81ezGAOGY29jHpS28XEyCGfKp+zTHeIVGvvKna2Dhl1up902wa6pypS+VIwqZ2LzJvlIFy9xjdRzmMIn+r8ZJ1ViSxetK/XHR/fLNsIbMUFLfdxslH3e9tG5crJm9htBa6Pw6idCznzdDJ/UI9MsZjw1fBcgV41go/BC9KvTSrynwJsy1vb5IJtTyZKgCDibePn4hT+FZ4wyy731p3HvKc5+p0zor/QI0XoR4A/h1sqquvGapz0hjiVbtLrVY007+fseijoPrqdcAY5MAfYpzF2eOwAP9+3WCZOgRzOcdjfWAvDFPDpV/g/Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++H7jz7vV/2L65drv1/6w9mpte+0/1/68drTWW7teC9bu1/669re1/3337bvTd/13N8b05z+r+vxuzfl59/nv9mfMZg==</latexit>

0 1 2 3 N

Prototype sentence
I ordered a burger
with fries

Edit vector

tn

zn

p(z)
<latexit sha1_base64="FjCuQsyldJJj37WYC+kn94uQBh0=">AAA1pXiclVtbc9vKkVaSzSar3WxOso/7glrZdZyUrLJ8jitb+xRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/2Z4ZDLoHhHwqqrKE+b6exly+nm6C8DCPIyHfvPm/n/38F//wy3/81a//af2f/+U3//rbb373+xuRTYsAroMszoq7oS8gjlK4lpGM4S4vwE+GMdwO7/cUf1tCIaIs7ct5Dp8SP0yjcRT4EqHb/NWgKJ/+8PmbjTdbb/SPt3qxXV1srFU/vc+/+/5/BqMsmCaQyiD2hfhh+00uPy38QkZBDMv1wVRA7gf3fgg/4GXqJyA+LfR4l95LREbeOCvwXyo9jfIeCz8RYp4M0TLx5UQ0OQW2cT9M5fg/Py2iNJ9KSANzo/E09mTmqcl7o6iAQMZzvPCDIsKxesHEL/xA4hKtr79UP95Z59Y73ekfefudg+Oz4/7x+dmVp6n1toFs4l81DbE5TJbowzv1i3tP4H1wgYWXjb3Az821mnEBYyiKKA3VoEZRGQlrNo7CaQE4oRQegyxJ/HS0GCAYw1guF4sBJN6rLl7/YblcsQlwH6CwVnu61WZXROGkdnapGm1WMsutTT/L2yyGmZRZYo12dWvFrpq3b8385yyG1mL4nEVgLYLnLEbWYqQscBuOcHaxmqHne2ivNh3GGCUjD9cmcX3gtQKXP2x/Qi/DsbexrZw0pz1bLgaJX4QoML9YHBzfNceC144JSqlp0j/fP9f3GUiYSS39RQE4ekX8l7mx67OjXcpJli8GnSbbeUC283mhwnggosR7wOsyn0RLHdl/wV+zlSXr5E+NXrnqJScg/a/3q7vNqm6jV62WD0+4Vq1Dce1yZffMzRuWs6cVy9lf9Om1YrlquGIz0kajdlLd6VWb64en5qyaFmphEW6gM43OGuhco/MGmmg0aapAo2nTdiqVOKY4pNmmZ2e8YpQ7RmrsDZOR6Yi38YexT2vXNFNdmVGbJ8RWnCHGR4fBeaDPOnMY4kkNm16cPULxOsBUtrU+wEjVpxWMN7YX5lz87wG2Fjo82rrjKRBJP97yDvCMFRLzkDpShToIkTceD6zHg6ZHTcvHzN5z4211V+FZIw+nVzXe2h4PU39EXTa+2/h+pdtm3cdefcddfa+nc2WSxVeXAxOKGXyVWZz1aHFgF8T0vrK9r1p6X9peOk8+ZnXy2qoXxtxd6JWpU9szS9N0OCkAmi6Zv43vVj3SqjHf36369lMPcBNU55YlgwczZ2vy/KQdP9M8h8JTfoybTuWm0+Zmxyv8R1r3hrPXr1/7ZRaNvKlQGT8ae3kmRITFmXGdxz5mpMr/86NTRUqOCapljoox3Subv3uSlaO92tHeTzrCOach6NLG2ArjQ8P1iFAqlrauXr9+ViY4Oj8OMyzKJknLPJEzo6uNvjpR5mplpjvW1U6LKyt4ez+cRO3r64dB3+m085OdVhYVCwZZzZypT6FmuOrqa5ti+jfV26v799z+dqb1DXDU6vrZAVeCgyhWYo3VBZYraKCuKn/jOMsKTesrw+vLygCpYbJYKXJkgYFQ1TmBHy/2mwalH0cjbvDZXBfJwlDLFZcgZHsHzSzrGUEuVOmYiyjOUl324dKiiyzxSr+IMImB1Tfmr4Up3NKsSNDriwFCL5Z2OYsG7RMzdJkhMYHLBMSMXGZEDLgMEDN2mTExocuExExcZkJM5DIRMV9c5gsx9y5zT0zsMvFSy7hIvEhgxOLH1tFcHXZmBze9L1MhvVGWfis99fkR5ThXJ4+zMV5S+U5d3yndNXOZjJjcZXJiHlzmgZjCZQpihMsIYqTLSGKmLjMlpnSZkphHl3kkZuYyM2LmLjMn5sllnpamQLMBgJk5q4/3sgqShQml4ZiFTT1uXeUxi6rqq3nGcXhIMIuNMiCYBUY5IphFRQkEs5AoxwSzeChDglkwlBOCWSSUU4JZGJRfCGYxUN4TzAKgjAmOGZwQnDCYLTRf4YxgJuYyJ5gpuXwgmMm4LAhmGi4FwYJvKsGyfU24dEuCmW7LR4KZaMsZwUyx5ZxgJtfyiWCr1U4M6jmUfohStOgWjOhaz2Uwyms9mcHIr/VsBqPB1tMZjBBbz2cwamw9ocFIsvWMBqPL1lMauWfPaTAKbT2pwci09awGo9XmaW25xOUSzj17EoORbutZDEa/racxGBG3nsdglNx6IoORc+uZDEbTracyGGG3nstg1N16MoOReOvZDEbnraczGLG3ns9gFP/8CY2xUERBXaEkOxQfOxQ2yS7BuwzeI3iPwfsE7zO4Q3CHwQcEHzD4kOBDBh8RfMTgY4KPGfye4PcMPiH4hMFdgrsMPiX4lMFnBJ8x+Jzgcwb3CO4x+ILgCwZfEnzJ4CuCrxjcJ7jP4GuCrxl8Q/ANg28JvmXwHcF3DP5A8AcGfyT44/PHqys6MKpjGt1h+tXSY9wu5/Zcbo9z+y63z7mOy3U4d+ByB5w7dLlDzh253BHnjl3umHPvXe49505c7oRzXZfrcu7U5U45d+ZyZ5w7d7lzzvVcrse5C5e74Nyly11y7srlrjjXd7k+565d7ppzNy53w7lbl7vl3J3L3XHug8t94NxHl7Oyv+ElRPkE+nMEfnZ9U/ctsxQW9vOsxZKpgQYJJY26Jla4Ww+rZ6MVoZ+mWriyZoZDg1B5oosTRKgo0SUJIlSKlNUAqQDR5QciVHboogMRKjZ0qYEIlRhlNUg2wi8GoXJCFxOIUBGhSwhEYrY8BqGCQZcLiKRsWQ2SsUUyCJUEuiBAhAoBXQYgQulfJ39EBNsHg1CqL6vdYntVGoTSuk7qiFAy16kcEUrhOoEjQolbp21E2opUtzot/TifqP3Wf2thlsNKM9WDeAPSJzB6YFFRsZ8MR6qHuSAiSyBUuP5LsFaqUqkF0CEi+JsgEYWJ6qr/Emz1XH9LUE1kseDjXyix2haKNaAWCnXEJrVQArUtFOiYWijOkFoozAm1cLhsrCjIL9RCMd6ztVkoEdYzXygB2hYuJltFFF/GlmShRGdbKLoHaqHgCrZSCyW0eoEWSmS2hQvNlhkFVlILxfVILRTWjFooqjm1UFBPy+obZky/M4Pr1Is6o5SrEy4ilGh1mkWE0qtOrohQUtUpFRFKpTqRIkIJVKdPRCht6qSJCCVLnSoRoRSpEyQilBh1WkSE0qFOhohQEtQpEBFKfTrxIUIJT6c7RCjN6SSHCCU3ndoQoZSmExoilMh0GkOE0pdOXohQ0tIpCxFKVTpRIUIJSqcnRCgt6aSECCUjnYoQoRSkExAiH9kOUroY8myR9Ops0WPZIuna0FdMtwr/enJVDCvuysSxVlEfUqHeu9iHIPYLQFFNdtQJhHc0NaAYR+oJKqRBNorSEJ3501ghYlxfJ8uFUA9/r0A+52CYxaOfcjOcLRfNLzcljs98U67TaeVPP7yupiZN2ZkKpn65azHSv9yzGEWA3LcYxYDsWIyiQB5YjOJAHlqMIkEeWYxiQR5bjKJBvrcYxYM8sRhFhOxajGJCnlqMokKeWYziQp5bjCJD9ixGsSEvLEbRIS8tRvEhryxGESL7FqMYkdcWoyiRNxajOJG3FqNIkXcWo1iRHyxG0SI/WswUaijkw8LPJ4YN7cffwPkUEu4ymHQR7jGYpBHuM5jUEXYYTAIJDxhMGgkPGUwyCY8YTEoJjxlMYgnfM5j0Ep4wmCQTdhlMqglPGUzCCc8YTNoJzxlM8gl7DCYFhRcMJhGFlwwmHYVXDCYphX0Gk5rCawaToMIbBpOmwlsGk6zCOwaTssIPDCZxhR8ZbD8I4NFWlWqifrgyZOISu4SStsQeoSQtsU+oVtZLb19/wTEV4PmeAOnhrWMYeZ1NbwiBr3A5iYT3mE3jEULYAk/or0OwlpwWnnpRLovRkXq7DGY51pb6O177TXuH7kiiFQeEkmbFIaEkWXFEKClWHBNKghXvCSW9ihNCSa6iSyipVZwSSmIVZ4SSVsU5oSRV0SOUlCouCCWhiktCSafiilCSqegTSioV14SSSMUNoaRRcUsoSVTcEUoKFR8IJYGKj4TWz2dSrAZBf7DwzZOZqjQEqgu67kcCVTTuUAsFvEstFO4etVCw+9RCMXWohSI6oBaK55BaKJojaqFYjqmFInlPLRTHCbVQFF1qoRhOqYUiOKMWbv45tXDTe9TCzb6gFm7yJbVwc6+ohZvapxZu5jW1cBNvqIWbd0st3LQ7auFmfaAWbtJHdr+q/qpqL7VlwLdMmjoMDxoV1fqNWAxtg256j5GcZFPpYRHkPWKiy6FwyySgOsmpkarby1oD2nClPARdREGjigJdRkGjjgJdSEGjkgJdSkGjlgJdTEGjmgJdTkGjngJdUEGjogJdUkGjpgJdVEGjqgJdVkGjrgJdWEGjsgJdWkGjtgJdXEGjugJdXkGjvgJdYEGjwgJdYkGjxgJdZEGjygJdZkGjzgJdaEGj0gJdakGj1gJdbEGj2gJdbkGj3gJdcEGj4gJdckGj5gJddEGj6gJddgGru/DzAyYiWUzBm6YjKOK5esNp5EvfCyGFAnOQakcClT6cqoTUfAPTVy8Jqhc1i2ShGzodKq+Q5FERYSJ0+tfv7w7nOgnqd0bUTTBrNnzb10kmvsTP7+4tHMset+wt2waTZCOIvzYRbVDPxLRW7lMZ9b5mlMsoHkFlOdCNevR1DzwmZBZMfKFeXvenMtOfq6BwRth4iTw3NvUYqy6rAxiBY2eaLXYFEnjoWDvTRC0E+smaaxz7eewHsKxfv+lWwNJ76VXX7vI2Xjpe8vrF5bqCveHTbbKXS57bG6dmkrM1bpBxsbRP41yigHBZP19rUoGkOapWNI6gaLoW2Vgm/owsLdC0w2SR6TeezKO3VS95PFWzf1LPB1z2pLvkrzuddFc28MYvaASq0fQv8Y9f4N4XGbO8WtmAvawkWjWUQG+zeFz4iXpMNXnMCixbhT8X3ovuj29fqFd99H/8mKbmlVWR4/4L/arZiwHEMbOxj0lferuYADHkU/VrjvEOiXrlTdXGximzVu+bZtNQ50xdKkcSNrV7kXmjDJS7x+g+ymEU+VuNl6yzIonVk/7lovvjm2ULmaWguO02Tj7qfm/buFwxeQujtdD9cRClYzlvhk7uF+qRMR4bvgqWK8CzVvgheFHqpVlV5kuYbXl7k0yo5clUARhMvH38RJzCt8IbZtn91rrzkOc8V6dzVvwRNV6EegD4d7Cprr5mqM5JY4hX7S61WtFM/37Goo+C6qvXAWOQA3+IcRZnj8MC/Pt1g2XqEMzlHI/1gb0wTA2XfoHzn+Dhv77++ZuN7eb/VFq9uHm7tf1ma/vi+40/71b/i+nXa/++9h9rr9a21/609ue1o7Xe2vVasHa/9te1v63977tv352+67+7MaY//1nV59/WnJ93n/8fNe3MbA==</latexit>N

Guu et al. 2018

Carnegie Mellon University

Previous ApproachPrototype
distributionp(t)Uniform<latexit sha1_base64="BqEu/64RuYCyn2jfc6RbHK9rVtg=">AAA1pXiclVtbc9vKkVay2d1Eye6ebB7zgorsOs6WrLJ8jmu38hRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/Sc8MBt0DQj61qrKE+b6exly+nm6C8DCPIyHfvPm/n/38H37xj//0z7/81fqvf/Mv//pv3/z2329ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9T/G0JhYiytC/nOXxK/DCNxlHgS4Ru81eDopR//PzNxputN/rHW73Yri421qqf3ufffv8/g1EWTBNIZRD7Qvyw/SaXnxZ+IaMghuX6YCog94N7P4Qf8DL1ExCfFnq8S+8lIiNvnBX4L5WeRnmPhZ8IMU+GaJn4ciKanALbuB+mcvxfnxZRmk8lpIG50XgaezLz1OS9UVRAIOM5XvhBEeFYvWDiF34gcYnW11+qH++sc+ud7vSPvP3OwfHZcf/4/OzK09R620A28a+ahtgcJkv04Z36xb0n8D64wMLLxl7g5+ZazbiAMRRFlIZqUKOojIQ1G0fhtACcUAqPQZYkfjpaDBCMYSyXi8UAEu9VF6//uFyu2AS4D1BYqz3darMronBSO7tUjTYrmeXWpp/lbRbDTMossUa7urViV83bt2b+cxZDazF8ziKwFsFzFiNrMVIWuA1HOLtYzdDzPbRXmw5jjJKRh2uTuD7wWoHLH7Y/oZfh2NvYVk6a054tF4PEL0IUmF8sDo7vmmPBa8cEpdQ06Z/vn+v7DCTMpJb+ogAcvSL+ZG7s+uxol3KS5YtBp8l2HpDtfF5gGD8NRJR4D3hd5pNoqSL76S/4a7ayZJ38qdErV73kBKT/9X51t1nVbfSq1fLhCdeqdSiuXa7snrl5w3L2tGI5U5ZPq5arhis2I200aifVnV61uX54as6qaaEWFuEGOtPorIHONTpvoIlGk6YKNJo2badSiWOKQ5ptenbGK0a5Y6TG3jAZmY54G38Y+7R2TTPVlRm1eUJsxRlifHQYnAf6rDOHIZ7UsOnF2SMUrwNMZVvrA4xUfVrBeGN7Yc7F/x5ga6HDo607ngKR9OMt7wDPWCExD6kjVaiDEHnj8cB6PGh61LR8zOw9N95WdxWeNfJwelXjre3xMPVH1GXju43vV7pt1n3s1Xfc1fd6OlcmWXx1OTChmMFXmcVZjxYHdkFM7yvb+6ql96XtpfPkY1Ynr616YczdhV6ZOrU9szRNh5MCoOmS+dv4btUjrRrz/d2qbz/1ADdBdW5ZMngwc7Ymz0/a8TPNcyg85ce46VRuOm1udrzCf6R1bzh7/fq1X2bRyJsKlfGjsZdnQkRYnBnXeexjRqr8Pz86VaTkmKBa5qgY072y+X9PsnK0Vzva+0lHOOc0BF3aGFthfGi4HhFKxdLW1evXz8oER+fHYYZF2SRpmSdyZnS10VcnylytzHTHutppcWUFb++Hk6h9ff0w6Duddn6y08qiYsEgq5kz9SnUDFddfW1TTP+ment1/57b3860vgGOWl0/O+BKcBDFSqyxusByBQ3UVeVvHGdZoWl9ZXh9WRkgNUwWK0WOLDAQqjon8OPFftOg9ONoxA0+m+siWRhqueIShGzvoJllPSPIhSodcxHFWarLPlxadJElXukXESYxsPrG/LUwhVuaFQl6fTFA6MXSLmfRoH1ihi4zJCZwmYCYkcuMiAGXAWLGLjMmJnSZkJiJy0yIiVwmIuaLy3wh5t5l7omJXSZeahkXiRcJjFj82Dqaq8PO7OCm92UqpDfK0m+lpz4/ohzn6uRxNsZLKt+p6zulu2YukxGTu0xOzIPLPBBTuExBjHAZQYx0GUnM1GWmxJQuUxLz6DKPxMxcZkbM3GXmxDy5zNPSFGg2ADAzZ/XxXlZBsjChNByzsKnHras8ZlFVfTXPOA4PCWaxUQYEs8AoRwSzqCiBYBYS5ZhgFg9lSDALhnJCMIuEckowC4PyC8EsBsp7glkAlDHBMYMTghMGs4XmK5wRzMRc5gQzJZcPBDMZlwXBTMOlIFjwTSVYtq8Jl25JMNNt+UgwE205I5gptpwTzORaPhFstdqJQT2H0g9RihbdghFd67kMRnmtJzMY+bWezWA02Ho6gxFi6/kMRo2tJzQYSbae0WB02XpKI/fsOQ1Goa0nNRiZtp7VYLTaPK0tl7hcwrlnT2Iw0m09i8Hot/U0BiPi1vMYjJJbT2Qwcm49k8FouvVUBiPs1nMZjLpbT2YwEm89m8HovPV0BiP21vMZjOKfP6ExFoooqCuUZIfiY4fCJtkleJfBewTvMXif4H0GdwjuMPiA4AMGHxJ8yOAjgo8YfEzwMYPfE/yewScEnzC4S3CXwacEnzL4jOAzBp8TfM7gHsE9Bl8QfMHgS4IvGXxF8BWD+wT3GXxN8DWDbwi+YfAtwbcMviP4jsEfCP7A4I8Ef3z+eHVFB0Z1TKM7TL9aeozb5dyey+1xbt/l9jnXcbkO5w5c7oBzhy53yLkjlzvi3LHLHXPuvcu959yJy51wrutyXc6dutwp585c7oxz5y53zrmey/U4d+FyF5y7dLlLzl253BXn+i7X59y1y11z7sblbjh363K3nLtzuTvOfXC5D5z76HJW9je8hCifQH+OwM+ub+q+ZZbCwn6etVgyNdAgoaRR18QKd+th9Wy0IvTTVAtX1sxwaBAqT3RxgggVJbokQYRKkbIaIBUguvxAhMoOXXQgQsWGLjUQoRKjrAbJRvjFIFRO6GICESoidAmBSMyWxyBUMOhyAZGULatBMrZIBqGSQBcEiFAhoMsARCj96+SPiGD7YBBK9WW1W2yvSoNQWtdJHRFK5jqVI0IpXCdwRChx67SNSFuR6lanpR/nE7Xf+m8tzHJYaaZ6EG9A+gRGDywqKvaT4Uj1MBdEZAmECtd/CdZKVSq1ADpEBH8TJKIwUV31X4KtnutvCaqJLBZ8/AslVttCsQbUQqGO2KQWSqC2hQIdUwvFGVILhTmhFg6XjRUF+YVaKMZ7tjYLJcJ65gslQNvCxWSriOLL2JIslOhsC0X3QC0UXMFWaqGEVi/QQonMtnCh2TKjwEpqobgeqYXCmlELRTWnFgrqaVl9w4zpd2ZwnXpRZ5RydcJFhBKtTrOIUHrVyRURSqo6pSJCqVQnUkQoger0iQilTZ00EaFkqVMlIpQidYJEhBKjTouIUDrUyRARSoI6BSJCqU8nPkQo4el0hwilOZ3kEKHkplMbIpTSdEJDhBKZTmOIUPrSyQsRSlo6ZSFCqUonKkQoQen0hAilJZ2UEKFkpFMRIpSCdAJC5CPbQUoXQ54tkl6dLXosWyRdG/qK6VbhX0+uimHFXZk41irqQyrUexf7EMR+ASiqyY46gfCOpgYU40g9QYU0yEZRGqIzfxorRIzr62S5EOrh7xXI5xwMs3j0U26Gs+Wi+eWmxPGZb8p1Oq386YfX1dSkKTtTwdQvdy1G+pd7FqMIkPsWoxiQHYtRFMgDi1EcyEOLUSTII4tRLMhji1E0yPcWo3iQJxajiJBdi1FMyFOLUVTIM4tRXMhzi1FkyJ7FKDbkhcUoOuSlxSg+5JXFKEJk32IUI/LaYhQl8sZiFCfy1mIUKfLOYhQr8oPFKFrkR4uZQg2FfFj4+cSwof34GzifQsJdBpMuwj0GkzTCfQaTOsIOg0kg4QGDSSPhIYNJJuERg0kp4TGDSSzhewaTXsITBpNkwi6DSTXhKYNJOOEZg0k74TmDST5hj8GkoPCCwSSi8JLBpKPwisEkpbDPYFJTeM1gElR4w2DSVHjLYJJVeMdgUlb4gcEkrvAjg+0HATzaqlJN1A9XhkxcYpdQ0pbYI5SkJfYJ1cp66e3rLzimAjzfEyA9vHUMI6+z6Q0h8BUuJ5HwHrNpPEIIW+AJ/XUI1pLTwlMvymUxOlJvl8Esx9pSf8drv2nv0B1JtOKAUNKsOCSUJCuOCCXFimNCSbDiPaGkV3FCKMlVdAkltYpTQkms4oxQ0qo4J5SkKnqEklLFBaEkVHFJKOlUXBFKMhV9Qkml4ppQEqm4IZQ0Km4JJYmKO0JJoeIDoSRQ8ZHQ+vlMitUg6A8WvnkyU5WGQHVB1/1IoIrGHWqhgHephcLdoxYKdp9aKKYOtVBEB9RC8RxSC0VzRC0UyzG1UCTvqYXiOKEWiqJLLRTDKbVQBGfUws0/pxZueo9auNkX1MJNvqQWbu4VtXBT+9TCzbymFm7iDbVw826phZt2Ry3crA/Uwk36yO5X1V9V7aW2DPiWSVOH4UGjolq/EYuhbdBN7zGSk2wqPSyCvEdMdDkUbpkEVCc5NVJ1e1lrQBuulIegiyhoVFGgyyho1FGgCyloVFKgSylo1FKgiyloVFOgyylo1FOgCypoVFSgSypo1FSgiypoVFWgyypo1FWgCytoVFagSyto1FagiytoVFegyyto1FegCyxoVFigSyxo1FigiyxoVFmgyyxo1FmgCy1oVFqgSy1o1Fqgiy1oVFugyy1o1FugCy5oVFygSy5o1Fygiy5oVF2gyy5gdRd+fsBEJIspeNN0BEU8V284jXzpeyGkUGAOUu1IoNKHU5WQmm9g+uolQfWiZpEsdEOnQ+UVkjwqIkyETv/6/d3hXCdB/c6IuglmzYZv+zrJxJf4+d29hWPZ45a9ZdtgkmwE8dcmog3qmZjWyn0qo97XjHIZxSOoLAe6UY++7oHHhMyCiS/Uy+v+VGb6cxUUzggbL5HnxqYeY9VldQAjcOxMs8WuQAIPHWtnmqiFQD9Zc41jP4/9AJb16zfdClh6L73q2l3exkvHS16/uFxXsDd8uk32cslze+PUTHK2xg0yLpb2aZxLFBAu6+drTSqQNEfVisYRFE3XIhvLxJ+RpQWadpgsMv3Gk3n0tuolj6dq9k/q+YDLnnSX/HWnk+7KBt74BY1ANZr+Jf7xC9z7ImOWVysbsJeVRKuGEuhtFo8LP1GPqSaPWYFlq/DnwnvR/fHtC/Wqj/6PH9PUvLIqctx/oV81ezGAOGY29jHpS28XEyCGfKp+zTHeIVGvvKna2Dhl1up902wa6pypS+VIwqZ2LzJvlIFy9xjdRzmMIn+r8ZJ1ViSxetK/XHR/fLNsIbMUFLfdxslH3e9tG5crJm9htBa6Pw6idCznzdDJ/UI9MsZjw1fBcgV41go/BC9KvTSrynwJsy1vb5IJtTyZKgCDibePn4hT+FZ4wyy731p3HvKc5+p0zor/QI0XoR4A/h1sqquvGapz0hjiVbtLrVY007+fseijoPrqdcAY5MAfYpzF2eOwAP9+3WCZOgRzOcdjfWAvDFPDpV/g/Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++H7jz7vV/2L65drv1/6w9mpte+0/1/68drTWW7teC9bu1/669re1/3337bvTd/13N8b05z+r+vxuzfl59/nv9mfMZg==</latexit>

0 1 2 3 N

Prototype sentence
I ordered a burger
with fries

Observed sentence
I ordered a pizzawith sauce

Edit vector

tn
xn

zn

p(z)
<latexit sha1_base64="FjCuQsyldJJj37WYC+kn94uQBh0=">AAA1pXiclVtbc9vKkVaSzSar3WxOso/7glrZdZyUrLJ8jitb+xRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/2Z4ZDLoHhHwqqrKE+b6exly+nm6C8DCPIyHfvPm/n/38F//wy3/81a//af2f/+U3//rbb373+xuRTYsAroMszoq7oS8gjlK4lpGM4S4vwE+GMdwO7/cUf1tCIaIs7ct5Dp8SP0yjcRT4EqHb/NWgKJ/+8PmbjTdbb/SPt3qxXV1srFU/vc+/+/5/BqMsmCaQyiD2hfhh+00uPy38QkZBDMv1wVRA7gf3fgg/4GXqJyA+LfR4l95LREbeOCvwXyo9jfIeCz8RYp4M0TLx5UQ0OQW2cT9M5fg/Py2iNJ9KSANzo/E09mTmqcl7o6iAQMZzvPCDIsKxesHEL/xA4hKtr79UP95Z59Y73ekfefudg+Oz4/7x+dmVp6n1toFs4l81DbE5TJbowzv1i3tP4H1wgYWXjb3Az821mnEBYyiKKA3VoEZRGQlrNo7CaQE4oRQegyxJ/HS0GCAYw1guF4sBJN6rLl7/YblcsQlwH6CwVnu61WZXROGkdnapGm1WMsutTT/L2yyGmZRZYo12dWvFrpq3b8385yyG1mL4nEVgLYLnLEbWYqQscBuOcHaxmqHne2ivNh3GGCUjD9cmcX3gtQKXP2x/Qi/DsbexrZw0pz1bLgaJX4QoML9YHBzfNceC144JSqlp0j/fP9f3GUiYSS39RQE4ekX8l7mx67OjXcpJli8GnSbbeUC283mhwnggosR7wOsyn0RLHdl/wV+zlSXr5E+NXrnqJScg/a/3q7vNqm6jV62WD0+4Vq1Dce1yZffMzRuWs6cVy9lf9Om1YrlquGIz0kajdlLd6VWb64en5qyaFmphEW6gM43OGuhco/MGmmg0aapAo2nTdiqVOKY4pNmmZ2e8YpQ7RmrsDZOR6Yi38YexT2vXNFNdmVGbJ8RWnCHGR4fBeaDPOnMY4kkNm16cPULxOsBUtrU+wEjVpxWMN7YX5lz87wG2Fjo82rrjKRBJP97yDvCMFRLzkDpShToIkTceD6zHg6ZHTcvHzN5z4211V+FZIw+nVzXe2h4PU39EXTa+2/h+pdtm3cdefcddfa+nc2WSxVeXAxOKGXyVWZz1aHFgF8T0vrK9r1p6X9peOk8+ZnXy2qoXxtxd6JWpU9szS9N0OCkAmi6Zv43vVj3SqjHf36369lMPcBNU55YlgwczZ2vy/KQdP9M8h8JTfoybTuWm0+Zmxyv8R1r3hrPXr1/7ZRaNvKlQGT8ae3kmRITFmXGdxz5mpMr/86NTRUqOCapljoox3Subv3uSlaO92tHeTzrCOach6NLG2ArjQ8P1iFAqlrauXr9+ViY4Oj8OMyzKJknLPJEzo6uNvjpR5mplpjvW1U6LKyt4ez+cRO3r64dB3+m085OdVhYVCwZZzZypT6FmuOrqa5ti+jfV26v799z+dqb1DXDU6vrZAVeCgyhWYo3VBZYraKCuKn/jOMsKTesrw+vLygCpYbJYKXJkgYFQ1TmBHy/2mwalH0cjbvDZXBfJwlDLFZcgZHsHzSzrGUEuVOmYiyjOUl324dKiiyzxSr+IMImB1Tfmr4Up3NKsSNDriwFCL5Z2OYsG7RMzdJkhMYHLBMSMXGZEDLgMEDN2mTExocuExExcZkJM5DIRMV9c5gsx9y5zT0zsMvFSy7hIvEhgxOLH1tFcHXZmBze9L1MhvVGWfis99fkR5ThXJ4+zMV5S+U5d3yndNXOZjJjcZXJiHlzmgZjCZQpihMsIYqTLSGKmLjMlpnSZkphHl3kkZuYyM2LmLjMn5sllnpamQLMBgJk5q4/3sgqShQml4ZiFTT1uXeUxi6rqq3nGcXhIMIuNMiCYBUY5IphFRQkEs5AoxwSzeChDglkwlBOCWSSUU4JZGJRfCGYxUN4TzAKgjAmOGZwQnDCYLTRf4YxgJuYyJ5gpuXwgmMm4LAhmGi4FwYJvKsGyfU24dEuCmW7LR4KZaMsZwUyx5ZxgJtfyiWCr1U4M6jmUfohStOgWjOhaz2Uwyms9mcHIr/VsBqPB1tMZjBBbz2cwamw9ocFIsvWMBqPL1lMauWfPaTAKbT2pwci09awGo9XmaW25xOUSzj17EoORbutZDEa/racxGBG3nsdglNx6IoORc+uZDEbTracyGGG3nstg1N16MoOReOvZDEbnraczGLG3ns9gFP/8CY2xUERBXaEkOxQfOxQ2yS7BuwzeI3iPwfsE7zO4Q3CHwQcEHzD4kOBDBh8RfMTgY4KPGfye4PcMPiH4hMFdgrsMPiX4lMFnBJ8x+Jzgcwb3CO4x+ILgCwZfEnzJ4CuCrxjcJ7jP4GuCrxl8Q/ANg28JvmXwHcF3DP5A8AcGfyT44/PHqys6MKpjGt1h+tXSY9wu5/Zcbo9z+y63z7mOy3U4d+ByB5w7dLlDzh253BHnjl3umHPvXe49505c7oRzXZfrcu7U5U45d+ZyZ5w7d7lzzvVcrse5C5e74Nyly11y7srlrjjXd7k+565d7ppzNy53w7lbl7vl3J3L3XHug8t94NxHl7Oyv+ElRPkE+nMEfnZ9U/ctsxQW9vOsxZKpgQYJJY26Jla4Ww+rZ6MVoZ+mWriyZoZDg1B5oosTRKgo0SUJIlSKlNUAqQDR5QciVHboogMRKjZ0qYEIlRhlNUg2wi8GoXJCFxOIUBGhSwhEYrY8BqGCQZcLiKRsWQ2SsUUyCJUEuiBAhAoBXQYgQulfJ39EBNsHg1CqL6vdYntVGoTSuk7qiFAy16kcEUrhOoEjQolbp21E2opUtzot/TifqP3Wf2thlsNKM9WDeAPSJzB6YFFRsZ8MR6qHuSAiSyBUuP5LsFaqUqkF0CEi+JsgEYWJ6qr/Emz1XH9LUE1kseDjXyix2haKNaAWCnXEJrVQArUtFOiYWijOkFoozAm1cLhsrCjIL9RCMd6ztVkoEdYzXygB2hYuJltFFF/GlmShRGdbKLoHaqHgCrZSCyW0eoEWSmS2hQvNlhkFVlILxfVILRTWjFooqjm1UFBPy+obZky/M4Pr1Is6o5SrEy4ilGh1mkWE0qtOrohQUtUpFRFKpTqRIkIJVKdPRCht6qSJCCVLnSoRoRSpEyQilBh1WkSE0qFOhohQEtQpEBFKfTrxIUIJT6c7RCjN6SSHCCU3ndoQoZSmExoilMh0GkOE0pdOXohQ0tIpCxFKVTpRIUIJSqcnRCgt6aSECCUjnYoQoRSkExAiH9kOUroY8myR9Ops0WPZIuna0FdMtwr/enJVDCvuysSxVlEfUqHeu9iHIPYLQFFNdtQJhHc0NaAYR+oJKqRBNorSEJ3501ghYlxfJ8uFUA9/r0A+52CYxaOfcjOcLRfNLzcljs98U67TaeVPP7yupiZN2ZkKpn65azHSv9yzGEWA3LcYxYDsWIyiQB5YjOJAHlqMIkEeWYxiQR5bjKJBvrcYxYM8sRhFhOxajGJCnlqMokKeWYziQp5bjCJD9ixGsSEvLEbRIS8tRvEhryxGESL7FqMYkdcWoyiRNxajOJG3FqNIkXcWo1iRHyxG0SI/WswUaijkw8LPJ4YN7cffwPkUEu4ymHQR7jGYpBHuM5jUEXYYTAIJDxhMGgkPGUwyCY8YTEoJjxlMYgnfM5j0Ep4wmCQTdhlMqglPGUzCCc8YTNoJzxlM8gl7DCYFhRcMJhGFlwwmHYVXDCYphX0Gk5rCawaToMIbBpOmwlsGk6zCOwaTssIPDCZxhR8ZbD8I4NFWlWqifrgyZOISu4SStsQeoSQtsU+oVtZLb19/wTEV4PmeAOnhrWMYeZ1NbwiBr3A5iYT3mE3jEULYAk/or0OwlpwWnnpRLovRkXq7DGY51pb6O177TXuH7kiiFQeEkmbFIaEkWXFEKClWHBNKghXvCSW9ihNCSa6iSyipVZwSSmIVZ4SSVsU5oSRV0SOUlCouCCWhiktCSafiilCSqegTSioV14SSSMUNoaRRcUsoSVTcEUoKFR8IJYGKj4TWz2dSrAZBf7DwzZOZqjQEqgu67kcCVTTuUAsFvEstFO4etVCw+9RCMXWohSI6oBaK55BaKJojaqFYjqmFInlPLRTHCbVQFF1qoRhOqYUiOKMWbv45tXDTe9TCzb6gFm7yJbVwc6+ohZvapxZu5jW1cBNvqIWbd0st3LQ7auFmfaAWbtJHdr+q/qpqL7VlwLdMmjoMDxoV1fqNWAxtg256j5GcZFPpYRHkPWKiy6FwyySgOsmpkarby1oD2nClPARdREGjigJdRkGjjgJdSEGjkgJdSkGjlgJdTEGjmgJdTkGjngJdUEGjogJdUkGjpgJdVEGjqgJdVkGjrgJdWEGjsgJdWkGjtgJdXEGjugJdXkGjvgJdYEGjwgJdYkGjxgJdZEGjygJdZkGjzgJdaEGj0gJdakGj1gJdbEGj2gJdbkGj3gJdcEGj4gJdckGj5gJddEGj6gJddgGru/DzAyYiWUzBm6YjKOK5esNp5EvfCyGFAnOQakcClT6cqoTUfAPTVy8Jqhc1i2ShGzodKq+Q5FERYSJ0+tfv7w7nOgnqd0bUTTBrNnzb10kmvsTP7+4tHMset+wt2waTZCOIvzYRbVDPxLRW7lMZ9b5mlMsoHkFlOdCNevR1DzwmZBZMfKFeXvenMtOfq6BwRth4iTw3NvUYqy6rAxiBY2eaLXYFEnjoWDvTRC0E+smaaxz7eewHsKxfv+lWwNJ76VXX7vI2Xjpe8vrF5bqCveHTbbKXS57bG6dmkrM1bpBxsbRP41yigHBZP19rUoGkOapWNI6gaLoW2Vgm/owsLdC0w2SR6TeezKO3VS95PFWzf1LPB1z2pLvkrzuddFc28MYvaASq0fQv8Y9f4N4XGbO8WtmAvawkWjWUQG+zeFz4iXpMNXnMCixbhT8X3ovuj29fqFd99H/8mKbmlVWR4/4L/arZiwHEMbOxj0lferuYADHkU/VrjvEOiXrlTdXGximzVu+bZtNQ50xdKkcSNrV7kXmjDJS7x+g+ymEU+VuNl6yzIonVk/7lovvjm2ULmaWguO02Tj7qfm/buFwxeQujtdD9cRClYzlvhk7uF+qRMR4bvgqWK8CzVvgheFHqpVlV5kuYbXl7k0yo5clUARhMvH38RJzCt8IbZtn91rrzkOc8V6dzVvwRNV6EegD4d7Cprr5mqM5JY4hX7S61WtFM/37Goo+C6qvXAWOQA3+IcRZnj8MC/Pt1g2XqEMzlHI/1gb0wTA2XfoHzn+Dhv77++ZuN7eb/VFq9uHm7tf1ma/vi+40/71b/i+nXa/++9h9rr9a21/609ue1o7Xe2vVasHa/9te1v63977tv352+67+7MaY//1nV59/WnJ93n/8fNe3MbA==</latexit>N

Guu et al. 2018

Carnegie Mellon University

Proposed Model:
Sparse Prototypes for Text Generacon(He et al. NeurIPS 2020)23

Carnegie Mellon University

Prototypes can be Sparse
Prototype Sentence

Data Examples
I ordered a pizzawith sauce

I ordered a turkey
with sauce
I ordered a burger
with fries
I ordered a sandwichwith fries and coke
I ordered a lobster rollwith oysters and spicy
sauce

Guu et al. 2018

Carnegie Mellon University

Sparse Neural Editor
Dirichlet
hyperparamθPrototype sentence
I ordered a burger
with fries

Observed sentence
I ordered a pizzawith sauceαEdit vector

tn
xn

zn

p(z)
<latexit sha1_base64="FjCuQsyldJJj37WYC+kn94uQBh0=">AAA1pXiclVtbc9vKkVaSzSar3WxOso/7glrZdZyUrLJ8jitb+xRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/2Z4ZDLoHhHwqqrKE+b6exly+nm6C8DCPIyHfvPm/n/38F//wy3/81a//af2f/+U3//rbb373+xuRTYsAroMszoq7oS8gjlK4lpGM4S4vwE+GMdwO7/cUf1tCIaIs7ct5Dp8SP0yjcRT4EqHb/NWgKJ/+8PmbjTdbb/SPt3qxXV1srFU/vc+/+/5/BqMsmCaQyiD2hfhh+00uPy38QkZBDMv1wVRA7gf3fgg/4GXqJyA+LfR4l95LREbeOCvwXyo9jfIeCz8RYp4M0TLx5UQ0OQW2cT9M5fg/Py2iNJ9KSANzo/E09mTmqcl7o6iAQMZzvPCDIsKxesHEL/xA4hKtr79UP95Z59Y73ekfefudg+Oz4/7x+dmVp6n1toFs4l81DbE5TJbowzv1i3tP4H1wgYWXjb3Az821mnEBYyiKKA3VoEZRGQlrNo7CaQE4oRQegyxJ/HS0GCAYw1guF4sBJN6rLl7/YblcsQlwH6CwVnu61WZXROGkdnapGm1WMsutTT/L2yyGmZRZYo12dWvFrpq3b8385yyG1mL4nEVgLYLnLEbWYqQscBuOcHaxmqHne2ivNh3GGCUjD9cmcX3gtQKXP2x/Qi/DsbexrZw0pz1bLgaJX4QoML9YHBzfNceC144JSqlp0j/fP9f3GUiYSS39RQE4ekX8l7mx67OjXcpJli8GnSbbeUC283mhwnggosR7wOsyn0RLHdl/wV+zlSXr5E+NXrnqJScg/a/3q7vNqm6jV62WD0+4Vq1Dce1yZffMzRuWs6cVy9lf9Om1YrlquGIz0kajdlLd6VWb64en5qyaFmphEW6gM43OGuhco/MGmmg0aapAo2nTdiqVOKY4pNmmZ2e8YpQ7RmrsDZOR6Yi38YexT2vXNFNdmVGbJ8RWnCHGR4fBeaDPOnMY4kkNm16cPULxOsBUtrU+wEjVpxWMN7YX5lz87wG2Fjo82rrjKRBJP97yDvCMFRLzkDpShToIkTceD6zHg6ZHTcvHzN5z4211V+FZIw+nVzXe2h4PU39EXTa+2/h+pdtm3cdefcddfa+nc2WSxVeXAxOKGXyVWZz1aHFgF8T0vrK9r1p6X9peOk8+ZnXy2qoXxtxd6JWpU9szS9N0OCkAmi6Zv43vVj3SqjHf36369lMPcBNU55YlgwczZ2vy/KQdP9M8h8JTfoybTuWm0+Zmxyv8R1r3hrPXr1/7ZRaNvKlQGT8ae3kmRITFmXGdxz5mpMr/86NTRUqOCapljoox3Subv3uSlaO92tHeTzrCOach6NLG2ArjQ8P1iFAqlrauXr9+ViY4Oj8OMyzKJknLPJEzo6uNvjpR5mplpjvW1U6LKyt4ez+cRO3r64dB3+m085OdVhYVCwZZzZypT6FmuOrqa5ti+jfV26v799z+dqb1DXDU6vrZAVeCgyhWYo3VBZYraKCuKn/jOMsKTesrw+vLygCpYbJYKXJkgYFQ1TmBHy/2mwalH0cjbvDZXBfJwlDLFZcgZHsHzSzrGUEuVOmYiyjOUl324dKiiyzxSr+IMImB1Tfmr4Up3NKsSNDriwFCL5Z2OYsG7RMzdJkhMYHLBMSMXGZEDLgMEDN2mTExocuExExcZkJM5DIRMV9c5gsx9y5zT0zsMvFSy7hIvEhgxOLH1tFcHXZmBze9L1MhvVGWfis99fkR5ThXJ4+zMV5S+U5d3yndNXOZjJjcZXJiHlzmgZjCZQpihMsIYqTLSGKmLjMlpnSZkphHl3kkZuYyM2LmLjMn5sllnpamQLMBgJk5q4/3sgqShQml4ZiFTT1uXeUxi6rqq3nGcXhIMIuNMiCYBUY5IphFRQkEs5AoxwSzeChDglkwlBOCWSSUU4JZGJRfCGYxUN4TzAKgjAmOGZwQnDCYLTRf4YxgJuYyJ5gpuXwgmMm4LAhmGi4FwYJvKsGyfU24dEuCmW7LR4KZaMsZwUyx5ZxgJtfyiWCr1U4M6jmUfohStOgWjOhaz2Uwyms9mcHIr/VsBqPB1tMZjBBbz2cwamw9ocFIsvWMBqPL1lMauWfPaTAKbT2pwci09awGo9XmaW25xOUSzj17EoORbutZDEa/racxGBG3nsdglNx6IoORc+uZDEbTracyGGG3nstg1N16MoOReOvZDEbnraczGLG3ns9gFP/8CY2xUERBXaEkOxQfOxQ2yS7BuwzeI3iPwfsE7zO4Q3CHwQcEHzD4kOBDBh8RfMTgY4KPGfye4PcMPiH4hMFdgrsMPiX4lMFnBJ8x+Jzgcwb3CO4x+ILgCwZfEnzJ4CuCrxjcJ7jP4GuCrxl8Q/ANg28JvmXwHcF3DP5A8AcGfyT44/PHqys6MKpjGt1h+tXSY9wu5/Zcbo9z+y63z7mOy3U4d+ByB5w7dLlDzh253BHnjl3umHPvXe49505c7oRzXZfrcu7U5U45d+ZyZ5w7d7lzzvVcrse5C5e74Nyly11y7srlrjjXd7k+565d7ppzNy53w7lbl7vl3J3L3XHug8t94NxHl7Oyv+ElRPkE+nMEfnZ9U/ctsxQW9vOsxZKpgQYJJY26Jla4Ww+rZ6MVoZ+mWriyZoZDg1B5oosTRKgo0SUJIlSKlNUAqQDR5QciVHboogMRKjZ0qYEIlRhlNUg2wi8GoXJCFxOIUBGhSwhEYrY8BqGCQZcLiKRsWQ2SsUUyCJUEuiBAhAoBXQYgQulfJ39EBNsHg1CqL6vdYntVGoTSuk7qiFAy16kcEUrhOoEjQolbp21E2opUtzot/TifqP3Wf2thlsNKM9WDeAPSJzB6YFFRsZ8MR6qHuSAiSyBUuP5LsFaqUqkF0CEi+JsgEYWJ6qr/Emz1XH9LUE1kseDjXyix2haKNaAWCnXEJrVQArUtFOiYWijOkFoozAm1cLhsrCjIL9RCMd6ztVkoEdYzXygB2hYuJltFFF/GlmShRGdbKLoHaqHgCrZSCyW0eoEWSmS2hQvNlhkFVlILxfVILRTWjFooqjm1UFBPy+obZky/M4Pr1Is6o5SrEy4ilGh1mkWE0qtOrohQUtUpFRFKpTqRIkIJVKdPRCht6qSJCCVLnSoRoRSpEyQilBh1WkSE0qFOhohQEtQpEBFKfTrxIUIJT6c7RCjN6SSHCCU3ndoQoZSmExoilMh0GkOE0pdOXohQ0tIpCxFKVTpRIUIJSqcnRCgt6aSECCUjnYoQoRSkExAiH9kOUroY8myR9Ops0WPZIuna0FdMtwr/enJVDCvuysSxVlEfUqHeu9iHIPYLQFFNdtQJhHc0NaAYR+oJKqRBNorSEJ3501ghYlxfJ8uFUA9/r0A+52CYxaOfcjOcLRfNLzcljs98U67TaeVPP7yupiZN2ZkKpn65azHSv9yzGEWA3LcYxYDsWIyiQB5YjOJAHlqMIkEeWYxiQR5bjKJBvrcYxYM8sRhFhOxajGJCnlqMokKeWYziQp5bjCJD9ixGsSEvLEbRIS8tRvEhryxGESL7FqMYkdcWoyiRNxajOJG3FqNIkXcWo1iRHyxG0SI/WswUaijkw8LPJ4YN7cffwPkUEu4ymHQR7jGYpBHuM5jUEXYYTAIJDxhMGgkPGUwyCY8YTEoJjxlMYgnfM5j0Ep4wmCQTdhlMqglPGUzCCc8YTNoJzxlM8gl7DCYFhRcMJhGFlwwmHYVXDCYphX0Gk5rCawaToMIbBpOmwlsGk6zCOwaTssIPDCZxhR8ZbD8I4NFWlWqifrgyZOISu4SStsQeoSQtsU+oVtZLb19/wTEV4PmeAOnhrWMYeZ1NbwiBr3A5iYT3mE3jEULYAk/or0OwlpwWnnpRLovRkXq7DGY51pb6O177TXuH7kiiFQeEkmbFIaEkWXFEKClWHBNKghXvCSW9ihNCSa6iSyipVZwSSmIVZ4SSVsU5oSRV0SOUlCouCCWhiktCSafiilCSqegTSioV14SSSMUNoaRRcUsoSVTcEUoKFR8IJYGKj4TWz2dSrAZBf7DwzZOZqjQEqgu67kcCVTTuUAsFvEstFO4etVCw+9RCMXWohSI6oBaK55BaKJojaqFYjqmFInlPLRTHCbVQFF1qoRhOqYUiOKMWbv45tXDTe9TCzb6gFm7yJbVwc6+ohZvapxZu5jW1cBNvqIWbd0st3LQ7auFmfaAWbtJHdr+q/qpqL7VlwLdMmjoMDxoV1fqNWAxtg256j5GcZFPpYRHkPWKiy6FwyySgOsmpkarby1oD2nClPARdREGjigJdRkGjjgJdSEGjkgJdSkGjlgJdTEGjmgJdTkGjngJdUEGjogJdUkGjpgJdVEGjqgJdVkGjrgJdWEGjsgJdWkGjtgJdXEGjugJdXkGjvgJdYEGjwgJdYkGjxgJdZEGjygJdZkGjzgJdaEGj0gJdakGj1gJdbEGj2gJdbkGj3gJdcEGj4gJdckGj5gJddEGj6gJddgGru/DzAyYiWUzBm6YjKOK5esNp5EvfCyGFAnOQakcClT6cqoTUfAPTVy8Jqhc1i2ShGzodKq+Q5FERYSJ0+tfv7w7nOgnqd0bUTTBrNnzb10kmvsTP7+4tHMset+wt2waTZCOIvzYRbVDPxLRW7lMZ9b5mlMsoHkFlOdCNevR1DzwmZBZMfKFeXvenMtOfq6BwRth4iTw3NvUYqy6rAxiBY2eaLXYFEnjoWDvTRC0E+smaaxz7eewHsKxfv+lWwNJ76VXX7vI2Xjpe8vrF5bqCveHTbbKXS57bG6dmkrM1bpBxsbRP41yigHBZP19rUoGkOapWNI6gaLoW2Vgm/owsLdC0w2SR6TeezKO3VS95PFWzf1LPB1z2pLvkrzuddFc28MYvaASq0fQv8Y9f4N4XGbO8WtmAvawkWjWUQG+zeFz4iXpMNXnMCixbhT8X3ovuj29fqFd99H/8mKbmlVWR4/4L/arZiwHEMbOxj0lferuYADHkU/VrjvEOiXrlTdXGximzVu+bZtNQ50xdKkcSNrV7kXmjDJS7x+g+ymEU+VuNl6yzIonVk/7lovvjm2ULmaWguO02Tj7qfm/buFwxeQujtdD9cRClYzlvhk7uF+qRMR4bvgqWK8CzVvgheFHqpVlV5kuYbXl7k0yo5clUARhMvH38RJzCt8IbZtn91rrzkOc8V6dzVvwRNV6EegD4d7Cprr5mqM5JY4hX7S61WtFM/37Goo+C6qvXAWOQA3+IcRZnj8MC/Pt1g2XqEMzlHI/1gb0wTA2XfoHzn+Dhv77++ZuN7eb/VFq9uHm7tf1ma/vi+40/71b/i+nXa/++9h9rr9a21/609ue1o7Xe2vVasHa/9te1v63977tv352+67+7MaY//1nV59/WnJ93n/8fNe3MbA==</latexit>

NCarnegie Mellon University

Sparse Neural Editor
Dirichlet
hyperparamSparse Prototype
distribution

0 1 2 3 NθPrototype sentence
I ordered a burger
with fries

Observed sentence
I ordered a pizzawith sauceαEdit vector

tn
xn

zn

p(z)
<latexit sha1_base64="FjCuQsyldJJj37WYC+kn94uQBh0=">AAA1pXiclVtbc9vKkVaSzSar3WxOso/7glrZdZyUrLJ8jitb+xRdqItFSZREXexDHxcINkFYuAkzhEgx3Od9zWv2l+2/2Z4ZDLoHhHwqqrKE+b6exly+nm6C8DCPIyHfvPm/n/38F//wy3/81a//af2f/+U3//rbb373+xuRTYsAroMszoq7oS8gjlK4lpGM4S4vwE+GMdwO7/cUf1tCIaIs7ct5Dp8SP0yjcRT4EqHb/NWgKJ/+8PmbjTdbb/SPt3qxXV1srFU/vc+/+/5/BqMsmCaQyiD2hfhh+00uPy38QkZBDMv1wVRA7gf3fgg/4GXqJyA+LfR4l95LREbeOCvwXyo9jfIeCz8RYp4M0TLx5UQ0OQW2cT9M5fg/Py2iNJ9KSANzo/E09mTmqcl7o6iAQMZzvPCDIsKxesHEL/xA4hKtr79UP95Z59Y73ekfefudg+Oz4/7x+dmVp6n1toFs4l81DbE5TJbowzv1i3tP4H1wgYWXjb3Az821mnEBYyiKKA3VoEZRGQlrNo7CaQE4oRQegyxJ/HS0GCAYw1guF4sBJN6rLl7/YblcsQlwH6CwVnu61WZXROGkdnapGm1WMsutTT/L2yyGmZRZYo12dWvFrpq3b8385yyG1mL4nEVgLYLnLEbWYqQscBuOcHaxmqHne2ivNh3GGCUjD9cmcX3gtQKXP2x/Qi/DsbexrZw0pz1bLgaJX4QoML9YHBzfNceC144JSqlp0j/fP9f3GUiYSS39RQE4ekX8l7mx67OjXcpJli8GnSbbeUC283mhwnggosR7wOsyn0RLHdl/wV+zlSXr5E+NXrnqJScg/a/3q7vNqm6jV62WD0+4Vq1Dce1yZffMzRuWs6cVy9lf9Om1YrlquGIz0kajdlLd6VWb64en5qyaFmphEW6gM43OGuhco/MGmmg0aapAo2nTdiqVOKY4pNmmZ2e8YpQ7RmrsDZOR6Yi38YexT2vXNFNdmVGbJ8RWnCHGR4fBeaDPOnMY4kkNm16cPULxOsBUtrU+wEjVpxWMN7YX5lz87wG2Fjo82rrjKRBJP97yDvCMFRLzkDpShToIkTceD6zHg6ZHTcvHzN5z4211V+FZIw+nVzXe2h4PU39EXTa+2/h+pdtm3cdefcddfa+nc2WSxVeXAxOKGXyVWZz1aHFgF8T0vrK9r1p6X9peOk8+ZnXy2qoXxtxd6JWpU9szS9N0OCkAmi6Zv43vVj3SqjHf36369lMPcBNU55YlgwczZ2vy/KQdP9M8h8JTfoybTuWm0+Zmxyv8R1r3hrPXr1/7ZRaNvKlQGT8ae3kmRITFmXGdxz5mpMr/86NTRUqOCapljoox3Subv3uSlaO92tHeTzrCOach6NLG2ArjQ8P1iFAqlrauXr9+ViY4Oj8OMyzKJknLPJEzo6uNvjpR5mplpjvW1U6LKyt4ez+cRO3r64dB3+m085OdVhYVCwZZzZypT6FmuOrqa5ti+jfV26v799z+dqb1DXDU6vrZAVeCgyhWYo3VBZYraKCuKn/jOMsKTesrw+vLygCpYbJYKXJkgYFQ1TmBHy/2mwalH0cjbvDZXBfJwlDLFZcgZHsHzSzrGUEuVOmYiyjOUl324dKiiyzxSr+IMImB1Tfmr4Up3NKsSNDriwFCL5Z2OYsG7RMzdJkhMYHLBMSMXGZEDLgMEDN2mTExocuExExcZkJM5DIRMV9c5gsx9y5zT0zsMvFSy7hIvEhgxOLH1tFcHXZmBze9L1MhvVGWfis99fkR5ThXJ4+zMV5S+U5d3yndNXOZjJjcZXJiHlzmgZjCZQpihMsIYqTLSGKmLjMlpnSZkphHl3kkZuYyM2LmLjMn5sllnpamQLMBgJk5q4/3sgqShQml4ZiFTT1uXeUxi6rqq3nGcXhIMIuNMiCYBUY5IphFRQkEs5AoxwSzeChDglkwlBOCWSSUU4JZGJRfCGYxUN4TzAKgjAmOGZwQnDCYLTRf4YxgJuYyJ5gpuXwgmMm4LAhmGi4FwYJvKsGyfU24dEuCmW7LR4KZaMsZwUyx5ZxgJtfyiWCr1U4M6jmUfohStOgWjOhaz2Uwyms9mcHIr/VsBqPB1tMZjBBbz2cwamw9ocFIsvWMBqPL1lMauWfPaTAKbT2pwci09awGo9XmaW25xOUSzj17EoORbutZDEa/racxGBG3nsdglNx6IoORc+uZDEbTracyGGG3nstg1N16MoOReOvZDEbnraczGLG3ns9gFP/8CY2xUERBXaEkOxQfOxQ2yS7BuwzeI3iPwfsE7zO4Q3CHwQcEHzD4kOBDBh8RfMTgY4KPGfye4PcMPiH4hMFdgrsMPiX4lMFnBJ8x+Jzgcwb3CO4x+ILgCwZfEnzJ4CuCrxjcJ7jP4GuCrxl8Q/ANg28JvmXwHcF3DP5A8AcGfyT44/PHqys6MKpjGt1h+tXSY9wu5/Zcbo9z+y63z7mOy3U4d+ByB5w7dLlDzh253BHnjl3umHPvXe49505c7oRzXZfrcu7U5U45d+ZyZ5w7d7lzzvVcrse5C5e74Nyly11y7srlrjjXd7k+565d7ppzNy53w7lbl7vl3J3L3XHug8t94NxHl7Oyv+ElRPkE+nMEfnZ9U/ctsxQW9vOsxZKpgQYJJY26Jla4Ww+rZ6MVoZ+mWriyZoZDg1B5oosTRKgo0SUJIlSKlNUAqQDR5QciVHboogMRKjZ0qYEIlRhlNUg2wi8GoXJCFxOIUBGhSwhEYrY8BqGCQZcLiKRsWQ2SsUUyCJUEuiBAhAoBXQYgQulfJ39EBNsHg1CqL6vdYntVGoTSuk7qiFAy16kcEUrhOoEjQolbp21E2opUtzot/TifqP3Wf2thlsNKM9WDeAPSJzB6YFFRsZ8MR6qHuSAiSyBUuP5LsFaqUqkF0CEi+JsgEYWJ6qr/Emz1XH9LUE1kseDjXyix2haKNaAWCnXEJrVQArUtFOiYWijOkFoozAm1cLhsrCjIL9RCMd6ztVkoEdYzXygB2hYuJltFFF/GlmShRGdbKLoHaqHgCrZSCyW0eoEWSmS2hQvNlhkFVlILxfVILRTWjFooqjm1UFBPy+obZky/M4Pr1Is6o5SrEy4ilGh1mkWE0qtOrohQUtUpFRFKpTqRIkIJVKdPRCht6qSJCCVLnSoRoRSpEyQilBh1WkSE0qFOhohQEtQpEBFKfTrxIUIJT6c7RCjN6SSHCCU3ndoQoZSmExoilMh0GkOE0pdOXohQ0tIpCxFKVTpRIUIJSqcnRCgt6aSECCUjnYoQoRSkExAiH9kOUroY8myR9Ops0WPZIuna0FdMtwr/enJVDCvuysSxVlEfUqHeu9iHIPYLQFFNdtQJhHc0NaAYR+oJKqRBNorSEJ3501ghYlxfJ8uFUA9/r0A+52CYxaOfcjOcLRfNLzcljs98U67TaeVPP7yupiZN2ZkKpn65azHSv9yzGEWA3LcYxYDsWIyiQB5YjOJAHlqMIkEeWYxiQR5bjKJBvrcYxYM8sRhFhOxajGJCnlqMokKeWYziQp5bjCJD9ixGsSEvLEbRIS8tRvEhryxGESL7FqMYkdcWoyiRNxajOJG3FqNIkXcWo1iRHyxG0SI/WswUaijkw8LPJ4YN7cffwPkUEu4ymHQR7jGYpBHuM5jUEXYYTAIJDxhMGgkPGUwyCY8YTEoJjxlMYgnfM5j0Ep4wmCQTdhlMqglPGUzCCc8YTNoJzxlM8gl7DCYFhRcMJhGFlwwmHYVXDCYphX0Gk5rCawaToMIbBpOmwlsGk6zCOwaTssIPDCZxhR8ZbD8I4NFWlWqifrgyZOISu4SStsQeoSQtsU+oVtZLb19/wTEV4PmeAOnhrWMYeZ1NbwiBr3A5iYT3mE3jEULYAk/or0OwlpwWnnpRLovRkXq7DGY51pb6O177TXuH7kiiFQeEkmbFIaEkWXFEKClWHBNKghXvCSW9ihNCSa6iSyipVZwSSmIVZ4SSVsU5oSRV0SOUlCouCCWhiktCSafiilCSqegTSioV14SSSMUNoaRRcUsoSVTcEUoKFR8IJYGKj4TWz2dSrAZBf7DwzZOZqjQEqgu67kcCVTTuUAsFvEstFO4etVCw+9RCMXWohSI6oBaK55BaKJojaqFYjqmFInlPLRTHCbVQFF1qoRhOqYUiOKMWbv45tXDTe9TCzb6gFm7yJbVwc6+ohZvapxZu5jW1cBNvqIWbd0st3LQ7auFmfaAWbtJHdr+q/qpqL7VlwLdMmjoMDxoV1fqNWAxtg256j5GcZFPpYRHkPWKiy6FwyySgOsmpkarby1oD2nClPARdREGjigJdRkGjjgJdSEGjkgJdSkGjlgJdTEGjmgJdTkGjngJdUEGjogJdUkGjpgJdVEGjqgJdVkGjrgJdWEGjsgJdWkGjtgJdXEGjugJdXkGjvgJdYEGjwgJdYkGjxgJdZEGjygJdZkGjzgJdaEGj0gJdakGj1gJdbEGj2gJdbkGj3gJdcEGj4gJdckGj5gJddEGj6gJddgGru/DzAyYiWUzBm6YjKOK5esNp5EvfCyGFAnOQakcClT6cqoTUfAPTVy8Jqhc1i2ShGzodKq+Q5FERYSJ0+tfv7w7nOgnqd0bUTTBrNnzb10kmvsTP7+4tHMset+wt2waTZCOIvzYRbVDPxLRW7lMZ9b5mlMsoHkFlOdCNevR1DzwmZBZMfKFeXvenMtOfq6BwRth4iTw3NvUYqy6rAxiBY2eaLXYFEnjoWDvTRC0E+smaaxz7eewHsKxfv+lWwNJ76VXX7vI2Xjpe8vrF5bqCveHTbbKXS57bG6dmkrM1bpBxsbRP41yigHBZP19rUoGkOapWNI6gaLoW2Vgm/owsLdC0w2SR6TeezKO3VS95PFWzf1LPB1z2pLvkrzuddFc28MYvaASq0fQv8Y9f4N4XGbO8WtmAvawkWjWUQG+zeFz4iXpMNXnMCixbhT8X3ovuj29fqFd99H/8mKbmlVWR4/4L/arZiwHEMbOxj0lferuYADHkU/VrjvEOiXrlTdXGximzVu+bZtNQ50xdKkcSNrV7kXmjDJS7x+g+ymEU+VuNl6yzIonVk/7lovvjm2ULmaWguO02Tj7qfm/buFwxeQujtdD9cRClYzlvhk7uF+qRMR4bvgqWK8CzVvgheFHqpVlV5kuYbXl7k0yo5clUARhMvH38RJzCt8IbZtn91rrzkOc8V6dzVvwRNV6EegD4d7Cprr5mqM5JY4hX7S61WtFM/37Goo+C6qvXAWOQA3+IcRZnj8MC/Pt1g2XqEMzlHI/1gb0wTA2XfoHzn+Dhv77++ZuN7eb/VFq9uHm7tf1ma/vi+40/71b/i+nXa/++9h9rr9a21/609ue1o7Xe2vVasHa/9te1v63977tv352+67+7MaY//1nV59/WnJ93n/8fNe3MbA==</latexit>

NCarnegie Mellon University

Learning and Inferenceθlog p(x)<latexit sha1_base64="9LOwO8rbjG5GIuD4j/8pNdIkd+w=">AAA1wniclVtbc9vKkVaS3SSrzSbnZB/3BRXZFWdLVlk+x5WtfYou1MWiJEqiLvahjwsEmyAs3IQZQqQY7nN+zb7u/pX9N+mZwaB7QMinVlW2MN/X05jL19NNEBrmcSTkmzf/97Of/+If/vGXv/r1P63/82/+5be/++bb39+IbFoEcB1kcVbcDX0BcZTCtYxkDHd5AX4yjOF2eL+n+NsSChFlaV/Oc/iU+GEajaPAlwh9/uYPgzgLvfzVoChnf/IGIeC/7ufFQMJMLjrd3fPl8vM3G2+23ugfb/Viu7rYWKt+ep+//f5vg1EWTBNIZRD7Qvyw/SaXnxZ+IaMghuX6YCog94N7P4Qf8DL1ExCfFnoyS+8lIiNvnBX4L5WeRnmPhZ8IMU+GaJn4ciKanALbuB+mcvwfnxZRmk8lpIG50XgaezLz1Mp4o6iAQMZzvPCDIsKxesHEL/xA4vqtr79UP95Z59Y73ekfefudg+Oz4/7x+dmVp6n1toFs4m81DbE5TJbowzv1i3tP4H1w9YWXjb3Az821mnEBYyiKKA3VoEZRGQlrNo7CaQE4oRQegyxJ/HS0GCAYw1guF4sBJN6rLl7/ablcsQlwH6CwVnu61WZXROGkdnapGm1WMsutTT/L2yyGmZRZYo12dWvFrpq3b8385yyG1mL4nEVgLYLnLEbWYqQscBuOcHaxmqHne2ivNh3GGEIjD9cmcX3gtQKXP2x/Qi/DsbexrZw0pz1bLgaJX4QoML9YHBzfNceC144JSqlp0j/fP9f30fGnpb8oAEeviP80N3Z9drRLOcnyxaDTZDsPyHYwmovyaSCixHvA6zKfREsV709/1UG/0il/avTKVS85Ael/vV/dbVZ1G71qtXx4wrVqHYprlyu7Z27esJw9rVjOlOXTquWq4YrNSBuN2kl1p1dtrh+emrNqWqiFRbiBzjQ6a6Bzjc4baKLRpKkCjaZN26lU4pjikGabnp3xilHuGKmxN0xGpiPexh/GPq1d00x1ZUZtnhBbcYYYHx0G54E+68xhiCc1bHpx9gjF6wDz3Nb6ACNVn1Yw3themHPxvwbYWujwaOuOp0Ak/XjLO8AzVkjMQ+pIFeogRN54PLAeD5oeNS0fM3vPjbfVXYVnjTycXtV4a3s8TP0Rddn4buP7lW6bdR979R139b2ezpVJFl9dDkwoZvBVZnHWo8WBXRDT+8r2vmrpfWl76Tz5mNXJa6teGHN3oVemTm3PLE3T4aQAaLpk/ja+W/VIq8Z8f7fq2089wE1QnVuWDB7MnK3J85N2/EzzHApP+TFuOpWbTpubHa/wH2ndG85ev37tl1k08qZCZfxo7OWZEBFWbsZ1HvuYkSr/z49OFSk5JqiWOSrGdK9s/t+TrBzt1Y72ftIRzjnFMlKVNsZWGB8arkeEUrG0dfX69bMywdH5cZhhUTZJWuaJnBldbfTViTJXKzPdsa52WlxZwdv74SRqX18/DPpOp52f7LSyqFgwyGrmTH0KNcNVV1/bFNO/qd5e3b/n9rczrW+Ao1bXzw64EhxEsRJrrC6wXEEDdVX5G8dZVmhaXxleX1YGSA2TxUqRIwsMhKrOCfx4sd80KP04GnGDz+a6SBaGWq64BCHbO2hmWc8IcqFKx1xEcZbqsg+XFl1kiVf6RYRJDKy+MX8tTOGWZkWCXl8MEHqxtMtZNGifmKHLDIkJXCYgZuQyI2LAZYCYscuMiQldJiRm4jITYiKXiYj54jJfiLl3mXtiYpeJl1rGReJFAiMWP9OO5uqwMzu46X2ZCumNsvSP0lOfH1GOc3XyOBvjJZXv1PWd0l0zl8mIyV0mJ+bBZR6IKVymIEa4jCBGuowkZuoyU2JKlymJeXSZR2JmLjMjZu4yc2KeXOZpaQo0GwCYmbP6eC+rIFmYUBqOWdjU49ZVHrOoqr6aZxyHhwSz2CgDgllglCOCWVSUQDALiXJMMIuHMiSYBUM5IZhFQjklmIVB+YVgFgPlPcEsAMqY4JjBCcEJg9lC8xXOCGZiLnOCmZLLB4KZjMuCYKbhUhAs+KYSLNvXhEu3JJjptnwkmIm2nBHMFFvOCWZyLZ8ItlrtxKCeQ+mHKEWLbsGIrvVcBqO81pMZjPxaz2YwGmw9ncEIsfV8BqPG1hMajCRbz2gwumw9pZF79pwGo9DWkxqMTFvPajBabZ7WlktcLuHcsycxGOm2nsVg9Nt6GoMRcet5DEbJrScyGDm3nslgNN16KoMRduu5DEbdrSczGIm3ns1gdN56OoMRe+v5DEbxz5/QGAtFFNQVSrJD8bFDYZPsErzL4D2C9xi8T/A+gzsEdxh8QPABgw8JPmTwEcFHDD4m+JjB7wl+z+ATgk8Y3CW4y+BTgk8ZfEbwGYPPCT5ncI/gHoMvCL5g8CXBlwy+IviKwX2C+wy+JviawTcE3zD4luBbBt8RfMfgDwR/YPBHgj8+f7y6ogOjOqbRHaZfLT3G7XJuz+X2OLfvcvuc67hch3MHLnfAuUOXO+Tckcsdce7Y5Y45997l3nPuxOVOONd1uS7nTl3ulHNnLnfGuXOXO+dcz+V6nLtwuQvOXbrcJeeuXO6Kc32X63Pu2uWuOXfjcjecu3W5W87dudwd5z643AfOfXQ5K/sbXkKUT6A/R+Bn1zd13zJLYWE/z1osmRpokFDSqGtihbv1sHo2WhH6aaqFK2tmODQIlSe6OEGEihJdkiBCpUhZDZAKEF1+IEJlhy46EKFiQ5caiFCJUVaDZCP8YhAqJ3QxgQgVEbqEQCRmy2MQKhh0uYBIypbVIBlbJINQSaALAkSoENBlACKU/nXyR0SwfTAIpfqy2i22V6VBKK3rpI4IJXOdyhGhFK4TOCKUuHXaRqStSHWr09KP84nab/27FmY5rDRTPYg3IH0CowcWFRX7yXCkepgLIrIEQoXr3wRrpSqVWgAdIoL/EySiMFFd9W+CrZ7rbwmqiSwWfPwLJVbbQrEG1EKhjtikFkqgtoUCHVMLxRlSC4U5oRYOl40VBfmFWijGe7Y2CyXCeuYLJUDbwsVkq4jiy9iSLJTobAtF90AtFFzBVmqhhFYv0EKJzLZwodkyo8BKaqG4HqmFwppRC0U1pxYK6mlZfcOM6XdmcJ16UWeUcnXCRYQSrU6ziFB61ckVEUqqOqUiQqlUJ1JEKIHq9IkIpU2dNBGhZKlTJSKUInWCRIQSo06LiFA61MkQEUqCOgUiQqlPJz5EKOHpdIcIpTmd5BCh5KZTGyKU0nRCQ4QSmU5jiFD60skLEUpaOmUhQqlKJypEKEHp9IQIpSWdlBChZKRTESKUgnQCQuQj20FKF0OeLZJenS16LFskXRv6iulW4V9ProphxV2ZONYq6kMq1HsX+xDEfgEoqsmOOoHwjqYGFONIPUGFNMhGURqiM38aK0SM6+tkuRDq4e8VyOccDLN49FNuhrPlovnlpsTxmW/KdTqt/OmH19XUpCk7U8HUL3ctRvqXexajCJD7FqMYkB2LURTIA4tRHMhDi1EkyCOLUSzIY4tRNMj3FqN4kCcWo4iQXYtRTMhTi1FUyDOLUVzIc4tRZMiexSg25IXFKDrkpcUoPuSVxShCZN9iFCPy2mIUJfLGYhQn8tZiFCnyzmIUK/KDxSha5EeLmUINhXxY+PnEsKH9+Bs4n0LCXQaTLsI9BpM0wn0GkzrCDoNJIOEBg0kj4SGDSSbhEYNJKeExg0ks4XsGk17CEwaTZMIug0k14SmDSTjhGYNJO+E5g0k+YY/BpKDwgsEkovCSwaSj8IrBJKWwz2BSU3jNYBJUeMNg0lR4y2CSVXjHYFJW+IHBJK7wI4PtBwE82qpSTdQPV4ZMXGKXUNKW2COUpCX2CdXKeunt6y84pgI83xMgPbx1DCOvs+kNIfAVLieR8B6zaTxCCFvgCf11CNaS08JTL8plMTpSb5fBLMfaUn/Ha79p79AdSbTigFDSrDgklCQrjgglxYpjQkmw4j2hpFdxQijJVXQJJbWKU0JJrOKMUNKqOCeUpCp6hJJSxQWhJFRxSSjpVFwRSjIVfUJJpeKaUBKpuCGUNCpuCSWJijtCSaHiA6EkUPGR0Pr5TIrVIOgPFr55MlOVhkB1Qdf9SKCKxh1qoYB3qYXC3aMWCnafWiimDrVQRAfUQvEcUgtFc0QtFMsxtVAk76mF4jihFoqiSy0Uwym1UARn1MLNP6cWbnqPWrjZF9TCTb6kFm7uFbVwU/vUws28phZu4g21cPNuqYWbdkct3KwP1MJN+sjuV9VfVe2ltgz4lklTh+FBo6JavxGLoW3QTe8xkpNsKj0sgrxHTHQ5FG6ZBFQnOTVSdXtZa0AbrpSHoIsoaFRRoMsoaNRRoAspaFRSoEspaNRSoIspaFRToMspaNRToAsqaFRUoEsqaNRUoIsqaFRVoMsqaNRVoAsraFRWoEsraNRWoIsraFRXoMsraNRXoAssaFRYoEssaNRYoIssaFRZoMssaNRZoAstaFRaoEstaNRaoIstaFRboMstaNRboAsuaFRcoEsuaNRcoIsuaFRdoMsuYHUXfn7ARCSLKXjTdARFPFdvOI186XshpFBgDlLtSKDSh1OVkJpvYPrqJUH1omaRLHRDp0PlFZI8KiJMhE7/+v3d4VwnQf3OiLoJZs2Gb/s6ycSX+PndvYVj2eOWvWXbYJJsBPHXJqIN6pmY1sp9KqPe14xyGcUjqCwHulGPvu6Bx4TMgokv1Mvr/lRm+nMVFM4IGy+R58amHmPVZXUAI3DsTLPFrkACDx1rZ5qohUA/WXONYz+P/QCW9es33QpYei+96tpd3sZLx0tev7hcV7A3fLpN9nLJc3vj1ExytsYNMi6W9mmcSxQQLuvna00qkDRH1YrGERRN1yIby8SfkaUFmnaYLDL9xpN59LbqJY+navZP6vmAy550l/x1p5Puygbe+AWNQDWa/iX+8gvc+yJjllcrG7CXlUSrhhLobRaPCz9Rj6kmj1mBZavw58J70f3x7Qv1qo/+w49pal5ZFTnuv9Cvmr0YQBwzG/uY9KW3iwkQQz5V/80x3iFRr7yp2tg4ZdbqfdNsGuqcqUvlSMKmdi8yb5SBcvcY3Uc5jCJ/q/GSdVYksXrSv1x0f3yzbCGzFBS33cbJR93vbRuXKyZvYbQWuj8OonQs583Qyf1CPTLGY8NXwXIFeNYKPwQvSr00q8p8CbMtb2+SCbU8mSoAg4m3j5+IU/ij8IZZdr+17jzkOc/V6ZwV/44aL0I9APw92FRXXzNU56QxxKt2l1qtaKb/f8aij4Lqq9cBY5ADf4hxFmePwwL8+3WDZeoQzOUcj/WBvTBMDZd+gfOf4OG/vv75m43t5l8qrV7cvN3afrO1ffH9xl92q79i+vXav639Ye3V2vban9f+sna01lu7XgvW/rb232v/s/a/7/bffXn38E4Y05//rOrzr2vOz7u//h01BNdw</latexit><latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//
sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

LELBO= Eq(t,z,✓|x) log p(x|t, z, ✓)DKL (q(t, z, ✓)||p(t, z, ✓))tn<latexit sha1_base64="c+j4YcI8u6msuBACbbZvrBomnR8=">AAA2I3iclVtbb9xKcpY3t402yZ7NvmSRF2JlY+2FLFg+x9ggwAKry+hije6ji33Gx+DM1HBo8SZ2DzUjnslLXvJH8mvyFuQlD/kB+Rep7mazqjmUDyLAEvv7qot9+aqrhkMPsigU8s2b/372sz/787/4y7/6+V+v/uJv/vbvfvnNr/7+WqTTfAhXwzRK89uBLyAKE7iSoYzgNsvBjwcR3AzudhR/U0AuwjTpyXkGn2I/SMJxOPQlQp+/+dc/ev3Yl5PBoOwsPpf3L/t5Idc9/P2Iv+UEpO/9qJqzVwuvH6WBlymTmQEblq+8117/qPuyzcsr78cfTd8m8erzN2tvNt7oH2/5YrO6WFupfs4+/+q7f+uP0uE0hkQOI1+I7zffZPJT6ecyHEawWO1PBWT+8M4P4Hu8TPwYxKdSr9bCe4HIyBunOf5LpKdR3qP0YyHm8QAt1cqIJqfANu77qRz/06cyTLKphGRobjSeRp5MPbX03ijMYSijOV74wzzEsXrDiZ/7Q4kbtLr6Qv14J50b73ird+DtdvYOTw57h6cnl56mVtsGso5/1TTE+iBeoA/v2M/vPIH3we0VXjr2hn5mrtWMcxhDnodJoAY1CotQWLNxGExzwAkl8DBM49hPRmUfwQjGclGWfYi9l128frVYLNkMcR8gt1Y7utVml4fBpHZ2oRptVjLNrE0vzdosBqmUaWyNtnVrya6at2/N/KcsBtZi8JTF0FoMn7IYWYuRssBtOMDZRWqGnu+hvdp0GGOMjjxcm9j1gdcKXHy/+Qm9DMbe2qZy0pz2bFFiqOYBCszPy73D2+ZY8NoxQSk1TXqnu6f6Pn0JM6mlX+aAo1fEP5sbuz472qWcpFnZ7zTZzj2ync+liue+CGPvHq+LbBIuVKA//mjOjWan7LHRK1O99GHw9X51t1nVbfSy1fL+EdeqdSiuXabsnrh5w3L2uGQ5U5aPy5bLhks2I200aifVnV62ub5/bM6qaaEWFuEGOtPorIHONTpvoLFG46YKNJo0badSiWOKQ5rhaV7NeMkoc4zU2BsmI9MRb+MPIp/WrmmmujKjNk+ILTlDjI8Og3NPn3XmMMSTGta9KH2A/PUQE+nGah8jVZ9WMF7bLM25+C99bJU6PNq64ykQSj/a8PbwjBUS85A6UoU6CJE3Hvesx72mR03Lh9Tec+1tdVfhWSMPp1c13toe91N/RF3Wvl37bqnbet3HXn3LXX2np3NpksVXlwMTihl8lVmc9WhxYBfE9L60vS9bel/YXjpPPqR18tqoF8bcXeiVqVPbE0vTdDjJAZoumb+1b5c90qox398u+/YTD3ATVOeWJYN7M2dr8vSkHT/TLIPcU36Mm07lptPmZsvL/Qda94az169f+0UajrypUBk/HHtZKkSIpaFxnUU+ZqTK/9OjU0VKhgmqZY6KMd0rm//3JCtHO7WjnZ90hHNOAtCljbEVxoeG6xGhVCxtXb1+/aRMcHR+FKRYlE3ilnkiZ0ZXG311oszV0ky3rKutFldW8PZ+OIna19cPg57TaesnOy0tKhYMspo5U59CzXDV1dc2xfRvqves7n/m9rczrW+Ao1bXTw64EhyEkRJrpC6wXEEDdVX5G0dpmmtaXxleX1YGSA3icqnIkTkGQlXnDP2o3G0aFH4UjrjBZ3Odx6WhFksuQcj2DppZ1DOCTKjSMRNhlCa67MOlRRdp7BV+HmISA6tvzF+lKdySNI/R6/M+Qs8XdjnzBu0TM3CZATFDlxkSM3KZETHgMkDM2GXGxAQuExAzcZkJMaHLhMR8cZkvxNy5zB0xkctECy3jPPZCgRGLH5pHc3XYmR1c975MhfRGafI76anPjyjHuTp5nI3x4sp34vpO6K6py6TEZC6TEXPvMvfE5C6TEyNcRhAjXUYSM3WZKTGFyxTEPLjMAzEzl5kRM3eZOTGPLvO4MAWaDQDMzGl9vBdVkJQmlAZjFjb1uHWVxyyqqq/mGcfhAcEsNoohwSwwihHBLCoKIJiFRDEmmMVDERDMgqGYEMwioZgSzMKg+EIwi4HijmAWAEVEcMTgmOCYwWyh+QqnBDMxFxnBTMnFPcFMxkVOMNNwIQgWfFMJlu1rwqVbEMx0WzwQzERbzAhmii3mBDO5Fo8EW612IlDPofRDlLxFt2BE13oug1Fe68kMRn6tZzMYDbaezmCE2Ho+g1Fj6wkNRpKtZzQYXbae0sg9eU6DUWjrSQ1Gpq1nNRitNk9ry8UuF3PuyZMYjHRbz2Iw+m09jcGIuPU8BqPk1hMZjJxbz2Qwmm49lcEIu/VcBqPu1pMZjMRbz2YwOm89ncGIvfV8BqP4p09ojIU8HNYVSrxF8bFFYRNvE7zN4B2Cdxi8S/AugzsEdxi8R/Aeg/cJ3mfwAcEHDD4k+JDB7wl+z+Ajgo8Y3CW4y+Bjgo8ZfELwCYNPCT5l8BnBZww+J/icwRcEXzD4kuBLBvcI7jH4iuArBl8TfM3gG4JvGHxL8C2DPxD8gcEfCf749PHqig6M6phGt5h+tfQYt825HZfb4dyuy+1yruNyHc7tudwe5/Zdbp9zBy53wLlDlzvk3HuXe8+5I5c74lzX5bqcO3a5Y86duNwJ505d7pRzZy53xrlzlzvn3IXLXXDu0uUuOddzuR7nrlzuinPXLnfNuRuXu+Hcrcvdcu6Dy33g3EeXs7K/5iVE8Qj6cwR+dn1T9y3SBEr7edZi8dRA/ZiSRl0TK9yth9Wz0YrQT1MtXFkzw4FBqDzRxQkiVJTokgQRKkWKaoBUgOjyAxEqO3TRgQgVG7rUQIRKjKIaJBvhF4NQOaGLCUSoiNAlBCIRWx6DUMGgywVEErasBknZIhmESgJdECBChYAuAxCh9K+TPyKC7YNBKNUX1W6xvSoMQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtRapbnRZ+lE3Ufuu/tTCLQaWZ6kG8AekTGD2wqKjIjwcj1cNcEJHGEChc/yVYK1Wp1ALoEBH8TZAIg1h11X8JtnquvyWoJlKWfPylEqttoViH1EKhjtikSiVQ20KBjqmF4gyohcKcUAuHy8aKgvxCLRTjHVubUomwnnmpBGhbuJhsFVF8KVuSUonOtlB099RCweVspUoltHqBSiUy28KFZsuMAiuoheJ6oBYKa0YtFNWcWiiox0X1DTOm35nBdepFnVHK1QkXEUq0Os0iQulVJ1dEKKnqlIoIpVKdSBGhBKrTJyKUNnXSRISSpU6ViFCK1AkSEUqMOi0iQulQJ0NEKAnqFIgIpT6d+BChhKfTHSKU5nSSQ4SSm05tiFBK0wkNEUpkOo0hQulLJy9EKGnplIUIpSqdqBChBKXTEyKUlnRSQoSSkU5FiFAK0gkIkY9sByldDHi2iM/qbHHGskXctaGvmG4V/vXkqhhW3KWJY62iHiRCvXexC8PIzwFFNdlSJxDe0dSAYhyqJ6iQDNNRmATozJ9GChHj+jpelEI9/L0E+ZSDQRqNfsrNYLYom19uShyf+aZcp9PKn354XU1NmrIzEUz9cttipH+5YzGKALlrMYoB2bEYRYHcsxjFgdy3GEWCPLAYxYI8tBhFg3xvMYoHeWQxigjZtRjFhDy2GEWFPLEYxYU8tRhFhjyzGMWGPLcYRYe8sBjFh7y0GEWI7FmMYkReWYyiRF5bjOJE3liMIkXeWoxiRX6wGEWL/GgxU6ihkPdzP5sYNrAff4fOp5Bgm8Gki2CHwSSNYJfBpI6gw2ASSLDHYNJIsM9gkklwwGBSSnDIYBJL8J7BpJfgiMEkmaDLYFJNcMxgEk5wwmDSTnDKYJJPcMZgUlBwzmASUXDBYNJRcMlgklLQYzCpKbhiMAkquGYwaSq4YTDJKrhlMCkr+MBgElfwkcH2gwAebVWpJuqHKwMmLrFNKGlL7BBK0hK7hGplvfB29RccUwGe7wmQHt46gpHXWfcGMPQVLieh8B7SaTRCCFvgCf11CNaS09xTL8qlETpSb5fBLMPaUn/Ha79p79AdSbRij1DSrNgnlCQrDgglxYpDQkmw4j2hpFdxRCjJVXQJJbWKY0JJrOKEUNKqOCWUpCrOCCWlinNCSajiglDSqbgklGQqeoSSSsUVoSRScU0oaVTcEEoSFbeEkkLFB0JJoOIjofXzmQSrQdAfLHzzZKYqDYHqgq77kUAVjVvUQgFvUwuFu0MtFOwutVBMHWqhiPaoheLZpxaK5oBaKJZDaqFI3lMLxXFELRRFl1oohmNqoQhOqIWbf0ot3PQzauFmn1MLN/mCWri5l9TCTe1RCzfzilq4idfUws27oRZu2i21cLM+UAs36SO7X1V/VbWX2jLgWyZNHYYHjYpq/UYshrZB172HUE7SqfSwCPIeMNFlkLtlElCd5NRI1e1lrQFtuFQegi6ioFFFgS6joFFHgS6koFFJgS6loFFLgS6moFFNgS6noFFPgS6ooFFRgS6poFFTgS6qoFFVgS6roFFXgS6soFFZgS6toFFbgS6uoFFdgS6voFFfgS6woFFhgS6xoFFjgS6yoFFlgS6zoFFngS60oFFpgS61oFFrgS62oFFtgS63oFFvgS64oFFxgS65oFFzgS66oFF1gS67gNVd+PkBE5HMp+BNkxHk0Vy94TType8FkECOOUi1Q4FKH0xVQmq+gemrlwTVi5p5XOqGTofKK8RZmIeYCJ3+9fu7g7lOgvqdEXUTzJoN3/Z1kokv8fO7ewvH8oxbni3aBhOnI4i+NhFtUM/EtJbuUxmdfc0ok2E0gsqyrxv16OseeEzIdDjxhXp53Z/KVH+ugtwZYeMl8szY1GOsuiwPYASOnWm22OVI4KFj7UwTtTDUT9Zc48jPIn8Ii/r1m24FLLwXXnXtLm/jpeMFr19crivYGz7dJnux4Lm9cWrGGVvjBhnlC/s0ziVyCBb187UmNZQ0R9UKxyHkTdciHcvYn5GlBZp2mCxS/caTefS27CWLpmr2j+r5gMsedRf8daej7tIGXvs5jUA1mv4l/vFz3Ps8ZZaXSxuwkxZEq4YS6E0ajXM/Vo+pJg9pjmWr8OfCe9794e1z9aqP/o8f08S8sioy3H+hXzV73ocoYjb2MekLbxsTIIZ8on7NMd4hVq+8qdrYOGXW6n3TdBronKlL5VDCunYvUm+UgnL3EN6FGYxCf6PxknWax5F60r8ouz+8WbSQaQKK22zj5IPu97aNyxSTtTBaC90f+mEylvNm6GR+rh4Z47Hhq2C5BDxrhR+AFyZeklZlvoTZhrczSYVanlQVgMOJt4ufiBP4nfAGaXq3seo85DnN1Omc5r9HjeeBHgD+7a+rq68ZqnPSGOJVu0utVjTTv5+w6KGgeup1wAhk3x9gnEXpwyAH/27VYKk6BDM5x2O9by8MU8OFn+P8J3j4r65+/mZts/k/lZYvrt9ubL7Z2Dz/bu1P29X/Yvr5yj+u/Hbl5crmyh9W/rRysHK2crUyXPnfZ7989g/PfvPu39/9x7v/fPdfxvRnz6o+v15xft79z/8B5J/4cQ==</latexit>

Approximate posteriorq(tn | xn)

q(zn | tn, xn)

xn
27

zn

N

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N
28

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>q(✓) = Dir(✓; )zn<latexit sha1_base64="6YuqFINaS8085MoDZB0qg9PpuvU=">AAA1y3iclVtbc9tKctbu5rKr3M4mj3lBrexan0RWWT7HtalsbdXqQl0sSqIk6mIf+rhAsgnCwk2YIUQKy7ylKq/5NXlNfkb+TXpmMOgeEPKpqMoS5vt6GnP5eroJwsMsCoV88+Z/f/bzX/zZn//FX/7yV+t/9dd/87d/982v//5GpLN8BNejNErzu6EvIAoTuJahjOAuy8GPhxHcDu/3FH9bQC7CNOnLRQafYj9Iwkk48iVCn7/554dXAzkF6X/r/cEbSJjLcj/MlxX4+8EwLgcRuhv7y28/f7PxZuuN/vFWL7ari4216qf3+dff/8dgnI5mMSRyFPlC/LD9JpOfSj+X4SiC5fpgJiDzR/d+AD/gZeLHID6VelZL7yUiY2+S5vgvkZ5GeY/Sj4VYxEO0jH05FU1OgW3cDzM5+ZdPZZhkMwnJyNxoMos8mXpqibxxmMNIRgu88Ed5iGP1RlM/90cSF3J9/aX68c46t97pTv/I2+8cHJ8d94/Pz648Ta23DWQT/6ppiM1hvEQf3qmf33sC74PbILx04o38zFyrGecwgTwPk0ANahwWobBmkzCY5YATSuBxlMaxn4zLAYIRTOSyLAcQe6+6eP3tcrliM8J9gNxa7elWm10eBtPa2aVqtFnJNLM2/TRrsximUqaxNdrVrRW7at6+NfOfsxhai+FzFiNrMXrOYmwtxsoCt+EIZxepGXq+h/Zq02GCsTT2cG1i1wdeK3D5w/Yn9DKceBvbyklz2vNlOYj9PECB+Xl5cHzXHAteOyYopaZJ/3z/XN9Hh6SWfpkDjl4R/2pu7PrsaJdymmbloNNkOw/Idj6Xg7x4Gogw9h7wusimIQY6Qn/CX/OVJetkT41emeqlD4av96u7zatu41etlg9PuFatQ3HtMmX3zM0blvOnFcu5snxatVw1XLEZa6NxO6nu9KrN9cNTc1ZNC7WwCDfQuUbnDXSh0UUDjTUaN1Wg0aRpO5NKHDMc0nzTszNeMcocIzX2hsnYdMTb+MPIp7VrmqmuzKjNE2IrzhDjo8PgPNBnnTkM8aSGTS9KHyF/PcKEt7U+wEjVpxVMNrZLcy7+2wBbpQ6Ptu54CoTSj7a8AzxjhcQ8pI5UoQ5C5I3HA+vxoOlR0/IxtffceFvdVXjWyMPpVY23tsfDzB9Tl43vNr5f6bZZ97FX33FX3+vpXJlk8dXlwIRiBl9lFmc9WhzYBTG9r2zvq5bel7aXzpOPaZ28tuqFMXcXemXq1PbM0jQdTnOApkvmb+O7VY+0asz3d6u+/cQD3ATVuWXJ4MHM2Zo8P2nHzyzLIPeUH+OmU7nptLnZ8XL/kda94ez169d+kYZjbyZUxg8nXpYKEWIJZ1xnkY8ZqfL//OhUkZJhgmqZo2JM98rm/z3JytFe7WjvJx3hnJMAdGljbIXxoeF6RCgVS1tXr18/KxMcnR8FKRZl07hlnsiZ0dVGX50oc7Uy0x3raqfFlRW8vR9Oovb19cOg73Ta+clOK4uKBYOsZs7Up1AzXHX1tU0x/Zvq7dX9e25/O9P6Bjhqdf3sgCvBQRgpsUbqAssVNFBXlb9JlKa5pvWV4fVlZYAUfvBYKXJkjoFQ1TkjPyr3mwaFH4VjbvDZXOdxaajliksQsr2DZpb1jCATqnTMRBiliS77cGnRRRp7hZ+HmMTA6hvzV2kKtyTNY/T6YoDQi6VdzrxB+8QMXWZIzMhlRsSMXWZMDLgMEDNxmQkxgcsExExdZkpM6DIhMV9c5gsx9y5zT0zkMtFSyziPvVBgxOKH2/FCHXZmBze9LzMhvXGa/FZ66vMjynGhTh5nY7y48p24vhO6a+oyKTGZy2TEPLjMAzG5y+TECJcRxEiXkcTMXGZGTOEyBTGPLvNIzNxl5sQsXGZBzJPLPC1NgWYDADNzWh/vRRUkpQml4YSFTT1uXeUxi6rqq3nGcXhIMIuNYkQwC4xiTDCLigIIZiFRTAhm8VAEBLNgKKYEs0goZgSzMCi+EMxioLgnmAVAEREcMTgmOGYwW2i+winBTMxFRjBTcvFAMJNxkRPMNFwIggXfVIJl+5pw6RYEM90WjwQz0RZzgpliiwXBTK7FE8FWq50I1HMo/RAlb9EtGNG1nstglNd6MoORX+vZDEaDraczGCG2ns9g1Nh6QoORZOsZDUaXrac0cs+e02AU2npSg5Fp61kNRqvN09pyscvFnHv2JAYj3dazGIx+W09jMCJuPY/BKLn1RAYj59YzGYymW09lMMJuPZfBqLv1ZAYj8dazGYzOW09nMGJvPZ/BKP75ExpjIQ9HdYUS71B87FDYxLsE7zJ4j+A9Bu8TvM/gDsEdBh8QfMDgQ4IPGXxE8BGDjwk+ZvB7gt8z+ITgEwZ3Ce4y+JTgUwafEXzG4HOCzxncI7jH4AuCLxh8SfAlg68IvmJwn+A+g68JvmbwDcE3DL4l+JbBdwTfMfgDwR8Y/JHgj88fr67owKiOaXSH6VdLj3G7nNtzuT3O7bvcPuc6Ltfh3IHLHXDu0OUOOXfkckecO3a5Y869d7n3nDtxuRPOdV2uy7lTlzvl3JnLnXHu3OXOOddzuR7nLlzugnOXLnfJuSuXu+Jc3+X6nLt2uWvO3bjcDeduXe6Wc3cud8e5Dy73gXMfXc7K/oaXEMUT6M8R+Nn1Td23SBMo7edZi8UzAw1iShp1Taxwtx5Wz0YrQj9NtXBlzQyHBqHyRBcniFBRoksSRKgUKaoBUgGiyw9EqOzQRQciVGzoUgMRKjGKapBshF8MQuWELiYQoSJClxCIRGx5DEIFgy4XEEnYshokZYtkECoJdEGACBUCugxAhNK/Tv6ICLYPBqFUX1S7xfaqMAildZ3UEaFkrlM5IpTCdQJHhBK3TtuItBWpbnVa+FE2Vfut/9bCLIaVZqoH8QakT2D0wKKizJe5Zf2tbk2kMQQK138J1kpVKrUAOkQEfxMkwiBWXfVfgq2e628JqomUJR9/qcRqWyjWEbVQqGM2qVIJ1LZQoBNqoTgDaqEwp9TC4bKxoiC/UAvFeM/WplQirGdeKgHaFi4mW0UUX8qWpFSisy0U3QO1UHA5W6lSCa1eoFKJzLZwodkyo8AKaqG4HqmFwppTC0W1oBYK6mlZfcOM6XducJ16UWeUcnXCRYQSrU6ziFB61ckVEUqqOqUiQqlUJ1JEKIHq9IkIpU2dNBGhZKlTJSKUInWCRIQSo06LiFA61MkQEUqCOgUiQqlPJz5EKOHpdIcIpTmd5BCh5KZTGyKU0nRCQ4QSmU5jiFD60skLEUpaOmUhQqlKJypEKEHp9IQIpSWdlBChZKRTESKUgnQCQuQj20FKF0OeLeJenS16LFvEXRv6iulW4V9ProphxV2ZONYq6kMi1HsX+zCK/BxQVNMddQLhHU0NKCaheoIKySgdh0mAzvxZpBAxqa/jZSnUw98rkM85GKbR+KfcDOfLsvnlpsTxmW/KdTqt/OmH19XUpCk7E8HUL3ctRvqXexajCJD7FqMYkB2LURTIA4tRHMhDi1EkyCOLUSzIY4tRNMj3FqN4kCcWo4iQXYtRTMhTi1FUyDOLUVzIc4tRZMiexSg25IXFKDrkpcUoPuSVxShCZN9iFCPy2mIUJfLGYhQn8tZiFCnyzmIUK/KDxSha5EeLmUINhXyY+9nUsIH9+DtyPoUEuwwmXQR7DCZpBPsMJnUEHQaTQIIDBpNGgkMGk0yCIwaTUoJjBpNYgvcMJr0EJwwmyQRdBpNqglMGk3CCMwaTdoJzBpN8gh6DSUHBBYNJRMElg0lHwRWDSUpBn8GkpuCawSSo4IbBpKnglsEkq+COwaSs4AODSVzBRwbbDwJ4tFWlmqgfrgyZuMQuoaQtsUcoSUvsE6qV9dLb119wzAR4vidAenjrCMZeZ9MbwshXuJyGwntMZ9EYIWyBJ/TXIVhLznJPvSiXRuhIvV0G8wxrS/0dr/2mvUN3JNGKA0JJs+KQUJKsOCKUFCuOCSXBiveEkl7FCaEkV9EllNQqTgklsYozQkmr4pxQkqroEUpKFReEklDFJaGkU3FFKMlU9AkllYprQkmk4oZQ0qi4JZQkKu4IJYWKD4SSQMVHQuvnMwlWg6A/WPjmyUxVGgLVBV33I4EqGneohQLepRYKd49aKNh9aqGYOtRCER1QC8VzSC0UzRG1UCzH1EKRvKcWiuOEWiiKLrVQDKfUQhGcUQs3/5xauOk9auFmX1ALN/mSWri5V9TCTe1TCzfzmlq4iTfUws27pRZu2h21cLM+UAs36SO7X1V/VbWX2jLgWyZNHYYHjYpq/UYshrZBN73HUE7TmfSwCPIeMdFlkLtlElCd5NRI1e1lrQFtuFIegi6ioFFFgS6joFFHgS6koFFJgS6loFFLgS6moFFNgS6noFFPgS6ooFFRgS6poFFTgS6qoFFVgS6roFFXgS6soFFZgS6toFFbgS6uoFFdgS6voFFfgS6woFFhgS6xoFFjgS6yoFFlgS6zoFFngS60oFFpgS61oFFrgS62oFFtgS63oFFvgS64oFFxgS65oFFzgS66oFF1gS67gNVd+PkBE5HMZ+DNkjHk0UK94TT2pe8FkECOOUi1Q4FKH85UQmq+gemrlwTVi5p5XOqGTofKK8RZmIeYCJ3+9fu7w4VOgvqdEXUTzJoN3/Z1kqkv8fO7ewvHsscte8u2wcTpGKKvTUQb1DMxrZX7VEa9rxllMozGUFkOdKMefd0DjwmZjqa+UC+v+zOZ6s9VkDsjbLxEnhmbeoxVl9UBjMGxM80WuxwJPHSsnWmiFkb6yZprHPlZ5I9gWb9+062ApffSq67d5W28dLzk9YvLdQV7w6fbZC+XPLc3Ts04Y2vcIKN8aZ/GuUQOwbJ+vtakRpLmqFrhJIS86VqkExn7c7K0QNMOk0Wq33gyj95WvWTRTM3+ST0fcNmT7pK/7nTSXdnAGz+nEahG07/EP36Oe5+nzPJqZQP20oJo1VACvU2jSe7H6jHV9DHNsWwV/kJ4L7o/vn2hXvXR//FjlphXVkWG+y/0q2YvBhBFzMY+Jn3p7WICxJBP1K8FxjvE6pU3VRsbp8xavW+azgKdM3WpHErY1O5F6o1TUO4ew/swg3HobzVesk7zOFJP+pdl98c3yxYyTUBx222cfNT93rZxmWKyFkZrofvjIEwmctEMnczP1SNjPDZ8FSxXgGet8APwwsRL0qrMlzDf8vamqVDLk6oCcDT19vETcQK/Fd4wTe+31p2HPOeZOp3T/J9Q43mgB4B/B5vq6muG6pw0hnjV7lKrFc3072cs+iiovnodMAI58IcYZ1H6OMzBv183WKoOwUwu8Fgf2AvD1HDh5zj/KR7+6+ufv9nYbv5PpdWLm7db22+2ti++3/jjbvW/mH659o9rv1l7tba99ru1P64drfXWrtdGa/+59l9r/732P+9O34l3T+/+ZEx//rOqzz+sOT/v/v3/AM3d200=</latexit>

q(zn | tn, xn)

xn

N
28

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N
29

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N
29

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N
29

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N

BERT
Data example x

29

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N

BERT

BERT

Data example x

Prototype xk

29Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

BERT

BERT

Data example x

Prototype xk

N Only the linear transformacon is learned while BERT is

ﬁxed. We use REINFORCE algorithm to propagate gradientsover discrete prototype variables
29

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)<latexit sha1_base64="eltIm+3ekJsYS/UYbPp4ywrrAN4=">AAA1qniclVvbcttKdtXM5DJRLnMmecwLKrJrnJSssnyOa1J5Gl2oi0VJlETdbPq4QHAThIWb0E2IFMM85wvymvxW/ia7u9HYu0HIp6IqS+i1djf6srr3IggP8zgS8t27//3Vr3/zZ3/+F3/5279a/+u/+du/+90Pv//7G5FNiwCugyzOiruhLyCOUriWkYzhLi/AT4Yx3A4f9hR/W0Ihoizty3kOXxI/TKNxFPgSocFgmCwGMYaP/OXXHzbebb3TP97qxXZ1sbFW/fS+/v6n/xyMsmCaQCqD2Bfi8/a7XH5Z+IWMghiW64OpgNwPHvwQPuNl6icgvix0p5fea0RG3jgr8F8qPY3yGgs/EWKeDDEy8eVENDkFtnGfp3L8r18WUZpPJaSBudF4Gnsy89QMeKOogEDGc7zwgyLCvnrBxC/8QOI8ra+/Vj/eWefWO93pH3n7nYPjs+P+8fnZlaep9baObOJfNQyxOUyW2IZ36hcPnsD74CwLLxt7gZ+bazXiAsZQFFEaqk6NojISNmwchdMCcEApPAVZkvjpaDFAMIaxXC4WA0i8N128/uflciUmwHWAwkbt6VJbXBGFk7qxS1Voi5JZbmP6Wd4WMcykzBIbtKtLK3HVuH0b5r8UMbQRw5ciAhsRvBQxshEjFYHLcISji9UIPd/DeLXoMMatMvJwbhK3DbxW4PLz9hdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9mbuy22dFNykmWLwadJtt5RLbzdTEoyueBiBLvEa/LfBIt3yjo3/HXbGXKOvlzo1auaskJSP/79epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYD5bGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWNHzd+Wqm2WdexVz/ypn7Sw7kyyeK704EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1t/LjaIs0aa/vH1bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCROjQTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvFxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjtQ1tTKSHdsUzstTVnB2/vhIOq2vn8Y9J1KO79YaWVS0TDIauRMfQo13VVX31sUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFI4eeOFZMjC9wIlc8J/Hix3wwo/Tga8YCv5rpIFoZarjQJQrZX0MyyHhHkQlnHXERxlmrbh1OLTWSJV/pFhEkMrL4xfy2McUuzIsFWXw0QerW001k0aJ+YocsMiQlcJiBm5DIjYsBlgJixy4yJCV0mJGbiMhNiIpeJiPnmMt+IeXCZB2Jil4mXWsZF4kUCdyx+dh3N1WFnVnDT+zYV0htl6R+kpz4/ohzn6uRxFsZLqrZTt+2U7pq5TEZM7jI5MY8u80hM4TIFMcJlBDHSZSQxU5eZElO6TEnMk8s8ETNzmRkxc5eZE/PsMs9LY9DsBsDMnNXHe1ltkoXZSsMx2zZ1v7XLYxGV66t5xnF4SDDbG2VAMNsY5YhgtitKIJhtiXJMMNsPZUgw2wzlhGC2E8opwWwblN8IZnugfCCYbYAyJjhmcEJwwmA20XyGM4KZmMucYKbk8pFgJuOyIJhpuBQEC76oBMv2OeHSLQlmui2fCGaiLWcEM8WWc4KZXMtngq1WOzGo51D6IUrRolswoms9l8Eor/VkBiO/1rMZjAZbT2cwQmw9n8GosfWEBiPJ1jMajC5bT2nkXjynwSi09aQGI9PWsxqMVpunteUSl0s49+JJDEa6rWcxGP22nsZgRNx6HoNRcuuJDEbOrWcyGE23nspghN16LoNRd+vJDEbirWczGJ23ns5gxN56PoNR/MsnNO6FIgpqh5Ls0P7YoW2T7BK8y+A9gvcYvE/wPoM7BHcYfEDwAYMPCT5k8BHBRww+JviYwR8J/sjgE4JPGNwluMvgU4JPGXxG8BmDzwk+Z3CP4B6DLwi+YPAlwZcMviL4isF9gvsMvib4msE3BN8w+JbgWwbfEXzH4HuC7xn8ieBPLx+vrujAqI5pdIfpV0uPcbuc23O5Pc7tu9w+5zou1+HcgcsdcO7Q5Q45d+RyR5w7drljzn10uY+cO3G5E851Xa7LuVOXO+Xcmcudce7c5c4513O5HucuXO6Cc5cud8m5K5e74lzf5fqcu3a5a87duNwN525d7pZzdy53x7l7l7vn3CeXs7K/4RaifAb9OQI/u76r65ZZCgv7edZiydRAg4SSRu2JFe76YfVstCL001QLV9EscGgQsifanCBCpkRbEkTIipRVB8mAaPuBCNkObToQIbOhrQYiZDHKqpOsh98MQnZCmwlEyERoC4FIzKbHIGQYtF1AJGXTapCMTZJByBJoQ4AIGQFtAxCh9K+TPyKCrYNBKNWX1WqxtSoNQmldJ3VEKJnrVI4IpXCdwBGhxK3TNiJtJtV1p6Uf5xO13vpvLcxyWGmmehBvQPoERg8sKsp8l7uov9StiSyBUOH6L8FaqUqlFsAGEcHfBIkoTFRV/Zdgq+f6W4JqIIsF7/9CidWWUKwBlVCoIzaohRKoLaFAx1RCcYZUQmFOqITdZX1FQX6jEorxgc3NQomwHvlCCdCWcDLZLKL4MjYlCyU6W0LRPVIJBVewmVooodUTtFAisyWcaDbNKLCSSiiuJyqhsGZUQlHNqYSCel5W3zBj+p0ZXKde1BmlXJ1wEaFEq9MsIpRedXJFhJKqTqmIUCrViRQRSqA6fSJCaVMnTUQoWepUiQilSJ0gEaHEqNMiIpQOdTJEhJKgToGIUOrTiQ8RSng63SFCaU4nOUQouenUhgilNJ3QEKFEptMYIpS+dPJChJKWTlmIUKrSiQoRSlA6PSFCaUknJUQoGelUhAilIJ2AEPnEVpDSxZBni6RXZ4seyxZJ1259xXSr7V8PrtrDirsy+1irqA+pUO9d7EMQ+wWgqCY76gTCOxoPKMaReoIKaZCNojTExvxprBAxrq+T5UKoh79XIF9qYJjFo19qZjhbLppfbkrsn/mmXKfTqj398LoamjS2MxVM/XLXYqR/uWcx2gFy32K0B2THYrQL5IHFaB/IQ4vRTpBHFqO9II8tRrtBfrQY7Qd5YjHaEbJrMdoT8tRitCvkmcVoX8hzi9HOkD2L0d6QFxaj3SEvLUb7Q15ZjHaI7FuM9oi8thjtEnljMdon8tZitFPkncVor8h7i9FukZ8sZowaCvmw8POJYUP78TdwPoWEuwwmXYR7DCZphPsMJnWEHQaTQMIDBpNGwkMGk0zCIwaTUsJjBpNYwo8MJr2EJwwmyYRdBpNqwlMGk3DCMwaTdsJzBpN8wh6DSUHhBYNJROElg0lH4RWDSUphn8GkpvCawSSo8IbBpKnwlsEkq/COwaSs8J7BJK7wE4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfb29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeE/ZNB4hhCXwhP46BL3ktPDUi3JZjA2pt8tglqO31N/x2m/aO3RHEq04IJQ0Kw4JJcmKI0JJseKYUBKs+Ego6VWcEEpyFV1CSa3ilFASqzgjlLQqzgklqYoeoaRUcUEoCVVcEko6FVeEkkxFn1BSqbgmlEQqbggljYpbQkmi4o5QUqi4J5QEKj4RWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUg+UgnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WPdUwkX6xO5X+a/Ke6klA75k0vgwPGjUrtZvxOLWNuim9xTJSTaVHpog7wkTXQ6Fa5OAfJLjkarby1oDOnDFHoI2UdBwUaBtFDR8FGgjBQ0nBdpKQcNLgTZT0HBToO0UNPwUaEMFDUcF2lJBw1OBNlXQcFWgbRU0fBVoYwUNZwXaWkHDW4E2V9BwV6DtFTT8FWiDBQ2HBdpiQcNjgTZZ0HBZoG0WNHwWaKMFDacF2mpBw2uBNlvQcFug7RY0/BZowwUNxwXackHDc4E2XdBwXaBtFzDfhZ8fMBHJYgreNB1BEc/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7nCuk6B+Z0TdBLNmo237OsnEl/j53b2FE9njkb1lW2eSbATx9waiA+qRmNLKfaqg3veCchnFI6giB7pQ976ugceEzIKJL9TL6/5UZvpzFRRODxsvkecmpu5jVWW1AyNw4kyxJa5AAg8dG2eKqIVAP1lzg2M/j/0AlvXrN90KWHqvverand7GS8dL7l9crivYGz7dJnu55Lm9cWomOZvjBhkXS/s0ziUKCJf187UmFUgaoypF4wiKZtMiG8vEn1GkBZpxmCwy/caTefS22koeT9Xon9XzAZc96S75604n3ZUFvPEL6oEqNNuX+McvcO2LjEVerSzAXlYSrQpKoLdZPC78RD2mmjxlBdpW4c+F96r78/tX6lUf/R8/pql5ZVXkuP5Cv2r2agBxzGLsY9LX3i4mQNzyqfo1x/0OiXrlTXlj0yiLVu+bZtNQ50xtlSMJm7p5kXmjDFRzT9FDlMMo8rcaL1lnRRKrJ/3LRffnd8sWMktBcdttnHzS9d63cbli8hZGa6H78yBKx3Le3Dq5X6hHxnhs+GqzXAGetcIPwYtSL80qmy9htuXtTTKhpidTBjCYePv4iTiFPwhvmGUPW+vOQ57zXJ3OWfEvqPEi1B3Av4NNdfW9QHVOmkC8am9SqxXD9O8XIvooqL56HTAGOfCHuM/i7GlYgP+wbrBMHYK5nOOxPrAXhqnh0i9w/BM8/NfXv/6wsd38n0qrFzfvt7bfbW1f/LTxp93qfzH9du0f1/5p7c3a9tof1/60drTWW7teC9bytf9a+++1//mw+eHyw/2Hzyb017+q6vzDmvPzYfR/rPTO/w==</latexit>

zn
q(zn | tn, xn)

xn

N
30

Carnegie Mellon University

The Approximate Posterior
θtn
q(tn | xn)

zn

Edit Operation

q(zn | tn, xn)

xn

N

=

=

$

$

—

<latexit sha1_base64="CkwhKTfPnHGnDGSq4dwPXVdXRrM=">AAA1r3iclVvbctvKlVUymUmizExOJo95QUV2xTMlqyyf45qpPEUX6mJREiVRF/vQxwHBTRAWbkI3QUoM53n+Ia+Zj5q/md3daOzdIORTUZUl9Fq7G31Z3XsRhId5HAn55s3//eSn//Czf/ynn//il+u/+ud/+ddff/Obf7sR2bQI4DrI4qy4G/oC4iiFaxnJGO7yAvxkGMPt8H5P8bclFCLK0r58zOFT4odpNI4CXyL0+ZtfD2IYyyIKJ9Ivimz2+ZuNN1tv9I+3erFdXWysVT+9z7/57n8GoyyYJpDKIPaF+H77TS4/LfxCRkEMy/XBVEDuB/d+CN/jZeonID4tdM+X3ktERt44K/BfKj2N8hoLPxHiMRliZOLLiWhyCmzjvp/K8X99WkRpPpWQBuZG42nsycxT0+CNogICGT/ihR8UEfbVCyZ+4QcSJ2t9/aX68c46t97pTv/I2+8cHJ8d94/Pz648Ta23dWQT/6phiM1hssQ2vFO/uPcE3genWnjZ2Av83FyrERcwhqKI0lB1ahSVkbBh4yicFoADSmEWZEnip6PFAEG1VMvFYgCJ96qL1/++XK7EBLgOUNioPV1qi9NLbsMuVaEtSma5jelneVvEMJMyS2zQri6txFXj9m2Y/1zE0EYMn4sIbETwXMTIRoxUBC7DEY4uViP0fA/j1aLDGPfLyMO5Sdw28FqBy++3P2Erw7G3sa0aaQ57vlwMEr8IUWB+sTg4vmv2Ba+dEJRSM6R/vn+u7zOQMJda+osCsPeK+KO5sdtmRzcpJ1m+GHSabOcB2c7nxaAonwYiSrwHvC7zSbR8paC/4K/5ypR18qdGrVzVkhOQ/tfr1dXmVbXRq9bIhyecq9auuHG5invm5o3I+dNK5FxFPq1GrgauxIx00KidVHd61db0w1NzVM0INbEIN9C5RucN9FGjjw000WjSVIFG02bsVCpxTLFL803PjnglKHeCVN8bISNTEW/jD2Of5q4ZpqqyoLaWEFtpDDHeO9ycB/qsM4chntSw6cXZDIrXASa1rfUB7lR9WsF4Y3thzsX/HmBpobdHW3U8BSLpx1veAZ6xArOaVEeqUAch8qbFA9viQbNFTctZZu+58ba6q/BskIfDqwpvbY2HqT+iKhvfbny3Um2zrmOvvuVNfaeHc2WSxVenAxOK6XyVWZz5aGnAToipfWVrX7XUvrS1dJ6cZXXy2qonxtxd6JmpU9szU9NscFIANJtk7W18u9oizRpr+9vVtv3UA1wEVbllyuDBjNmGPD9op51pnkPhqXZMM52qmU5bMzte4c9o3huNvX792i+zaORNhcr40djLMyEitGmm6Tz2MSNV7T/fO2VSckxQLWNUjKlexfzdg6wa2qsb2vvRhnDMaQja2phYYdrQcN0jlIqlbVOvXz8rE+ydH4cZmrJJ0jJO5Ezv6qCvDpQ1tTLSHdvUTktTVvD2fjiIuq2vHwZ9p9LOj1ZamVQ0DLIaOVOfQk131dXXFsXUb6q3V9fvufXtSOsbYK/V9bMdrgQHUazEGqsLtCsYoK6q9sZxlhWa1leG15dVAFLDZLFicmSBG6HyOYEfL/abAaUfRyMe8NlcF8nCUMuVJkHI9gqaWdYjglwo65iLKM5SbftwarGJLPFKv4gwiYHVN+avhTFuaVYk2OqLAUIvlnY6iwbtEzN0mSExgcsExIxcZkQMuAwQM3aZMTGhy4TETFxmQkzkMhExX1zmCzH3LnNPTOwy8VLLuEi8SOCOxQ+wo0d12JkV3PS+TIX0Rln6B+mpz48ox0d18jgL4yVV26nbdkp3zVwmIyZ3mZyYB5d5IKZwmYIY4TKCGOkykpipy0yJKV2mJGbmMjNi5i4zJ+bRZR6JeXKZp6UxaHYDYGbO6uO9rDbJwmyl4Zhtm7rf2uWxiMr11TzjODwkmO2NMiCYbYxyRDDbFSUQzLZEOSaY7YcyJJhthnJCMNsJ5ZRgtg3KLwSzPVDeE8w2QBkTHDM4IThhMJtoPsMZwUzMZU4wU3L5QDCTcVkQzDRcCoIFX1SCZfuccOmWBDPdljOCmWjLOcFMseUjwUyu5RPBVqudGNRzKP0QpWjRLRjRtZ7LYJTXejKDkV/r2QxGg62nMxghtp7PYNTYekKDkWTrGQ1Gl62nNHLPntNgFNp6UoORaetZDUarzdPaconLJZx79iQGI93WsxiMfltPYzAibj2PwSi59UQGI+fWMxmMpltPZTDCbj2Xwai79WQGI/HWsxmMzltPZzBibz2fwSj++RMa90IRBbVDSXZof+zQtkl2Cd5l8B7BewzeJ3ifwR2COww+IPiAwYcEHzL4iOAjBh8TfMzg9wS/Z/AJwScM7hLcZfApwacMPiP4jMHnBJ8zuEdwj8EXBF8w+JLgSwZfEXzF4D7BfQZfE3zN4BuCbxh8S/Atg+8IvmPwB4I/MPgjwR+fP15d0YFRHdPoDtOvlh7jdjm353J7nNt3uX3OdVyuw7kDlzvg3KHLHXLuyOWOOHfscsece+9y7zl34nInnOu6XJdzpy53yrkzlzvj3LnLnXOu53I9zl243AXnLl3uknNXLnfFub7L9Tl37XLXnLtxuRvO3brcLefuXO6Ocx9c7gPnPrqclf0NtxDlE+jPEfjZ9U1dt8xSWNjPsxZLpgYaJJQ0ak+scNcPq2ejFaGfplq4imaBQ4OQPdHmBBEyJdqSIEJWpKw6SAZE2w9EyHZo04EImQ1tNRAhi1FWnWQ9/GIQshPaTCBCJkJbCERiNj0GIcOg7QIiKZtWg2RskgxClkAbAkTICGgbgAilf538ERFsHQxCqb6sVoutVWkQSus6qSNCyVynckQohesEjgglbp22EWkzqa47Lf04n6j11n9rYZbDSjPVg3gD0icwemBRUbGfDEeqhrkgIksgVLj+S7BWqlKpBbBBRPA3QSIKE1VV/yXY6rn+lqAayGLB+79QYrUlFGtAJRTqiA1qoQRqSyjQMZVQnCGVUJgTKmF3WV9RkF+ohGK8Z3OzUCKsR75QArQlnEw2iyi+jE3JQonOllB0D1RCwRVsphZKaPUELZTIbAknmk0zCqykEoprRiUU1pxKKKpHKqGgnpbVN8yYfucG16kXdUYpVydcRCjR6jSLCKVXnVwRoaSqUyoilEp1IkWEEqhOn4hQ2tRJExFKljpVIkIpUidIRCgx6rSICKVDnQwRoSSoUyAilPp04kOEEp5Od4hQmtNJDhFKbjq1IUIpTSc0RCiR6TSGCKUvnbwQoaSlUxYilKp0okKEEpROT4hQWtJJCRFKRjoVIUIpSCcgRD6yFaR0MeTZIunV2aLHskXStVtfMd1q+9eDq/aw4q7MPtYq6kMq1HsX+xDEfgEoqsmOOoHwjsYDinGknqBCGmSjKA2xMX8aK0SM6+tkuRDq4e8VyOcaGGbx6MeaGc6Xi+aXmxL7Z74p1+m0ak8/vK6GJo3tTAVTv9y1GOlf7lmMdoDctxjtAdmxGO0CeWAx2gfy0GK0E+SRxWgvyGOL0W6Q7y1G+0GeWIx2hOxajPaEPLUY7Qp5ZjHaF/LcYrQzZM9itDfkhcVod8hLi9H+kFcWox0i+xajPSKvLUa7RN5YjPaJvLUY7RR5ZzHaK/KDxWi3yI8WM0YNhXxY+PnEsKH9+Bs4n0LCXQaTLsI9BpM0wn0GkzrCDoNJIOEBg0kj4SGDSSbhEYNJKeExg0ks4XsGk17CEwaTZMIug0k14SmDSTjhGYNJO+E5g0k+YY/BpKDwgsEkovCSwaSj8IrBJKWwz2BSU3jNYBJUeMNg0lR4y2CSVXjHYFJW+IHBJK7wI4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfT29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeLNsGo8QwhJ4Qn8dgl5yWnjqRbksxobU22Uwz9Fb6u947TftHbojiVYcEEqaFYeEkmTFEaGkWHFMKAlWvCeU9CpOCCW5ii6hpFZxSiiJVZwRSloV54SSVEWPUFKquCCUhCouCSWdiitCSaaiTyipVFwTSiIVN4SSRsUtoSRRcUcoKVR8IJQEKj4SWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUjeUwnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WB+ohIv0kd2v8l+V91JLBnzJpPFheNCoXa3fiMWtbdBNbxbJSTaVHpogb4aJLofCtUlAPsnxSNXtZa0BHbhiD0GbKGi4KNA2Cho+CrSRgoaTAm2loOGlQJspaLgp0HYKGn4KtKGChqMCbamg4alAmypouCrQtgoavgq0sYKGswJtraDhrUCbK2i4K9D2Chr+CrTBgobDAm2xoOGxQJssaLgs0DYLGj4LtNGChtMCbbWg4bVAmy1ouC3Qdgsafgu04YKG4wJtuaDhuUCbLmi4LtC2C5jvws8PmIhkMQVvmo6giB/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7vBRJ0H9zoi6CWbNRtv2dZKJL/Hzu3sLJ7LHI3vLts4k2Qjirw1EB9QjMaWV+1RBva8F5TKKR1BFDnSh7n1dA48JmQUTX6iX1/2pzPTnKiicHjZeIs9NTN3HqspqB0bgxJliS1yBBB46Ns4UUQuBfrLmBsd+HvsBLOvXb7oVsPReetW1O72Nl46X3L+4XFewN3y6TfZyyXN749RMcjbHDTIulvZpnEsUEC7r52tNKpA0RlWKxhEUzaZFNpaJP6dICzTjMFlk+o0n8+httZU8nqrRP6nnAy570l3y151OuisLeOMX1ANVaLYv8Y9f4NoXGYu8WlmAvawkWhWUQG+zeFz4iXpMNZllBdpW4T8K70X3h7cv1Ks++j9+TFPzyqrIcf2FftXsxQDimMXYx6QvvV1MgLjlU/XrEfc7JOqVN+WNTaMsWr1vmk1DnTO1VY4kbOrmReaNMlDNzaL7KIdR5G81XrLOiiRWT/qXi+4Pb5YtZJaC4rbbODnT9d62cbli8hZGa6H7wyBKx/KxuXVyv1CPjPHY8NVmuQI8a4UfghelXppVNl/CfMvbm2RCTU+mDGAw8fbxE3EKfxDeMMvut9adhzznuTqds+I/UONFqDuAfweb6uprgeqcNIF41d6kViuG6d/PRPRRUH31OmAMcuAPcZ/F2WxYgH+/brBMHYK5fMRjfWAvDFPDpV/g+Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++G7jT7vV/2L6xdrv1n6/9mpte+0/1/60drTWW7teC9ama39d+9va/77bfnf77od3fzahP/1JVee3a87Pu+j/AQxz0L4=</latexit><latexit<latexit sha1_base64="CkwhKTfPnHGnDGSq4dwPXVdXRrM=">AAA1r3iclVvbctvKlVUymUmizExOJo95QUV2xTMlqyyf45qpPEUX6mJREiVRF/vQxwHBTRAWbkI3QUoM53n+Ia+Zj5q/md3daOzdIORTUZUl9Fq7G31Z3XsRhId5HAn55s3//eSn//Czf/ynn//il+u/+ud/+ddff/Obf7sR2bQI4DrI4qy4G/oC4iiFaxnJGO7yAvxkGMPt8H5P8bclFCLK0r58zOFT4odpNI4CXyL0+ZtfD2IYyyIKJ9Ivimz2+ZuNN1tv9I+3erFdXWysVT+9z7/57n8GoyyYJpDKIPaF+H77TS4/LfxCRkEMy/XBVEDuB/d+CN/jZeonID4tdM+X3ktERt44K/BfKj2N8hoLPxHiMRliZOLLiWhyCmzjvp/K8X99WkRpPpWQBuZG42nsycxT0+CNogICGT/ihR8UEfbVCyZ+4QcSJ2t9/aX68c46t97pTv/I2+8cHJ8d94/Pz648Ta23dWQT/6phiM1hssQ2vFO/uPcE3genWnjZ2Av83FyrERcwhqKI0lB1ahSVkbBh4yicFoADSmEWZEnip6PFAEG1VMvFYgCJ96qL1/++XK7EBL<latexit
sha1_base64="CkwhKTfPnHGnDGSq4dwPXVdXRrM=">AAA1r3iclVvbctvKlVUymUmizExOJo95QUV2xTMlqyyf45qpPEUX6mJREiVRF/vQxwHBTRAWbkI3QUoM53n+Ia+Zj5q/md3daOzdIORTUZUl9Fq7G31Z3XsRhId5HAn55s3//eSn//Czf/ynn//il+u/+ud/+ddff/Obf7sR2bQI4DrI4qy4G/oC4iiFaxnJGO7yAvxkGMPt8H5P8bclFCLK0r58zOFT4odpNI4CXyL0+ZtfD2IYyyIKJ9Ivimz2+ZuNN1tv9I+3erFdXWysVT+9z7/57n8GoyyYJpDKIPaF+H77TS4/LfxCRkEMy/XBVEDuB/d+CN/jZeonID4tdM+X3ktERt44K/BfKj2N8hoLPxHiMRliZOLLiWhyCmzjvp/K8X99WkRpPpWQBuZG42nsycxT0+CNogICGT/ihR8UEfbVCyZ+4QcSJ2t9/aX68c46t97pTv/I2+8cHJ8d94/Pz648Ta23dWQT/6phiM1hssQ2vFO/uPcE3genWnjZ2Av83FyrERcwhqKI0lB1ahSVkbBh4yicFoADSmEWZEnip6PFAEG1VMvFYgCJ96qL1/++XK7EBLgOUNioPV1qi9NLbsMuVaEtSma5jelneVvEMJMyS2zQri6txFXj9m2Y/1zE0EYMn4sIbETwXMTIRoxUBC7DEY4uViP0fA/j1aLDGPfLyMO5Sdw28FqBy++3P2Erw7G3sa0aaQ57vlwMEr8IUWB+sTg4vmv2Ba+dEJRSM6R/vn+u7zOQMJda+osCsPeK+KO5sdtmRzcpJ1m+GHSabOcB2c7nxaAonwYiSrwHvC7zSbR8paC/4K/5ypR18qdGrVzVkhOQ/tfr1dXmVbXRq9bIhyecq9auuHG5invm5o3I+dNK5FxFPq1GrgauxIx00KidVHd61db0w1NzVM0INbEIN9C5RucN9FGjjw000WjSVIFG02bsVCpxTLFL803PjnglKHeCVN8bISNTEW/jD2Of5q4ZpqqyoLaWEFtpDDHeO9ycB/qsM4chntSw6cXZDIrXASa1rfUB7lR9WsF4Y3thzsX/HmBpobdHW3U8BSLpx1veAZ6xArOaVEeqUAch8qbFA9viQbNFTctZZu+58ba6q/BskIfDqwpvbY2HqT+iKhvfbny3Um2zrmOvvuVNfaeHc2WSxVenAxOK6XyVWZz5aGnAToipfWVrX7XUvrS1dJ6cZXXy2qonxtxd6JmpU9szU9NscFIANJtk7W18u9oizRpr+9vVtv3UA1wEVbllyuDBjNmGPD9op51pnkPhqXZMM52qmU5bMzte4c9o3huNvX792i+zaORNhcr40djLMyEitGmm6Tz2MSNV7T/fO2VSckxQLWNUjKlexfzdg6wa2qsb2vvRhnDMaQja2phYYdrQcN0jlIqlbVOvXz8rE+ydH4cZmrJJ0jJO5Ezv6qCvDpQ1tTLSHdvUTktTVvD2fjiIuq2vHwZ9p9LOj1ZamVQ0DLIaOVOfQk131dXXFsXUb6q3V9fvufXtSOsbYK/V9bMdrgQHUazEGqsLtCsYoK6q9sZxlhWa1leG15dVAFLDZLFicmSBG6HyOYEfL/abAaUfRyMe8NlcF8nCUMuVJkHI9gqaWdYjglwo65iLKM5SbftwarGJLPFKv4gwiYHVN+avhTFuaVYk2OqLAUIvlnY6iwbtEzN0mSExgcsExIxcZkQMuAwQM3aZMTGhy4TETFxmQkzkMhExX1zmCzH3LnNPTOwy8VLLuEi8SOCOxQ+wo0d12JkV3PS+TIX0Rln6B+mpz48ox0d18jgL4yVV26nbdkp3zVwmIyZ3mZyYB5d5IKZwmYIY4TKCGOkykpipy0yJKV2mJGbmMjNi5i4zJ+bRZR6JeXKZp6UxaHYDYGbO6uO9rDbJwmyl4Zhtm7rf2uWxiMr11TzjODwkmO2NMiCYbYxyRDDbFSUQzLZEOSaY7YcyJJhthnJCMNsJ5ZRgtg3KLwSzPVDeE8w2QBkTHDM4IThhMJtoPsMZwUzMZU4wU3L5QDCTcVkQzDRcCoIFX1SCZfuccOmWBDPdljOCmWjLOcFMseUjwUyu5RPBVqudGNRzKP0QpWjRLRjRtZ7LYJTXejKDkV/r2QxGg62nMxghtp7PYNTYekKDkWTrGQ1Gl62nNHLPntNgFNp6UoORaetZDUarzdPaconLJZx79iQGI93WsxiMfltPYzAibj2PwSi59UQGI+fWMxmMpltPZTDCbj2Xwai79WQGI/HWsxmMzltPZzBibz2fwSj++RMa90IRBbVDSXZof+zQtkl2Cd5l8B7BewzeJ3ifwR2COww+IPiAwYcEHzL4iOAjBh8TfMzg9wS/Z/AJwScM7hLcZfApwacMPiP4jMHnBJ8zuEdwj8EXBF8w+JLgSwZfEXzF4D7BfQZfE3zN4BuCbxh8S/Atg+8IvmPwB4I/MPgjwR+fP15d0YFRHdPoDtOvlh7jdjm353J7nNt3uX3OdVyuw7kDlzvg3KHLHXLuyOWOOHfscsece+9y7zl34nInnOu6XJdzpy53yrkzlzvj3LnLnXOu53I9zl243AXnLl3uknNXLnfFub7L9Tl37XLXnLtxuRvO3brcLefuXO6Ocx9c7gPnPrqclf0NtxDlE+jPEfjZ9U1dt8xSWNjPsxZLpgYaJJQ0ak+scNcPq2ejFaGfplq4imaBQ4OQPdHmBBEyJdqSIEJWpKw6SAZE2w9EyHZo04EImQ1tNRAhi1FWnWQ9/GIQshPaTCBCJkJbCERiNj0GIcOg7QIiKZtWg2RskgxClkAbAkTICGgbgAilf538ERFsHQxCqb6sVoutVWkQSus6qSNCyVynckQohesEjgglbp22EWkzqa47Lf04n6j11n9rYZbDSjPVg3gD0icwemBRUbGfDEeqhrkgIksgVLj+S7BWqlKpBbBBRPA3QSIKE1VV/yXY6rn+lqAayGLB+79QYrUlFGtAJRTqiA1qoQRqSyjQMZVQnCGVUJgTKmF3WV9RkF+ohGK8Z3OzUCKsR75QArQlnEw2iyi+jE3JQonOllB0D1RCwRVsphZKaPUELZTIbAknmk0zCqykEoprRiUU1pxKKKpHKqGgnpbVN8yYfucG16kXdUYpVydcRCjR6jSLCKVXnVwRoaSqUyoilEp1IkWEEqhOn4hQ2tRJExFKljpVIkIpUidIRCgx6rSICKVDnQwRoSSoUyAilPp04kOEEp5Od4hQmtNJDhFKbjq1IUIpTSc0RCiR6TSGCKUvnbwQoaSlUxYilKp0okKEEpROT4hQWtJJCRFKRjoVIUIpSCcgRD6yFaR0MeTZIunV2aLHskXStVtfMd1q+9eDq/aw4q7MPtYq6kMq1HsX+xDEfgEoqsmOOoHwjsYDinGknqBCGmSjKA2xMX8aK0SM6+tkuRDq4e8VyOcaGGbx6MeaGc6Xi+aXmxL7Z74p1+m0ak8/vK6GJo3tTAVTv9y1GOlf7lmMdoDctxjtAdmxGO0CeWAx2gfy0GK0E+SRxWgvyGOL0W6Q7y1G+0GeWIx2hOxajPaEPLUY7Qp5ZjHaF/LcYrQzZM9itDfkhcVod8hLi9H+kFcWox0i+xajPSKvLUa7RN5YjPaJvLUY7RR5ZzHaK/KDxWi3yI8WM0YNhXxY+PnEsKH9+Bs4n0LCXQaTLsI9BpM0wn0GkzrCDoNJIOEBg0kj4SGDSSbhEYNJKeExg0ks4XsGk17CEwaTZMIug0k14SmDSTjhGYNJO+E5g0k+YY/BpKDwgsEkovCSwaSj8IrBJKWwz2BSU3jNYBJUeMNg0lR4y2CSVXjHYFJW+IHBJK7wI4PtBwE82iqrJuqHK0MmLrFLKGlL7BFK0hL7hGplvfT29RccUwGe7wmQHt46hpHX2fSGEPgKl5NIeLNsGo8QwhJ4Qn8dgl5yWnjqRbksxobU22Uwz9Fb6u947TftHbojiVYcEEqaFYeEkmTFEaGkWHFMKAlWvCeU9CpOCCW5ii6hpFZxSiiJVZwRSloV54SSVEWPUFKquCCUhCouCSWdiitCSaaiTyipVFwTSiIVN4SSRsUtoSRRcUcoKVR8IJQEKj4SWj+fSdENgv5g4ZsnM5U1BPIFXfcjgTKNO1RCAe9SCYW7RyUU7D6VUEwdKqGIDqiE4jmkEormiEoolmMqoUjeUwnFcUIlFEWXSiiGUyqhCM6ohIt/TiVc9B6VcLEvqISLfEklXNwrKuGi9qmEi3lNJVzEGyrh4t1SCRftjkq4WB+ohIv0kd2v8l+V91JLBnzJpPFheNCoXa3fiMWtbdBNbxbJSTaVHpogb4aJLofCtUlAPsnxSNXtZa0BHbhiD0GbKGi4KNA2Cho+CrSRgoaTAm2loOGlQJspaLgp0HYKGn4KtKGChqMCbamg4alAmypouCrQtgoavgq0sYKGswJtraDhrUCbK2i4K9D2Chr+CrTBgobDAm2xoOGxQJssaLgs0DYLGj4LtNGChtMCbbWg4bVAmy1ouC3Qdgsafgu04YKG4wJtuaDhuUCbLmi4LtC2C5jvws8PmIhkMQVvmo6giB/VG04jX/peCCkUmINUORKo9OFUJaTmG5i+eklQvahZJAtd0OlQtQpJHhURJkKnfv3+7vBRJ0H9zoi6CWbNRtv2dZKJL/Hzu3sLJ7LHI3vLts4k2Qjirw1EB9QjMaWV+1RBva8F5TKKR1BFDnSh7n1dA48JmQUTX6iX1/2pzPTnKiicHjZeIs9NTN3HqspqB0bgxJliS1yBBB46Ns4UUQuBfrLmBsd+HvsBLOvXb7oVsPReetW1O72Nl46X3L+4XFewN3y6TfZyyXN749RMcjbHDTIulvZpnEsUEC7r52tNKpA0RlWKxhEUzaZFNpaJP6dICzTjMFlk+o0n8+httZU8nqrRP6nnAy570l3y151OuisLeOMX1ANVaLYv8Y9f4NoXGYu8WlmAvawkWhWUQG+zeFz4iXpMNZllBdpW4T8K70X3h7cv1Ks++j9+TFPzyqrIcf2FftXsxQDimMXYx6QvvV1MgLjlU/XrEfc7JOqVN+WNTaMsWr1vmk1DnTO1VY4kbOrmReaNMlDNzaL7KIdR5G81XrLOiiRWT/qXi+4Pb5YtZJaC4rbbODnT9d62cbli8hZGa6H7wyBKx/KxuXVyv1CPjPHY8NVmuQI8a4UfghelXppVNl/CfMvbm2RCTU+mDGAw8fbxE3EKfxDeMMvut9adhzznuTqds+I/UONFqDuAfweb6uprgeqcNIF41d6kViuG6d/PRPRRUH31OmAMcuAPcZ/F2WxYgH+/brBMHYK5fMRjfWAvDFPDpV/g+Cd4+K+vf/5mY7v5P5VWL27ebm2/2dq++G7jT7vV/2L6xdrv1n6/9mpte+0/1/60drTWW7teC9ama39d+9va/77bfnf77od3fzahP/1JVee3a87Pu+j/AQxz0L4=</latexit>

Prototype tnIorderedsweet

potatofries

Data xnI

orderedaburger?

30<latexit sha1_base64="8Vaw+NtjnY8DDN1Idw+YV9AmdXw=">AAA1p3iclVtZc9tKdtZMthlNljvJ47ygIrvGk5JVlu91zVSeRgu1WJRESdTmS18XCB6CsLAJ3YRIcZjnvOc1+WH5NzndjcY5DUK+FVXZQn/f6UYvX/f5CELDPI6EfPfuf3/xy7/667/527/71a/Xf/P3//CP//Tdb//5RmTTIoDrIIuz4m7oC4ijFK5lJGO4ywvwk2EMt8OHPcXfllCIKEv7cp7D58QP02gcBb5E6H4ASS7nAuSX7zbebb3TP97qxXZ1sbFW/fS+/PaH/xyMsmCaQCqD2Bfix+13ufy88AsZBTEs1wdTAbkfPPgh/IiXqZ+A+LzQPV56rxEZeeOswH+p9DTKayz8RIh5MsTIxJcT0eQU2Mb9OJXjP31eRGk+lZAG5kbjaezJzFPD90ZRAYGM53jhB0WEffWCiV/4gcRJWl9/rX68s86td7rTP/L2OwfHZ8f94/OzK09T620d2cTfahhic5gssQ3v1C8ePIH3wSkWXjb2Aj8312rEBYyhKKI0VJ0aRWUkbNg4CqcF4IBSeAqyJPHT0WKAYAxjuVwscLG8N128/sNyuRIT4DpAYaP2dKktrojCSd3YpSq0RckstzH9LG+LGGZSZokN2tWllbhq3L4N81+KGNqI4UsRgY0IXooY2YiRisBlOMLRxWqEnu9hvFp0GOM+GXk4N4nbBl4rcPnj9mdsZTj2NrZVI81hz5aLQeIXIQrMLxYHx3fNvuC1E4JSaob0z/fP9X0GEmZSS39RAPZeEf9ubuy22dFNykmWLwadJtt5RLbzZTEoyueBiBLvEa/LfBIt3yjoL/jfbGXKOvlzo1auaskJSP/b9epqs6ra6E1r5OMzzlVrV9y4XMW9cPNG5Ox5JXKmIp9XI1cDV2JGOmjUTqo7vWlr+vG5OapmhJpYhBvoTKOzBjrX6LyBJhpNmirQaNqMnUoljil2abbp2RGvBOVOkOp7I2RkKuJt/GHs09w1w1RVFtTWEmIrjSHGe4eb80CfdeYwxJMaNr04e4LibYDJbGt9gDtVn1Yw3themHPxPwZYWujt0VYdT4FI+vGWd4BnrJCYh9SRKtRBiLxp8cC2eNBsUdPyKbP33Hhf3VV4NsjD4VWF97bG49QfUZWN7zd+WKm2WdexV9/zpn7Qw7kyyeKb04EJxXS+yizOfLQ0YCfE1L6yta9aal/aWjpPPmV18tqqJ8bcXeiZqVPbC1PTbHBSADSbZO1tfL/aIs0aa/v71bb91ANcBFW5Zcrg0YzZhrw8aKedaZ5D4al2TDOdqplOWzM7XuE/0bw3Gnv79q1fZtHImwqV8aOxl2dCRGjPTNN57GNGqtp/uXfKpOSYoFrGqBhTvYr5fw+yamivbmjvZxvCMachaGtjYoVpQ8N1j1AqlrZNvX37okywd34cZmjKJknLOJEzvauDvjlQ1tTKSHdsUzstTVnB2/vhIOq2vn0Y9J1KOz9baWVS0TDIauRMfQo13VVX31oUU7+p3l5dv+fWtyOtb4C9VtcvdrgSHESxEmusLtCuYIC6qtobx1lWaFpfGV5fVgFIDZPFismRBW6EyucEfrzYbwaUfhyNeMAXc10kC0MtV5oEIdsraGZZjwhyoaxjLqI4S7Xtw6nFJrLEK/0iwiQGVt+YvxbGuKVZkWCrrwYIvVra6SwatE/M0GWGxAQuExAzcpkRMeAyQMzYZcbEhC4TEjNxmQkxkctExHx1ma/EPLjMAzGxy8RLLeMi8SKBOxY/uI7m6rAzK7jpfZ0K6Y2y9PfSU58fUY5zdfI4C+MlVdup23ZKd81cJiMmd5mcmEeXeSSmcJmCGOEyghjpMpKYqctMiSldpiTmyWWeiJm5zIyYucvMiXl2meelMWh2A2Bmzurjvaw2ycJspeGYbZu639rlsYjK9dU84zg8JJjtjTIgmG2MckQw2xUlEMy2RDkmmO2HMiSYbYZyQjDbCeWUYLYNyq8Esz1QPhDMNkAZExwzOCE4YTCbaD7DGcFMzGVOMFNy+Ugwk3FZEMw0XAqCBV9UgmX7nHDplgQz3ZZPBDPRljOCmWLLOcFMruUzwVarnRjUcyj9EKVo0S0Y0bWey2CU13oyg5Ff69kMRoOtpzMYIbaez2DU2HpCg5Fk6xkNRpetpzRyL57TYBTaelKDkWnrWQ1Gq83T2nKJyyWce/EkBiPd1rMYjH5bT2MwIm49j8EoufVEBiPn1jMZjKZbT2Uwwm49l8Gou/VkBiPx1rMZjM5bT2cwYm89n8Eo/uUTGvdCEQW1Q0l2aH/s0LZJdgneZfAewXsM3id4n8EdgjsMPiD4gMGHBB8y+IjgIwYfE3zM4I8Ef2TwCcEnDO4S3GXwKcGnDD4j+IzB5wSfM7hHcI/BFwRfMPiS4EsGXxF8xeA+wX0GXxN8zeAbgm8YfEvwLYPvCL5j8D3B9wz+RPCnl49XV3RgVMc0usP0q6XHuF3O7bncHuf2XW6fcx2X63DuwOUOOHfocoecO3K5I84du9wx5z663EfOnbjcCee6Ltfl3KnLnXLuzOXOOHfucuec67lcj3MXLnfBuUuXu+Tclctdca7vcn3OXbvcNeduXO6Gc7cud8u5O5e749y9y91z7pPLWdnfcAtRPoP+HIGfXd/VdcsshYX9PGuxZGqgQUJJo/bECnf9sHo2WhH6aaqFq2gWODQI2RNtThAhU6ItCSJkRcqqg2RAtP1AhGyHNh2IkNnQVgMRshhl1UnWw68GITuhzQQiZCK0hUAkZtNjEDIM2i4gkrJpNUjGJskgZAm0IUCEjIC2AYhQ+tfJHxHB1sEglOrLarXYWpUGobSukzoilMx1KkeEUrhO4IhQ4tZpG5E2k+q609KP84lab/27FmY5rDRTPYg3IH0CowcWFRX7yXCkapgLIrIEQoXr3wRrpSqVWgAbRAT/J0hEYaKq6t8EWz3X3xJUA1kseP8XSqy2hGINqIRCHbFBLZRAbQkFOqYSijOkEgpzQiXsLusrCvIrlVCMD2xuFkqE9cgXSoC2hJPJZhHFl7EpWSjR2RKK7pFKKLiCzdRCCa2eoIUSmS3hRLNpRoGVVEJxPVEJhTWjEopqTiUU1POy+oYZ0+/M4Dr1os4o5eqEiwglWp1mEaH0qpMrIpRUdUpFhFKpTqSIUALV6RMRSps6aSJCyVKnSkQoReoEiQglRp0WEaF0qJMhIpQEdQpEhFKfTnyIUMLT6Q4RSnM6ySFCyU2nNkQopemEhgglMp3GEKH0pZMXIpS0dMpChFKVTlSIUILS6QkRSks6KSFCyUinIkQoBekEhMgntoKULoY8WyS9Olv0WLZIunbrK6Zbbf96cNUeVtyV2cdaRX1IhXrvYh+C2C8ARTXZUScQ3tF4QDGO1BNUSINsFKUhNuZPY4WIcX2dLBdCPfy9AvlSA8MsHv1cM8PZctH8clNi/8w35TqdVu3ph9fV0KSxnalg6pe7FiP9yz2L0Q6Q+xajPSA7FqNdIA8sRvtAHlqMdoI8shjtBXlsMdoN8qPFaD/IE4vRjpBdi9GekKcWo10hzyxG+0KeW4x2huxZjPaGvLAY7Q55aTHaH/LKYrRDZN9itEfktcVol8gbi9E+kbcWo50i7yxGe0XeW4x2i/xkMWPUUMiHhZ9PDBvaj7+B8ykk3GUw6SLcYzBJI9xnMKkj7DCYBBIeMJg0Eh4ymGQSHjGYlBIeM5jEEn5kMOklPGEwSSbsMphUE54ymIQTnjGYtBOeM5jkE/YYTAoKLxhMIgovGUw6Cq8YTFIK+wwmNYXXDCZBhTcMJk2FtwwmWYV3DCZlhfcMJnGFnxhsPwjg0VZZNVE/XBkycYldQklbYo9QkpbYJ1Qr67W3r7/gmArwfE+A9PDWMYy8zqY3hMBXuJxEwnvKpvEIISyBJ/TXIeglp4WnXpTLYmxIvV0Gsxy9pf6O137T3qE7kmjFAaGkWXFIKElWHBFKihXHhJJgxUdCSa/ihFCSq+gSSmoVp4SSWMUZoaRVcU4oSVX0CCWligtCSajiklDSqbgilGQq+oSSSsU1oSRScUMoaVTcEkoSFXeEkkLFPaEkUPGJ0Pr5TIpuEPQHC988mamsIZAv6LofCZRp3KESCniXSijcPSqhYPephGLqUAlFdEAlFM8hlVA0R1RCsRxTCUXykUoojhMqoSi6VEIxnFIJRXBGJVz8cyrhoveohIt9QSVc5Esq4eJeUQkXtU8lXMxrKuEi3lAJF++WSrhod1TCxbqnEi7SJ3a/yn9V3kstGfAlk8aH4UGjdrV+Ixa3tkE3vadITrKp9NAEeU+Y6HIoXJsE5JMcj1TdXtYa0IEr9hC0iYKGiwJto6Dho0AbKWg4KdBWChpeCrSZgoabAm2noOGnQBsqaDgq0JYKGp4KtKmChqsCbaug4atAGytoOCvQ1goa3gq0uYKGuwJtr6Dhr0AbLGg4LNAWCxoeC7TJgobLAm2zoOGzQBstaDgt0FYLGl4LtNmChtsCbbeg4bdAGy5oOC7Qlgsangu06YKG6wJtu4D5Lvz8gIlIFlPwpukIiniu3nAa+dL3QkihwBykypFApQ+nKiE138D01UuC6kXNIlnogk6HqlVI8qiIMBE69ev3d4dznQT1OyPqJpg1G23b10kmvsTP7+4tnMgej+wt2zqTZCOIvzUQHVCPxJRW7lMF9b4VlMsoHkEVOdCFuvd1DTwmZBZMfKFeXvenMtOfq6Bweth4iTw3MXUfqyqrHRiBE2eKLXEFEnjo2DhTRC0E+smaGxz7eewHsKxfv+lWwNJ77VXX7vQ2Xjpecv/icl3B3vDpNtnLJc/tjVMzydkcN8i4WNqncS5RQLisn681qUDSGFUpGkdQNJsW2Vgm/owiLdCMw2SR6TeezKO31VbyeKpG/6yeD7jsSXfJX3c66a4s4I1fUA9Uodm+xF9+gWtfZCzyamUB9rKSaFVQAr3N4nHhJ+ox1eQpK9C2Cn8uvFfdn96/Uq/66D/8mKbmlVWR4/oL/arZqwHEMYuxj0lfe7uYAHHLp+q/Oe53SNQrb8obm0ZZtHrfNJuGOmdqqxxJ2NTNi8wbZaCae4oeohxGkb/VeMk6K5JYPelfLro/vVu2kFkKittu4+STrve+jcsVk7cwWgvdnwZROpbz5tbJ/UI9MsZjw1eb5QrwrBV+CF6UemlW2XwJsy1vb5IJNT2ZMoDBxNvHT8Qp/F54wyx72Fp3HvKc5+p0zop/Q40Xoe4A/h5sqqtvBapz0gTiVXuTWq0Ypv9/IaKPguqr1wFjkAN/iPsszp6GBfgP6wbL1CFo/i6p/gMlw9Rw6Rc4/gke/uvrX77b2G7+pdLqxc37re13W9sXP2z8ebf6K6Zfrf1u7V/X3qxtr/1x7c9rR2u9teu1YC1Z+6+1/177nw9/+HD+4ebDnQn95S+qOv+y5vx88P8PehDN2A==</latexit><latexit

Embed
input toinference
network

Carnegie Mellon University

Experiments

31

Carnegie Mellon University

Experiments• Datasets- MSCOCO image capcon (40K training)- Yelp restaurant reviews (17M training)Carnegie Mellon University

Perplexity Results
Yelp Large

prototypes: 779

Yelp Medium

prototypes: 1.5k

Yelp Large

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

Yelp Medium

Yelp Large

40

PPL

30
20
10
0

prototypes: 779

prototypes: 1.5k

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

Yelp Medium

Yelp Large

40

PPL

30

34.2

20
10
0

prototypes: 779

prototypes: 1.5k

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

Yelp Medium

Yelp Large

40

PPL

30

34.2
26.9

20
10
0

prototypes: 779

prototypes: 1.5k

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

Yelp Medium

Yelp Large

40

PPL

30

34.2
26.9

20

20.2

10
0

prototypes: 779

prototypes: 1.5k

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

25

Yelp Medium

40
30
PPL

18.75
12.5

10

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

25

20

30

12.5

10

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

40

PPL

18.75

Yelp Medium

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

25

20

30

18.6

12.5

10

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

40

PPL

18.75

Yelp Medium

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

20

18.6

12.5

30

40
20

10

0

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

40

60
PPL

18.75

80

PPL

25

Yelp Medium

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

25

Yelp Medium

80

40

74.7
18.6

12.5

30
PPL

20

60
PPL

18.75

40
20

10

0

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Perplexity Results
Yelp Large

25

Yelp Medium

80

40

74.7
18.6

12.5

30

61.9
PPL

20

60
PPL

18.75

40
20

10

0

0

0

Neural Editor
Sparse(Guu et al. 2018)Neural Editor

prototypes: 779

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 1.5k

34.2
26.9

20

6.25

NLM

Yelp Large

20.2

NLM

Neural Editor
Sparse(Guu et al. 2018) Neural Editor

prototypes: 17M

2K

Carnegie Mellon University

Processing Speed Results
Yelp Large

prototypes: 17M

2K

Carnegie Mellon University

Processing Speed Results
Yelp Large

Sent/Sec

1000

1

0.01

NLM

Neural Editor(Guu et al. 2018)

prototypes: 17M

SparseNeural Editor

2K

Carnegie Mellon University

Processing Speed Results
Yelp Large
1000

Sent/Sec

272
1

0.01

NLM

Neural Editor(Guu et al. 2018)

prototypes: 17M

SparseNeural Editor

2K

Carnegie Mellon University

Processing Speed Results
Yelp Large
1000

Sent/Sec

272
1

0.01

0.1
NLM

Neural Editor(Guu et al. 2018)

prototypes: 17M

SparseNeural Editor

2K

Carnegie Mellon University

Processing Speed Results
Yelp Large
1000

Sent/Sec

272
1

0.01

108
0.1

NLM

Neural Editor(Guu et al. 2018)

prototypes: 17M

SparseNeural Editor

2K

Carnegie Mellon University

How Does Sparsity Aﬀect
Prototypes?
Dense prototypes see more semancc overlap, sparse prototypes are more
syntaccc.

Table 2: Qualitacve examples of prototypes when using denser and sparser prototype supports.

Carnegie Mellon University

How Does Sparsity Aﬀect
Prototypes?
Dense prototypes see more overlapping content words.
Model

Overall

NOUN

DET

AUX

PRON

ADJ

VERB

31K
prototypes

91.2K

14.4K

9.6K

9.3K

9.0K

7.2K

6.4K

1.5K
prototypes

74.7K

9.9K

8.5K

8.2K

7.3K

5.6K

4.4K

Relative
Change-18.1%-31.3%-11.5%-11.8%-18.9%-22.2%-31.3K

Table 1: Number of matching tokens between examples and their prototypes. Results are reported
in cluster of POS tags.

Carnegie Mellon University

Example Generacon Results

37Carnegie Mellon University

Conclusion

38

Carnegie Mellon University

Summary• Prototype support can be sparse to improve eﬃciency — superior overprevious neural editor model with only 1‰ of datastore and 1‰ of test cme• Prototypes tend to drop semancc but keep syntaccc informacon when theygrow sparser

Carnegie Mellon University

Future Direccons?
• MANY other promising non-parametric models! How do they play withsparsity?
• What is the interaccon between big non-parametric models and small
parametric models? Do we need 100 billion parameters if we have Google
search?
Code available, try it out!

hops://github.com/jxhe/sparse-text-prototype40