Gaussian Process Generative
Models for Language and Robotics
Daichi Mochihashi
The Institute of Statistical Mathematics
daichi@ism.ac.jp
NAIST Data Science Lecture2021-10-25 (Mon)(based on Tutorial at CoRL 2019)Self introductionl PhD NAIST, 2005
l B.Sc. University of Tokyo,

1998
l Associate Professor at
The Institute of Statistical
Mathematics (ISM), Tokyol Research fields:

Natural Language Processing andBayesian Machine Learning

Overviewl Brief introduction to Gaussian processes (GP)– Very flexible nonlinear regression from Bayesianpoint of viewl Recognizing “Motion” (run, throw, bend, ...)

through a combination of:– Gaussian processes + semi-Markov HMM +Hierarchical Dirichlet processes (IROS 2018)– Variational Antoencoder for high-dimensionalsignals (IROS 2019)l GP Neural Networks for Text Visualization

Part 1:
What is a Gaussian process?

Gaussian processesl Gaussian process…stochastic process to3
generate randomfunctions
2
1

f

0
-1
-2
-3
-1

0

1

2

3

4

x

– function : mapping from x 7! y
– As a function of time t, it represents a trajectory

Gaussian processes (2)l Gaussian process on two-dimensional inputs＝Random surface function
l Similarly for higher-dimensional inputs x

Gaussian processes (3)l Bayesian learning of a function: given datapoint,

posterior distribution of functions is obtainedl Blue: Expectationl High variance(light blue) forarea with no datal Ordinary modelcannot represent
variances ofprediction

GPML

l GPML (“Gaussian Processes for Machine Learning”)• Published in 2006
• For intermediate and expert
users who need moremathematical constructions• Textbook is downloadable
for free:
http://www.gaussianprocess.org/gpml/

Introductory textbook
l “Gaussian process andMachine Learning”,
Kodansha MLP series,
2019 in Japan– 55 Amazon Ratings so farl From linear model toGaussian processes
l Also covers unsupervised
learning and variational
inferencel Why deer?

Simple regressionl Simplest prediction:y = a + bx

l How to determine a and b?

Minimization of errorl Minimize error ✏ between the observation yn and<latexit sha1_base64="R2/EbKBXP4sp6l4utDPeEJgVDcU=">AAAHrnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJiRN1USV12waK/6S7SRNorwARy4cygUkDojH4MILcOgjII5F4sKB8dpRPnZSbCXenfn9xzO7613uWWYQMnY+N79w5eo1bfH60o2bt24vr6zeKQduyzdEyXAt169wPRCW6YhSaIaWqHi+0G1uiQPefBn5D9rCD0zXeR12PVGz9RPHrJuGHqKpUhVeYFquc7ySYhkmrzW1kU0aKUiuvLu6sAVVeAMuGNACGwQ4EGLbAh0CvI8gCww8tNWgjzYfW6b0CxjAEmpbSAkkdLQ28f8Ee0eJ1cF+FDOQagPfYuHPR+UapNkP9oVdsO/sK/vJ/s6M1Zcxoly6+OSxVnjHy2/vFf/8V2XjM4TGSHWJgiN9eU0h1OGZrMXE2jxpiao04vjt3vuL4vNCuv+QfWK/sL6P7Jx9wwqd9m/j874onGH0tBxvgZGq0ManASkc4z72hrly9PWldZDkM+InFa7s+3ibmGMD1brsudCZGeGUeNcpwXUJrktwPYLrEVyH4DoEVye4OsE1Ca5JcBsEt0Fw2wS3TXA7BLdDcBWCqxDcIcEdEhwjOKZwPJnfaC1T88qTeY396nxy+SUM/YLw27jqR0RV9lXKwxzNCS62UGSgkAFJRrlFK3ycHdpUOhqlIT+uGNkHhMqS+wZHy7Ru3EMp80rF+RkVF+XXak9VMrKqit2JrEaa3amcxlV7Sf4G+izcnfYUYnNqRXHsbypUgaAKCvVK7jYpyEm6jntQtHfGszk8NeI84l2J9uSiuHiGZadPLLVRzmWyjzO5/Sep9RfJabYI9+EBPML4T2EdtnD8S/Kkegdn8EFjWlmraccxOj+XaO7CxKU1/gHgsJHv</latexit><latexit sha1_base64="9LohLkALeohWoe9L6zCvtAqTdE0=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatWqygdpU5oq8ppNasWxLdtJmkR5ASSu9MAJJA6Ix+DCC3DoIyCOReLCgfHakZPstNhKvDvz+49ndte73LVMP2DsdGZ27tLlK9r81YVr12/cXFxavrXrO23PEGXDsRyvwnVfWKYtyoEZWKLiekJvcUvs8eaL0L/XEZ5vOvaroOeKw5besM26aegBmkq9ml1bSrEMk9eK2sjGjRTEV95ZntuAKrwBBwxoQwsE2BBg2wIdfLwPIAsMXLQdwgBtHrZM6RcwhAXUtpESSOhobeJ/A3sHsdXGfhjTl2oD32Lhz0PlCqTZD/aFnbHv7Cv7yf6eG2sgY4S59PDJI61wa4tv75T+/FfVwmcAR4nqAgVH+uKaAqjDU1mLibW50hJWaUTxO/2Ts9KzYnpwn31iv7C+j+yUfcMK7c5v43NBFD9g9LQcb4GRqtDBpwEpHOMB9ka5cvQNpHUY55PwkwpH9j28TczxCNW67DnQPTfCMfGuY4LrEVyP4PoE1ye4LsF1Ca5OcHWCaxJck+DWCG6N4DYJbpPgtghui+AqBFchuH2C2yc4RnBM4Xg8v+FapuaVx/Ma+dX55PJLGPkF4W/hqk+IquyrlIs5mhNcZKFIXyF9kgxzC1f4ODuyqXQ4SiN+XJHYh4TKkvsGR8u0btxDKfNKxflzKi7Jr7U1VUliVRXbE1klmu2pnMZVO3H+Bvos3J12FGJ9akVx7K8rVJGgigr1Uu42KchJuo57ULh3RrM5OjWiPKJdifbkwrh4hmWnTyy1sZvLZB9lcoXHqdXn8Wk2D3fhHjzA+E9gFTZw/MuYQwPewXs40R5qBa2ivY7Q2ZlYcxsmLs34BwY5j60=</latexit>

prediction ŷn = a+b xn : ✏ = yn<latexit sha1_base64="4V8ILK3KXNRdVgJP2wuNf1nln6c=">AAAHxHichZXNbtNAEMenLZBSPtrCBQkOCVEQElW0CUggJKQWKGrVqkoT2qaqq8hrNo0Vx7Zs51vhwJEX4MAJJCQQJ3gFLrwAhz4C4lgkLhwYrx3lw5NiK/HuzO8/ntld73Lb0F2PseOp6ZkzZ8/FZs/PXbh46fL8wuKVXdeqO5rY0SzDcopcdYWhm2LH0z1DFG1HqDVuiD1efeL79xrCcXXLfO61bXFYU49Mvaxrqoem0sINpaJ63XavZMaVxCMlEVeVxB0lwZWlVgndSZZm8opHG5mwkYTwylmLM2ugwAuwQIM61ECACR62DVDBxfsAMsDARtshdNHmYEuXfgE9mENtHSmBhIrWKv4fYe8gtJrY92O6Uq3hWwz8OaiMQ4r9YJ/YCfvOPrOf7O/EWF0Zw8+ljU8eaIVdmn99rfDnv6oaPj2oDFSnKDjSp9fkQRkeyFp0rM2WFr9KLYjf6Lw5KTzMp7q32Hv2C+t7x47ZN6zQbPzWPmyL/FuMnpLjLTCSAg18apDEMe5ir58rR19XWnthPgN+VGHJvoO3jjlWUK3KngXNiRFaxLtaBNcmuDbBdQiuQ3BNgmsSXJngygRXJbgqwa0Q3ArBrRPcOsFtENwGwRUJrkhw+wS3T3CM4FiE4+H8+muZmlcezmvgj84nl19C3y8Ifw1X/YBQZD9K2ZijPsIFFop0I6RLkn5u/gofZvu2KO2PUp8fVgzsPUJlyH2Do2VcN+yhlLlIxbkJFRfk11obq2RgjSo2R7IaaDbHchpWbYX5a+gzcHfaihCrYyuKY381QuUJKh+hnsrdJglZSZdxD/L3zmA2+6dGkEewK9GerB8Xz7DM+IkVbexm05m76ez2veTy4/A0m4XrcBNuY/z7sAxrOP47mMMr+Ahf4GvsWcyIubF6gE5PhZqrMHLFXv4DEueYlQ==</latexit><latexit sha1_base64="7VSOK3jGYq0LXE1W6jZdmhzP5vM=">AAAH1XichZW/b9NAFMdfWyCl/GgKCxKLRRRUhFpdAhIICakFKrVqVbUpaVI1VeQzl9aKf8l207pRNsTAwsDAwAQSA+IvYKUL/wBD/wTEWCQWBp7PjpzEL8VW4rv3Pt/n9+7Od9wxdM9n7GRkdOzc+QuZ8YsTly5fuTqZnbq26dn7ribKmm3YbpWrnjB0S5R93TdE1XGFanJDVHjzaeivtITr6bb13A8csWOqu5be0DXVR1M9O1MTjqcbtqU8VoK6pcwotT3VbwedemKZVpW7Cj+sW3fq2RybZfJS0o1C3MhBfK3ZU2OLUIMXYIMG+2CCAAt8bBuggof3NhSAgYO2HWijzcWWLv0COjCB2n2kBBIqWpv4v4u97dhqYT+M6Um1hm8x8OeiUoE8+8E+s1P2nX1hP9nfobHaMkaYS4BPHmmFU598fWPjz39VJj592EtUZyg40mfX5EMDHspadKzNkZawSi2K3zp6d7rxqJRv32Yf2S+s7wM7YcdYodX6rX1aF6X3GD0vx1tgpBq08KlBDse4jb1urhx9bWntxPkkfL/Cln0Xbx1z3EO1Kns2HAyNcEi865DgAoILCO6I4I4I7oDgDgiuQXANgmsSXJPg5glunuCWCG6J4JYJbpngqgRXJbgtgtsiOEZwLMXxeH7DtUzNK4/nNfKn55PLL6HrF4TfxFWfEDXZT1MO5qj3cZGFIr0U6ZFkmFu4wnvZri1Nh6PU5XsVib1DqAy5b3C0DOp6PZRyLVXx2pCKN+TXag5UkljTipW+rBLNykBOvarVOH8NfQbuTqspYmFgRXHsL6SoEkGVUtQzudvkoCjpBu5B4d4ZzWb31IjyiHYl2lMM4+IZVhg8sdKNzeJs4d5scf1+bu5JfJqNw024BdMY/wHMwSKOfxlzeAtf4RscZyqZTuZl5lWEjo7EmuvQd2Xe/ANEUp6N</latexit>ŷn = yn(a + bxn )yn<latexit sha1_base64="9LohLkALeohWoe9L6zCvtAqTdE0=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatWqygdpU5oq8ppNasWxLdtJmkR5ASSu9MAJJA6Ix+DCC3DoIyCOReLCgfHakZPstNhKvDvz+49ndte73LVMP2DsdGZ27tLlK9r81YVr12/cXFxavrXrO23PEGXDsRyvwnVfWKYtyoEZWKLiekJvcUvs8eaL0L/XEZ5vOvaroOeKw5besM26aegBmkq9ml1bSrEMk9eK2sjGjRTEV95ZntuAKrwBBwxoQwsE2BBg2wIdfLwPIAsMXLQdwgBtHrZM6RcwhAXUtpESSOhobeJ/A3sHsdXGfhjTl2oD32Lhz0PlCqTZD/aFnbHv7Cv7yf6eG2sgY4S59PDJI61wa4tv75T+/FfVwmcAR4nqAgVH+uKaAqjDU1mLibW50hJWaUTxO/2Ts9KzYnpwn31iv7C+j+yUfcMK7c5v43NBFD9g9LQcb4GRqtDBpwEpHOMB9ka5cvQNpHUY55PwkwpH9j28TczxCNW67DnQPTfCMfGuY4LrEVyP4PoE1ye4LsF1Ca5OcHWCaxJck+DWCG6N4DYJbpPgtghui+AqBFchuH2C2yc4RnBM4Xg8v+FapuaVx/Ma+dX55PJLGPkF4W/hqk+IquyrlIs5mhNcZKFIXyF9kgxzC1f4ODuyqXQ4SiN+XJHYh4TKkvsGR8u0btxDKfNKxflzKi7Jr7U1VUliVRXbE1klmu2pnMZVO3H+Bvos3J12FGJ9akVx7K8rVJGgigr1Uu42KchJuo57ULh3RrM5OjWiPKJdifbkwrh4hmWnTyy1sZvLZB9lcoXHqdXn8Wk2D3fhHjzA+E9gFTZw/MuYQwPewXs40R5qBa2ivY7Q2ZlYcxsmLs34BwY5j60=</latexit>ŷn = a + bxn<latexit sha1_base64="RpKJQO0vcFbVMReJuW2uL+4jlmM=">AAAHuHichZXNbtNAEMenLZBSPprCBYlLRRSEhBRtAhKoElILVGrVqkoT0qZqquA1m8SKY1u282nlBXgBDnABqUiIx+DCC3DoIyCOReLCgfHakZN4Umwl3p35/cczu+tdbuma4zJ2Nje/cOnylcTi1aVr12/cXE6u3DpwzLatipJq6qZd5oojdM0QJVdzdVG2bKG0uC4OefOF7z/sCNvRTOOV27fESUupG1pNUxUXTdVkstJQXK8/rBrPlIe8V0VTimWYvFbjjWzYSEF45c2VhS2owBswQYU2tECAAS62dVDAwfsYssDAQtsJeGizsaVJv4AhLKG2jZRAQkFrE//r2DsOrQb2/ZiOVKv4Fh1/NipXIc1+sC/snH1nX9lP9ndmLE/G8HPp45MHWmFVl9/eKf75r6qFTxcakeoCBUf64ppcqMFTWYuGtVnS4lepBvE7g3fnxbVC2rvPPrFfWN9Hdsa+YYVG57d6ui8K7zF6Wo63wEgV6OBThRSOsYe9Ua4cfZ60DsN8In5SYcq+jbeGOTZQrcieCd2ZEXrEu3oE1ye4PsENCG5AcF2C6xJcjeBqBNckuCbBbRDcBsFtE9w2we0Q3A7BlQmuTHBHBHdEcIzgWIzj4fz6a5maVx7Oa+CPzyeXX8LILwh/C1d9RFRkP05ZmKM2wQUWinRipEOSfm7+Ch9nR7Y47Y/SiB9XRPYhodLlvsHRMq0b91DKfKzi/IyKi/JrbU1VElnjit2JrCLN7lRO46q9MH8VfTruTnsxYnNqRXHsb8aoAkEVYtRLudukICfpGu5B/t4ZzObo1AjyCHYl2pPz4+IZlp0+seKNg1wm+yiT23+cWn8enmaLcBfuwQOM/wTWYQvHv4Q5dOADnMLnxFridaKe0AJ0fi7U3IaJK2H/A+LHlWE=</latexit>

Simple regression (2)l Given data D = {(x1 , y1 ), (x2 , y2 ), · · · , (xN , yN )} ,l Prediction ŷn for each xn is a linear functionŷn = a + bxn
l Difference between the observation isyn ŷn = yn (a + bxn )– We want to minimize this error!

Simple regression (3)l For n=1,2,...,N,error＝ yn ŷn → Minimize sum of errors
l Errors can be negative, thus take a square oferrors (Least squares):
N
N
X
X
E=
(yn ŷn )2 =(yn (a + bxn ))2
n=1

n=1

We want a,b to minimize this E

Simple regressionl Since derivative of E is zero at minimal point,
N
@E
@ X
=
(yn (a + bxn ))2@a@a n=1
N@ X 2=(yn + a2 + b2 x2n 2ayn
@a n=1

2abxn + 2bxn yn ) = 0

N
@E
@ X
=
(yn (a + bxn ))2@b@b n=1
N@ X 2=(yn + a2 + b2 x2n 2ayn 2abxn + 2bxn yn ) = 0@b n=1P 2PPPl Yieldsxyn
n xn
n x n ynPa = n n Pn 2
N n xn ( n xn ) 2
PPP
N n x n yn
n xn
n yn
P 2
P
b=
N n xn ( n xn ) 2

Simple regression (example)
y = 1.77 + 0.42x

l Formula gives

a = 1.77, b= 0.42

Simplest case:
Data D={(3,2),(2,4),(-1,1)}

Multiple regressionl x is multi-dimensional? à Multiple regression

When x = (x1 , x2 , · · · , xD )T,

y = w 0 + w 1 x1 + w 2 x2 + · · · + w D xD

l Squared error is(yŷ)2 = (y(w0 + w1 x1 + w2 x2 + · · · + wD xD ))2l Minimize this error:

PN→ Take gradients of E = n=1 (yn ŷn )2 w.r.t.
w0 , w1 , · · · , wD to equate 0 and solve
linear equations.

Vector-Matrix forml When we set x = (1, x1 , x2 , · · · , xD )and w = (w0 , w1 , w2 , · · · , wD ) ,ŷ = w0 + w1 x1 + w2 x2 + · · · + wD xD
0 1= (w0 , w1 , w2 , · · · , wD ) 1
B x1 CB C
B .. C
@ . A
T

=w x

xD

Vector-Matrix form (2)l Aligning vertically over n=1,2,...,N gives0l Namely,

Can writeŷ = Xw

0 T1
w x1
x1
ŷ1
B ŷ2 C B wT x2 C B xT2 C
B C B
C B C
B .. C = B .. C = B .. C w@ . A @ . A @ . A
ŷN

1

0

T

w T xN

1xTN

CalledDesign matrix

Vector-Matrix form (3)
E=

N
X

n=1

2

(yn ŷn ) = (y1 ŷ1 , · · · , yN1y1 ŷ1
B
C
..
@
A
.
yN

l Therefore

E = (yŷN )0Xw)T (yXw)= yT (yXw)(Xw)T (y= yT y2wT (XT y) + wT XT Xw

Xw)

ŷN

Analytical solution of multiple regression
E = yT y

2wT (XT y) + wT XT Xw

l taking derivative wrt w ,@E=@w
l Thus2XT y + 2XT Xw = 0(Normal equation)
XT Xw = XT y
) w = (XT X) 1 XT y .

Solution of multiple regression

Multiple regression examplel Given data below:

0
1
B1
X=B
@1
1

1
1
3
2

1

!2
1C
C
0A
2

Multiple regression example (2)l Therefore, optimal weight vector isy = 1.202 0.016x1 +1.209x2

More complex regression?

l Data often exhibit nonlinear relationships

More complex regression function!

Linear regression
y = w0 + w1 x + w2 sin(x)y = w 0 + w 1 x + w 2 x2
01
0 1
w0 w1 w21
w0 w1 w2
1
@ x A
@xA
=
=
sin(x)x2|{z} | {z }|{z} | {z }
(x)
wT
(x)
wT

l Both written as linear function of weights w

Ty=w(x) … Linear regression–l Combining this with sigmoid function yields
logistic regression:y = (wT (x))<latexit sha1_base64="R6dNB6qxcdhKJVVsyuqO/fHCSek=">AAAHf3ichZXPTxNBFMdfQV3EHxS9mHBpbEC4NNNq4o/EBFQSCYRAS6GEYrOzzJZNt+2mu/2dHrz6D3jwpIkxxov+DV78BzzwJxiPmHjx4NvZbfpjHrBNOzPvfb6v783MznDHtlyPsZPIxOSly1e0qavT167fuDkTnb2161brNUNkjapdreW47grbqoisZ3m2yDk1oZe5LfZ46bnv32uImmtVKzte2xGHZb1YsUzL0D00FaJz7djTWN61imV9Md9ovtrJc+fYwm5raakQjbMEk09M7STDThzCZ6s6O7kAeTiCKhhQhzIIqICHfRt0cPFzAElg4KDtELpoq2HPkn4BPZhGbR0pgYSO1hL+FnF0EForOPZjulJt4L/Y+K2hMgbz7Cf7zE7ZD/aF/WL/zozVlTH8XNrY8kArnMLMmzuZvxeqyth6cDxQnaPgSJ9fkwcmPJK1WFibIy1+lUYQv9F5e5p5kp7vLrAP7DfW956dsO9YYaXxx/i4LdLvZPQj1JjYNrA1II4z3MVRP1OOvq609sJsBnyLIFsE1ya4NsF1CK5DcE2CaxKcSXAmwZUIrkRwKwS3QnBrBLdGcOsEt05wOYLLEdw+we0THCM4pnA8XF9/H1LrysN1DfzqenK5i/t+QfjLuGMHRF6OVcrBHK0RLrBQpKuQLkn6uXlY/zDbt6m0P0t9flgxsPcIlS3feY6Wcd2wh1JuKRVvnVFxBq1FuZLD9MCqKjZGshpoNsZyGlZthvkb6LPxZNlUiNWxHcVxvKpQaYJKK9QLedrEISVpE89n/9wLVrN/4gd5BKcS7Un5cfH+SY7fNmpnN5VI3k+kth/El5+FN9EUzMFdWMT4D2EZXuL8ZzGH1/AJvsI3LaLd0xIaC9CJSKi5DSOP9vg/UIWKmA==</latexit>Linear regression
y = wT (x) + ✏<latexit sha1_base64="bs6b/TPeYDqCG1ev59MSCKeS3gY=">AAAGC3ichZTPa9RQEMenq9Faf7TViyBCMFQqwvJ2FRRBaNWCpVDabbfd0tSSpC/dsNkkJNlss8uevAmePXhS8FC82Zt4E8F/wEP/BPFYwYsHJy/RbTfT+kLy5s18vpM34WV0z7aCkLH9ocKp09KZs8PnRs5fuHhpdGz88krgtnyDVw3Xdv2argXcthxeDa3Q5jXP51pTt/mq3nicxFcj7geW6yyHscc3mtq2Y5mWoYXo2hy7HssPZTVqP1tWvbo1qUY7t+Tbssq9wLKTuMKKTAw5b5QyQ4FsLLjjhQaosAUuGNCCJnBwIETbBg0CvNahBAw89G1AF30+WpaIc+jBCGpbSHEkNPQ28LmNq/XM6+A6yRkItYFvsfH2USnDBPvGdtkB+8res+/s97G5uiJHspcYZz3Vcm9z9MXVpV//VTVxDqHeV52g0JE+uaYQTLgvarGwNk94kiqNNH/UeXWw9KAy0b3J3rIfWN8bts8+Y4VO9NN4t8grr0X2LdSYOEewg5nUf3vU0dtFXy/bRZ+LCS4muA7BdQiuTXBtgjMJziS4BsE1CG6a4KYJbpbgZglujuDmCK5GcDWCWyO4NYJjBMdynI4npI4npSvspphTD0UGOTIgyeSNHJ8aUof5vr9HqGzxP+joGdQdjuSV8xltYNTGMz6fI2YGvoWO65kcVSGoSo56Agr2GwXKgjaxUyR/YPo1/vaedB8J1zsmUk7yYicsDfa9vLFSLpbuFMuLd5WpR1lPHIZrcAMmMf89mIKnsABV3MNz2IUPsCe9lPakj9KnFC0MZZorcGRIX/4AFGRGGg==</latexit>l For explanation, assume no noise:y = wT (x) = w1 1 (x) + w2 2 (x) + · · · + wM<latexit sha1_base64="qfnZkBAMEH7UAXxQiMf73aO2kFM=">AAAGN3ichZS/b9NAFMdfA4ESfjSFBYklImpVhBRdDBIIqVILVCKqWrVp06aqi2U7l8aKY1u24zSJ8g/wDzAwgcSAGJiZWfgHOnShGwIxoSKxMPB8NqTEr8VRfO++9/k+37POT3NMw/MZOxhLnTmbPnd+/ELm4qXLVyayk1c3PLvt6ryi26btVjXV46Zh8Ypv+CavOi5XW5rJN7Xmo3B9M+CuZ9jWut91+E5L3bWMuqGrPkpKttTNTc/m5KDzdF12GsaMHOzdyszmOkoxnCpFIdzuKJKYStFU1mu276G6JNQloSrZPCswceWSQTEO8hBfK/Zkqgky1MAGHdrQAg4W+BiboIKHv20oAgMHtR3oo+ZiZIh1DgPIoLeNFEdCRbWJ912cbceqhfMwpyfcOj7FxL+LzhxMsX32hh2xj+wt+8p+nZirL3KEe+niqEVe7igTz66v/fyvq4WjD42h6xSHhvTpNflQh/uiFgNrc4QSVqlH+YPe86O1B+Wp/jR7xb5hfS/ZAfuAFVrBD/31Ki+/ENlr6KnjGMAeZpL/7lFDtY/aIN7FkOsSXJfgegTXI7gOwXUIrk5wdYJrElyT4OYJbp7gSgRXIrhFglskuCrBVQlui+C2CI4RHEtwGp6QBp6UvohbYowUivQSpEeS4RM53lWkjvNDfUC4TPE9aKiM+o6vJJ3LMa3jqolnfDlBLIy8Cw3nCwmqTFDlBPUY8thv8iAJuo6dIvwCo7fxp/dE+wi5wQkrUpgXO2FxtO8lgw2pULxTkFbv5ucexj1xHG7ATZjB/PdgDp7AClRwD+9gHz7BYfp9+jD9Of0lQlNjseca/HOlv/8G4QpWPw==</latexit>M (x)l For given pairs {(x1 , y1 ), (x2 , y2 ), · · · , (xN , yN ), }<latexit sha1_base64="SXjg6lWcE8Yn2JPWtc6LmsrYPQ4=">AAAGIHichZTPbtNAEManBUMJf5rCBYlLRVTUSFG0NkggTi1QiapS1KakTVVXke2sWyuObdlO2sTKC/ACIHGiEgfEO3DhQG6IA4fyBohjkbhwYLw2pMTTslG8429+33jXWo/u2VYQMnY0MXnuvHTh4tSl3OUrV69N52eubwRuxzd4zXBt16/rWsBty+G10AptXvd8rrV1m2/qrcdxfrPL/cBynWdhz+M7bW3XsUzL0EKUGnlFjXLzavegIZd6DblYErGCsVIsqUbTDYNSkq+gVimWcuqgkS+wMhNjNhvIaVCAdKy6M5MtUKEJLhjQgTZwcCDE2AYNAvxtgwwMPNR2IELNx8gSeQ4DyKG3gxRHQkO1hdddvNtOVQfv45qBcBv4FBv/PjpnYY59YW/ZMRuyd+wb+3VqrUjUiNfSw1lPvNxrTD+/uf7zv642ziHsjVxnOHSkz95TCCY8EHuxcG+eUOJdGkn9bv/F8frD6lx0hx2y77i/1+yIfcAdOt0fxps1Xn0lqjfRY+LchQOspP5do45qhNogXcWI6xFcj+D6BNcnuH2C2yc4k+BMgmsRXIvgFglukeCWCW6Z4FYIboXg6gRXJ7gtgtsiOEZwLMPpeEL28KREIm6LOVEoMsiQAUnGT+R41ZA6yY/0AeGyxfegozLuO5nJOispbWDWxjNeyRBLY+9Cx/ulDFUlqGqGegIF7DcFUARtYqeIv8DkbfzpPck6Ym5wSkaJ62InlMf7XjbYUMry3bKydq+w8CjtiVNwC27DPNa/DwvwFFahhmt4Ce/hIwylQ2kofZI+J+jkROq5Af8M6etvXyVMVQ==</latexit>

this gives y = w
0<latexit sha1_base64="Jyt0lmui7Y6egaOMjzGc1AFfx9c=">AAAIInicpZRNa9RAGMefxpet8aWtXgQvi0tLeymzqaAIQqsWLKVLu7Xtlo6EJDvbDZtkQ5LNbhryBfwCgp4UPYhfQvCivYl4KPgFxGMFLx588tLudne2IiYkM/M8v/8/z0ySUW1Ddz1CDkaEM2fPnc+NXhAvXrp8ZWx84uqm22w5GtvQmkbTqaiKywzdYhue7hmsYjtMMVWDbamNB3F+y2eOqzetx15gsyemsmvpNV1TPAzJE8IcbVlV5qiOorFwUqQq29WtUHEcJYhCLToK2KbiOXonEgO5mKc0H8hS3FC/2vTcNFASKbOqXZJSkZpqsxNGaSL1FCM5pB7reK4XGIwiXHc9p+WhFSbvif9YDrXrulycpn5HLs7kp/LJWOoZa0mBWWLlKBHX1lVKfUppmFJKlX6Wic/jFeg1LPUZloYZlmb+c9FUM6SrdT2K+At37NtOX1s7fW1tee7k22vLK32FDHki9duRKI8XyCxJjvxgp5h1CpAdq80JoQEUqtAEDVpgAgMLPOwboICL5w4UgYCNsScQYszBnp7kGUQgoraFFENCwWgD77s42smiFo5jTzdRa/gUAy8HlXmYJF/JW3JIPpJ35Dv5PdQrTDziWgJs1VTLbHns6fX1X39Vmdh6UO+qTlGoSJ8+Jw9qcCeZi45zs5NIPEst9ff3nh2u3y1PhlPkFfmB83tJDsgHnKHl/9TerLHyi8S9ipoatj500Ike16hiNMRYlFXR5QIOF3C4PQ63x+HaHK7N4WocrsbhGhyuweEWONwCh1vicEscbpnDLXO4CoercLhtDrfN4QiHIwOcil9IHb+UMOmbSZtGeKQ7QLpcMn4iw7uCVC/fjUcclZH8DypG+nW9mUFlKaM1zBr4jZcGiMW+tVBxvDhAlTlUeYB6CAXcbwogJXQNd4r4D0xX42jvSeuIuWhIRop9cScs9u97g51NabY4Nyut3SrM38/2xFG4ATdhGv1vwzw8glXYAE14LrwXPgn7ude5/dzn3JcUFUYyzTU4ceS+/QFo/gR/</latexit>|

1<latexit sha1_base64="9LhEnJl2Iv9QqBTLXGNwuk8pBHU=">AAAG6HichZTNbtNAEMenAUwpH03LBYlLSlTEKdoEJBASUgtUompVpQlpU9VVZLvr1MSxLdtxvpQX4MStQkhIVEKAeAwuvACHnjgjjkXiwoHx2lXSeNpuFO/uzO8/mVlnR3VMw/MZO5xIXbh4Sbo8eWXq6rXrN6bTM7Mbnt1yNV7RbNN2q6ricdOweMU3fJNXHZcrTdXkm2rjWejfDLjrGbb10u86fKep1C1DNzTFR1MtPSsH3Yw890Sey8hqcc+Qg3YtnWU5JkYmucjHiyzEo2jPpH6CDLtggwYtaAIHC3xcm6CAh59tyAMDB2070EebiytD+DkMYAq1LaQ4EgpaG/is4247tlq4D2N6Qq3hr5j4dVGZgXn2g31hR+w7+8p+sX+nxuqLGGEuXZzVSMud2vTrW+W/56qaOPuwN1SdoVCRPrsmH3R4JGoxsDZHWMIqtSh+0Ns/Kj8uzffvsgP2G+v7wA7ZN6zQCv5oH9d56Z2IvosaHecAZw2yeMJ93B1nqqKvL6yDOJsh3yHIDsF1Ca5LcD2C6xFcm+DaBKcTnE5wDYJrENwiwS0S3DLBLRPcCsGtEFyV4KoEt0VwWwTHCI4luPD/1xJkuIoULYJyMIZxgossFOklSI8kw7w4PhWkRvmhfUCoTHHLVLSM60Y9lLKYqKF4Sg1ltNbF2Y3SQ2tSsXoiq6FmdSynUdVanL+GPhPv8lqCWBp7hyrulxJUiaBKCeq5uN9ZKAhax44Ydpro/Rz32CiPqA/QnkIYFzt+fry/JxcbhVz+fq6w/iC78DTu/ZNwG+7APYz/EBbgBZ5/BXPowHv4BJ+lV9IbaV96G6GpiVhzE04M6eA/hBFt1A==</latexit>0y1
1 (x1 )
B y2 CB 1 (x2 )B C
BB .. C
B ..
=
@ . A
@ .
yN
1 (xN ){zy}|

2 (x2 )······

2 (xN )···

2 (x1 ){z

M (x1 )

1 0

1

w1
C B
C
M (x2 ) C B w2 C
C B w3 C..
C
A B.
B .. C
@ . A
(x )

M

N

wM
} | {z }w

From Linear regression to Gaussian
processes
01

y1
B y2 CB C
B .. C@ . A =

0yN

<latexit sha1_base64="Jyt0lmui7Y6egaOMjzGc1AFfx9c=">AAAIInicpZRNa9RAGMefxpet8aWtXgQvi0tLeymzqaAIQqsWLKVLu7Xtlo6EJDvbDZtkQ5LNbhryBfwCgp4UPYhfQvCivYl4KPgFxGMFLx588tLudne2IiYkM/M8v/8/z0ySUW1Ddz1CDkaEM2fPnc+NXhAvXrp8ZWx84uqm22w5GtvQmkbTqaiKywzdYhue7hmsYjtMMVWDbamNB3F+y2eOqzetx15gsyemsmvpNV1TPAzJE8IcbVlV5qiOorFwUqQq29WtUHEcJYhCLToK2KbiOXonEgO5mKc0H8hS3FC/2vTcNFASKbOqXZJSkZpqsxNGaSL1FCM5pB7reK4XGIwiXHc9p+WhFSbvif9YDrXrulycpn5HLs7kp/LJWOoZa0mBWWLlKBHX1lVKfUppmFJKlX6Wic/jFeg1LPUZloYZlmb+c9FUM6SrdT2K+At37NtOX1s7fW1tee7k22vLK32FDHki9duRKI8XyCxJjvxgp5h1CpAdq80JoQEUqtAEDVpgAgMLPOwboICL5w4UgYCNsScQYszBnp7kGUQgoraFFENCwWgD77s42smiFo5jTzdRa/gUAy8HlXmYJF/JW3JIPpJ35Dv5PdQrTDziWgJs1VTLbHns6fX1X39Vmdh6UO+qTlGoSJ8+Jw9qcCeZi45zs5NIPEst9ff3nh2u3y1PhlPkFfmB83tJDsgHnKHl/9TerLHyi8S9ipoatj500Ike16hiNMRYlFXR5QIOF3C4PQ63x+HaHK7N4WocrsbhGhyuweEWONwCh1vicEscbpnDLXO4CoercLhtDrfN4QiHIwOcil9IHb+UMOmbSZtGeKQ7QLpcMn4iw7uCVC/fjUcclZH8DypG+nW9mUFlKaM1zBr4jZcGiMW+tVBxvDhAlTlUeYB6CAXcbwogJXQNd4r4D0xX42jvSeuIuWhIRop9cScs9u97g51NabY4Nyut3SrM38/2xFG4ATdhGv1vwzw8glXYAE14LrwXPgn7ude5/dzn3JcUFUYyzTU4ceS+/QFo/gR/</latexit>

|

{z
y

}

|

B
B
B
@

1 (x1 )

2 (x1 )

1 (x2 )2 (x2 )······

2 (xN )···..
.
1 (xN )

M (x1 )

NwM
} | {z }w

l When we assume w ⇠ N (0, ↵I) like a ridge<latexit sha1_base64="0ugoZSeXTRWFGDC2zQnXG7NSqzU=">AAAGDnichZS/b9NAFMdfA4ZSfjSFBYnFIioqEoouAQnE1AKViCpFbUraVHVV2e65seJfsh2HxMrKwMTGwAQSA4KVboiBpf8AQ/8ExFgkFgaez4aU+LVcFN/z9z7f53vW+WmeZQYhYwcThVOnpTNnJ89Nnb9w8dJ0cebyWuB2fZ03dddy/ZamBtwyHd4MzdDiLc/nqq1ZfF3rPEzW1yPuB6brPAn7Ht+y1V3HNExdDVHaLspK1JOVwLTlWNFVS64P55SI3VJUy2urSlS7uV0ssTITQ84HlSwoQTaW3ZlCBxTYARd06IINHBwIMbZAhQB/m1ABBh5qWxCj5mNkinUOQ5hCbxcpjoSKagevu3i3makO3ic5A+HW8SkW/n10yjDLvrJ37JDts/fsG/t1bK5Y5Ej20sdZS73c255+fnX1539dNs4htEeuExwa0ifXFIIB90QtJtbmCSWpUk/zR4OXh6v3G7PxDfaGfcf6XrMD9gUrdKIf+tsV3nglsu+gx8A5gqeYSfm7Rw3VGLVhtosR1ye4PsENCG5AcD2C6xGcQXAGwXUIrkNwCwS3QHA1gqsR3BLBLRFci+BaBLdBcBsExwiO5TgNT0gbT0osYlvMqUKRQY4MSDJ5IseritRRfqQPCZclvgcNlXHf0ZW8s57ROq5aeMbrOWJx7F1oeL+YoxoE1chRj6CE/aYEVUEb2CmSLzB9G396T7qPhBses1JN8mInrIz3vXywVi1XbperK3dK8w+ynjgJ1+A6zGH+uzAPj2EZmriHZ/ABPsKe9ELakz5Jn1O0MJF5rsA/Q9r/DVgcR2o=</latexit>

regression,y = w also obeys Gaussian distribution<latexit sha1_base64="TZsRjRcG03pow2e3HX9JDfNKxv8=">AAAGBHichZRNaxNRFIZPY0dr/WiqIIKb1FBxFW6ioAhCqxYshZKmpk3plDB3eqcZMl/MTCYmY7Yu/AMuXLXgQtqduHXjxj/goj9BXFZw48IzH5qaOW1vyNwz733eM/cMdw53DN3zGTscy50bl85fmLg4eenylatT+elra57dcVVRV23Ddhtc8YShW6Lu674hGo4rFJMbYp23n0br64FwPd22Xvg9R2yZyo6la7qq+Cg18zfkoFeQZx7LMwWZm6FcbekDOeg280VWYvEoZINyGhQhHVV7OtcGGbbBBhU6YIIAC3yMDVDAw98mlIGBg9oWhKi5GOnxuoABTKK3g5RAQkG1jdcdvNtMVQvvo5xe7FbxKQb+XXQWYJZ9Yx/YEfvK9tl39vvEXGGcI9pLD2eeeIXTnHpzc/XXmS4TZx9aQ9cpDo706TX5oMHDuBYda3NiJapSTfIH/bdHq49qs+Edtsd+YH277JB9wQqt4Kf6fkXU3sXZt9Gj4RzAS8wk/9sjRzVEbZDuYsj1CK5HcH2C6xNcl+C6BKcRnEZwbYJrE9w8wc0T3CLBLRLcEsEtEVyD4BoEt0FwGwTHCI5lOI4npIUnJYxjM54ThSK9DOmRZPREgVcFqeP8UB8QLiP+Hjgqo77jK1nnckqruGrgGV/OEAsj74Lj/UKGqhFULUM9gyL2myJUYlrDThF9gcnb+Nt7kn1E3OCElUqUFzthebTvZYO1Sql8r1RZuV+ce5L2xAm4BbfhLuZ/AHPwHKpQxz28gj3YhwPptXQgfZQ+JWhuLPVch/+G9PkPSLhDlQ==</latexit>

y=<latexit sha1_base64="FyuaKtBxh9emcwi+JTrxDxzfQQo=">AAAGqnichZQ9b9NgEMevAUIpL01hQWKxiIqKVEVPAhIICakFKlG1itKEtIniKrKdx6kV27FsJyGx/AWYkRiYQGJAbIysZeALMPQjIMYisTBwfoG8+No6ip977n7/e+6c+GRL1xyXsaO51LnzF9IX5y8tXL5y9dpiZun6rtPt2QqvKl29a9dkyeG6ZvKqq7k6r1k2lwxZ53ty52kQ3+tz29G65gt3aPF9Q2qbmqopkouuZiYn9ofCY0GUDU8sHWi+2B8IoqMZgicqki4U/RUM9VZFuaK1DeluM5NlORZeQtLIx0YW4qvUXUp9BhFa0AUFemAABxNctHWQwMFPA/LAwELfPnjos9HSwjgHHxZQ20OKIyGht4P3Nu4asdfEfZDTCdUKnqLj10alAMvsO/vIjtk39on9YH9OzOWFOYJahrjKkZZbzcVXNyu/z1QZuLpwMFadopCRPr0nF1R4GPaiYW9W6Am6VKL8/dGb48qj8rJ3h71nP7G/d+yIHWKHZv+X8mGHl9+G2VuoUXHtw0vMJP6vUUavhz4/rmLMDQluSHAjghsR3IDgBgSnEpxKcB2C6xDcOsGtE9wmwW0S3BbBbRFcjeBqBFcnuDrBMYJjCS74P/VCMrAiRY+gLMyhTXGRhyKdBOmQZFAXx7uE1CQ/9vuESg/fGhk9s7rJCKWsYBXt8IlMqsbepGJ76qyxZnvmpElVMa5KwZiOb1wxQWzM/DIy7jcSVJmgygnqGWRx+mWhENIqzq1gHkRP/d8kjOoIOP+ESCHIi3M5PzuFk8ZuIZe/lyvs3M+uPYkn9DzcgtuwgvkfwBo8hxJUsYbX8AUO4Wt6NV1O19ONCE3NxZobMHWlW38BeLZmxg==</latexit>l What are µ, ⌃ ?
<latexit sha1_base64="KIiSdEurKix/WIElK9/aEZpVHjg=">AAAGhnichZTLbtNAFIZPA5hQLm1hg8SmIipigaJJABWxSoFKVK2qNiFtqqaqPO44WPFNvqQkUV6AB4AFCy4SC8SOV2DDC7DoIyCWrdQNC47HBifxaesonjP/fP+ZM058uGsafsDYwUTu3PkLysX8pcnLV65em5qeub7hO6GnibrmmI7X4KovTMMW9cAITNFwPaFa3BSbvP00Wt/sCM83HPtF0HXFjqW2bEM3NDVAqdnkVnivyWtGy1J3pwusyOQ1mw1KSVCA5FpzZnLfoAl74IAGIVggwIYAYxNU8PGzDSVg4KK2A33UPIwMuS5gAJPoDZESSKiotvHewtl2oto4j3L60q3hLiZ+PXTOwhz7yb6wQ/aDfWW/2J8Tc/VljqiWLo489gp3d+r1zdrxmS4LxwBepq5THBzp088UgA6P5FkMPJsrleiUWpy/03t7WHtcnevfYZ/YbzzfR3bAvuMJ7c6R9nldVN/J7Hvo0XHswCvM1PxfI0e1j9ogqSLlugTXJbgewfUIbp/g9glOJzid4NoE1ya4BYJbILglglsiuGWCWya4BsE1CG6L4LYIjhEcy3DR/ymUZBTFjpCgXMxhjHCxQpF+hvRJMqpL4F1FaphP9QHhMuVbw1EZ9w2vUM4aVtGST2TYlapZx8rIXqlnZWynYddqUpWGaya+casZYnHsl+E4X8xQVYKqZqhnUMDuV4CypHXsW1E/iJ/6v04Y1xFxgxNWylFe7Mul8S6cDTbKxdL9Ynn9QaHyJOnQebgFt+Eu5p+HCjyHNahjDS68gffwQckrReWhMh+juYnEcwNGLqXyF2HXWWg=</latexit>w ⇠ N (µ, ⌃)

1

w1
C B
C
M (x2 ) C B w2 C
C B w3 C..
C
A B.
B .. C
@ . A
(x )

M

{z

1 0

From Linear regression to Gaussian
processes (2)y=<latexit sha1_base64="FyuaKtBxh9emcwi+JTrxDxzfQQo=">AAAGqnichZQ9b9NgEMevAUIpL01hQWKxiIqKVEVPAhIICakFKlG1itKEtIniKrKdx6kV27FsJyGx/AWYkRiYQGJAbIysZeALMPQjIMYisTBwfoG8+No6ip977n7/e+6c+GRL1xyXsaO51LnzF9IX5y8tXL5y9dpiZun6rtPt2QqvKl29a9dkyeG6ZvKqq7k6r1k2lwxZ53ty52kQ3+tz29G65gt3aPF9Q2qbmqopkouuZiYn9ofCY0GUDU8sHWi+2B8IoqMZgicqki4U/RUM9VZFuaK1DeluM5NlORZeQtLIx0YW4qvUXUp9BhFa0AUFemAABxNctHWQwMFPA/LAwELfPnjos9HSwjgHHxZQ20OKIyGht4P3Nu4asdfEfZDTCdUKnqLj10alAMvsO/vIjtk39on9YH9OzOWFOYJahrjKkZZbzcVXNyu/z1QZuLpwMFadopCRPr0nF1R4GPaiYW9W6Am6VKL8/dGb48qj8rJ3h71nP7G/d+yIHWKHZv+X8mGHl9+G2VuoUXHtw0vMJP6vUUavhz4/rmLMDQluSHAjghsR3IDgBgSnEpxKcB2C6xDcOsGtE9wmwW0S3BbBbRFcjeBqBFcnuDrBMYJjCS74P/VCMrAiRY+gLMyhTXGRhyKdBOmQZFAXx7uE1CQ/9vuESg/fGhk9s7rJCKWsYBXt8IlMqsbepGJ76qyxZnvmpElVMa5KwZiOb1wxQWzM/DIy7jcSVJmgygnqGWRx+mWhENIqzq1gHkRP/d8kjOoIOP+ESCHIi3M5PzuFk8ZuIZe/lyvs3M+uPYkn9DzcgtuwgvkfwBo8hxJUsYbX8AUO4Wt6NV1O19ONCE3NxZobMHWlW38BeLZmxg==</latexit>l µ = E[y] = E[ w] =<latexit sha1_base64="/t/3zq/todGA29ZKM9aEbicUm6Q=">AAAG0HichZTPaxNREMenUddafzTVi+ClGCriIbxEQRGEVg1YWkqamDYlKWF3+zZdstld9leahCBee/LmwZOCB/HuVcGD/gMe+ieIxwpePDj7djU/dlJfyL55M5/vZGY3O4pt6K7H2NFM6tTpM9LZ2XNz5y9cvDSfXri85Vq+o/KKahmWU1Vklxu6ySue7hm8ajtcbisG31Zaj8L4dsAdV7fMp17X5rttuWnqmq7KHroa6Vt1pe0/qBdq9aC7K3aluK/Xgw4ehBVGwkPAGukMyzKxFpNGLjYyEK+itZD6AnXYAwtU8KENHEzw0DZABhc/NcgBAxt9u9BHn4OWLuIcBjCHWh8pjoSM3hZem3iqxV4Tz2FOV6hV/BUDvw4qF2GJfWPv2DH7yt6z7+z31Fx9kSOspYu7Emm53Zg/vFr+9V9VG3cP9oeqExQK0if35IEG90QvOvZmC0/YpRrlD3ovj8v3S0v9G+wN+4H9vWZH7DN2aAY/1bebvPRKZN9DjYZ7AAeYqf6vRgW9ffQN4iqGXJfgugTXI7gewXUIrkNwGsFpBNciuBbBrRDcCsGtEtwqwa0R3BrBVQmuSnA7BLdDcIzgWIIL/0++IEMrUvgEZWMOfYyLPBTpJkiXJMO6OF5lpEb5oX9AqAzx1ijomdSNRihlMdFDcUoPZfQ2xb0bpYfepGJ9rKqhZn2iplHVRly/ijED382NBFGYeIYKngsJqkRQpQT1GDI4JzOQF7SGEy6cHNHz+TszozpCbjAlkg/z4gTPTc7rpLGVz+ZuZ/ObdzLLD+NZPgvX4DrcxPx3YRme4P2vYA0v4AN8hE9SSTqQnknPIzQ1E2uuwNiSDv8AY+NueQ==</latexit>Tl ⌃ = E[yy ]w ⇠ N (µ, ⌃)E[w] = 0E[y]E[y]T= E[( w)( w)T ] = E[ wwT<latexit sha1_base64="g9zmrExpS2X/hKOQPPBX/ENy+T0=">AAAHH3ichZTPbxJREMcHtFhRC9WLiRciaVMPkgeaaEyatCqJTZuG8qOlYYHsbh+wYYENu4BA+Af8Bzx4qokH45/hoZ6MFw+cPBuPNfHiwdm3q/zYoS7Lvnkzn+8ws8uOYuiaaTE29vkvXV4KXFm+Grx2/cZKKLx689Bsddoqz6ktvdXOK7LJda3Jc5Zm6TxvtLncUHR+pNSf2fGjLm+bWquZtfoGLzbkalOraKpsoasc7ktKRqs25PVNKVmQun08S9nifWdTdJdSNihJQYFsSEqqpknd3r2JhYLIZsRmXQ+epazY2CFHKutGTXYAJ1AOR1mMiSPiNeKuEQX3SLVW/WcgwQm0QIUONIBDEyy0dZDBxE8B4sDAQF8Rhuhro6WJOIcRBFHbQYojIaO3jtcq7gqut4l7O6cp1Cr+io7fNiojsMa+svfsnH1iH9h39nthrqHIYdfSx1VxtNwoh17dzvz6r6qBqwW1ieoChYL0xT1ZUIHHohcNezOEx+5SdfJ3B6/PM0/Sa8N19pb9wP5O2Zh9xA6b3Z/quwOefiOyn6CmgmsXXmIm6V+NCnqH6Bu5VUy4PsH1CW5AcAOC6xFcj+AqBFchuDrB1Qlum+C2CW6H4HYIbpfgdgkuT3B5gjsmuGOCYwTHPJz9f+oI0rYcRYegDMyhzXCOhyJND2mSpF0Xx6uM1DQ/8Y8IlS7eGgU987rpCKVMeXpILeghg96quHfT9MTrVezNVDXR7M3VNK3ad+tXMabju7nvIZJzz1DBfdJDpQkq7aGeQxTnZBQSgq7ghLMnh/N8/s5Mpw6bGy2IJOy8OMHj8/PaaxwmYvEHscTBw+jWU3eWL8MduAsbmP8RbMELvP85rOGbb8m34gsFTgNngc+BLw7q97maWzBzBMZ/AL3liP0=</latexit>

=↵

T

]

T

l Therefore,y ⇠ N (0, ↵T)<latexit sha1_base64="tPU7qB8si21A8HAZNbD5Mlp1f7Y=">AAAGyXichZQ9b9NQFIZPA5hSPtrCgsQSERUVCUU3AQnE1AKVqFJFadK0qepS2e51YsWxje24TaJMTPAHGJhAYkAMrDAzwB9g6E9AjEViYeD42pAPn5Qb5d7r9z7vyTl2fFTHNDyfsaOp1KnTZ6Sz0+dmzl+4eGl2bv7ypme3XY1XNdu03ZqqeNw0LF71Dd/kNcflSks1+ZbafBiebwXc9Qzb2vA7Dt9tKXXL0A1N8VHam8vIQScte0YrLRcX5YDdkhXTaSiyWmoYYnqycRMplmVipJObXLzJQDxK9nzqC8iwDzZo0IYWcLDAx70JCnj42YEcMHBQ24Ueai7uDHHOoQ8z6G0jxZFQUG3iXMernVi18DqM6Qm3hr9i4tdFZxoW2Df2jh2zr+w9+85+T4zVEzHCXDq4qpGXO3uzL65Wfv3X1cLVh8bAdYJDRfrkmnzQ4Z6oxcDaHKGEVWpR/KD78rhyv7zQu8HesB9Y32t2xD5jhVbwU3u7zsuvRPR99Oi4BnCIkeR/Oaqo9lDrx1kMuA7BdQiuS3BdgjsguAOC0wlOJ7gmwTUJbpnglgluleBWCa5AcAWCqxFcjeC2CW6b4BjBsQQX/p/aggx3kaNNUA7GMEa4SKFIL0F6JBnmxXFWkBrmB3qfcJnirVFRGfcNn1DOUqKG0oQaKqjWxb0bpgdq0rE2ktXAszaW07CrGOev4ZmJ72YxQayMPUMVr1cSVJmgygnqEWSwT2YgL2gdO1zYOaLn87dnRnmEXH/CST6Mix08N96vk5vNfDZ3O5tfv5NZehD38mm4BtdhEePfhSV4jPe/ijk8hw/wET5JBempdCh1IzQ1FXuuwMiQnv0Bo5JrIg==</latexit>– This is called a Gaussian process!
( y is jointly Gaussian)<latexit sha1_base64="9onwr1SjRAkM4Oz8ZrAFhA7TSpc=">AAAG13ichZS/b9NQEMevAdxSfrSFBYmlIipiil4CEhVTC1SialWlCWkTtVVlu8/BimNb/tU6VsSGEBIDCx2YQGJA/Bks/AMMnZgRY5FYGDg/u3ISX9sXxe+9u8/3cvecd4pt6K7H2NFY4cLFS9L4xOXJK1evXZ+anrmx4Vq+o/KGahmW01Rklxu6yRue7hm8aTtc7ioG31Q6T2L/ZsAdV7fM515o852u3DZ1TVdlD0317SDcnS6yEhNjNr8op4sipKNqzRR+wjbsgQUq+NAFDiZ4uDZABhc/W1AGBjbadiBCm4MrXfg59GEStT5SHAkZrR18tnG3lVpN3McxXaFW8VcM/DqonIU59oN9YcfsO/vKfrF/p8aKRIw4lxBnJdFye3fqza3633NVXZw9eJGpzlAoSJ9dkwcazItadKzNFpa4SjWJH/QOj+uPanPRXfaJ/cb6PrIj9g0rNIM/6ud1Xvsgou+hRsM5wFmFIp5whLuTTBX0RcLaT7PJ+AOCPCC4kOBCgusRXI/g9glun+A0gtMIrkNwHYJbJLhFglsmuGWCWyG4FYJrElyT4FoE1yI4RnAsx8X/P1+Q8SpR+ARlYwx9iEssFOnmSJck47w4PmWkBvnM3idUhrhlClpGdYMeSlnN1VA9pYY6Wtvi7AbpzJpXrA5llWlWR3IaVK2l+avoM/Aur+WIpZF3qOB+KUfVCKqWo56K+12EiqA17Ihxp0nez0mPTfJI+gDtqcRxseOXR/t7frFRKZXvlyrrD4oLj9PePwG34Q7cw/gPYQGe4fk3MIc2vIX3cCi1pJfSK+l1ghbGUs1NGBrSu/8Ku2hu</latexit>From Linear regression to Gaussian
processes (3)<latexit sha1_base64="Bh0brQhk5Uwrt1Iraovpk86ERr0=">AAAG1XichZRNb9NAEIanAUwpH03hgsQlIipqJRQ2AQmEhNQClahSVWnSNKnqEtnuOrHi2JbtuCRRbogDF06IAyeQOCB+AkdA8Ac49CcgjkXiwoHxR8mHJ2Wj7K7ffd7JjB2PbOma4zJ2MJU4cfKUcHr6zMzZc+cvzCbnLm45ZttWeFkxddOuypLDdc3gZVdzdV61bC61ZJ1X5OYD/7zicdvRTGPT7Vh8tyXVDU3VFMlFqZa8IXqdlOhorZS4viB67Loo6VZDEuVCQwumx5uL946OvPxiLZlmGRaMVHyTjTZpiEbBnEt8BRH2wAQF2tACDga4uNdBAgc/O5AFBhZqu9BDzcadFpxz6MMMettIcSQkVJs41/FqJ1INvPZjOoFbwV/R8WujMwXz7Dt7zw7ZN/aB/WB/JsbqBTH8XDq4yqGXW7XZ55dLv//rauHqQmPgOsYhI318TS6ocCeoRcParEDxq1TC+F731WHpbnG+d429ZT+xvjfsgH3CCg3vl/JugxdfB9H30KPi6sETjCT+y1FGtYdaP8piwHUIrkNwXYLrEtw+we0TnEpwKsE1Ca5JcMsEt0xwqwS3SnB5gssTXJXgqgS3TXDbBMcIjsU4///UDkh/FzraBGVhDG2ECxWKdGKkQ5J+XhxnCalhfqD3CZcevDUyKuO+4RPKWYjVUJhQQwnVenDvhumBGnesjWQ18KyN5TTsWo/yV/BMx3dzPUasjD1DGa9XYlSRoIox6iGksU+mIRfQKnY4v3OEz+eoZ4Z5+Fx/wknOj4sdPDver+ObrVwmezOT27iVXrof9fJpuAJXYQHj34YleIT3v4w5vISP8Bm+CBWhLzwVnoVoYiryXIKRIbz4Czlsbx0=</latexit>l Where

0

B
B
T
= ↵B
@

K=↵

|

<latexit sha1_base64="MLgbvjmVw0M+nLehXGNG8Tfasu0=">AAAIaHiclZRLTxNRFMdv64M6PgBdqDExEwoIC5vbYgIxIQGVRAIhtRQo4UIzM73Tjp1XZqalZTJ8AHeuXLjSxIXxY7jxC7hg5VpdYqILF565U2zpXECn6dxzz/n9z5wzjyPbuuZ6GB8kkufOX7g4kLokXL5y9drg0PD1dddqOApdUyzdckqy5FJdM+map3k6LdkOlQxZpxty/XEY32hSx9Uss+i1bbptSFVTUzVF8sBVHk68JE2q+EuBKMyKRNLtmiQS2fBJvqYFf42dojArRFGBNMwKdWRHUqgvEJlWNdO3DclztFYgkGbF8lyREIGostXy90eJbNe0CXaRVlDWJneKo/sBAxgqEGpWuvqg7BOPtjzXa+u0W0jQd9UmFGJ6lvH/BZhnFTAWSykSheUcF6OUfRn958HkKHEaOt26n81MU2Pbx7YX+FOZGWoEQShT/r3TnWJQHkrjDGaHGDeyHSONOkfeGk5+QQRVkIUU1EAGoshEHtg6kpALvy2URRjZ4NtGPvgcsDQWpyhAAmgbQFEgJPDW4VyF3VbHa8I+zOkytQJX0eHvgFJEY/gzfo8P8Sf8AX/Fv0/M5bMcYS1tWOVIS+3y4Itbqz/PVBmweqjWVZ2ikIE+vScPqWiG9aJBbzbzhF0qUf7m3qvD1YeFMX8cv8Xfob83+AB/hA7N5g/l3TNaeM2yV0CjwtqEVUFpuMM+7I4qlSHmM2/QqabLtzhki8O1OVybw+1xuD0Ot8vhdjmcyuFUDlfncHUON8/h5jncIodb5HBLHG6Jw5U4XInDbXK4TQ6HORyOceH712BkaEWKBoeyIYd2jIs8PNKNkS6XDOuicJaA6uW7/oCj0tlXJoOnX9cb4SnzsR7yJ/SwCt4qu3e9dNcbVywfq6qrWe6rqVe10qlfgZgO3/JKjFjoe4Yy7BdiVIFDFWLUE/Z9p1GO0SpMxHDSRM/naMZGdURzgB/JhXlh4mf753vcWM9lslOZ3LMH6blHndmfQnfQCJqA/NNoDj2F+7+GlMSv5N3kveTEwLfUUOpm6naEJhMdzQ107EiN/AEGUvs1</latexit>Ty ⇠ N (0, ↵
..
.
(xi )
..
.
{z

T

) = N (0, K)
1

0

(xj )

}|

{z

C
C B· · ·C@A· · ·CAT

Kij = (xi )T (xj ) = k(xi , xj ) : kernel function<latexit sha1_base64="tMmEUGiyufcc0IVRrxhQu3wDnsk=">AAAHCHichZTPTxNBFMcfVSviD4peTLyADQYS00yricaEBFQSSQkpLYUSis3uMluWbnc3u9vSdtN/gH9AE0+aeDBevBkvHvQg/4AHTp6NR0y8ePDt7EJL94HbdGfmO5/v63uz3Sdbuua4jB0Mxc6dvxC/OHxp5PKVq9dGE2PXVx2zYSu8qJi6aZdkyeG6ZvCiq7k6L1k2l+qyztfk2hN/f63JbUczjRW3bfHNulQ1NFVTJBelSiKTrXjaTrc8MVOeKFvb2lS52apo089Xjhc70zO1QL0bLCuJJEsxcY1HJ+lwkoTwypljsR9Qhi0wQYEG1IGDAS7OdZDAwc8GpIGBhdomeKjZONPEPocujKC3gRRHQkK1hvcqrjZC1cC1H9MRbgV/Rcevjc5xmGTf2Tt2yPbZe/aT/T01lidi+Lm0cZQDL7cqo3s3C3/+66rj6MJ2z3WGQ0b67JpcUOGhqEXD2iyh+FUqQfxm58Vh4VF+0rvD3rBfWN9rdsC+YoVG87fydpnnX4noW+hRcWziqEAST9jD1VGmMu55Qu2G2fT4FkG2CK5NcG2C6xBch+B2CW6X4FSCUwmuRnA1gpsjuDmCWyC4BYLLElyW4EoEVyK4dYJbJzhGcCzC+f+/hiD9WeBoEJSFMbQTXKBQpBMhHZL08+J4l5Dq53t6l3Dp4i2TURn09e9QzlykhtwpNRRQrYqz66d7atSxeCKrnmdxIKd+11KYv4J7Or7LSxFifuAZyriej1B5gspHqKfi/U5CRtAqdkS/0wTP56jHBnkEfYDeyfhxseOnB/t7dLKaSaXvpTLL95Ozj8PePwy34DZMYfwHMAvP8PyLmMNL+AzfYD++F/8Q/xj/FKCxodBzA05c8S//APlWetM=</latexit>l Now, w are integrated out in a Bayesian way<latexit sha1_base64="iEXwe3orMtJvmJFSerrW6KMAYXc=">AAAG13ichZS/b9NQEMevAUwpP9rCgsRSYRUxRS8BCcTUApWoWlVpQtpEbVXZ7nOw4tiWf6WJFbEhhMTAQgcmkBgQfwYL/wBDJ2bEWCQWBs7PrpzE1/ZF8Xvv7vO93D3nneqYhuczdjhROHf+gnRx8tLU5StXr03PzF7f8OzA1Xhds03bbaiKx03D4nXf8E3ecFyudFSTb6rtJ7F/M+SuZ9jWc7/n8J2O0rIM3dAUH0217bC7OyOzIhNjLr8opQsZ0lGxZws/YRv2wAYNAugABwt8XJuggIefLSgBAwdtOxChzcWVIfwcBjCF2gApjoSC1jY+W7jbSq0W7uOYnlBr+Csmfl1UzsE8+8G+sCP2nX1lv9i/E2NFIkacSw9nNdFyZ3f6zc3a3zNVHZx9eJGpTlGoSJ9ekw86PBS1GFibIyxxlVoSP+wfHNUeVeejO+wT+431fWSH7BtWaIV/tM/rvPpBRN9DjY5ziLMGMp5whLvjTFX0RcI6SLPJ+H2C3Ce4HsH1CK5PcH2C6xJcl+B0gtMJrk1wbYJbJLhFglsmuGWCWyG4FYJrEFyD4JoE1yQ4RnAsx8X/v0CQ8SpRBATlYAxjhEssFOnlSI8k47w4PhWkhvnMPiBUprhlKlrGdcMeSlnJ1VA5oYYaWlvi7IbpzJpXrI5klWlWx3IaVq2l+WvoM/Eur+WIpbF3qOJ+KUdVCaqao56K+y1DWdA6dsS40yTv57jHJnkkfYD2lOO42PFL4/09v9goF0v3iuX1+/LC47T3T8ItuA13Mf4DWIBneP51zKEFb+E9HEhN6aX0SnqdoIWJVHMDRob07j/9ymhs</latexit>

1}

Random draw from a Gaussian process
3
2
1

f

0
-1
-2
-3
-1

0

1

2

x

l Why such smooth functions?

34

Multivariate Gaussian distribution1✓µ = E[x]

p
exp
|⌃|

⌃ = E[xxT ]E[x]E[x]Tl p(x|µ, ⌃) = p

(

2⇡)D

1
(x µ)T ⌃ 1 (x µ)2(mean vector)(covariance matrix)◆

Random draws from MVN

l If correlation is high, similar values are sampled– Minus for negative correlation, 0 for independent

Random draws from MVN (2)

0
1
1 0 0
⌃ = @0 1 0A
0 0 1

0

1
0
1
1 0.5 0.2
1 0.8 0.6
⌃ = @0.5 1 0.5A ⌃ = @0.8 1 0.8A
0.2 0.5 1
0.6 0.8 1l Three dimensional caseIntuitive explanationl Assume grid of input points

X = (x1 , x2 , · · · , xN )and create a covariance matrix K between them:e.g.

Kij = k(xi , xj )kernel functionl k(xi,xj): kernel function. For example:

k(x, x0 ) = exp( |x(RBF kernel)

x0 |2 /✓)l Same as those used in SVM or others(GP is a Bayesian version of the kernel methods)

Intuitive explanation (2)l Draw from correlated multivariate Gaussian:

Sample from MVN

Covariance matrix

Intuitive explanation (3)l Draw from correlated multivariate Gaussian:

Sample from MVN

Covariance matrix

Intuitive explanation (4)l Draw from correlated multivariate Gaussian:

Sample from MVN

Covariance matrix

Kernels and random draws from GP(a) Linear kernel: xT x0<latexit sha1_base64="o6T+JObDly2gsz3JjoTGT6gQrVY=">AAAHPXichZU7bxNBEMcnCRwhPJxAg0QTYQWorLVBAlElQCSiRJFjx4mtOES3lz1z8r10d3b8kL8ALQUFUIBEgfgQFDR8AYp8AAoEoglSGgrm9s7y4ybJWd7HzO8/nrld73LXNPyAscOJyalz55UL0xdnLl2+cjU1O3dty3caniZKmmM6XpmrvjANW5QCIzBF2fWEanFTbPP6k9C/3RSebzj2ZtB2xa6l1mxDNzQ1QFOl2mw938Tmzt5smmWYfOaTg2w8SEP85J25qRRUYR8c0KABFgiwIcCxCSr4+NmBLDBw0bYLXbR5ODKkX0APZlDbQEogoaK1jm0NZzux1cZ5GNOXag1/xcSvh8p5WGDf2Sd2xL6xz+wn+3dirK6MEebSxp5HWuHupV7eKB6fqbKwD+DFQHWKgiN9ek0B6PBQ1mJgba60hFVqUfxm5/VR8VFhoXubfWC/sL737JB9xQrt5l/t44YovJHR91GjY9/EXoM0vuEuzvqZcvR1pbUXZzPgWwTZIrg2wbUJrkNwHYI7ILgDgtMJTie4OsHVCW6J4JYIboXgVghuleBWCa5McGWCqxBcheAYwbEEx+P1Dfchta48XtfIn1xPLndx3y8Iv4U7dkBU5TxJuZijMcJFFor0E6RPkmHdAlsVqWF+YO8RKlP+izlaxnXDHkqZT9SQP6GGIlprcm2G6YE1qVgbyWqgWRvLaVi1Huevoc/Es2I9QSyP7RGO8+UEVSCoQoJ6Ks+PNOQkreOJG55k0fr0z/Aoj+icoT25MC7eKNnx+yM52MplsvcyuY376cXH8d0yDTfhFtzF+A9gEZ7h+y9hDha8grfwTvmi/FB+K38idHIi1lyHkUc5/g+ASH7T</latexit>(c) Exponential kernel:
exp( |x x0 |/✓)
<latexit sha1_base64="1AbHwsjEwq76AgEDgI8bFbo/+zg=">AAAHUnichZXNbtNAEMcnDSSlFJLCBQkOLVGhHBo2AQnEqQUqUbWq0oS0qeoqst11asWJLdtJ89FcOPICHDiBhATiBK/AhQeAQx8BcSwSEuLAeO0oH56WjeLdnfn9JzPeza5iGbrjMnYcmYieOx+LT16Yujh96XIiOXNlyzEbtsqLqmmYdkmRHW7odV50ddfgJcvmck0x+LZSfeL5t5vcdnSz/txtW3yvJlfquqarsoumcvKGxFvWwuKR1GxJc4vSHPa3j+5K7gF35TvlZIqlmWiz4UEmGKQgaDlzJpoACfbBBBUaUAMOdXBxbIAMDn52IQMMLLTtQRdtNo504efQgynUNpDiSMhoreKzgrPdwFrHuRfTEWoVf8XAr43KWZhn39kHdsK+so/sB/t7aqyuiOHl0sZe8bXcKideXiv8/q+qhr0LBwPVGQoF6bNrckGDh6IWHWuzhMWrUvXjNzuvTgqP8vPdW+wt+4n1vWHH7AtWWG/+Ut9t8vxrEX0fNRr2TexVSOEb7uKsn6mCvq6w9oJsBnyLIFsE1ya4NsF1CK5DcIcEd0hwGsFpBFcluCrBLRPcMsGtEtwqwa0R3BrBlQiuRHA7BLdDcIzgWIhTgvX19iG1rkqwrr4/vJ6K2MV9Pyf8NdyxA0IS8zBlYY76COdbKNIJkQ5JenVzfMpIDfMDe49QGeJfrKBlXDfsoZS5UA25U2oooLUi1maYHljDivWRrAaa9bGchlUbQf4q+gw8KzZCxMrYHlFwvhKi8gSVD1FPxfmRgqygNTxxvZPMX5/+Ge7n4Z8ztCfrxcUbJTN+f4QHW9l05l46u3k/tfQ4uFsm4TrchAWM/wCW4Bm+/yLm8ALewyf4HPsW+xOPxKM+OhEJNFdhpMWn/wFDpoL6</latexit>(b) RBF kernel: exp( |x x0 |2 /✓)<latexit sha1_base64="gvHVh+BUU+e79I0DGerEJnj52qg=">AAAHVHichZXNbtNAEMenDU1KgSaFCxISaokK5dCwCUggTi1QiapVlSakTVWXyHbXqRUntmwnzUdz48QLcOAEEgfgCG/AhRcAqY+AOBaJSw+M147y4WnZKN7dmd9/MuPd7CqWoTsuY8dj45ELE9HY5MWpS5evTMcTM1e3HLNuq7ygmoZpFxXZ4YZe4wVXdw1etGwuVxWDbyuVp55/u8FtRzdrL9yWxfeqcrmma7oqu2gqJW5KvGktLB5JjaY0tyjNYX/n6GXmnuQecFe+W0okWYqJNhsepINBEoKWNWcicZBgH0xQoQ5V4FADF8cGyODgZxfSwMBC2x500GbjSBd+Dl2YQm0dKY6EjNYKPss42w2sNZx7MR2hVvFXDPzaqJyFefaDfWQn7Dv7zH6x0zNjdUQML5cW9oqv5VYp/vp6/u9/VVXsXTjoq85RKEifX5MLGjwStehYmyUsXpWqH7/RfnOSf5yb79xm79lvrO8dO2bfsMJa44/6YZPn3oro+6jRsG9gr0IS33AHZ71MFfR1hLUbZNPnmwTZJLgWwbUIrk1wbYI7JLhDgtMITiO4CsFVCG6Z4JYJbpXgVglujeDWCK5IcEWC2yG4HYJjBMdCnBKsr7cPqXVVgnX1/eH1VMQu7vk54a/iju0TkpiHKQtz1Ic430KRToh0SNKrm+NTRmqQ79u7hMoQ/2IFLaO6QQ+lzIZqyJ5RQx6tZbE2g3TfGlasD2XV16yP5DSo2gjyV9Fn4FmxESJWRvaIgvOVEJUjqFyIeibOjyRkBK3hieudZP769M5wPw//nKE9GS8u3ijp0fsjPNjKpNL3U5nNB8mlJ8HdMgk34BYsYPyHsATP8f0XMIdX8Am+wNfoz+hpLBKb8NHxsUBzDYZabPofgQ6Dng==</latexit>

0(d) Periodic kernel: exp(cos(|x x |/✓))<latexit sha1_base64="nWN1lFOasZk+WeTAOxc9j8VeytU=">AAAHV3ichZXNbtNAEMenLTSlfCSBCxKXhqiQHho2AQnEqQUqUbWq0oS0qeoqst11asWJLdtJ89GckXgBDpxA4oAQV3gALrwAhzwC4lgkLkgwXjvKh6dlo3h3Z37/yYx3s6tYhu64jPWnpmcuXJyNzF2av3zl6rVoLH59xzEbtsqLqmmYdkmRHW7odV50ddfgJcvmck0x+K5Sfer5d5vcdnSz/sJtW/ygJlfquqarsoumciwh8ZaVklTTSZ1IzZaUWJYS2N89uSe5R9yVl5bKsSRLM9EWwoNMMEhC0HJmfCYKEhyCCSo0oAYc6uDi2AAZHPzsQwYYWGg7gC7abBzpws+hB/OobSDFkZDRWsVnBWf7gbWOcy+mI9Qq/oqBXxuVC7DIvrMP7JR9Yx/ZD/bnzFhdEcPLpY294mu5VY6+uln4/V9VDXsXjoaqcxQK0ufX5IIGj0QtOtZmCYtXperHb3ZenxYe5xe7d9g79hPre8v67CtWWG/+Ut9v8/wbEf0QNRr2TexVSOIb7uJskKmCvq6w9oJshnyLIFsE1ya4NsF1CK5DcMcEd0xwGsFpBFcluCrBrRLcKsGtE9w6wW0Q3AbBlQiuRHB7BLdHcIzgWIhTgvX19iG1rkqwrr4/vJ6K2MUDPyf8NdyxQ0IS8zBlYY76GOdbKNIJkQ5JenVzfMpIjfJDe49QGeJfrKBlUjfqoZS5UA25M2oooLUi1maUHlrDis2xrIaazYmcRlVbQf4q+gw8K7ZCxNrEHlFwvhai8gSVD1HPxPmRhKygNTxxvZPMX5/BGe7n4Z8ztCfrxcUbJTN5f4QHO9l05n46u/0gufIkuFvm4BbchhTGfwgr8BzffxFzeAmf4DN8me3P/o3gLemj01OB5gaMtUj8H2LEhPE=</latexit>Ordinary usage: GPRl Gaussian process regression (GPR)– Given a set of pairs (x,y), predict y for new x*

Part 2:
Language, Robotics, andGaussian processes

Language and Roboticsl (Intellectual) Robotics is not only aroundphysical movements, but symbolic actions arenecessary for high-level behavior– Closely tied with languages

What is “word”?
l Children are not taught

about “words” :learn only fromexperiencesl Robots should be flexible enough to learn“words” automatically from their environmentl Novel words, colloquial expressions, dialects,
and child languages .. should be robust
in a practical viewpoint

What is “word”?
l Problem: want to find “words” from a sequenceof acoustic signals
à “she sings the song”
l Easier version: find words from phonologicalsequences of child’s speech (Goldwater+ 2006)
WAtsDIs à WAts DIs
WAtIzIt à WAt Iz It
yuwanttusiD6bUk à yu want tu si D6 bUk(CHILDES corpus)

Word segmentationl Word segmentation and morpheme analysis are

crucial tool to analyze Asian languages
山花貞夫・新民連会長は十六日の記者会見で
à 山花 貞夫 ・ 新 民連 会長 は 十六 日 の 記者 会見 で今后⼀段时期，不但居民会更多地选择国债à 今后 ⼀段 时期 ， 不但 居民 会 更 多 地 选择 国债
l Usually done by supervised learning, but can bemade unsupervised (Mochihashi+ ACL 2009,
Uchiumi+ ACL 2015)!

Western languages
l Case of Latin (Scripta continua)– English was also originally written without spaces
Vergil text, aroundAD 141

Unsupervised word segmentation(Mochihashi+ ACL 2009)

first,shedreamedoflittlealiceherself,andonceagainthetinyhandswereclaspedup
onherknee,andthebrighteagereyeswerelookingupintohersshecouldhearthevery
tonesofhervoice,andseethatqueerlittletossofherheadtokeepbackthewanderingh
airthatwouldalwaysgetintohereyesandstillasshelistened,orseemedtolisten,thew
holeplacearoundherbecamealivethestrangecreaturesofherlittlesister'sdream.thelonggrassrustledatherfeetasthewhiterabbithurriedbythefrightenedmousesplashedhiswaythroughtheneighbouringpoolshecouldheartherattleoftheteacupsasthemafirst, she dream ed of little alice herself ,and once again the tiny hand s wereclasped upon her knee ,and the bright eager eyes were looking up into hers -shecould hearthe very tone s of her voice , and see that queer little toss ofherhead to keep back the wandering hair that would always get into hereyes -and still as she listened , or seemed to listen , thewhole place a round her became
alive the strange creatures of her little sister 'sdream. thelong grass rustled ather
feet as thewhitera bbit hurried by -- the frightened mouse splashed his way
through the neighbour ing pool -- shecould hearthe rattle ofthe tea cups…

‫‪Arabic unsupervised segmentation‬‬‫‪l Arabic AFP news, 40,000 sentences‬‬‫‪.‬اﻟﻔﻠﺳطﯾﻧﯾﺑﺳﺑﺑﺗظﺎھرةﻻﻧﺻﺎرﺣرﻛﺔاﻟﻣﻘﺎوﻣﺔاﻻﺳﻼﻣﯾﺔﺣﻣﺎس‬‫واذاﺗﺣﻘﻘذﻟﻛﻔﺎﻧﻛﯾﺳﻠوﻓﺳﻛﯾﯾﻛوﻧﻘدﺣﺎزﺛﻼﺛﺟواﺋزﻛﺑرﯨﻔﯾﺎﺑرزﺛﻼﺛﺔ‬‫‪Google‬ﺎﻟﺻﺣﯾﺔ‬‫‪translate:‬ﻠﯨﻠوازﻣﮭﻣ‬‫‪.‬اﻟﻣﺣﻠﯾﺔواﻟدوﻟﯾﺔﻟﻠﺣﺻوﻟﻌ‬‫”‪“Filstinebsbptazahrplansarhrkpalmquaompalaslamiphamas.‬ﮭو‪+‬ﻗﺎﺋد‬‫‪+".‬ﻣﺎﯾﺳﻣﯨب‪+‬اﻟﺳﻠطﺔاﻟﻔﻠﺳطﯾﻧﯾﺔ‪+‬ﻻﯾﺗﻣﺗﻌﺑﻠﻘب‪+‬رﺋﯾس‪+‬ﺑﻠ‬‫اﻋﻠﻧﺗﺷرطﺔﺟﻧوﺑﺎﻓرﯾﻘﯾﺎاﻟﯾوﻣﺎﻻﺛﻧﯾﻧﺎﻧﻣﺎﻻﯾﻘل‬‫‪NPYLM‬ﺳﺗﻐرﻗﺎﻋدادھﺧﻣﺳﺔاﻋوام‪".‬ﺗﺎرﯾﺧﻲ‬‫وﻗﺎﻟﺗداﻧﯾﯾﻠﺗوﻣﺳوﻧﺎﻟﺗﯾﻛﺗﺑﺗﺎﻟﺳﯾﻧﺎرﯾو‪.‬وﻗدا‬‫‪ .‬اﻟﻔﻠﺳطﯾﻧﻲ ﺑﺳﺑب ﺗظﺎھرة ل اﻧﺻﺎر ﺣرﻛﺔ اﻟﻣﻘﺎوﻣﺔ اﻻﺳﻼﻣﯾﺔ ﺣﻣﺎس‬‫و اذا ﺗﺣﻘق ذﻟك ف ان ﻛﯾﺳﻠوﻓﺳﻛﻲ ﯾﻛون ﻗد ﺣﺎز ﺛﻼث ﺟواﺋز ﻛﺑرﯨﻔﯾﺎﺑرز ﺛﻼﺛﺔ‬‫‪ Google‬اﻟﺻﺣﯾﺔ‬‫‪ translate:‬ﻟوازم ھم‬‫‪ .‬اﻟﻣﺣﻠﯾﺔو اﻟدوﻟﯾﺔ ﻟﻠﺣﺻوﻟﻌﻠﻰ‬‫‪“Palestinian‬ﻔﻠﺳطﯾﻧﯾﺔ‬‫‪supporters‬ب ‪ +‬اﻟﺳﻠطﺔ اﻟ‬‫‪of‬ﺋد ‪ +‬ﻣﺎﯾﺳﻣﻰ‬‫‪ +‬ﻗﺎ‬‫‪event‬ھو‬‫‪+‬بل‬‫‪the‬ب‪ +of‬رﺋﯾس‬‫‪Islamic‬ب ﻟﻘ‬‫‪ + " .‬ﻻﯾﺗﻣﺗﻊ‬‫‪the‬‬‫‪because‬‬‫‪Resistance‬اﻧﻣﺎﻻﯾﻘل‬‫‪ Movement,‬اﻟﯾوم اﻻﺛﻧﯾن‬‫”‪Hamas.‬ﺔ ﺟﻧوﺑﺎﻓرﯾﻘﻲ ا‬‫ا ﻋﻠ ن ت ﺷ ر ط‬‫وﻗد اﺳﺗﻐرق اﻋداد ه ﺧﻣﺳﺔاﻋوام ‪ .‬و ﻗﺎل ت دان ﯾﯾل ﺗوﻣﺳون اﻟﺗﻲ " ﺗﺎرﯾﺧﻲ‬NPYLM: Nested Char-Word HPYLM
“”
PY

PY

“は”

“”

PY
PY

PY

“教”
“国”
“時”

“が”

“彼 が” “教会 が”

“会”

“臨”
Char HPYLM
Word HPYLMl Nested Character-Word n-gram model (HPYLM)– i.e. Bayesian hierarchical Markov model

Inference with Dynamic Programming:
forward variable

l “Inducing Word and Part-of-speech with Pitman-Yor

Hidden Semi-Markov models”, ACL 2015l Forward filtering-backward sampling MCMC
to escape from local minima

Learning with Multimodal information
Multimodal Information
Thisisabottle

SpeechRecognition

User’s utterances

Language modelThis / is / a / bottle

Mutualleaning
Object concept model

Observing, grasping, and shaking

Concepts,
word meanings
BottleDrink
Plushie
Soft

Multimodal
information“Mutual Learning of an Object Concept and Language Model
based on MLDA and NPYLM”,
Nakamura et al., IROS 2014

Word discovery from acoustic signals
Language model(Word bigram
model withletter bigram
model)(Taniguchi+ 2016)
Earlier and more complete
than NLP (eg. Goldwater+17)!

Word sequence

Phoneme sequence

HDP-HLM:
Hierarchical
Dirichlet process
Hierarchical LM

Acoustic model(phoneme model)

Word discovery from acoustic signals (2)üüüFive artificial words {aioi, aue, ao, ie, uo} prepared by connecting five
Japanese vowels.
30 sentences (25 two-word and 5 three-word sentences) are preparedand each sentence is recorded twice by four Japanese speakers.
MFCC (frame size =25ms, shift = 10ms, frame rate 100hz)* HDP-HLM are trained separately for each speaker.
MFCC

Phonemeü The inference procedure
could gradually estimate theboundaries of words andex) ao-ie-ao
phonemes.

Word

Part 2:
Motion segmentation and Gaussian
processes

Learning “Motion” from Movements

観測値

Movements
観測シーン

Observation

単位系列1

Motion 1

単位系列2

Motion 2

時間
Timel Crucial for high-level recognition and planningof actions of robots .. word segmentation for robots– Planning for preparing dishes: wash->cut->boil->...

Model of movements

観測値

Movements
観測シーン

Observation

単位系列1

Motion 1

単位系列2

Motion 2

時間
Timel Observations are continuousà Stochastic models

for trajectories are necessaryl We leverage the Gaussian processes

Actual data of movementsl Time series from two angles (knee and shoulder)l How to induce “motion” from this data?

Gaussian process hidden semi-Markov model

Only the observationl Hidden state (motion class):cj ⇠ Mult(c|cj 1 )l Generate trajectory: xj ⇠ GP(x | Xcj )l a kind of HMM, but segmentations are unknown

Application to Roboticsl Hidden semi-Markov model with Gaussian process

observations to model the time series of joint
angles, Forward-Backward Bayesian learning(infer “words”)
l Segmentation results using simple arm motions :Segmentation result

Dynamic programming MCMC
l Forward filtering:l Backward sampling:Lattice representation
BOS

1

2

3

4

5

6

EOS

p(x[3 : 5]|x[1 : 2])

segment
lengthl Semi-Markov HMM (Murphy 02, Ostendorf 96)– We do not know “correct path” beforehand (as HMM)
l Draw high-probability path via dynamic programming

The first experimentl Unsupervised segmentation of arm movements

measured by Kinectl 2D coordinates (x,y) of the right hand
– Assume x and y are independently generated:
x ~ GP(c), y ~ GP(c)l Motions involved:– Move the hand to right– Raise the hand high– Raise the hand slightly

Results} Correctly recovered thethree motions} Learned

Gaussian
processes:

1000
1000

1000

500

500

500

0

𝑥

-500

-500
-500

-1000
-1000

𝑦

0

0

-500

0

500

1000

-1000
-1000

-500

0

500

1000

-1000
-1000

-500

0

5

Motion segmentationl Correctly recognized Karate motions from observations

Inferred motions (excerpt)l Class 0: Right punch with an additional stepl Class 6: Left lower guardingComparison with other methodsl GP-HSMM vs. HDP-HMM, HDP-HMM+NPYLM
Ground truth
Proposed
HDP-HMMHDP-HMM
NPYLM

Error

Error
0

20

40

60

0

20

40

60

0

Error
(confuse right punchwith right
guard)
20
40
60

Ground truth
ProposedError (1 class
1 motion）

HDP-HMMHDP-HMM
+
NPYLM

0

20

40

60

Number of Motions
l Number of hidden states (motions) can also be

estimated using hierarchical Dirichlet processes– HDP-GP-HSMM– “Sequence Pattern Extraction by Segmenting Time
Series Data Using GP-HSMM with Hierarchical
Dirichlet Process”, Nagano+, IROS 2018
Groundtruth: 11

Number of Motions (2)l We leveraged infinite HMM (Beal+ 2001, Teh+2006) with our semi-Markov structurel Since the state space is infinite-dimensional,
also used a Beam sampling (van Gael+2007)
for slice sampling+dynamic programming– Internally,using a stick-breaking representation
for possibly infinite state spaces

High-dimensional regime
l Actually, robot movements are quite high-dimensional– In our case, # of joint angles = 93
– Cannot apply the method to whole data

Strategy (Our work at IROS 2019)l Solution: dimensionality reductionl Linear PCA à NG
l GPLVM (Gaussian process LVM) à OK, butinference is not stablel Using VAE as a surrogate of GPLVM

Simultaneous optimization

Segmentation runs in
the latent spaceNotel In this case, each cluster (=motion) has its own

VAE for compression– VAE priors are different for each cluster

Graphical modell We observe only s (high-dimensional time series)

Hierarchical Dirichlet
process-variational
autoencoder-Gaussian
process-hidden semiMarkov model(Long!)=
HVGH
Observation

Experimentsl Dance exercises which include four and sevenunit motions (labels are not used in learning)Results (1)l Exercise containing four unit motions:

VAE as a preprocessing

Results (2)l Exercise containing seven unit motions:

VAE as a preprocessing

Estimated latent space in VAE

l VAE is separately learned for compressionl Motions (in color) are mixed and not separated

Estimated latent space in VAE (2)l Simultaneous learning in HVGHl Motions (in color) are clearly separated

Zoology and Brain science (Marmoset)l Can recognize ape’s motions automaticallyl joint work with Koki Mimura (NCAP), JSAI 2019

Part 3:
Nonparametric Bayesian
Deep Visualization (NPDV)Joint work with Bridgestone Corporation; appeared at
IBISML 43 (2021)

79

Data Visualizationl Visualize high-dimensional data by mapping theminto 2D or 3D– Important first step for exploratory data analysis
ID

Y1

Y2

…

Y_D

１
２
・・・
N

l Principal Component Analysis (PCA) has a strong

limitation à Data often have nonlinear structures

NPDV: NP Bayesian Deep Visualizationl Simultaneously

optimize
Y→X, X →V
l Y→X：NN-iWMM
Neural network
using GP +infinite mixture(no NN weights)
l X →V：visualization via
t-SNE (van der
Maaten 2009)

Infinite Warped Mixture Model (iWMM)(Iwata+ 2013)∞-Gaussian
mixturemodel

InferenceX

Y<latexit sha1_base64="T1QB0mKOGtjE1L6Pq0zjYHc4ECM=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJqRNaarIa9aNib9kO0mTKC/AGYkDAgkkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613umoYfMHY6Mzt34eIlZf7ywpWr164vLi3f2POdlqeJiuaYjlflqi9MwxaVwAhMUXU9oVrcFPu8+TT077eF5xuO/TzouuLIUo9tQzc0NUDTi5qlBg2u96uD+lKG5Zi8VtKNfNzIQHwVneW5DajBS3BAgxZYIMCGANsmqODjfQh5YOCi7Qj6aPOwZUi/gAEsoLaFlEBCRWsT/4+xdxhbbeyHMX2p1vAtJv48VK5Alv1gX9gZ+86+sp/s79RYfRkjzKWLTx5phVtffH2r/Oe/KgufATQS1TkKjvT5NQWgwyNZi4G1udISVqlF8du9t2flx6Vs/y77xH5hfR/ZKfuGFdrt39rnXVF6h9GzcrwFRqpBG58aZHCM+9gb5srR15fWQZxPwo8rHNn38DYwxwaqVdlzoDM1wgnxrhOC6xJcl+B6BNcjuA7BdQhOJzid4JoE1yS4NYJbI7hNgtskuC2C2yK4KsFVCe6A4A4IjhEcS3E8nt9wLVPzyuN5jfzp+eTySxj6BeG3cNUnRE3205SLORpjXGShSD9F+iQZ5hau8FF2aEvT4SgN+VFFYh8QKlPuGxwtk7pRD6UspiouTqm4LL9Wa6KSxJpWbI9llWi2J3IaVe3E+WvoM3F32kkR6xMrimN/PUWVCKqUop7J3SYDBUnruAeFe2c0m8NTI8oj2pVoTyGMi2dYfvLESjf2Crn8/Vxh90Fm9Ul8ms3DbbgD9zD+Q1iFDRz/CuZgwxt4Dx+UglJV6ooaobMzseYmjF3Kq3/peZLL</latexit><latexit sha1_base64="p+MlzADJVLtSx0+TRZ5ORfiBDS4=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJqRNaKrKa9aNib9kO0mTKC/AGYkDAgkkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613umoYfMHY6Mzt34eIlZf7ywpWr164vLi3f2POdlqeJiuaYjlflqi9MwxaVwAhMUXU9oVrcFPu8+TT077eF5xuO/TzouuLQUo9tQzc0NUDTi7qlBg2u92uDo6UMyzF5raQb+biRgfgqOstzG1CHl+CABi2wQIANAbZNUMHH+wDywMBF2yH00eZhy5B+AQNYQG0LKYGEitYm/h9j7yC22tgPY/pSreFbTPx5qFyBLPvBvrAz9p19ZT/Z36mx+jJGmEsXnzzSCvdo8fWt8p//qix8BtBIVOcoONLn1xSADo9kLQbW5kpLWKUWxW/33p6VH5ey/bvsE/uF9X1kp+wbVmi3f2ufd0XpHUbPyvEWGKkObXxqkMEx7mNvmCtHX19aB3E+CT+ucGTfw9vAHBuoVmXPgc7UCCfEu04IrktwXYLrEVyP4DoE1yE4neB0gmsSXJPg1ghujeA2CW6T4LYIbovgqgRXJbgawdUIjhEcS3E8nt9wLVPzyuN5jfzp+eTySxj6BeG3cNUnRF3205SLORpjXGShSD9F+iQZ5hau8FF2aEvT4SgN+VFFYh8QKlPuGxwtk7pRD6UspiouTqm4LL9Wa6KSxJpWbI9llWi2J3IaVe3E+WvoM3F32kkR6xMrimN/PUWVCKqUop7J3SYDBUnruAeFe2c0m8NTI8oj2pVoTyGMi2dYfvLESjf2Crn8/Vxh90Fm9Ul8ms3DbbgD9zD+Q1iFDRz/CuZgwxt4Dx+UglJVjhQ1QmdnYs1NGLuUV/8A8L2SzA==</latexit>l Infinite Gaussian mixture in latent spaceà Map to observation through Gaussian process
l d’th dimension of observation Y distributes
according to GP on X: Yd ⇠ GP(µ, KX )<latexit sha1_base64="0wQNDylLUmvjz1qx3x5EqYX9jIc=">AAAH0HichZW/b9NAFMdfWyCl/GgKCxJLRFRUJFRdAhKIqQUqWqWq0oS2qeoq8rmXxoodW7aTNLEixNqBlYEJJAbEygoLC/8AQ/8ExFgkFgaez47ywy+trcR3732+z+/dne+4beiux9jJxOTUhYuXEtOXZ65cvXZ9Njl3Y9u1Go4mtjTLsJwSV11h6HWx5emeIUq2I1STG2KH154F/p2mcFzdqr/02rbYN9XDul7RNdVDUzm5sFs+SCmubqZ8xTFTL/LdBYWbvmI2uvdzZcVUvSqv+KXuvXIyzRaZvFLxRiZqpCG68tbc1CoocAAWaNAAEwTUwcO2ASq4eO9BBhjYaNsHH20OtnTpF9CFGdQ2kBJIqGit4f8h9vYiax37QUxXqjV8i4E/B5UpmGc/2Sd2yn6wz+wX+zc2li9jBLm08clDrbDLs8e3in/PVZn49KDaV52h4EifXZMHFXgsa9GxNltagiq1MH6z8/a0+KQw799lH9hvrO89O2HfscJ684/2cVMU3mH0eTneAiMp0MSnBmkcYx97vVw5+nxp7Ub59PlhhSX7Dt465lhFtSp7FrTGRjgi3nVEcG2CaxNch+A6BNciuBbBVQiuQnA1gqsR3DLBLRPcGsGtEVyO4HIEVyK4EsHtEtwuwTGCYzGOR/MbrGVqXnk0r6E/Pp9cfgk9vyD8Jq76PqHIfpyyMUd9iAstFOnGSJckg9yCFT7I9mxxOhilHj+o6Nu7hMqQ+wZHy6hu0EMp87GK82MqLsqv1RyppG+NK9aHsupr1kdyGlRtRPlr6DNwd9qIESsjK4pjfyVGFQiqEKOey90mDVlJV3APCvbOcDZ7p0aYR7gr0Z5sEBfPsMzoiRVvbGcXMw8Ws5sP00tPo9NsGm7DHVjA+I9gCVZx/LcwhzfwBb7Ct0QhcZR4lXgdopMTkeYmDF2J4/8EH56b</latexit>Neural network ＝ Gaussian processl Single hidden-layer NN with input layer x<latexit sha1_base64="05i1Qsam0Fm9F/5SK3jUPFG0cuo=">AAAHp3ichZXNbtNAEMenLeBSPtrCBYlLRRQEl2gTkECcWqBSq1YlHyRN1VSV12xaK45t2c638gJwBXHgBBIHxGNw4QU49BEQxyJx4cB47chJdlpsJd6d+f3HM7vrXe5aph8wdjIzO3fh4iVt/vLClavXri8uLd+o+E7LM0TZcCzHq3LdF5Zpi3JgBpaoup7Qm9wSu7zxLPTvtoXnm479Mui54qCpH9lm3TT0AE2F7uFSimWYvFbURjZupCC+8s7y3AbU4BU4YEALmiDAhgDbFujg470PWWDgou0ABmjzsGVKv4AhLKC2hZRAQkdrA/+PsLcfW23shzF9qTbwLRb+PFSuQJr9YF/YKfvOvrKf7O+ZsQYyRphLD5880gr3cPH1rdKf/6qa+AzgOFGdo+BIn19TAHV4LGsxsTZXWsIqjSh+u//+tPSkmB7cZZ/YL6zvIzth37BCu/3b+FwQxQ8YPS3HW2CkGrTxaUAKx3iAvVGuHH0DaR3G+ST8pMKRfQ9vE3M8RrUuew50zozQJd7VJbgewfUIrk9wfYLrEFyH4OoEVye4BsE1CG6N4NYIbpPgNglui+C2CK5KcFWC2yO4PYJjBMcUjsfzG65lal55PK+RX51PLr+EkV8Q/iau+oSoyb5KuZijOcFFFor0FdInyTC3cIWPsyObSoejNOLHFYl9SKgsuW9wtEzrxj2UMq9UnD+j4pL8WptTlSRWVbE9kVWi2Z7KaVy1E+dvoM/C3WlHIdanVhTH/rpCFQmqqFDP5W6Tgpyk67gHhXtnNJujUyPKI9qVaE8ujItnWHb6xFIblVwm+yCTKzxMrT6NT7N5uA134B7GfwSrsIHjX8YcBLyBt/BOu6+90CpaNUJnZ2LNTZi4NP0f102Oyw==</latexit>

1z
l i’th output i (x) is written as :<latexit sha1_base64="gMVPXwKsUrY/Py15IDMa+Rz/xkI=">AAAHrnichZVLb9NAEMenLeBSHm3hgsSlIgoql2gTkECcWqBSq1ZVmpA0VRMir7tprPgl23krX4AjFw7lAhIHxMfgwhfg0I+AOBaJCwfGa0d5eFJsJd6d+f3HM7vrXe4Yuuczdj43v3Dl6jVl8frSjZu3bi+vrN4penbT1URBsw3bLXHVE4ZuiYKv+4YoOa5QTW6IQ954GfgPW8L1dNt67XcdUTHVU0uv6Zrqo6nUe5Ou6uudR9WVBEsxea3FG+mokYDoytqrC9tQhhOwQYMmmCDAAh/bBqjg4X0MaWDgoK0CfbS52NKlX8AAllDbREogoaK1gf+n2DuOrBb2g5ieVGv4FgN/LirXIMl+sC/sgn1nX9lP9ndmrL6MEeTSxScPtcKpLr+9l//zX5WJTx/qI9UlCo705TX5UINnshYda3OkJahSC+O3eu8v8s9zyf5D9on9wvo+snP2DSu0Wr+1zwcid4bRk3K8BUYqQwufGiRwjPvYG+bK0deX1kGUz4ifVNiy7+KtY451VKuyZ0N7ZoQO8a4OwXUJrktwPYLrEVyb4NoEVyO4GsE1CK5BcJsEt0lwOwS3Q3C7BLdLcCWCKxHcEcEdERwjOBbjeDS/wVqm5pVH8xr64/PJ5Zcw9AvCb+KqHxFl2Y9TDuaoT3ChhSK9GOmRZJBbsMLH2aEtTgejNOTHFSP7gFAZct/gaJnWjXsoZTZWcXZGxXn5tZpTlYysccXeRFYjzd5UTuOq/Sh/DX0G7k77MWJrakVx7G/FqBxB5WLUK7nbJCAj6RruQcHeGc7m8NQI8wh3JdqTCeLiGZaePrHijWImlX6cyhw8SWy8iE6zRbgPD2Ad4z+FDdjG8S/Ik+odnMEHhSlFpaJUQ3R+LtLchYlLqf8Dic+RMw==</latexit>

zi1 (x) = b1i +N1X<latexit sha1_base64="be+5Foc0l/Cr4ohUtAHWc5/6x0A=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfN1Vl9JsQyT11qykY0aKYiugr26sAU1eAM2aNAGEwRY4GPbABU8vA8hCwwctB3BAG0utnTpFzCEJdS2kRJIqGht4X8Te4eR1cJ+ENOTag3fYuDPReUapNkP9oWds+/sK/vJ/s6MNZAxglx6+OShVjj15bd3yn/+qzLx6cNxrLpAwZG+uCYfGvBU1qJjbY60BFVqYfxO//S8/KyUHtxnn9gvrO8jO2PfsEKr81v7XBSlDxg9LcdbYKQadPCpQQrHeIC9Ua4cfQNpHUb5xPykwpZ9F28dczxGtSp7NnRnRjgh3nVCcD2C6xFcn+D6BNcluC7BNQiuQXAtgmsR3AbBbRDcNsFtE9wOwe0QXJXgqgR3QHAHBMcIjiU4Hs1vsJapeeXRvIb+5Hxy+SWM/ILwm7jqY6Im+0nKwRz1CS60UKSXID2SDHILVvg4O7Il6WCURvy4IrYPCZUh9w2OlmnduIdSFhIVF2ZUXJZfqzlVSWxNKnYnsoo1u1M5javyUf4a+gzcnfIJYnNqRXHsbyaoEkGVEtRLudukICfpBu5Bwd4Zzubo1AjzCHcl2pML4uIZlp0+sZKNvVwm+yiTKz5OrT+PTrNFuAv34AHGfwLrsIXjX8EcmvAO3sOp8lApKlXldYjOz0Wa2zBxKdo/CzuPRA==</latexit>

Wij1 x1j (x), x1j (x) =

b0j +

j=1

<latexit sha1_base64="LMYTwKSxLTup9cVMeObg85dJ7Rc=">AAAIIXichZXNbtNAEMcnlJISoB9wQeJiERW1AlWbgARCqtQClVq1qtqUtqma1orNpnFix5btpEks9wF4AQ6cAHFAPAYXuCEkDhUHzohjkbhwYLx2yIenxVbi3Znffzyzu95VLF1zXMaOE+eGzg9fSI5cTF26fGV0bHzi6pZj1m2Vb6qmbtp5pehwXavxTVdzdZ63bF40FJ1vK9XHgX+7wW1HM2tP3ZbF94ziQU0raWrRRZM83mrvZ2RtqjktzUpK0JRuSwWnbsheZTbj73urcsaXttHhaRW/ic8Ksneko6NUp4PCglXWpILOS+5UStlncuVfkGoQZBkDMAxY9aWmXE0VbO2g7E7L42k2w8QlxRuZqJGG6FozJ4YWoQDPwAQV6mAAhxq42NahCA7eu5ABBhba9sBDm40tTfg5+JBCbR0pjkQRrVX8P8DebmStYT+I6Qi1im/R8WejUoJJ9pW9YyfsI3vPfrA/p8byRIwglxY+lVDLLXns+fWN3/9VGfh0odxVnaFQkD67JhdK8EDUomFtlrAEVaph/Eb7xcnGw9ykd4u9Zj+xvlfsmH3ACmuNX+rbdZ57idEnxXhzjFSABj5VSOMYe9jr5KqgzxNWP8qny/crTNG38dYwxzKqi6JnwuGpEZrEu5oE1yK4FsG1Ca5NcIcEd0hwJYIrEVyV4KoEN09w8wS3RHBLBLdMcMsElye4PMHtENwOwTGCYzFOieY3WMvUvCrRvIb++Hwq4kvo+DnhN3DVd4mC6McpC3PU+rjQQpFOjHRIMsgtWOG9bMcWp4NR6vC9iq7dJ1S62DcUtAzqej2Uci1W8dopFW+Ir9UYqKRrjStW+rLqalYGcupVrUb5q+jTcXdajRELAytKwf5CjMoRVC5GPRG7TRqygi7hHhTsneFsdk6NMI9wV6I92SAunmGZwRMr3tjKzmTuzmTX76XnHkWn2QjcgJswhfHvwxws4vhvYg7fE8OJ0cRY8k3yU/Jz8kuInktEmmvQdyW//QWPvLeb</latexit>

z1

N2<latexit sha1_base64="G6XmFjgUM/Lat9F/OGdVLLxN0U0=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfP1XH0lxTJMXmvJRjZqpCC6CvbqwhbU4A3YoEEbTBBggY9tA1Tw8D6ELDBw0HYEA7S52NKlX8AQllDbRkogoaK1hf9N7B1GVgv7QUxPqjV8i4E/F5VrkGY/2Bd2zr6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eKf/5r8rEpw/HseoCBUf64pp8aMBTWYuOtTnSElSphfE7/dPz8rNSenCffWK/sL6P7Ix9wwqtzm/tc1GUPmD0tBxvgZFq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zPEY1ars2dCdGeGEeNcJwfUIrkdwfYLrE1yX4LoE1yC4BsG1CK5FcBsEt0Fw2wS3TXA7BLdDcFWCqxLcAcEdEBwjOJbgeDS/wVqm5pVH8xr6k/PJ5Zcw8gvCb+Kqj4ma7CcpB3PUJ7jQQpFegvRIMsgtWOHj7MiWpINRGvHjitg+JFSG3Dc4WqZ14x5KWUhUXJhRcVl+reZUJbE1qdidyCrW7E7lNK7KR/lr6DNwd8oniM2pFcWxv5mgSgRVSlAv5W6TgpykG7gHBXtnOJujUyPMI9yVaE8uiItnWHb6xEo29nKZ7KNMrvg4tf48Os0W4S7cgwcY/wmswxaOfwVzaMI7eA+nykOlqFSV1yE6PxdpbsPEpWj/ABnBj0Y=</latexit><latexit sha1_base64="BMNGLV1GIjQKiMie4pR3ONbwDZc=">AAAHqXichZVLb9NAEMenLeBSHm3hgsSlIgpCQoo2AQnEqQUqtWpV5UHalL7kNZvUil+ynaRJlC+AxJUeOIHEAfExuPAFOPQjII5F4sKB8dpRHjspthLvzvz+45nd9S73LDMIGTubmp65dPmKNnt17tr1GzfnFxZvbQduwzdE2XAt169wPRCW6YhyaIaWqHi+0G1uiR1efxH5d5rCD0zXeRW2PXFg6zXHrJqGHqKp1DnMHi2kWIbJa0ltZJNGCpIr7y7OrME+vAEXDGiADQIcCLFtgQ4B3nuQBQYe2g6gizYfW6b0C+jBHGobSAkkdLTW8b+Gvb3E6mA/ihlItYFvsfDno3IJ0uwH+8LO2Xf2lf1kfyfG6soYUS5tfPJYK7yj+bd3Sn/+q7LxGcLxQHWBgiN9cU0hVOGprMXE2jxpiao04vjNzul56Vkx3b3PPrFfWN9Hdsa+YYVO87fxuSCKHzB6Wo63wEj70MSnASkc4y72+rly9HWltZfkM+BHFa7s+3ibmOMxqnXZc6E1McIJ8a4TgmsTXJvgOgTXIbgWwbUIrkpwVYKrE1yd4FYIboXg1gluneA2CG6D4CoEVyG4XYLbJThGcEzheDK/0Vqm5pUn8xr71fnk8kvo+wXht3HVD4h92VcpD3M0R7jYQpGBQgYkGeUWrfBhtm9T6WiU+vywYmDvESpL7hscLeO6YQ+lzCsV5ydUXJJfqz1WycCqKjZHshpoNsdyGlZtJfkb6LNwd9pSiNWxFcWxv6pQRYIqKtRLudukICfpKu5B0d4Zz2b/1IjziHcl2pOL4uIZlh0/sdTGdi6TfZTJFR6nlp8np9ks3IV78ADjP4FlWMPxL2MONXgH7+FUe6gVtIr2OkanpxLNbRi5NOMfSyWPcA==</latexit><latexit sha1_base64="R8ZqzoF4SDVnDSmgi4CIMGySDBw=">AAAHuHichZXNbtNAEMenLeBSPprCBYlLIAriVG0CEqinFqjUqlWVJqRN1ZTgNevUimNbtpM0ifICvAAHuIBUJMRjcOEFOPQREMciceHAeO3ISTwpthLvzvz+45nd9S53TMPzGTubmZ27dPmKMn914dr1GzcXU0u39jy75WqirNmm7Va46gnTsETZN3xTVBxXqE1uin3eeB7499vC9Qzbeul3HXHUVOuWoRua6qOplkpVTaH7quvaneq9NH+Vq6UybJnJK51s5KJGBqKrYC/NbUAV3oANGrSgCQIs8LFtggoe3oeQAwYO2o6gjzYXW4b0CxjAAmpbSAkkVLQ28L+OvcPIamE/iOlJtYZvMfHnojINWfaDfWHn7Dv7yn6yv1Nj9WWMIJcuPnmoFU5t8e2d0p//qpr49OE4Vl2g4EhfXJMPOjyVtRhYmyMtQZVaGL/de3deWilm+w/YJ/YL6/vIztg3rNBq/9ZOd0XxPUbPyvEWGKkKbXxqkMEx7mNvmCtHX19aB1E+MT+usGXfxdvAHI9RrcqeDZ2pEU6Id50QXJfgugTXI7gewXUIrkNwOsHpBNcguAbBrRHcGsFtEtwmwW0R3BbBVQiuQnAHBHdAcIzgWILj0fwGa5maVx7Na+hPzieXX8LQLwh/E1d9TFRlP0k5mKMxxoUWivQSpEeSQW7BCh9lh7YkHYzSkB9VxPYBoTLlvsHRMqkb9VDKQqLiwpSKS/JrbU5UEluTiu2xrGLN9kROo6qdKH8NfSbuTjsJYn1iRXHsryeoIkEVE9QLudtkIC9pHfegYO8MZ3N4aoR5hLsS7ckHcfEMy02eWMnGXn4592g5v/s4s/osOs3m4S7ch4cY/wmswgaOfxlzaMMHOIXPyoryWqkrRojOzkSa2zB2Ke4/DTCU2g==</latexit>

b1<latexit sha1_base64="QXvz4mzIdmqYGLWxMRKRtVXlQuE=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqklUCcWqBSq1ZVmpImpSmV16wbK/6S7SRNorwADwAHDnxIHBCPwYUX4NBHQByLxIUD47UjJ/Gk2Eq8O/P7j2d217vcNQ0/YOxsanrm0uUryuzVuWvXb9ycX1i8te87TU8TZc0xHa/KVV+Yhi3KgRGYoup6QrW4KSq88TT0V1rC8w3Hfh50XHFkqSe2oRuaGqCpVrPUoM71XqX/Mn+8kGE5Jq+ldCMfNzIQX0VncWYDavAKHNCgCRYIsCHAtgkq+HgfQh4YuGg7gh7aPGwZ0i+gD3OobSIlkFDR2sD/E+wdxlYb+2FMX6o1fIuJPw+VS5BlP9gXds6+s6/sJ/s7MVZPxghz6eCTR1rhHs+/vrP3578qC58B1BPVBQqO9MU1BaDDI1mLgbW50hJWqUXxW92353uPS9neffaJ/cL6PrIz9g0rtFu/tc+7ovQOo2fleAuMVIMWPjXI4Bj3sDfIlaOvJ639OJ+EH1U4su/hbWCOdVSrsudAe2KEU+JdpwTXIbgOwXUJrktwbYJrE5xOcDrBNQiuQXBrBLdGcJsEt0lwWwS3RXBVgqsS3AHBHRAcIziW4ng8v+FapuaVx/Ma+dPzyeWXMPALwm/hqk+ImuynKRdzNEa4yEKRfor0STLMLVzhw+zAlqbDURrww4rE3idUptw3OFrGdcMeSllMVVycUPGe/FqtsUoSa1qxPZJVotkey2lYtRPnr6HPxN1pJ0Wsj60ojv31FFUiqFKKeiZ3mwwUJK3jHhTundFsDk6NKI9oV6I9hTAunmH58RMr3dgv5PLLucLuSmb1SXyazcJduAcPMP5DWIUNHP8y5uDCG3gPH5QV5YWiKlqETk/FmtswcinmP07xk20=</latexit>

x1<latexit sha1_base64="EvynLVwq2k3n8W/w6hPV9wqJLEA=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfP1bH0lxTJMXmvJRjZqpCC6CvbqwhbU4A3YoEEbTBBggY9tA1Tw8D6ELDBw0HYEA7S52NKlX8AQllDbRkogoaK1hf9N7B1GVgv7QUxPqjV8i4E/F5VrkGY/2Bd2zr6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eKf/5r8rEpw/HseoCBUf64pp8aMBTWYuOtTnSElSphfE7/dPz8rNSenCffWK/sL6P7Ix9wwqtzm/tc1GUPmD0tBxvgZFq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zPEY1ars2dCdGeGEeNcJwfUIrkdwfYLrE1yX4LoE1yC4BsG1CK5FcBsEt0Fw2wS3TXA7BLdDcFWCqxLcAcEdEBwjOJbgeDS/wVqm5pVH8xr6k/PJ5Zcw8gvCb+Kqj4ma7CcpB3PUJ7jQQpFegvRIMsgtWOHj7MiWpINRGvHjitg+JFSG3Dc4WqZ14x5KWUhUXJhRcVl+reZUJbE1qdidyCrW7E7lNK7KR/lr6DNwd8oniM2pFcWxv5mgSgRVSlAv5W6TgpykG7gHBXtnOJujUyPMI9yVaE8uiItnWHb6xEo29nKZ7KNMrvg4tf48Os0W4S7cgwcY/wmswxaOfwVzaMI7eA+nykOlqFSV1yE6PxdpbsPEpWj/ABJ+j0U=</latexit><latexit sha1_base64="Gbam75vABvGioZzco2NfSg9FmmE=">AAAHqXichZVLb9NAEMenLeBSHm3hgsSlIgpCQoo2AQnEqQUqtWpV5UHalKatvGaTWvFLtvNWvgASV3rgBBIHxMfgwhfg0I+AOBaJCwfGa0dO4kmxlXh35vcfz+yud7lj6J7P2NnM7Nyly1eU+asL167fuLm4tHxr17ObribKmm3YboWrnjB0S5R93TdExXGFanJD7PHGi8C/1xKup9vWK7/riENTrVt6TddUH02lzlH2eCnFMkxeK8lGNmqkILry9vLcBlThDdigQRNMEGCBj20DVPDwPoAsMHDQdgh9tLnY0qVfwAAWUNtESiChorWB/3XsHURWC/tBTE+qNXyLgT8XlSuQZj/YF3bOvrOv7Cf7OzVWX8YIcunik4da4Rwvvr1T+vNflYlPH05i1QUKjvTFNflQg6eyFh1rc6QlqFIL47d6p+elZ8V0/z77xH5hfR/ZGfuGFVqt39rngih+wOhpOd4CI1WhhU8NUjjGfewNc+Xo60vrIMon5scVtuy7eOuY4wmqVdmzoT01Qod4V4fgugTXJbgewfUIrk1wbYKrEVyN4BoE1yC4NYJbI7hNgtskuC2C2yK4CsFVCG6f4PYJjhEcS3A8mt9gLVPzyqN5Df3J+eTySxj6BeE3cdXHRFX2k5SDOepjXGihSC9BeiQZ5Bas8FF2aEvSwSgN+VFFbB8QKkPuGxwtk7pRD6XMJyrOT6m4JL9Wc6KS2JpUbI9lFWu2J3IaVe1E+WvoM3B32kkQ6xMrimN/PUEVCaqYoF7K3SYFOUnXcA8K9s5wNoenRphHuCvRnlwQF8+w7OSJlWzs5jLZR5lc4XFq9Xl0ms3DXbgHDzD+E1iFDRz/MuZQh3fwHk6Vh0pBqSivQ3R2JtLchrFL0f4BPJuPbg==</latexit>

W0<latexit sha1_base64="nnbc5hsCBtXrsdTmvR4Ug0hlEGg=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqklUCcWqBSq1ZVmpImpSmV16wbK/6S7SRNorwADwAHDnxIHBCPwYUX4NBHQByLxIUD47UjJ/Gk2Eq8O/P7j2d217vcNQ0/YOxsanrm0uUryuzVuWvXb9ycX1i8te87TU8TZc0xHa/KVV+Yhi3KgRGYoup6QrW4KSq88TT0V1rC8w3Hfh50XHFkqSe2oRuaGqCpVrPUoM71XqX/kh0vZFiOyWsp3cjHjQzEV9FZnNmAGrwCBzRoggUCbAiwbYIKPt6HkAcGLtqOoIc2D1uG9Avowxxqm0gJJFS0NvD/BHuHsdXGfhjTl2oN32Liz0PlEmTZD/aFnbPv7Cv7yf5OjNWTMcJcOvjkkVa4x/Ov7+z9+a/KwmcA9UR1gYIjfXFNAejwSNZiYG2utIRValH8Vvft+d7jUrZ3n31iv7C+j+yMfcMK7dZv7fOuKL3D6Fk53gIj1aCFTw0yOMY97A1y5ejrSWs/zifhRxWO7Ht4G5hjHdWq7DnQnhjhlHjXKcF1CK5DcF2C6xJcm+DaBKcTnE5wDYJrENwawa0R3CbBbRLcFsFtEVyV4KoEd0BwBwTHCI6lOB7Pb7iWqXnl8bxG/vR8cvklDPyC8Fu46hOiJvtpysUcjREuslCknyJ9kgxzC1f4MDuwpelwlAb8sCKx9wmVKfcNjpZx3bCHUhZTFRcnVLwnv1ZrrJLEmlZsj2SVaLbHchpW7cT5a+gzcXfaSRHrYyuKY389RZUIqpSinsndJgMFSeu4B4V7ZzSbg1MjyiPalWhPIYyLZ1h+/MRKN/YLufxyrrC7kll9Ep9ms3AX7sEDjP8QVmEDx7+MObjwBt7DB2VFeaGoihah01Ox5jaMXIr5D0euk2w=</latexit>

N0<latexit sha1_base64="be+5Foc0l/Cr4ohUtAHWc5/6x0A=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfN1Vl9JsQyT11qykY0aKYiugr26sAU1eAM2aNAGEwRY4GPbABU8vA8hCwwctB3BAG0utnTpFzCEJdS2kRJIqGht4X8Te4eR1cJ+ENOTag3fYuDPReUapNkP9oWds+/sK/vJ/s6MNZAxglx6+OShVjj15bd3yn/+qzLx6cNxrLpAwZG+uCYfGvBU1qJjbY60BFVqYfxO//S8/KyUHtxnn9gvrO8jO2PfsEKr81v7XBSlDxg9LcdbYKQadPCpQQrHeIC9Ua4cfQNpHUb5xPykwpZ9F28dczxGtSp7NnRnRjgh3nVCcD2C6xFcn+D6BNcluC7BNQiuQXAtgmsR3AbBbRDcNsFtE9wOwe0QXJXgqgR3QHAHBMcIjiU4Hs1vsJapeeXRvIb+5Hxy+SWM/ILwm7jqY6Im+0nKwRz1CS60UKSXID2SDHILVvg4O7Il6WCURvy4IrYPCZUh9w2OlmnduIdSFhIVF2ZUXJZfqzlVSWxNKnYnsoo1u1M5javyUf4a+gzcnfIJYnNqRXHsbyaoEkGVEtRLudukICfpBu5Bwd4Zzubo1AjzCHcl2pML4uIZlp0+sZKNvVwm+yiTKz5OrT+PTrNFuAv34AHGfwLrsIXjX8EcmvAO3sOp8lApKlXldYjOz0Wa2zBxKdo/CzuPRA==</latexit>x<latexit sha1_base64="05i1Qsam0Fm9F/5SK3jUPFG0cuo=">AAAHp3ichZXNbtNAEMenLeBSPtrCBYlLRRQEl2gTkECcWqBSq1YlHyRN1VSV12xaK45t2c638gJwBXHgBBIHxGNw4QU49BEQxyJx4cB47chJdlpsJd6d+f3HM7vrXe5aph8wdjIzO3fh4iVt/vLClavXri8uLd+o+E7LM0TZcCzHq3LdF5Zpi3JgBpaoup7Qm9wSu7zxLPTvtoXnm479Mui54qCpH9lm3TT0AE2F7uFSimWYvFbURjZupCC+8s7y3AbU4BU4YEALmiDAhgDbFujg470PWWDgou0ABmjzsGVKv4AhLKC2hZRAQkdrA/+PsLcfW23shzF9qTbwLRb+PFSuQJr9YF/YKfvOvrKf7O+ZsQYyRphLD5880gr3cPH1rdKf/6qa+AzgOFGdo+BIn19TAHV4LGsxsTZXWsIqjSh+u//+tPSkmB7cZZ/YL6zvIzth37BCu/3b+FwQxQ8YPS3HW2CkGrTxaUAKx3iAvVGuHH0DaR3G+ST8pMKRfQ9vE3M8RrUuew50zozQJd7VJbgewfUIrk9wfYLrEFyH4OoEVye4BsE1CG6N4NYIbpPgNglui+C2CK5KcFWC2yO4PYJjBMcUjsfzG65lal55PK+RX51PLr+EkV8Q/iau+oSoyb5KuZijOcFFFor0FdInyTC3cIWPsyObSoejNOLHFYl9SKgsuW9wtEzrxj2UMq9UnD+j4pL8WptTlSRWVbE9kVWi2Z7KaVy1E+dvoM/C3WlHIdanVhTH/rpCFQmqqFDP5W6Tgpyk67gHhXtnNJujUyPKI9qVaE8ujItnWHb6xFIblVwm+yCTKzxMrT6NT7N5uA134B7GfwSrsIHjX8YcBLyBt/BOu6+90CpaNUJnZ2LNTZi4NP0f102Oyw==</latexit>

0
Wjk
xk

k=1

W1
N1

N
K0
X

<latexit sha1_base64="pnBwoIFlxOZzvBwgvbEjhepf8Hc=">AAAHuHichZXNbtNAEMenLeBSPprCBYlLIAriVG0CEqinFqjUqlWVJqRN1ZTgNevUimNbtpM0ifICvAAHuIBUJMRjcOEFOPQREMciceHAeO3ISTwpthLvzvz+45nd9S53TMPzGTubmZ27dPmKMn914dr1GzcXU0u39jy75WqirNmm7Va46gnTsETZN3xTVBxXqE1uin3eeB7499vC9Qzbeul3HXHUVOuWoRua6qOplkpVTaH7quvaneq9NH/FaqkMW2bySicbuaiRgegq2EtzG1CFN2CDBi1oggALfGyboIKH9yHkgIGDtiPoo83FliH9AgawgNoWUgIJFa0N/K9j7zCyWtgPYnpSreFbTPy5qExDlv1gX9g5+86+sp/s79RYfRkjyKWLTx5qhVNbfHun9Oe/qiY+fTiOVRcoONIX1+SDDk9lLQbW5khLUKUWxm/33p2XVorZ/gP2if3C+j6yM/YNK7Tav7XTXVF8j9GzcrwFRqpCG58aZHCM+9gb5srR15fWQZRPzI8rbNl38TYwx2NUq7JnQ2dqhBPiXScE1yW4LsH1CK5HcB2C6xCcTnA6wTUIrkFwawS3RnCbBLdJcFsEt0VwFYKrENwBwR0QHCM4luB4NL/BWqbmlUfzGvqT88nllzD0C8LfxFUfE1XZT1IO5miMcaGFIr0E6ZFkkFuwwkfZoS1JB6M05EcVsX1AqEy5b3C0TOpGPZSykKi4MKXikvxamxOVxNakYnssq1izPZHTqGonyl9Dn4m7006CWJ9YURz76wmqSFDFBPVC7jYZyEtaxz0o2DvD2RyeGmEe4a5Ee/JBXDzDcpMnVrKxl1/OPVrO7z7OrD6LTrN5uAv34SHGfwKrsIHjX8Yc2vABTuGzsqK8VuqKEaKzM5HmNoxdivsPBe2U2Q==</latexit>

b

0

!

Connection
Wij` ⇠ N (0, w /N` )<latexit sha1_base64="k9z/fC3sUS8igmMXyuUyMOpPBzU=">AAAH2HichZXNbtNAEMenLZBSPprCBYmLRRRUJFQ2AQnEqQUqtWoVpQlpUpoS2WaTmtixZTtJEysSNwRHDnDgBBIHxCtwQyBegEMfAXEsEhcOjNeO8uFJsZV4d+b3H8/srncVS9ccl7HDqemZEydPxWZPz505e+78fHzhwrZjNm2VF1RTN+2SIjtc1xq84GquzkuWzWVD0XlRqd/3/cUWtx3NbDx0OxbfM+RaQ6tqquyiqRJnxYqnPe099spc13tS2dEMySursi5leovsOvZrhlxpSzekTMVHrlXiCbbExCVFG6mwkYDwypoLM2tQhidgggpNMIBDA1xs6yCDg/cupICBhbY98NBmY0sTfg49mENtEymOhIzWOv7XsLcbWhvY92M6Qq3iW3T82aiUIMl+sI/siH1nn9hP9ndiLE/E8HPp4FMJtNyqzL+8lP/zX5WBTxf2B6pjFArSx9fkQhXuiFo0rM0SFr9KNYjf6r45yt/NJb2r7D37hfW9Y4fsC1bYaP1WP2zx3FuMnhTjzTFSGVr4VCGBY+xhr5+rgj5PWHthPgN+VGGKvo23hjnuo1oWPRPaEyMcEO86ILgOwXUIrktwXYJrE1yb4KoEVyW4OsHVCW6F4FYIbp3g1glug+A2CK5EcCWC2yG4HYJjBMcinBLOr7+WqXlVwnkN/NH5VMSX0Pdzwm/gqh8QZdGPUhbmqI1wgYUinQjpkKSfm7/Ch9m+LUr7o9TnhxUDe49Q6WLfUNAyrhv2UMpspOLshIrz4ms1xioZWKOKzZGsBprNsZyGVZkwfxV9Ou5OmQixOraiFOyvRqgcQeUi1AOx2yQgLegq7kH+3hnMZv/UCPIIdiXak/bj4hmWGj+xoo3t9FLq5lJ661Zi+V54ms3CZbgCixj/NizDGo5/AXN4DZ/hK3yLPYo9iz2PvQjQ6alQcxFGrtirfwavoUY=</latexit>

Biasb`i ⇠ N (0,<latexit sha1_base64="YenEqkRjQZTRC/KLFgp+uDfhZ/o=">AAAHynichZXNbtNAEMenLZBSPprCBYlL1CioSKjaBCQQpxao1KpRlCakTdW0kddsUit2bNlO0sTKjQPiBThwAokDQuIKdy68AIc+AuJYJC4cGK8d5cOTYivx7szvP57ZXe9yS9ccl7HTmdm5CxcvxeYvL1y5eu36Ynzpxq5jtmxVlFRTN+0yVxyha01RcjVXF2XLForBdbHHG099/15b2I5mNp+7XUscGkq9qdU0VXHRVI0v86OK0PWqlqg4mpHwKqqiJ3L9FXYP+3VDqfK71XiSrTJ5JaKNdNhIQnjlzaW5TajACzBBhRYYIKAJLrZ1UMDB+wDSwMBC2yF4aLOxpUm/gD4soLaFlEBCQWsD/+vYOwitTez7MR2pVvEtOv5sVCYgxX6wj+yMfWef2E/2d2osT8bwc+nikwdaYVUXX98q/vmvysCnC8dD1TkKjvT5NblQg0eyFg1rs6TFr1IN4rd7b86Kjwsp7w57z35hfe/YKfuGFTbbv9UPO6LwFqOn5HgLjFSBNj5VSOIYe9gb5MrR50lrP8xnyI8rTNm38dYwx2NUK7JnQmdqhBPiXScE1yW4LsH1CK5HcB2C6xBcjeBqBNcguAbBrRPcOsFtEdwWwW0T3DbBlQmuTHD7BLdPcIzgWITj4fz6a5maVx7Oa+CPzieXX8LALwi/gat+SFRkP0pZmKM2xgUWinQipEOSfm7+Ch9lB7Yo7Y/SgB9VDO19QqXLfYOjZVI36qGU+UjF+SkVF+XXakxUMrRGFdmxrIaa7EROo6pcmL+KPh13p1yE2JhYURz7GxGqQFCFCPVM7jZJyEi6hnuQv3cGszk4NYI8gl2J9mT8uHiGpSdPrGhjN7Oavr+a2XmQXHsSnmbzcBuWYQXjP4Q12MTxL2EOr+AzfIGvsWzMjnVjXoDOzoSamzB2xV7+A8oIm6E=</latexit>b)

Neural network ＝ Gaussian process (2)zi1 (x) = b1i +N1
X

Wij1 x1j (x), x1j (x) =

b0j +

j=1

<latexit sha1_base64="LMYTwKSxLTup9cVMeObg85dJ7Rc=">AAAIIXichZXNbtNAEMcnlJISoB9wQeJiERW1AlWbgARCqtQClVq1qtqUtqma1orNpnFix5btpEks9wF4AQ6cAHFAPAYXuCEkDhUHzohjkbhwYLx2yIenxVbi3Znffzyzu95VLF1zXMaOE+eGzg9fSI5cTF26fGV0bHzi6pZj1m2Vb6qmbtp5pehwXavxTVdzdZ63bF40FJ1vK9XHgX+7wW1HM2tP3ZbF94ziQU0raWrRRZM83mrvZ2RtqjktzUpK0JRuSwWnbsheZTbj73urcsaXttHhaRW/ic8Ksneko6NUp4PCglXWpILOS+5UStlncuVfkGoQZBkDMAxY9aWmXE0VbO2g7E7L42k2w8QlxRuZqJGG6FozJ4YWoQDPwAQV6mAAhxq42NahCA7eu5ABBhba9sBDm40tTfg5+JBCbR0pjkQRrVX8P8DebmStYT+I6Qi1im/R8WejUoJJ9pW9YyfsI3vPfrA/p8byRIwglxY+lVDLLXns+fWN3/9VGfh0odxVnaFQkD67JhdK8EDUomFtlrAEVaph/Eb7xcnGw9ykd4u9Zj+xvlfsmH3ACmuNX+rbdZ57idEnxXhzjFSABj5VSOMYe9jr5KqgzxNWP8qny/crTNG38dYwxzKqi6JnwuGpEZrEu5oE1yK4FsG1Ca5NcIcEd0hwJYIrEVyV4KoEN09w8wS3RHBLBLdMcMsElye4PMHtENwOwTGCYzFOieY3WMvUvCrRvIb++Hwq4kvo+DnhN3DVd4mC6McpC3PU+rjQQpFOjHRIMsgtWOG9bMcWp4NR6vC9iq7dJ1S62DcUtAzqej2Uci1W8dopFW+Ir9UYqKRrjStW+rLqalYGcupVrUb5q+jTcXdajRELAytKwf5CjMoRVC5GPRG7TRqygi7hHhTsneFsdk6NMI9wV6I92SAunmGZwRMr3tjKzmTuzmTX76XnHkWn2QjcgJswhfHvwxws4vhvYg7fE8OJ0cRY8k3yU/Jz8kuInktEmmvQdyW//QWPvLeb</latexit>

K
X

0
Wjk
xk

k=1

`
l Sum of independent weights Wij
and biases b`i

!

1
(x))
à Joint probability p(z11 (x), z21 (x), · · · , zN
2
has a multivariate Gaussian distribution
by Central Limit Theorem … Gaussian process<latexit sha1_base64="RTLnOWpg/EhMklPO2JK2XL9+7Vk=">AAAHrnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJiRN1YTI624aK/6S7SRNorwARy4cygUkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613uGLrnM3Y+N79w5eo1ZfH60o2bt24vr6zeKXt229VESbMN261w1ROGbomSr/uGqDiuUE1uiAPeehn4DzrC9XTbeu33HFEz1RNLb+ia6qOpwt9UhWHU9fpKimWYvNaSjWzUSEF05e3VhS2owjHYoEEbTBBggY9tA1Tw8D6CLDBw0FaDAdpcbOnSL2AIS6htIyWQUNHawv8T7B1FVgv7QUxPqjV8i4E/F5VrkGY/2Bd2wb6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eK/75r8rEpw/NWHWJgiN9eU0+NOCZrEXH2hxpCarUwvid/vuL4vNCevCQfWK/sL6P7Jx9wwqtzm/t874onGH0tBxvgZGq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zLGJalX2bOjOjHBKvOuU4HoE1yO4PsH1Ca5LcF2CaxBcg+BaBNciuA2C2yC4bYLbJrgdgtshuArBVQjukOAOCY4RHEtwPJrfYC1T88qjeQ39yfnk8ksY+QXhN3HVx0RV9pOUgznqE1xooUgvQXokGeQWrPBxdmRL0sEojfhxRWwfEipD7hscLdO6cQ+lzCcqzs+ouCi/VnOqktiaVOxOZBVrdqdyGlftRflr6DNwd9pLEJtTK4pjfzNBFQiqkKBeyd0mBTlJN3APCvbOcDZHp0aYR7gr0Z5cEBfPsOz0iZVslHOZ7ONMbv9Jav1FdJotwn14AI8w/lNYhy0c/5I8qd7BGXxQmFJWako9ROfnIs1dmLiU5j9fH5G6</latexit><latexit sha1_base64="QZeFkeb+TOw6EdD5Qdv1djkx3bs=">AAAHsXichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqkSCBOLVCpVasqTUiTKCnBazatiWNbtpM0sfIC3BEHJBBIHBCPwYUX4NBHQByLxIUD47UjJ/G02Eq8O/P7j2d217vcNnTXY+xkZnbuwsVLyvzlhStXr11fXFq+se9aHUcTJc0yLKfCVVcYuilKnu4ZomI7Qm1zQ5R560ngL3eF4+qW+czr2+KgrR6aelPXVA9NtfLzujCMhq+/GjaWUizD5LWSbGSjRgqiK28tz21CHV6CBRp0oA0CTPCwbYAKLt41yAIDG20H4KPNwZYu/QKGsIDaDlICCRWtLfw/xF4tsprYD2K6Uq3hWwz8OahcgTT7wb6wU/adfWU/2d8zY/kyRpBLH5881Aq7sfj6VvHPf1VtfHpwFKvOUXCkz6/JgyY8lLXoWJstLUGVWhi/O3h7WnxUSPt32Sf2C+v7yE7YN6zQ7P7WPu+JwjuMnpbjLTBSHbr41CCFY+xjb5QrR58vrcMon5ifVFiy7+CtY45HqFZlz4LemRGOiXcdE1yf4PoENyC4AcH1CK5HcE2CaxJci+BaBLdOcOsEt0VwWwS3TXDbBFchuArBVQmuSnCM4FiC49H8BmuZmlcezWvoT84nl1/CyC8IfxtXfUzUZT9J2ZijPsGFFop0E6RLkkFuwQofZ0e2JB2M0ogfV8T2IaEy5L7B0TKtG/dQynyi4vwZFRfl19qeqiS2JhU7E1nFmp2pnMZVu1H+GvoM3J12E8TG1Iri2N9IUAWCKiSop3K3SUFO0k3cg4K9M5zN0akR5hHuSrQnF8TFMyw7fWIlG/u5THY1k9u7n1p7HJ1m83Ab7sA9jP8A1mATx78kz5w38B4+KKtKVXmh8BCdnYk0N2HiUlr/AKT6ky8=</latexit><latexit sha1_base64="SAPDs7BXEVQdJa5ckIkKTIutV50=">AAAH0HichZW/b9NAFMdfW8Cl/GgKCxJLRRSUSqg6ByQQUwtUatUqShPSpmpKZLuX1ooTW7bzWxFi7cDKwAQSA2JlhYWFf4ChfwJiLBILA89nR/lxL8VW4rv3Pt/n9+7Od7pjmZ7P2OnU9MyFi5eU2ctzV65euz4fW7ix49l11+B5w7Zst6BrHrfMGs/7pm/xguNyrapbfFevPA38uw3ueqZde+63HX5Q1Y5qZtk0NB9NpVjSSXZeqCU12Vq6FzRSQaNoHNq+J/rddCnVQ9tSKRZny0xci3JDjRpxiK6MvTCzDkU4BBsMqEMVONTAx7YFGnh474MKDBy0HUAXbS62TOHn0IM51NaR4khoaK3g/xH29iNrDftBTE+oDXyLhT8XlYuQYD/YR3bGvrNP7Cf7OzFWV8QIcmnjUw+13CnNn9zK/fmvqopPH44HqnMUOtLn1+RDGR6JWkyszRGWoEojjN/ovDnLPc4munfZe/YL63vHTtk3rLDW+G182ObZtxg9IcabY6QiNPBpQBzHuIu9fq46+rrC2ovyGfCjClv0XbxNzPEY1Zro2dCcGKFFvKtFcG2CaxNch+A6BNckuCbBlQmuTHAVgqsQ3CrBrRLcBsFtENwmwW0SXIHgCgS3R3B7BMcIjkmcHs1vsJapedWjeQ398nzq4kvo+znhr+KqHxBF0ZcpB3M0R7jQQpGeRHokGeQWrPBhtm+T6WCU+vywYmDvESpL7Bs6WsZ1wx5KmZEqzkyoOCe+1upYJQOrrNgayWqg2RrLaViVjvI30Gfh7pSWiLWxFaVjf02isgSVlahnYreJQ0rQZdyDgr0znM3+qRHmEe5KtCcVxMUzTB0/seTGTmpZvb+c2n4QX3kSnWazcBvuQBLjP4QVWMfxz2MOr+EzfIGvSlZpKS+VVyE6PRVpbsLIpZz8Ay6NnJU=</latexit>– Mean: clearly 0– Covariance is
K 1 (x, x0 ) ⌘ E [zi1 (x)zi1 (x0 )]=<latexit sha1_base64="5icDyexRZernUEjoOYXqCCw41x0=">AAAIOnichZXPTxNREMenoBbrD0AvJiamsQFKJOS1mmhMTEAkgUAIFIESFpp96ytsuttdu9vf6T/g0YsHT5p4MP4L3rz4D3jg6o1whEQPHpx9u7U/doq7affNzOc7O/Pe7ltuG7rjMnYcGRq+dPlKdORq7Nr1GzdHx8ZvbTtWuaSJLc0yrFKWq44w9KLYcnXXEFm7JFSTG2KHFxa8+E5FlBzdKr5067bYN9XDop7XNdVFV27s7cpBKlmbqU1NxycV8bqsV2KKqbpHnDcXW8rMXuMgldOTtengOjW9H1eU2OSzuOLoh6aa4wfp+IO2UUWjR10L1LV/6lh8sHTBLyQ3lmCzTB7x8CAVDBIQHOvW+PASKPAKLNCgDCYIKIKLYwNUcPDcgxQwsNG3D030lXCky7iAFsRQW0ZKIKGit4D/h2jtBd4i2l5OR6o1vIuBvxIq4zDBfrDP7Ix9Z1/YCfszMFdT5vBqqeOV+1ph50bf3Nn89V+ViVcXjjqqCxQc6Yt7ciEPT2QvOvZmS4/XpebnrzTenW0+zUw0J9lHdor9fWDH7Bt2WKyca582ROY9Zp+Q8y0wkwIVvGqQwDluotWulWOsKb2toJ4O36uwpF3CU8caj1CtSsuC6sAMNeJeNYKrE1yd4BoE1yC4KsFVCS5PcHmCKxBcgeDmCW6e4JYJbpngVghuheCyBJcluF2C2yU4RnAsxPFgfb1nmVpXHqyrHw+vJ5dvQjsuiLiJT32HUKQdpmysUe/hfA9FOiHSIUmvNu8J72bbvjDtzVKb71Z0/C1CZch9g6OnX9cdoZTroY7XB3S8Kd9Ws6+TjjesWO2pqqNZ7aupW7UW1K9hzMDdaS1ELPY9URztxRCVIahMiHohd5sEpCWdxz3I2zv91Wx/Nfw6/F2JjqS9vPgNS/V/scKD7fRs6uFseuNRYu558DUbgbtwH5KY/zHMwRLO/xbW8DtyLzIVSUa/Rn9GT6KnPjoUCTS3oeeInv8Fo9K/sA==</latexit>

2
b +

2
1
1 0
w E [xi (x)xi (x )] =

2
b +

2
0
w C(x, x )NNGP (Neural Network Gaussian process)l Assume `<latexit sha1_base64="vqdzNAQOmaqadAvlqEJ7xT5gIv4=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLSxTEhWgTkECcWqBSq1ZVmpA2pa4i22xaE3/JdpImUV6AMxIHBBJIHBCPwYUX4NBHQByLxIUD47UjJ9lpayvx7szvP57ZXe/qnmUGIWPHE5NTFy5eUqYvz1y5eu367Nz8je3AbfoGrxiu5fpVXQu4ZTq8EpqhxauezzVbt/iO3ngW+Xda3A9M13kRdjy+b2sHjlk3DS1E00uVW5a6eF9dzNfmMizHxLUgN/JJIwPJVXTnp1ZBhVfgggFNsIGDAyG2LdAgwHsP8sDAQ9s+9NDmY8sUfg59mEFtEymOhIbWBv4fYG8vsTrYj2IGQm3gWyz8+ahcgCz7yb6yE/aDfWO/2L9TY/VEjCiXDj71WMu92uybW+W/56psfIZwmKrOUOhIn11TCHV4LGoxsTZPWKIqjTh+q/vupPyklO3dZZ/Zb6zvEztm37FCp/XH+LLFS+8xelaMN8dIKrTwaUAGx7iHvUGuOvp6wtpP8kn5UYUr+j7eJuZ4iGpN9FxonxrhiHjXEcF1CK5DcF2C6xJcm+DaBFcnuDrBNQiuQXDLBLdMcGsEt0Zw6wS3TnBVgqsS3C7B7RIcIzgmcXoyv9FapuZVT+Y19svzqYsvYeDnhN/GVZ8SqujLlIc5miNcbKHIQCIDkoxyi1b4MDuwyXQ0SgN+WJHa+4TKEvuGjpZx3bCHUhaliounVFwWX6s9VklqlRUbI1mlmo2xnIZVm0n+Bvos3J02JWJlbEXp2F+RqBJBlSTqudhtMlAQdB33oGjvjGdzcGrEecS7Eu0pRHHxDMuPn1hyY7uQyz/IFbYeZpaeJqfZNNyGO3AP4z+CJVjF8a9gDg68hQ/wUSkoVaWmaDE6OZFobsLIpbz+D1y3kZ4=</latexit>

1 layer output zj` 1 is GP:zi` (x) = b`i +NX̀<latexit sha1_base64="SoxEhCxzgAAaMSMN6eFb2Lc4u2Q=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEhWgTKoE4tUClVq2qNCFtSlMi22xaE3/JdpImVl6AB4ADBz4kDojH4MILcOgjII5F4sKB8dqRk+y0tZV4d+b3H8/srnc11zT8gLHjicmpCxcvKdOXZ65cvXZ9dm7+xrbvtDydV3THdLyqpvrcNGxeCYzA5FXX46qlmXxHaz6N/Dtt7vmGYz8Pui7ft9QD22gYuhqgqdZ7Gda4ad7P9+uv63MZlmPiWpAb+aSRgeQqOvNTq1CDV+CADi2wgIMNAbZNUMHHew/ywMBF2z6EaPOwZQg/hz7MoLaFFEdCRWsT/w+wt5dYbexHMX2h1vEtJv48VC5Alv1kX9kJ+8G+sV/s36mxQhEjyqWLTy3Wcrc+++ZW+e+5KgufARymqjMUGtJn1xRAAx6JWgyszRWWqEo9jt/uvTspPy5lw7vsM/uN9X1ix+w7Vmi3/+hftnjpPUbPivHmGKkGbXzqkMExDrE3yFVDXyis/SSflB9VOKLv4W1gjoeoVkXPgc6pEY6Idx0RXJfgugTXI7gewXUIrkNwDYJrEFyT4JoEt0xwywS3RnBrBLdOcOsEVyW4KsHtEtwuwTGCYxKnJfMbrWVqXrVkXmO/PJ+a+BIGfk74LVz1KVETfZlyMUdjhIstFOlLpE+SUW7RCh9mBzaZjkZpwA8rUnufUJli39DQMq4b9lDKolRx8ZSKy+JrtcYqSa2yYmMkq1SzMZbTsGozyV9Hn4m706ZErIytKA37KxJVIqiSRD0Tu00GCoJu4B4U7Z3xbA5OjTiPeFeiPYUoLp5h+fETS25sF3L5B7nC1mJm6Ulymk3DbbgD9zD+Q1iCVRz/Cubgwlv4AB+VReWFoip6jE5OJJqbMHIp5n+DCJNR</latexit>

Wij` x`j (x), x`j (x) = (zj` 1 (x))

j=1<latexit sha1_base64="D+0iXPO1Z5TDOh3tf0nEKC4ghVE=">AAAIF3ichZXfT9NQFMfP+OFw/gD0xcSXxgUDEcjdNNGYkIBKAoEQGAIjFJa13kFZuzZtN7Y15Q/wHzDGJ018MP4RPpgYXn3ggT/B+IiJD/rg6W2X/egB22y955zP9/Sce9tbxdI1x2XsLNHXPzB4JTl0NXXt+o2bwyOjtzYds2qrfEM1ddPOK0WH61qFb7iaq/O8ZfOioeh8Syk/D+JbNW47mll56TYsvmsU9ytaSVOLLroKI2ZzT+a6XtDG6xPSjKRElvRAkp2qUfAOZzL+nrdSCNx+aisMe9qhL09K9dA6ROmkdHyc6rAlzCVbB9p4c88LnFMZX/gnCiNpNs3EIcUHmWiQhuhYNUf7F0CGV2CCClUwgEMFXBzrUAQHzx3IAAMLfbvgoc/GkSbiHHxIobaKFEeiiN4y/u+jtRN5K2gHOR2hVvEuOv5sVEowxk7ZJ3bOTthn9oP9vTCXJ3IEtTTwqoRabhWGX99Z//1flYFXFw7aqksUCtKX9+RCCZ6IXjTszRKeoEs1zF9rvjlff5ob8+6zD+wn9veenbGv2GGl9kv9uMZz7zD7mJhvjplkqOFVhTTOsYdWq1YFY57w+lE9bb5bYQrbxlPDGg9QXRSWCUcXZqgT96oTXIPgGgTXJLgmwR0R3BHBlQiuRHBlgisT3BzBzRHcIsEtEtwSwS0RXJ7g8gS3TXDbBMcIjsU4JVrf4Fmm1lWJ1jWMx9dTEW9CK86JuIFPfZuQhR2nLKxR6+JCD0U6MdIhyaC24AnvZFu+OB3MUovvVLT9PqHSxb6hoKdX1xmhlKuxjlcv6HhdvK1GTydtb1yx3FVVW7PcU1OnaiWqX8WYjrvTSoyY73miFLTnY1SOoHIx6oXYbdKQFXQJ96Bg7wxXs/XVCOsIdyU6kg3y4jcs0/vFig82s9OZh9PZtUfp2WfR12wI7sI9GMf8j2EWFnD+N7CGU/iTGEgMJt8mvyS/JU9CtC8RaW5D15H8/g92B7a7</latexit>l ` -layer Mean is 0, variance is<latexit sha1_base64="LeHFiym2DY6K2nLXvH/KvCo2or0=">AAAHqnichZXNbtNAEMenLeBSPtrCBYlLRRTEAVWbgATi1AKVWrWq0oS0iZKq8rqbxoq/ZDtJkygvwIFrkTiBxAHxGFx4AQ59BMSxSFw4MF47ysdOiq3EuzO//3hmd73LPcsMQsbOZ2bnrly9ps1fX7hx89btxaXlO/uB2/QNUTRcy/VLXA+EZTqiGJqhJUqeL3SbW+KAN15F/oOW8APTdd6EHU8c2vqJY9ZMQw8jU1VY1tFSiq0yea2ojUzSSEFy5dzluU2owjG4YEATbBDgQIhtC3QI8K5ABhh4aDuEHtp8bJnSL6APC6htIiWQ0NHawP8T7FUSq4P9KGYg1Qa+xcKfj8oVSLMf7Au7YN/ZV/aT/Z0aqydjRLl08MljrfCOFt/eK/z5r8rGZwj1oeoSBUf68ppCqMFzWYuJtXnSElVpxPFb3bOLwot8uveQfWK/sL6P7Jx9wwqd1m/j857If8DoaTneAiNVoYVPA1I4xj3sDXLl6OtJaz/JZ8iPK1zZ9/E2Mcc6qnXZc6E9NcIp8a5TgusQXIfgugTXJbg2wbUJrkZwNYJrEFyD4NYJbp3gtghui+C2CW6b4EoEVyK4MsGVCY4RHFM4nsxvtJapeeXJvMZ+dT65/BIGfkH4bVz1Q6Iq+yrlYY7mGBdbKDJQyIAko9yiFT7KDmwqHY3SgB9VDO19QmXJfYOjZVI36qGUOaXi3JSKC/JrtScqGVpVxc5YVkPNzkROo6rdJH8DfRbuTrsKsTGxojj2NxQqT1B5hXotd5sUZCVdwz0o2jvj2RycGnEe8a5Ee7JRXDzDMpMnltrYz65mnqxm956m1l4mp9k83IcH8AjjP4M12MTxL2IOdXgHZ/Bee6zltbJWidHZmURzF8Yu7fgfjLCQCg==</latexit>

K ` (x, x0 ) ⌘ E [zi` (x)zi` (x0 )]=<latexit sha1_base64="dRe8k7P6/VXC7507Sd2oVEolzP4=">AAAIenichZXNbtNAEMcnLdAQPtrCBYlL2qhtoqbVJlSiQkJqgYpWqao0pTRVnES22SRW7DjYzqeVF+AFOHACiQPiMbhwRuLQR0Aci8SlB8ZrR07ibbGVeHfm9x/P7K53paaqmBYhZ6Gp6WvXb8yEb0Zu3b5zd3Zu/t5rU28ZMj2WdVU38pJoUlVp0GNLsVSabxpU1CSVnkj1547/pE0NU9Ebr6xekxY1sdpQKoosWmgqz/3IlASqqvFusruSiCwL9G1LaUcFTbRqkmTvDIRkoc+IshLvJvzmSqIYFYTI8tOoYCpVTSxLpXR0ddjplNJCMuJHKdv9ku1IhYU1YSE1KCsOqUVtwdCiL7ODOElmxoDEICKotGIVhGZNiXviNUeIWSQ4xhW0Gkq1ZhUj5bkYWSfsigYbKa8RA+/K6vPTuyDAG9BBhhZoQKEBFrZVEMHEuwApINBEWxFstBnYUpifwgAiqG0hRZEQ0VrH/yr2Cp61gX0npsnUMr5FxZ+ByigskZ/kCzkn38lX8otcXBrLZjGcXHr4lFwtbZZn3z04+vtflYZPC2q+6gqFhPTVNVlQgU1Wi4K1NZnFqVJ247f778+PnuSW7GXyifzG+j6SM/INK2y0/8ifD2nuA0ZfYuNNMZIAbXzKEMMxtrE3zFVCn82sAy8fnx9X6Kxv4K1gjjVUi6ynQ+fSCF3Ou7ocrsfhehyuz+H6HK7D4TocrsLhKhyuzuHqHG6bw21zuD0Ot8fhMhwuw+HyHC7P4U453CmHIxyOBDjJm19nLfPmVfLm1fUH51NiX8LQTzl+DVe9TwisH6SamKMyxrkWHmkGSJNLOrk5K3yUHdqCtDNKQ35U4dsHHJXK9g0JLZO6UQ9PmQ1UnL2k4iP2tWoTlfjWoGJ/LCtfsz+R06jqwMtfRp+Ku9NBgNiZWFES9ncCVI5D5QLUC7bbxCDN6AruQc7e6c7m8NRw83B3Jb4n7cTFMyw1eWIFG6/T66lH6+nDjdjWM+80C8NDWIQ4xn8MW7CL438McmgzVApVQ7WZi/BiOBFeddGpkKe5D2NXeOMf9ezVvA==</latexit>

2
b +

2
w Ezi` 1 ⇠GP(0,K ` 1 )⇥(zi` 1 (x))(zi` 1 (x0 ))– This expectation can be computed by:(1) GP regression (2) Numerical approximation(3) Analytical solution for specific φ like ReLU⇤

NNGP (Neural Network Gaussian process)l When φ is ReLU (Cho&Saul 2009, Lee+ 2017) :K ` (x, x0 ) =

2
b +

`
1
✓x,x
0 = cos<latexit sha1_base64="OtJcMBUNis/jw/ks9eudqB2RHqY=">AAAI9HichZVLb9NAEMcnPIN5lMIFiYtF1NIIWm0CCISExKtSUVDVB32gbhPZZtNYtWNju2lay1+AL8ABLoA4ICQOfAUufAEkOMINcQSJCwfGa5uk8bTYSrw78/uPZ3bXu7prmX7A2JfCvv0HDh46XDyiHD12/MTQyeFTi76z4RliwXAsx1vWNV9YZlssBGZgiWXXE5qtW2JJX78T+5c6wvNNp/0g2HLFqq2ttc2maWgBmhrDhVqtzoVljXUvds+XldEbKvfNNVtr6PWqekHlTU8zwtS0Wa9GYZW7ZqRw/7EXhLV6GGvHK1EsL/d3z8fhIpXzlXE2cUV0V5VRlbd8VzNEeFnYEQ9MW/gKt0QzGMNwZpsHLRFo/0I0wjijCHNQxvCV6rhKA2WVG45P+xTumWutoKxwrqREDGTeuFjU1kNUqFkmsuD+MYnCrNjMVs6aSZG91zROltgEk5eab1TSRgnSa8YZ3j8FHB6BAwZsgA0C2hBg2wINfLxXoAIMXLStQog2D1um9AuIQEHtBlICCQ2t6/i/hr2V1NrGfhzTl2oD32Lhz0OlCiPsE3vDfrKP7C37zv7sGiuUMeJctvCpJ1rhNoaenJn//V+Vjc8AWj3VHgod6b1rCqAJ12QtJtbmSktcpZHE72w//Tl/fW4kHGUv2Q+s7wX7wj5ghe3OL+P1rJh7htFH5HgLjMShg08DSjjGIfayXHX0hdIapfn0+J0KR/Y9vE3MsYVqTfYc2Nw1Qpd4V5fgtghui+C2CW6b4DYJbpPgmgTXJLh1glsnuFsEd4vg7hHcPYKrEVyN4JYJbpngHhLcQ4JjBMdynJ7Ob7yWqXnV03lN/Pn51OWXkPkF4bdx1fcILvt5ysUczR1cYqFIP0f6JBnnFq/wfjaz5el4lDK+X9GzR4TKkvuGjpZBXb+HUs7kKp7ZpeJ5+bXaA5X0rHnF/R1Z9TT3B3LqV02n+Rvos3B3ms4RkwMrSsf+ZI6aI6i5HHVX7jYlqEq6iXtQvHcms5mdGkkeya5Ee6pxXDzDKoMnVr6xWJ2oXJqozl4u3bydnmZFOAvnYAzjX4WbMIXjvwBG4X3hc+Fr4VuxU3xefFV8naD7CqnmNOy4iu/+Am4oApU=</latexit>

2 qw

K ` 1 (x, x)K ` 1 (x0 , x0 )

2⇡ ⇣
` 1
⇥ sin ✓x,x
0 + (⇡K ` (x, x0 )

` 1
` 1
✓x,x
)
cos
✓
0
x,x0
!

p
K ` (x, x)K ` (x0 , x0 )⌘

Generative model of NN-iWMM
l ∞-Gaussian mixture ∞-mixture of Gaussians

generates latentsl Fromand kernelparameters b , w ,compute NNGPL
kernel matrix KNN
l Generate data Y
for each dimension dfrom NNGP<latexit sha1_base64="0nTLzTnzfKiLkMSsbAJxvm/LH/A=">AAAHunichZXNbtNAEMenLeBSPpqWCxKXiCiIA4o2AQmEOLRApVatqjQhbaqmirzuJrXiL9nOt/ICvAAHJCSQECAegwsvwKGPgDgWiQsHxmtH+fCk2Eq8O/P7j2d217vcMXTPZ+xsbn7h0uUryuLVpWvXb9xcTqys7nt209VESbMN2y1z1ROGbomSr/uGKDuuUE1uiAPeeBH4D1rC9XTbeuV3HXFsqnVLr+ma6qOpmliteHrdVKv8QTJqtauJFMsweSXjjWzUSEF05e2VhU2owAnYoEETTBBggY9tA1Tw8D6CLDBw0HYMfbS52NKlX8AAllDbREogoaK1gf917B1FVgv7QUxPqjV8i4E/F5VJSLMf7As7Z9/ZV/aT/Z0Zqy9jBLl08clDrXCqy69vF//8V2Xi04fTkeoCBUf64pp8qMETWYuOtTnSElSphfFbvTfnxaeFdP8e+8B+YX3v2Rn7hhVard/axz1ReIvR03K8BUaqQAufGqRwjPvYG+bK0deX1kGUz4ifVNiy7+KtY46nqFZlz4b2zAgd4l0dgusSXJfgegTXI7g2wbUJrkZwNYJrEFyD4NYJbp3gtghui+C2CW6b4MoEVya4Q4I7JDhGcCzG8Wh+g7VMzSuP5jX0x+eTyy9h6BeE38RVPyIqsh+nHMxRn+BCC0V6MdIjySC3YIWPs0NbnA5GaciPK0b2AaEy5L7B0TKtG/dQynys4vyMiovyazWnKhlZ44qdiaxGmp2pnMZVu1H+GvoM3J12Y8TG1Iri2N+IUQWCKsSol3K3SUFO0jXcg4K9M5zN4akR5hHuSrQnF8TFMyw7fWLFG/u5TPZhJrf3KLX2PDrNFuEO3IX7GP8xrMEmjn8Jc+jAO/gEn5VnCld0pRGi83OR5hZMXIr/D5pGles=</latexit><latexit sha1_base64="vsKCCkC3krRXUGjXYPDIRWN2O0k=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRBTEKdqklUCcWqBSq0RRmpA2pSmRbTatFX/JdpImVl6AB4ADBz4kDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613F1jXXY+xsbn7h0uUr0uLVpWvXb9xcXlm9tedaHUflNdXSLaeuyC7XNZPXPM3Ted12uGwoOt9X2k8D/36XO65mmc+9vs2PDPnY1FqaKntoahReFpt+wzFSpdKwuZJmWSauVLKRixppiK6ytbqwDQ14BRao0AEDOJjgYVsHGVy8DyEHDGy0HYGPNgdbmvBzGMISajtIcSRktLbx/xh7h5HVxH4Q0xVqFd+i489BZQoy7Af7ws7Zd/aV/WR/Z8byRYwglz4+lVDL7eby6zvVP/9VGfj04CRWXaBQkL64Jg9a8EjUomFttrAEVaph/O7g7Xn1cSXj32ef2C+s7yM7Y9+wQrP7W/28yyvvMHpGjDfHSA3o4lOFNI6xj71Rrgr6fGEdRvnE/KTCEn0Hbw1zPEG1LHoW9GZGOCXedUpwfYLrE9yA4AYE1yO4HsG1CK5FcG2CaxPcJsFtEtwOwe0QXIHgCgRXJ7g6wR0Q3AHBMYJjCU6J5jdYy9S8KtG8hv7kfCriSxj5OeE3cNXHREP0k5SNOWoTXGihSDdBuiQZ5Bas8HF2ZEvSwSiN+HFFbB8SKl3sGwpapnXjHkpZTlRcnlFxVXytxlQlsTWpKE5kFWuKUzmNq0pR/ir6dNydSglia2pFKdjfSlAVgqokqGdit0lDXtAt3IOCvTOczdGpEeYR7kq0Jx/ExTMsN31iJRt7+WxuLZvfXU9vPIlOs0W4C/fgAcZ/CBuwjeNfwxxseAPv4YO0Lr2QZEkN0fm5SHMbJi5J/wdQ5ZME</latexit>

NNGP kernel

L-layer ∞-Unit NN
Unsupervised NN with no units,
no numerical weights

Joint modeling with t-SNE
l Recap:l Problem: t-SNE has no clear generative modelà How to combine NN-iWMM and t-SNE?

Joint modeling with t-SNE: RegBayesl Posterior p(θ|Y) equals to following optimization:(A. Zellner, 1988)
Θ：パラメータPl RegBayes：Bayesian model with posterior

constraint (J. Zhu+, 2014)– Optimal distribution
under the given constraint

Joint distribution of θ and Y

NPDV Formulationl Joint distribution of NPDV (q*(θ) and NN-iWMM,

t-SNE)l Specifically,

NN-iWMM

t-SNE

NN-iWMMt-SNE between X and V– V : visual coordinates,: NNGP kernel
hyperparameter– λ : hyperparameter ※ λ=ND in this study

NPDV training
l Objective : lower bound of likelihood using

variational distribution Q– Re-parametrization trick to generate randomly
fromto approximate (*)– Gradient base learning from the approximation

GP Neural Networks and ARDl Since there is no NN

weights in NPDV,
we can directly
optimize hidden X
for observation Y
l ARD (automatic relevancedetermination) prior on X
à We can learn hidden dimensionality of X!
– This is impossible for ordinary neural networks
with numerous weights to learn

GP Neural Networks and ARDl Input: “mammoth” 3D data mapped non-linearlyfor high dimensions– NPDV correctly recovered mammoth, witheffectively three dimensions!

Hyperparameter settings
rameters, however, some osidered in practice. For themensions and varitional mix
sufficiently large values, we
informative latent dimension
ARD mechanism and iGMM⇢, the number of inducingbalance term , we confirm
accuracy than the existing m
M = 50, ⇢ = 30, ⌘ = 0.01
attention to only the number

5

Simulati

We present the qualitative pra simulation experiment. Th
generated by nonlinear tra
1

Experimentsl Text corpus: 20newsgroups (English)、Livedoor

News corpus (Japanese)– 20newsgroups: USENET internet news text on 20
newsgroups, 18000 documents– Livedoor: Livedoor news articles, 9 categories,
7300 documents
l tf.idf preprocessing, compress into 1000 dims

to visualize in 2D
l Baseline: t-SNE (Maaten+ 2009), Deep K-means(Phard+ 2020)

Results (20newsgroups: English)lk-NN classification accuracy (Bold: SOTA, *：Next to SOTA)

Clearer idendification of clusters & SOTA accuracy

Results (Livedoor: Japanese)lk-NN classification accuracy (Bold: SOTA, *：Next to SOTA)

Unsupervised visualization of sentences(t-SNE)l Directly visualize sentences via t-SNE→ Prone to noise, cannot consider meanings

Unrelated clustering of sentences that belong to “cooking” cluster under NPDV

Unsupervised visualization of sentences(NPDV)l Internally estimates ∞-GMM to give “label” oneach sentence
Clusters obtained through NPDV
Simultaneously estimates latent clustering and visualization

Data:
Kyoto corpus(5000 articles
from Mainichi
newspaper
1995)○ NPDV estimates semantically meaningful clusters

Summaryl Gentle introduction to Gaussian processesl Learning “Motions” from movements

through a Gaussian process generative model
for robots (from research on NLP)
– GP+HDP+Hidden semi-Markov Model (IROS 2018)– HVGH: GP+HDP+Hidden SMM+VAE (IROS 2019)
for high-dimensional observations(segmentation on latent Gaussian processes)l Analytical Gaussian process neural networkwith no neural weights– NPDV: Visualization w/ combination with t-SNE

“Language and Robotics”
l Advanced Robotics (2019),“Survey on frontiers of language and robotics”.
T. Taniguchi, D. Mochihashi, T. Nagai, et al.

https://www.tandfonline.com/doi/full/10.1080/01691864.2019.1632223