Nonparametric Bayesian
Deep Visualization
Haruya Ishizuka @Bridgestone Corporation, Japan
Daichi Mochihashi @The Institute of Statistical Mathematics,
Japan
ECML-PKDD 2022
統計数理セミナー 2022-11-30

Benefits of the proposed NPDV• Intuitive understanding of internal structures using latent clusters• Using a Bayesian neural network with very few parameters
thanks to Neural Network Gaussian processes

NPDV
1/30

t-SNE (2008)
Security Classification

PaCMAP (2021)

Introduction

2/30

Security Classification

High-dimensional data visualization• Requires dimensionality reduction• Will enhance knowledge discovery in various domains,e.g. RNA analysisn

Y1 Y2 … YD

１
２
・・・
N

Dimensionality
Reduction

V1

V2

１
２
・・・
N

Not plottable
when D > 3

3/30

n

Plottable

Security Classification

Issues of existing methods
１．Inaccurate visualization Y has an intrinsic structure on latent manifold
２．Low interpretability on possible clusters
n Y1 … Y100

embed into
100D space

１

Visualize in
3D space・・・N

intrinsic structure:
3D manifold

4/30

Security Classification

Alternative approach: neural clustering models• Estimate latent coordinates and clusters jointly② Visualization① Neural clustering model(Auto-encoder + clustering)
n

Y1

… YD

n X1

1

1

…

…

N

N

… XQ

K

• Cons.1: Large # of hyperparameters• Cons.2: X only will not always be suitable for visualization
5/30

Security Classification

Neural network infinite Warped Mixture Model (NN-iWMM)• Reduce some hyperparameters by Nonparametric Bayes techniques
×: to be optimized ✔:no need to be optimized

6/30

Neural clustering

NN-iWMM

Components# of layers

×

×

-

# of units

×✔

# of latent dims×✔

# of clusters

×✔

Security Classification

ARD-NNGP∞-GMMInfinite Warped Mixture Model (iWMM) (Iwata+ 2013)∞-Gaussian
mixturemodel

InferenceX

Y<latexit sha1_base64="T1QB0mKOGtjE1L6Pq0zjYHc4ECM=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJqRNaarIa9aNib9kO0mTKC/AGYkDAgkkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613umoYfMHY6Mzt34eIlZf7ywpWr164vLi3f2POdlqeJiuaYjlflqi9MwxaVwAhMUXU9oVrcFPu8+TT077eF5xuO/TzouuLIUo9tQzc0NUDTi5qlBg2u96uD+lKG5Zi8VtKNfNzIQHwVneW5DajBS3BAgxZYIMCGANsmqODjfQh5YOCi7Qj6aPOwZUi/gAEsoLaFlEBCRWsT/4+xdxhbbeyHMX2p1vAtJv48VK5Alv1gX9gZ+86+sp/s79RYfRkjzKWLTx5phVtffH2r/Oe/KgufATQS1TkKjvT5NQWgwyNZi4G1udISVqlF8du9t2flx6Vs/y77xH5hfR/ZKfuGFdrt39rnXVF6h9GzcrwFRqpBG58aZHCM+9gb5srR15fWQZxPwo8rHNn38DYwxwaqVdlzoDM1wgnxrhOC6xJcl+B6BNcjuA7BdQhOJzid4JoE1yS4NYJbI7hNgtskuC2C2yK4KsFVCe6A4A4IjhEcS3E8nt9wLVPzyuN5jfzp+eTySxj6BeG3cNUnRE3205SLORpjXGShSD9F+iQZ5hau8FF2aEvT4SgN+VFFYh8QKlPuGxwtk7pRD6UspiouTqm4LL9Wa6KSxJpWbI9llWi2J3IaVe3E+WvoM3F32kkR6xMrimN/PUWVCKqUop7J3SYDBUnruAeFe2c0m8NTI8oj2pVoTyGMi2dYfvLESjf2Crn8/Vxh90Fm9Ul8ms3DbbgD9zD+Q1iFDRz/CuZgwxt4Dx+UglJV6ooaobMzseYmjF3Kq3/peZLL</latexit><latexit sha1_base64="p+MlzADJVLtSx0+TRZ5ORfiBDS4=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJqRNaKrKa9aNib9kO0mTKC/AGYkDAgkkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613umoYfMHY6Mzt34eIlZf7ywpWr164vLi3f2POdlqeJiuaYjlflqi9MwxaVwAhMUXU9oVrcFPu8+TT077eF5xuO/TzouuLQUo9tQzc0NUDTi7qlBg2u92uDo6UMyzF5raQb+biRgfgqOstzG1CHl+CABi2wQIANAbZNUMHH+wDywMBF2yH00eZhy5B+AQNYQG0LKYGEitYm/h9j7yC22tgPY/pSreFbTPx5qFyBLPvBvrAz9p19ZT/Z36mx+jJGmEsXnzzSCvdo8fWt8p//qix8BtBIVOcoONLn1xSADo9kLQbW5kpLWKUWxW/33p6VH5ey/bvsE/uF9X1kp+wbVmi3f2ufd0XpHUbPyvEWGKkObXxqkMEx7mNvmCtHX19aB3E+CT+ucGTfw9vAHBuoVmXPgc7UCCfEu04IrktwXYLrEVyP4DoE1yE4neB0gmsSXJPg1ghujeA2CW6T4LYIbovgqgRXJbgawdUIjhEcS3E8nt9wLVPzyuN5jfzp+eTySxj6BeG3cNUnRF3205SLORpjXGShSD9F+iQZ5hau8FF2aEvT4SgN+VFFYh8QKlPuGxwtk7pRD6UspiouTqm4LL9Wa6KSxJpWbI9llWi2J3IaVe3E+WvoM3F32kkR6xMrimN/PUWVCKqUop7J3SYDBUnruAeFe2c0m8NTI8oj2pVoTyGMi2dYfvLESjf2Crn8/Vxh90Fm9Ul8ms3DbbgD9zD+Q1iFDRz/CuZgwxt4Dx+UglJVjhQ1QmdnYs1NGLuUV/8A8L2SzA==</latexit>• !"#$"$%&'()*++$)"',$-%*.&'$"'/)%&"%'+0)1&
à 2)0'%3'34+&.5)%$3"'+0)1&'%6.3*76'()*++$)"'0.31&++&+• 89%6 8$,&"+$3"'3#'34+&.5)%$3"':'8$+%.$4*%&+')113.8$"7'%3'(;'3"'<=
Yd ⇠ GP(µ, KX )<latexit sha1_base64="0wQNDylLUmvjz1qx3x5EqYX9jIc=">AAAH0HichZW/b9NAFMdfWyCl/GgKCxJLRFRUJFRdAhKIqQUqWqWq0oS2qeoq8rmXxoodW7aTNLEixNqBlYEJJAbEygoLC/8AQ/8ExFgkFgaez47ywy+trcR3732+z+/dne+4beiux9jJxOTUhYuXEtOXZ65cvXZ9Njl3Y9u1Go4mtjTLsJwSV11h6HWx5emeIUq2I1STG2KH154F/p2mcFzdqr/02rbYN9XDul7RNdVDUzm5sFs+SCmubqZ8xTFTL/LdBYWbvmI2uvdzZcVUvSqv+KXuvXIyzRaZvFLxRiZqpCG68tbc1CoocAAWaNAAEwTUwcO2ASq4eO9BBhjYaNsHH20OtnTpF9CFGdQ2kBJIqGit4f8h9vYiax37QUxXqjV8i4E/B5UpmGc/2Sd2yn6wz+wX+zc2li9jBLm08clDrbDLs8e3in/PVZn49KDaV52h4EifXZMHFXgsa9GxNltagiq1MH6z8/a0+KQw799lH9hvrO89O2HfscJ684/2cVMU3mH0eTneAiMp0MSnBmkcYx97vVw5+nxp7Ub59PlhhSX7Dt465lhFtSp7FrTGRjgi3nVEcG2CaxNch+A6BNciuBbBVQiuQnA1gqsR3DLBLRPcGsGtEVyO4HIEVyK4EsHtEtwuwTGCYzGOR/MbrGVqXnk0r6E/Pp9cfgk9vyD8Jq76PqHIfpyyMUd9iAstFOnGSJckg9yCFT7I9mxxOhilHj+o6Nu7hMqQ+wZHy6hu0EMp87GK82MqLsqv1RyppG+NK9aHsupr1kdyGlRtRPlr6DNwd9qIESsjK4pjfyVGFQiqEKOey90mDVlJV3APCvbOcDZ7p0aYR7gr0Z5sEBfPsMzoiRVvbGcXMw8Ws5sP00tPo9NsGm7DHVjA+I9gCVZx/LcwhzfwBb7Ct0QhcZR4lXgdopMTkeYmDF2J4/8EH56b</latexit>

7/30

Security Classification

Nonparametric Bayeisna Deep Visualization: Overview• Integrate NN-iWMM and a visualization method into a Bayesian model- Infer X so as to be optimal to estimate V

8/30

Security Classification

Proposed NPDV

9/30

Security Classification

Neural Network = Gaussian process (Neal 1994)• >$"7/&'6$88&"'/)?&.'@@'A$%6'$"0*%+'/)?&.'• $9%6 3*%0*%'B!C-D$+'A.$%%&"')+=zi1 (x) = b1i +N1
X

Wij1 x1j (x), x1j (x) =

b0j +

j=1

<latexit sha1_base64="LMYTwKSxLTup9cVMeObg85dJ7Rc=">AAAIIXichZXNbtNAEMcnlJISoB9wQeJiERW1AlWbgARCqtQClVq1qtqUtqma1orNpnFix5btpEks9wF4AQ6cAHFAPAYXuCEkDhUHzohjkbhwYLx2yIenxVbi3Znffzyzu95VLF1zXMaOE+eGzg9fSI5cTF26fGV0bHzi6pZj1m2Vb6qmbtp5pehwXavxTVdzdZ63bF40FJ1vK9XHgX+7wW1HM2tP3ZbF94ziQU0raWrRRZM83mrvZ2RtqjktzUpK0JRuSwWnbsheZTbj73urcsaXttHhaRW/ic8Ksneko6NUp4PCglXWpILOS+5UStlncuVfkGoQZBkDMAxY9aWmXE0VbO2g7E7L42k2w8QlxRuZqJGG6FozJ4YWoQDPwAQV6mAAhxq42NahCA7eu5ABBhba9sBDm40tTfg5+JBCbR0pjkQRrVX8P8DebmStYT+I6Qi1im/R8WejUoJJ9pW9YyfsI3vPfrA/p8byRIwglxY+lVDLLXns+fWN3/9VGfh0odxVnaFQkD67JhdK8EDUomFtlrAEVaph/Eb7xcnGw9ykd4u9Zj+xvlfsmH3ACmuNX+rbdZ57idEnxXhzjFSABj5VSOMYe9jr5KqgzxNWP8qny/crTNG38dYwxzKqi6JnwuGpEZrEu5oE1yK4FsG1Ca5NcIcEd0hwJYIrEVyV4KoEN09w8wS3RHBLBLdMcMsElye4PMHtENwOwTGCYzFOieY3WMvUvCrRvIb++Hwq4kvo+DnhN3DVd4mC6McpC3PU+rjQQpFOjHRIMsgtWOG9bMcWp4NR6vC9iq7dJ1S62DcUtAzqej2Uci1W8dopFW+Ir9UYqKRrjStW+rLqalYGcupVrUb5q+jTcXdajRELAytKwf5CjMoRVC5GPRG7TRqygi7hHhTsneFsdk6NMI9wV6I92SAunmGZwRMr3tjKzmTuzmTX76XnHkWn2QjcgJswhfHvwxws4vhvYg7fE8OJ0cRY8k3yU/Jz8kuInktEmmvQdyW//QWPvLeb</latexit>

N2z<latexit sha1_base64="G6XmFjgUM/Lat9F/OGdVLLxN0U0=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfP1XH0lxTJMXmvJRjZqpCC6CvbqwhbU4A3YoEEbTBBggY9tA1Tw8D6ELDBw0HYEA7S52NKlX8AQllDbRkogoaK1hf9N7B1GVgv7QUxPqjV8i4E/F5VrkGY/2Bd2zr6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eKf/5r8rEpw/HseoCBUf64pp8aMBTWYuOtTnSElSphfE7/dPz8rNSenCffWK/sL6P7Ix9wwqtzm/tc1GUPmD0tBxvgZFq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zPEY1ars2dCdGeGEeNcJwfUIrkdwfYLrE1yX4LoE1yC4BsG1CK5FcBsEt0Fw2wS3TXA7BLdDcFWCqxLcAcEdEBwjOJbgeDS/wVqm5pVH8xr6k/PJ5Zcw8gvCb+Kqj4ma7CcpB3PUJ7jQQpFegvRIMsgtWOHj7MiWpINRGvHjitg+JFSG3Dc4WqZ14x5KWUhUXJhRcVl+reZUJbE1qdidyCrW7E7lNK7KR/lr6DNwd8oniM2pFcWxv5mgSgRVSlAv5W6TgpykG7gHBXtnOJujUyPMI9yVaE8uiItnWHb6xEo29nKZ7KNMrvg4tf48Os0W4S7cgwcY/wmswxaOfwVzaMI7eA+nykOlqFSV1yE6PxdpbsPEpWj/ABnBj0Y=</latexit>

1W<latexit sha1_base64="R8ZqzoF4SDVnDSmgi4CIMGySDBw=">AAAHuHichZXNbtNAEMenLeBSPprCBYlLIAriVG0CEqinFqjUqlWVJqRN1ZTgNevUimNbtpM0ifICvAAHuIBUJMRjcOEFOPQREMciceHAeO3ISTwpthLvzvz+45nd9S53TMPzGTubmZ27dPmKMn914dr1GzcXU0u39jy75WqirNmm7Va46gnTsETZN3xTVBxXqE1uin3eeB7499vC9Qzbeul3HXHUVOuWoRua6qOplkpVTaH7quvaneq9NH+Vq6UybJnJK51s5KJGBqKrYC/NbUAV3oANGrSgCQIs8LFtggoe3oeQAwYO2o6gjzYXW4b0CxjAAmpbSAkkVLQ28L+OvcPIamE/iOlJtYZvMfHnojINWfaDfWHn7Dv7yn6yv1Nj9WWMIJcuPnmoFU5t8e2d0p//qpr49OE4Vl2g4EhfXJMPOjyVtRhYmyMtQZVaGL/de3deWilm+w/YJ/YL6/vIztg3rNBq/9ZOd0XxPUbPyvEWGKkKbXxqkMEx7mNvmCtHX19aB1E+MT+usGXfxdvAHI9RrcqeDZ2pEU6Id50QXJfgugTXI7gewXUIrkNwOsHpBNcguAbBrRHcGsFtEtwmwW0R3BbBVQiuQnAHBHdAcIzgWILj0fwGa5maVx7Na+hPzieXX8LQLwh/E1d9TFRlP0k5mKMxxoUWivQSpEeSQW7BCh9lh7YkHYzSkB9VxPYBoTLlvsHRMqkb9VDKQqLiwpSKS/JrbU5UEluTiu2xrGLN9kROo6qdKH8NfSbuTjsJYn1iRXHsryeoIkEVE9QLudtkIC9pHfegYO8MZ3N4aoR5hLsS7ckHcfEMy02eWMnGXn4592g5v/s4s/osOs3m4S7ch4cY/wmswgaOfxlzaMMHOIXPyoryWqkrRojOzkSa2zB2Ke4/DTCU2g==</latexit>

b1

1<latexit sha1_base64="QXvz4mzIdmqYGLWxMRKRtVXlQuE=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqklUCcWqBSq1ZVmpImpSmV16wbK/6S7SRNorwADwAHDnxIHBCPwYUX4NBHQByLxIUD47UjJ/Gk2Eq8O/P7j2d217vcNQ0/YOxsanrm0uUryuzVuWvXb9ycX1i8te87TU8TZc0xHa/KVV+Yhi3KgRGYoup6QrW4KSq88TT0V1rC8w3Hfh50XHFkqSe2oRuaGqCpVrPUoM71XqX/Mn+8kGE5Jq+ldCMfNzIQX0VncWYDavAKHNCgCRYIsCHAtgkq+HgfQh4YuGg7gh7aPGwZ0i+gD3OobSIlkFDR2sD/E+wdxlYb+2FMX6o1fIuJPw+VS5BlP9gXds6+s6/sJ/s7MVZPxghz6eCTR1rhHs+/vrP3578qC58B1BPVBQqO9MU1BaDDI1mLgbW50hJWqUXxW92353uPS9neffaJ/cL6PrIz9g0rtFu/tc+7ovQOo2fleAuMVIMWPjXI4Bj3sDfIlaOvJ639OJ+EH1U4su/hbWCOdVSrsudAe2KEU+JdpwTXIbgOwXUJrktwbYJrE5xOcDrBNQiuQXBrBLdGcJsEt0lwWwS3RXBVgqsS3AHBHRAcIziW4ng8v+FapuaVx/Ma+dPzyeWXMPALwm/hqk+ImuynKRdzNEa4yEKRfor0STLMLVzhw+zAlqbDURrww4rE3idUptw3OFrGdcMeSllMVVycUPGe/FqtsUoSa1qxPZJVotkey2lYtRPnr6HPxN1pJ0Wsj60ojv31FFUiqFKKeiZ3mwwUJK3jHhTundFsDk6NKI9oV6I9hTAunmH58RMr3dgv5PLLucLuSmb1SXyazcJduAcPMP5DWIUNHP8y5uDCG3gPH5QV5YWiKlqETk/FmtswcinmP07xk20=</latexit>

x1<latexit sha1_base64="EvynLVwq2k3n8W/w6hPV9wqJLEA=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfP1bH0lxTJMXmvJRjZqpCC6CvbqwhbU4A3YoEEbTBBggY9tA1Tw8D6ELDBw0HYEA7S52NKlX8AQllDbRkogoaK1hf9N7B1GVgv7QUxPqjV8i4E/F5VrkGY/2Bd2zr6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eKf/5r8rEpw/HseoCBUf64pp8aMBTWYuOtTnSElSphfE7/dPz8rNSenCffWK/sL6P7Ix9wwqtzm/tc1GUPmD0tBxvgZFq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zPEY1ars2dCdGeGEeNcJwfUIrkdwfYLrE1yX4LoE1yC4BsG1CK5FcBsEt0Fw2wS3TXA7BLdDcFWCqxLcAcEdEBwjOJbgeDS/wVqm5pVH8xr6k/PJ5Zcw8gvCb+Kqj4ma7CcpB3PUJ7jQQpFegvRIMsgtWOHj7MiWpINRGvHjitg+JFSG3Dc4WqZ14x5KWUhUXJhRcVl+reZUJbE1qdidyCrW7E7lNK7KR/lr6DNwd8oniM2pFcWxv5mgSgRVSlAv5W6TgpykG7gHBXtnOJujUyPMI9yVaE8uiItnWHb6xEo29nKZ7KNMrvg4tf48Os0W4S7cgwcY/wmswxaOfwVzaMI7eA+nykOlqFSV1yE6PxdpbsPEpWj/ABJ+j0U=</latexit><latexit sha1_base64="Gbam75vABvGioZzco2NfSg9FmmE=">AAAHqXichZVLb9NAEMenLeBSHm3hgsSlIgpCQoo2AQnEqQUqtWpV5UHalKatvGaTWvFLtvNWvgASV3rgBBIHxMfgwhfg0I+AOBaJCwfGa0dO4kmxlXh35vcfz+yud7lj6J7P2NnM7Nyly1eU+asL167fuLm4tHxr17ObribKmm3YboWrnjB0S5R93TdExXGFanJD7PHGi8C/1xKup9vWK7/riENTrVt6TddUH02lzlH2eCnFMkxeK8lGNmqkILry9vLcBlThDdigQRNMEGCBj20DVPDwPoAsMHDQdgh9tLnY0qVfwAAWUNtESiChorWB/3XsHURWC/tBTE+qNXyLgT8XlSuQZj/YF3bOvrOv7Cf7OzVWX8YIcunik4da4Rwvvr1T+vNflYlPH05i1QUKjvTFNflQg6eyFh1rc6QlqFIL47d6p+elZ8V0/z77xH5hfR/ZGfuGFVqt39rngih+wOhpOd4CI1WhhU8NUjjGfewNc+Xo60vrIMon5scVtuy7eOuY4wmqVdmzoT01Qod4V4fgugTXJbgewfUIrk1wbYKrEVyN4BoE1yC4NYJbI7hNgtskuC2C2yK4CsFVCG6f4PYJjhEcS3A8mt9gLVPzyqN5Df3J+eTySxj6BeE3cdXHRFX2k5SDOepjXGihSC9BeiQZ5Bas8FF2aEvSwSgN+VFFbB8QKkPuGxwtk7pRD6XMJyrOT6m4JL9Wc6KS2JpUbI9lFWu2J3IaVe1E+WvoM3B32kkQ6xMrimN/PUEVCaqYoF7K3SYFOUnXcA8K9s5wNoenRphHuCvRnlwQF8+w7OSJlWzs5jLZR5lc4XFq9Xl0ms3DXbgHDzD+E1iFDRz/MuZQh3fwHk6Vh0pBqSivQ3R2JtLchrFL0f4BPJuPbg==</latexit><latexit sha1_base64="pnBwoIFlxOZzvBwgvbEjhepf8Hc=">AAAHuHichZXNbtNAEMenLeBSPprCBYlLIAriVG0CEqinFqjUqlWVJqRN1ZTgNevUimNbtpM0ifICvAAHuIBUJMRjcOEFOPQREMciceHAeO3ISTwpthLvzvz+45nd9S53TMPzGTubmZ27dPmKMn914dr1GzcXU0u39jy75WqirNmm7Va46gnTsETZN3xTVBxXqE1uin3eeB7499vC9Qzbeul3HXHUVOuWoRua6qOplkpVTaH7quvaneq9NH/FaqkMW2bySicbuaiRgegq2EtzG1CFN2CDBi1oggALfGyboIKH9yHkgIGDtiPoo83FliH9AgawgNoWUgIJFa0N/K9j7zCyWtgPYnpSreFbTPy5qExDlv1gX9g5+86+sp/s79RYfRkjyKWLTx5qhVNbfHun9Oe/qiY+fTiOVRcoONIX1+SDDk9lLQbW5khLUKUWxm/33p2XVorZ/gP2if3C+j6yM/YNK7Tav7XTXVF8j9GzcrwFRqpCG58aZHCM+9gb5srR15fWQZRPzI8rbNl38TYwx2NUq7JnQ2dqhBPiXScE1yW4LsH1CK5HcB2C6xCcTnA6wTUIrkFwawS3RnCbBLdJcFsEt0VwFYKrENwBwR0QHCM4luB4NL/BWqbmlUfzGvqT88nllzD0C8LfxFUfE1XZT1IO5miMcaGFIr0E6ZFkkFuwwkfZoS1JB6M05EcVsX1AqEy5b3C0TOpGPZSykKi4MKXikvxamxOVxNakYnssq1izPZHTqGonyl9Dn4m7006CWJ9YURz76wmqSFDFBPVC7jYZyEtaxz0o2DvD2RyeGmEe4a5Ee/JBXDzDcpMnVrKxl1/OPVrO7z7OrD6LTrN5uAv34SHGfwKrsIHjX8Yc2vABTuGzsqK8VuqKEaKzM5HmNoxdivsPBe2U2Q==</latexit>b0<latexit sha1_base64="nnbc5hsCBtXrsdTmvR4Ug0hlEGg=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqklUCcWqBSq1ZVmpImpSmV16wbK/6S7SRNorwADwAHDnxIHBCPwYUX4NBHQByLxIUD47UjJ/Gk2Eq8O/P7j2d217vcNQ0/YOxsanrm0uUryuzVuWvXb9ycX1i8te87TU8TZc0xHa/KVV+Yhi3KgRGYoup6QrW4KSq88TT0V1rC8w3Hfh50XHFkqSe2oRuaGqCpVrPUoM71XqX/kh0vZFiOyWsp3cjHjQzEV9FZnNmAGrwCBzRoggUCbAiwbYIKPt6HkAcGLtqOoIc2D1uG9Avowxxqm0gJJFS0NvD/BHuHsdXGfhjTl2oN32Liz0PlEmTZD/aFnbPv7Cv7yf5OjNWTMcJcOvjkkVa4x/Ov7+z9+a/KwmcA9UR1gYIjfXFNAejwSNZiYG2utIRValH8Vvft+d7jUrZ3n31iv7C+j+yMfcMK7dZv7fOuKL3D6Fk53gIj1aCFTw0yOMY97A1y5ejrSWs/zifhRxWO7Ht4G5hjHdWq7DnQnhjhlHjXKcF1CK5DcF2C6xJcm+DaBKcTnE5wDYJrENwawa0R3CbBbRLcFsFtEVyV4KoEd0BwBwTHCI6lOB7Pb7iWqXnl8bxG/vR8cvklDPyC8Fu46hOiJvtpysUcjREuslCknyJ9kgxzC1f4MDuwpelwlAb8sCKx9wmVKfcNjpZx3bCHUhZTFRcnVLwnv1ZrrJLEmlZsj2SVaLbHchpW7cT5a+gzcXfaSRHrYyuKY389RZUIqpSinsndJgMFSeu4B4V7ZzSbg1MjyiPalWhPIYyLZ1h+/MRKN/YLufxyrrC7kll9Ep9ms3AX7sEDjP8QVmEDx7+MObjwBt7DB2VFeaGoihah01Ox5jaMXIr5D0euk2w=</latexit><latexit sha1_base64="be+5Foc0l/Cr4ohUtAHWc5/6x0A=">AAAHqXichZXNbtNAEMenLeBSPtrCBYlLRRSEhBRtAhKIUwtUatUqygdpU5oq8ppNasVfsp2kSZQXQOJKD5xA4oB4DC68AIc+AuJYJC4cGK8dOYknxVbi3Znffzyzu97ljqF7PmNnc/MLly5fURavLl27fuPm8srqrT3PbruaqGi2YbtVrnrC0C1R8XXfEFXHFarJDbHPWy8C/35HuJ5uW6/8niOOTLVp6Q1dU300lfN1Vl9JsQyT11qykY0aKYiugr26sAU1eAM2aNAGEwRY4GPbABU8vA8hCwwctB3BAG0utnTpFzCEJdS2kRJIqGht4X8Te4eR1cJ+ENOTag3fYuDPReUapNkP9oWds+/sK/vJ/s6MNZAxglx6+OShVjj15bd3yn/+qzLx6cNxrLpAwZG+uCYfGvBU1qJjbY60BFVqYfxO//S8/KyUHtxnn9gvrO8jO2PfsEKr81v7XBSlDxg9LcdbYKQadPCpQQrHeIC9Ua4cfQNpHUb5xPykwpZ9F28dczxGtSp7NnRnRjgh3nVCcD2C6xFcn+D6BNcluC7BNQiuQXAtgmsR3AbBbRDcNsFtE9wOwe0QXJXgqgR3QHAHBMcIjiU4Hs1vsJapeeXRvIb+5Hxy+SWM/ILwm7jqY6Im+0nKwRz1CS60UKSXID2SDHILVvg4O7Il6WCURvy4IrYPCZUh9w2OlmnduIdSFhIVF2ZUXJZfqzlVSWxNKnYnsoo1u1M5javyUf4a+gzcnfIJYnNqRXHsbyaoEkGVEtRLudukICfpBu5Bwd4Zzubo1AjzCHcl2pML4uIZlp0+sZKNvVwm+yiTKz5OrT+PTrNFuAv34AHGfwLrsIXjX8EcmvAO3sOp8lApKlXldYjOz0Wa2zBxKdo/CzuPRA==</latexit>

10/30<latexit sha1_base64="k9z/fC3sUS8igmMXyuUyMOpPBzU=">AAAH2HichZXNbtNAEMenLZBSPprCBYmLRRRUJFQ2AQnEqQUqtWoVpQlpUpoS2WaTmtixZTtJEysSNwRHDnDgBBIHxCtwQyBegEMfAXEsEhcOjNeO8uFJsZV4d+b3H8/srncVS9ccl7HDqemZEydPxWZPz505e+78fHzhwrZjNm2VF1RTN+2SIjtc1xq84GquzkuWzWVD0XlRqd/3/cUWtx3NbDx0OxbfM+RaQ6tqquyiqRJnxYqnPe099spc13tS2dEMySursi5leovsOvZrhlxpSzekTMVHrlXiCbbExCVFG6mwkYDwypoLM2tQhidgggpNMIBDA1xs6yCDg/cupICBhbY98NBmY0sTfg49mENtEymOhIzWOv7XsLcbWhvY92M6Qq3iW3T82aiUIMl+sI/siH1nn9hP9ndiLE/E8HPp4FMJtNyqzL+8lP/zX5WBTxf2B6pjFArSx9fkQhXuiFo0rM0SFr9KNYjf6r45yt/NJb2r7D37hfW9Y4fsC1bYaP1WP2zx3FuMnhTjzTFSGVr4VCGBY+xhr5+rgj5PWHthPgN+VGGKvo23hjnuo1oWPRPaEyMcEO86ILgOwXUIrktwXYJrE1yb4KoEVyW4OsHVCW6F4FYIbp3g1glug+A2CK5EcCWC2yG4HYJjBMcinBLOr7+WqXlVwnkN/NH5VMSX0Pdzwm/gqh8QZdGPUhbmqI1wgYUinQjpkKSfm7/Ch9m+LUr7o9TnhxUDe49Q6WLfUNAyrhv2UMpspOLshIrz4ms1xioZWKOKzZGsBprNsZyGVZkwfxV9Ou5OmQixOraiFOyvRqgcQeUi1AOx2yQgLegq7kH+3hnMZv/UCPIIdiXak/bj4hmWGj+xoo3t9FLq5lJ661Zi+V54ms3CZbgCixj/NizDGo5/AXN4DZ/hK3yLPYo9iz2PvQjQ6alQcxFGrtirfwavoUY=</latexit>

Bias<latexit sha1_base64="YenEqkRjQZTRC/KLFgp+uDfhZ/o=">AAAHynichZXNbtNAEMenLZBSPprCBYlL1CioSKjaBCQQpxao1KpRlCakTdW0kddsUit2bNlO0sTKjQPiBThwAokDQuIKdy68AIc+AuJYJC4cGK8d5cOTYivx7szvP57ZXe9yS9ccl7HTmdm5CxcvxeYvL1y5eu36Ynzpxq5jtmxVlFRTN+0yVxyha01RcjVXF2XLForBdbHHG099/15b2I5mNp+7XUscGkq9qdU0VXHRVI0v86OK0PWqlqg4mpHwKqqiJ3L9FXYP+3VDqfK71XiSrTJ5JaKNdNhIQnjlzaW5TajACzBBhRYYIKAJLrZ1UMDB+wDSwMBC2yF4aLOxpUm/gD4soLaFlEBCQWsD/+vYOwitTez7MR2pVvEtOv5sVCYgxX6wj+yMfWef2E/2d2osT8bwc+nikwdaYVUXX98q/vmvysCnC8dD1TkKjvT5NblQg0eyFg1rs6TFr1IN4rd7b86Kjwsp7w57z35hfe/YKfuGFTbbv9UPO6LwFqOn5HgLjFSBNj5VSOIYe9gb5MrR50lrP8xnyI8rTNm38dYwx2NUK7JnQmdqhBPiXScE1yW4LsH1CK5HcB2C6xBcjeBqBNcguAbBrRPcOsFtEdwWwW0T3DbBlQmuTHD7BLdPcIzgWITj4fz6a5maVx7Oa+CPzieXX8LALwi/gat+SFRkP0pZmKM2xgUWinQipEOSfm7+Ch9lB7Yo7Y/SgB9VDO19QqXLfYOjZVI36qGU+UjF+SkVF+XXakxUMrRGFdmxrIaa7EROo6pcmL+KPh13p1yE2JhYURz7GxGqQFCFCPVM7jZJyEi6hnuQv3cGszk4NYI8gl2J9mT8uHiGpSdPrGhjN7Oavr+a2XmQXHsSnmbzcBuWYQXjP4Q12MTxL2EOr+AzfIGvsWzMjnVjXoDOzoSamzB2xV7+A8oIm6E=</latexit>x<latexit sha1_base64="05i1Qsam0Fm9F/5SK3jUPFG0cuo=">AAAHp3ichZXNbtNAEMenLeBSPtrCBYlLRRQEl2gTkECcWqBSq1YlHyRN1VSV12xaK45t2c638gJwBXHgBBIHxGNw4QU49BEQxyJx4cB47chJdlpsJd6d+f3HM7vrXe5aph8wdjIzO3fh4iVt/vLClavXri8uLd+o+E7LM0TZcCzHq3LdF5Zpi3JgBpaoup7Qm9wSu7zxLPTvtoXnm479Mui54qCpH9lm3TT0AE2F7uFSimWYvFbURjZupCC+8s7y3AbU4BU4YEALmiDAhgDbFujg470PWWDgou0ABmjzsGVKv4AhLKC2hZRAQkdrA/+PsLcfW23shzF9qTbwLRb+PFSuQJr9YF/YKfvOvrKf7O+ZsQYyRphLD5880gr3cPH1rdKf/6qa+AzgOFGdo+BIn19TAHV4LGsxsTZXWsIqjSh+u//+tPSkmB7cZZ/YL6zvIzth37BCu/3b+FwQxQ8YPS3HW2CkGrTxaUAKx3iAvVGuHH0DaR3G+ST8pMKRfQ9vE3M8RrUuew50zozQJd7VJbgewfUIrk9wfYLrEFyH4OoEVye4BsE1CG6N4NYIbpPgNglui+C2CK5KcFWC2yO4PYJjBMcUjsfzG65lal55PK+RX51PLr+EkV8Q/iau+oSoyb5KuZijOcFFFor0FdInyTC3cIWPsyObSoejNOLHFYl9SKgsuW9wtEzrxj2UMq9UnD+j4pL8WptTlSRWVbE9kVWi2Z7KaVy1E+dvoM/C3WlHIdanVhTH/rpCFQmqqFDP5W6Tgpyk67gHhXtnNJujUyPKI9qVaE8ujItnWHb6xFIblVwm+yCTKzxMrT6NT7N5uA134B7GfwSrsIHjX8YcBLyBt/BOu6+90CpaNUJnZ2LNTZi4NP0f102Oyw==</latexit>

Security Classification!

Connection
Wij` ⇠ N (0, w /N` )b`i ⇠ N (0,

W0
N0

0
Wjk
xk

k=1

<latexit sha1_base64="BMNGLV1GIjQKiMie4pR3ONbwDZc=">AAAHqXichZVLb9NAEMenLeBSHm3hgsSlIgpCQoo2AQnEqQUqtWpV5UHalL7kNZvUil+ynaRJlC+AxJUeOIHEAfExuPAFOPQjII5F4sKB8dpRHjspthLvzvz+45nd9S73LDMIGTubmp65dPmKNnt17tr1GzfnFxZvbQduwzdE2XAt169wPRCW6YhyaIaWqHi+0G1uiR1efxH5d5rCD0zXeRW2PXFg6zXHrJqGHqKp1DnMHi2kWIbJa0ltZJNGCpIr7y7OrME+vAEXDGiADQIcCLFtgQ4B3nuQBQYe2g6gizYfW6b0C+jBHGobSAkkdLTW8b+Gvb3E6mA/ihlItYFvsfDno3IJ0uwH+8LO2Xf2lf1kfyfG6soYUS5tfPJYK7yj+bd3Sn/+q7LxGcLxQHWBgiN9cU0hVOGprMXE2jxpiao04vjNzul56Vkx3b3PPrFfWN9Hdsa+YYVO87fxuSCKHzB6Wo63wEj70MSnASkc4y72+rly9HWltZfkM+BHFa7s+3ibmOMxqnXZc6E1McIJ8a4TgmsTXJvgOgTXIbgWwbUIrkpwVYKrE1yd4FYIboXg1gluneA2CG6D4CoEVyG4XYLbJThGcEzheDK/0Vqm5pUn8xr71fnk8kvo+wXht3HVD4h92VcpD3M0R7jYQpGBQgYkGeUWrfBhtm9T6WiU+vywYmDvESpL7hscLeO6YQ+lzCsV5ydUXJJfqz1WycCqKjZHshpoNsdyGlZtJfkb6LNwd9pSiNWxFcWxv6pQRYIqKtRLudukICfpKu5B0d4Zz2b/1IjziHcl2pOL4uIZlh0/sdTGdi6TfZTJFR6nlp8np9ks3IV78ADjP4FlWMPxL2MONXgH7+FUe6gVtIr2OkanpxLNbRi5NOMfSyWPcA==</latexit>

N1

K
X

b)

Neural Network = Gaussian process (2)zi1 (x) = b1i +N1
X

Wij1 x1j (x), x1j (x) =

b0j +

j=1

<latexit sha1_base64="LMYTwKSxLTup9cVMeObg85dJ7Rc=">AAAIIXichZXNbtNAEMcnlJISoB9wQeJiERW1AlWbgARCqtQClVq1qtqUtqma1orNpnFix5btpEks9wF4AQ6cAHFAPAYXuCEkDhUHzohjkbhwYLx2yIenxVbi3Znffzyzu95VLF1zXMaOE+eGzg9fSI5cTF26fGV0bHzi6pZj1m2Vb6qmbtp5pehwXavxTVdzdZ63bF40FJ1vK9XHgX+7wW1HM2tP3ZbF94ziQU0raWrRRZM83mrvZ2RtqjktzUpK0JRuSwWnbsheZTbj73urcsaXttHhaRW/ic8Ksneko6NUp4PCglXWpILOS+5UStlncuVfkGoQZBkDMAxY9aWmXE0VbO2g7E7L42k2w8QlxRuZqJGG6FozJ4YWoQDPwAQV6mAAhxq42NahCA7eu5ABBhba9sBDm40tTfg5+JBCbR0pjkQRrVX8P8DebmStYT+I6Qi1im/R8WejUoJJ9pW9YyfsI3vPfrA/p8byRIwglxY+lVDLLXns+fWN3/9VGfh0odxVnaFQkD67JhdK8EDUomFtlrAEVaph/Eb7xcnGw9ykd4u9Zj+xvlfsmH3ACmuNX+rbdZ57idEnxXhzjFSABj5VSOMYe9jr5KqgzxNWP8qny/crTNG38dYwxzKqi6JnwuGpEZrEu5oE1yK4FsG1Ca5NcIcEd0hwJYIrEVyV4KoEN09w8wS3RHBLBLdMcMsElye4PMHtENwOwTGCYzFOieY3WMvUvCrRvIb++Hwq4kvo+DnhN3DVd4mC6McpC3PU+rjQQpFOjHRIMsgtWOG9bMcWp4NR6vC9iq7dJ1S62DcUtAzqej2Uci1W8dopFW+Ir9UYqKRrjStW+rLqalYGcupVrUb5q+jTcXdajRELAytKwf5CjMoRVC5GPRG7TRqygi7hHhTsneFsdk6NMI9wV6I92SAunmGZwRMr3tjKzmTuzmTX76XnHkWn2QjcgJswhfHvwxws4vhvYg7fE8OJ0cRY8k3yU/Jz8kuInktEmmvQdyW//QWPvLeb</latexit>

K
X

0
Wjk
xk

k=1

Wij`
b`i
• >*,'3#'$"8&0&"8&"%'A&$76%+'''''''')"8'4$)+&+
1(x))
à E3$"%'0.34)4$/$%? p(z11 (x), z21 (x), · · · , zN26)+')',*/%$5).$)%&'()*++$)"'8$+%.$4*%$3"'
4?'F&"%.)/'G$,$%'H6&3.&, … Gaussian process• 2&)"='1/&)./?'I• F35).$)"1&'$+<latexit sha1_base64="RTLnOWpg/EhMklPO2JK2XL9+7Vk=">AAAHrnichZXNbtNAEMenLeBSPtrCBYlLRRTEKdoEJBCnFqjUqlWVJiRN1YTI624aK/6S7SRNorwARy4cygUkDojH4MILcOgjII5F4sKB8dqRk3hSbCXenfn9xzO7613uGLrnM3Y+N79w5eo1ZfH60o2bt24vr6zeKXt229VESbMN261w1ROGbomSr/uGqDiuUE1uiAPeehn4DzrC9XTbeu33HFEz1RNLb+ia6qOpwt9UhWHU9fpKimWYvNaSjWzUSEF05e3VhS2owjHYoEEbTBBggY9tA1Tw8D6CLDBw0FaDAdpcbOnSL2AIS6htIyWQUNHawv8T7B1FVgv7QUxPqjV8i4E/F5VrkGY/2Bd2wb6zr+wn+zsz1kDGCHLp4ZOHWuHUl9/eK/75r8rEpw/NWHWJgiN9eU0+NOCZrEXH2hxpCarUwvid/vuL4vNCevCQfWK/sL6P7Jx9wwqtzm/t874onGH0tBxvgZGq0MGnBikc4wH2Rrly9A2kdRjlE/OTClv2Xbx1zLGJalX2bOjOjHBKvOuU4HoE1yO4PsH1Ca5LcF2CaxBcg+BaBNciuA2C2yC4bYLbJrgdgtshuArBVQjukOAOCY4RHEtwPJrfYC1T88qjeQ39yfnk8ksY+QXhN3HVx0RV9pOUgznqE1xooUgvQXokGeQWrPBxdmRL0sEojfhxRWwfEipD7hscLdO6cQ+lzCcqzs+ouCi/VnOqktiaVOxOZBVrdqdyGlftRflr6DNwd9pLEJtTK4pjfzNBFQiqkKBeyd0mBTlJN3APCvbOcDZHp0aYR7gr0Z5cEBfPsOz0iZVslHOZ7ONMbv9Jav1FdJotwn14AI8w/lNYhy0c/5I8qd7BGXxQmFJWako9ROfnIs1dmLiU5j9fH5G6</latexit><latexit sha1_base64="QZeFkeb+TOw6EdD5Qdv1djkx3bs=">AAAHsXichZXNbtNAEMenLeBSPtrCBYlLRRTEKdqkSCBOLVCpVasqTUiTKCnBazatiWNbtpM0sfIC3BEHJBBIHBCPwYUX4NBHQByLxIUD47UjJ/G02Eq8O/P7j2d217vcNnTXY+xkZnbuwsVLyvzlhStXr11fXFq+se9aHUcTJc0yLKfCVVcYuilKnu4ZomI7Qm1zQ5R560ngL3eF4+qW+czr2+KgrR6aelPXVA9NtfLzujCMhq+/GjaWUizD5LWSbGSjRgqiK28tz21CHV6CBRp0oA0CTPCwbYAKLt41yAIDG20H4KPNwZYu/QKGsIDaDlICCRWtLfw/xF4tsprYD2K6Uq3hWwz8OahcgTT7wb6wU/adfWU/2d8zY/kyRpBLH5881Aq7sfj6VvHPf1VtfHpwFKvOUXCkz6/JgyY8lLXoWJstLUGVWhi/O3h7WnxUSPt32Sf2C+v7yE7YN6zQ7P7WPu+JwjuMnpbjLTBSHbr41CCFY+xjb5QrR58vrcMon5ifVFiy7+CtY45HqFZlz4LemRGOiXcdE1yf4PoENyC4AcH1CK5HcE2CaxJci+BaBLdOcOsEt0VwWwS3TXDbBFchuArBVQmuSnCM4FiC49H8BmuZmlcezWvoT84nl1/CyC8IfxtXfUzUZT9J2ZijPsGFFop0E6RLkkFuwQofZ0e2JB2M0ogfV8T2IaEy5L7B0TKtG/dQynyi4vwZFRfl19qeqiS2JhU7E1nFmp2pnMZVu1H+GvoM3J12E8TG1Iri2N9IUAWCKiSop3K3SUFO0k3cg4K9M5zN0akR5hHuSrQnF8TFMyw7fWIlG/u5THY1k9u7n1p7HJ1m83Ab7sA9jP8A1mATx78kz5w38B4+KKtKVXmh8BCdnYk0N2HiUlr/AKT6ky8=</latexit><latexit sha1_base64="SAPDs7BXEVQdJa5ckIkKTIutV50=">AAAH0HichZW/b9NAFMdfW8Cl/GgKCxJLRRSUSqg6ByQQUwtUatUqShPSpmpKZLuX1ooTW7bzWxFi7cDKwAQSA2JlhYWFf4ChfwJiLBILA89nR/lxL8VW4rv3Pt/n9+7Od7pjmZ7P2OnU9MyFi5eU2ctzV65euz4fW7ix49l11+B5w7Zst6BrHrfMGs/7pm/xguNyrapbfFevPA38uw3ueqZde+63HX5Q1Y5qZtk0NB9NpVjSSXZeqCU12Vq6FzRSQaNoHNq+J/rddCnVQ9tSKRZny0xci3JDjRpxiK6MvTCzDkU4BBsMqEMVONTAx7YFGnh474MKDBy0HUAXbS62TOHn0IM51NaR4khoaK3g/xH29iNrDftBTE+oDXyLhT8XlYuQYD/YR3bGvrNP7Cf7OzFWV8QIcmnjUw+13CnNn9zK/fmvqopPH44HqnMUOtLn1+RDGR6JWkyszRGWoEojjN/ovDnLPc4munfZe/YL63vHTtk3rLDW+G182ObZtxg9IcabY6QiNPBpQBzHuIu9fq46+rrC2ovyGfCjClv0XbxNzPEY1Zro2dCcGKFFvKtFcG2CaxNch+A6BNckuCbBlQmuTHAVgqsQ3CrBrRLcBsFtENwmwW0SXIHgCgS3R3B7BMcIjkmcHs1vsJapedWjeQ398nzq4kvo+znhr+KqHxBF0ZcpB3M0R7jQQpGeRHokGeQWrPBhtm+T6WCU+vywYmDvESpL7Bs6WsZ1wx5KmZEqzkyoOCe+1upYJQOrrNgayWqg2RrLaViVjvI30Gfh7pSWiLWxFaVjf02isgSVlahnYreJQ0rQZdyDgr0znM3+qRHmEe5KtCcVxMUzTB0/seTGTmpZvb+c2n4QX3kSnWazcBvuQBLjP4QVWMfxz2MOr+EzfIGvSlZpKS+VVyE6PRVpbsLIpZz8Ay6NnJU=</latexit>

K 1 (x, x0 ) ⌘ E [zi1 (x)zi1 (x0 )]=<latexit sha1_base64="5icDyexRZernUEjoOYXqCCw41x0=">AAAIOnichZXPTxNREMenoBbrD0AvJiamsQFKJOS1mmhMTEAkgUAIFIESFpp96ytsuttdu9vf6T/g0YsHT5p4MP4L3rz4D3jg6o1whEQPHpx9u7U/doq7affNzOc7O/Pe7ltuG7rjMnYcGRq+dPlKdORq7Nr1GzdHx8ZvbTtWuaSJLc0yrFKWq44w9KLYcnXXEFm7JFSTG2KHFxa8+E5FlBzdKr5067bYN9XDop7XNdVFV27s7cpBKlmbqU1NxycV8bqsV2KKqbpHnDcXW8rMXuMgldOTtengOjW9H1eU2OSzuOLoh6aa4wfp+IO2UUWjR10L1LV/6lh8sHTBLyQ3lmCzTB7x8CAVDBIQHOvW+PASKPAKLNCgDCYIKIKLYwNUcPDcgxQwsNG3D030lXCky7iAFsRQW0ZKIKGit4D/h2jtBd4i2l5OR6o1vIuBvxIq4zDBfrDP7Ix9Z1/YCfszMFdT5vBqqeOV+1ph50bf3Nn89V+ViVcXjjqqCxQc6Yt7ciEPT2QvOvZmS4/XpebnrzTenW0+zUw0J9lHdor9fWDH7Bt2WKyca582ROY9Zp+Q8y0wkwIVvGqQwDluotWulWOsKb2toJ4O36uwpF3CU8caj1CtSsuC6sAMNeJeNYKrE1yd4BoE1yC4KsFVCS5PcHmCKxBcgeDmCW6e4JYJbpngVghuheCyBJcluF2C2yU4RnAsxPFgfb1nmVpXHqyrHw+vJ5dvQjsuiLiJT32HUKQdpmysUe/hfA9FOiHSIUmvNu8J72bbvjDtzVKb71Z0/C1CZch9g6OnX9cdoZTroY7XB3S8Kd9Ws6+TjjesWO2pqqNZ7aupW7UW1K9hzMDdaS1ELPY9URztxRCVIahMiHohd5sEpCWdxz3I2zv91Wx/Nfw6/F2JjqS9vPgNS/V/scKD7fRs6uFseuNRYu558DUbgbtwH5KY/zHMwRLO/xbW8DtyLzIVSUa/Rn9GT6KnPjoUCTS3oeeInv8Fo9K/sA==</latexit>

11/30

2
b +

2
1
1 0
E
[x
(x)x
w
i
i (x )] =Security Classification

2
b +

2
0
C(x,
x
)
w

!

NNGP (Neural Network Gaussian Process) (Lee+ 2017)
zj` 1
` 1
• J++*,&'''''''''/)?&.'3*%0*%'''''''''$+'(;=NX̀
zi` (x) = b`i +Wij` x`j (x), x`j (x) = (zj` 1 (x))<latexit sha1_base64="vqdzNAQOmaqadAvlqEJ7xT5gIv4=">AAAHsHichZXNbtNAEMenLeBSPtrCBYlLSxTEhWgTkECcWqBSq1ZVmpA2pa4i22xaE3/JdpImUV6AMxIHBBJIHBCPwYUX4NBHQByLxIUD47UjJ9lpayvx7szvP57ZXe/qnmUGIWPHE5NTFy5eUqYvz1y5eu367Nz8je3AbfoGrxiu5fpVXQu4ZTq8EpqhxauezzVbt/iO3ngW+Xda3A9M13kRdjy+b2sHjlk3DS1E00uVW5a6eF9dzNfmMizHxLUgN/JJIwPJVXTnp1ZBhVfgggFNsIGDAyG2LdAgwHsP8sDAQ9s+9NDmY8sUfg59mEFtEymOhIbWBv4fYG8vsTrYj2IGQm3gWyz8+ahcgCz7yb6yE/aDfWO/2L9TY/VEjCiXDj71WMu92uybW+W/56psfIZwmKrOUOhIn11TCHV4LGoxsTZPWKIqjTh+q/vupPyklO3dZZ/Zb6zvEztm37FCp/XH+LLFS+8xelaMN8dIKrTwaUAGx7iHvUGuOvp6wtpP8kn5UYUr+j7eJuZ4iGpN9FxonxrhiHjXEcF1CK5DcF2C6xJcm+DaBFcnuDrBNQiuQXDLBLdMcGsEt0Zw6wS3TnBVgqsS3C7B7RIcIzgmcXoyv9FapuZVT+Y19svzqYsvYeDnhN/GVZ8SqujLlIc5miNcbKHIQCIDkoxyi1b4MDuwyXQ0SgN+WJHa+4TKEvuGjpZx3bCHUhaliounVFwWX6s9VklqlRUbI1mlmo2xnIZVm0n+Bvos3J02JWJlbEXp2F+RqBJBlSTqudhtMlAQdB33oGjvjGdzcGrEecS7Eu0pRHHxDMuPn1hyY7uQyz/IFbYeZpaeJqfZNNyGO3AP4z+CJVjF8a9gDg68hQ/wUSkoVaWmaDE6OZFobsLIpbz+D1y3kZ4=</latexit><latexit sha1_base64="SoxEhCxzgAAaMSMN6eFb2Lc4u2Q=">AAAHsnichZXNbtNAEMenLeBSPtrCBYlLRRTEhWgTKoE4tUClVq2qNCFtSlMi22xaE3/JdpImVl6AB4ADBz4kDojH4MILcOgjII5F4sKB8dqRk+y0tZV4d+b3H8/srnc11zT8gLHjicmpCxcvKdOXZ65cvXZ9dm7+xrbvtDydV3THdLyqpvrcNGxeCYzA5FXX46qlmXxHaz6N/Dtt7vmGYz8Pui7ft9QD22gYuhqgqdZ7Gda4ad7P9+uv63MZlmPiWpAb+aSRgeQqOvNTq1CDV+CADi2wgIMNAbZNUMHHew/ywMBF2z6EaPOwZQg/hz7MoLaFFEdCRWsT/w+wt5dYbexHMX2h1vEtJv48VC5Alv1kX9kJ+8G+sV/s36mxQhEjyqWLTy3Wcrc+++ZW+e+5KgufARymqjMUGtJn1xRAAx6JWgyszRWWqEo9jt/uvTspPy5lw7vsM/uN9X1ix+w7Vmi3/+hftnjpPUbPivHmGKkGbXzqkMExDrE3yFVDXyis/SSflB9VOKLv4W1gjoeoVkXPgc6pEY6Idx0RXJfgugTXI7gewXUIrkNwDYJrEFyT4JoEt0xwywS3RnBrBLdOcOsEVyW4KsHtEtwuwTGCYxKnJfMbrWVqXrVkXmO/PJ+a+BIGfk74LVz1KVETfZlyMUdjhIstFOlLpE+SUW7RCh9mBzaZjkZpwA8rUnufUJli39DQMq4b9lDKolRx8ZSKy+JrtcYqSa2yYmMkq1SzMZbTsGozyV9Hn4m706ZErIytKA37KxJVIqiSRD0Tu00GCoJu4B4U7Z3xbA5OjTiPeFeiPYUoLp5h+fETS25sF3L5B7nC1mJm6Ulymk3DbbgD9zD+Q1iCVRz/Cubgwlv4AB+VReWFoip6jE5OJJqbMHIp5n+DCJNR</latexit>

j=1<latexit sha1_base64="D+0iXPO1Z5TDOh3tf0nEKC4ghVE=">AAAIF3ichZXfT9NQFMfP+OFw/gD0xcSXxgUDEcjdNNGYkIBKAoEQGAIjFJa13kFZuzZtN7Y15Q/wHzDGJ018MP4RPpgYXn3ggT/B+IiJD/rg6W2X/egB22y955zP9/Sce9tbxdI1x2XsLNHXPzB4JTl0NXXt+o2bwyOjtzYds2qrfEM1ddPOK0WH61qFb7iaq/O8ZfOioeh8Syk/D+JbNW47mll56TYsvmsU9ytaSVOLLroKI2ZzT+a6XtDG6xPSjKRElvRAkp2qUfAOZzL+nrdSCNx+aisMe9qhL09K9dA6ROmkdHyc6rAlzCVbB9p4c88LnFMZX/gnCiNpNs3EIcUHmWiQhuhYNUf7F0CGV2CCClUwgEMFXBzrUAQHzx3IAAMLfbvgoc/GkSbiHHxIobaKFEeiiN4y/u+jtRN5K2gHOR2hVvEuOv5sVEowxk7ZJ3bOTthn9oP9vTCXJ3IEtTTwqoRabhWGX99Z//1flYFXFw7aqksUCtKX9+RCCZ6IXjTszRKeoEs1zF9rvjlff5ob8+6zD+wn9veenbGv2GGl9kv9uMZz7zD7mJhvjplkqOFVhTTOsYdWq1YFY57w+lE9bb5bYQrbxlPDGg9QXRSWCUcXZqgT96oTXIPgGgTXJLgmwR0R3BHBlQiuRHBlgisT3BzBzRHcIsEtEtwSwS0RXJ7g8gS3TXDbBMcIjsU4JVrf4Fmm1lWJ1jWMx9dTEW9CK86JuIFPfZuQhR2nLKxR6+JCD0U6MdIhyaC24AnvZFu+OB3MUovvVLT9PqHSxb6hoKdX1xmhlKuxjlcv6HhdvK1GTydtb1yx3FVVW7PcU1OnaiWqX8WYjrvTSoyY73miFLTnY1SOoHIx6oXYbdKQFXQJ96Bg7wxXs/XVCOsIdyU6kg3y4jcs0/vFig82s9OZh9PZtUfp2WfR12wI7sI9GMf8j2EWFnD+N7CGU/iTGEgMJt8mvyS/JU9CtC8RaW5D15H8/g92B7a7</latexit>• ` K/)?&.'2&)"'$+'IL'5).$)"1&'$+<latexit sha1_base64="LeHFiym2DY6K2nLXvH/KvCo2or0=">AAAHqnichZXNbtNAEMenLeBSPtrCBYlLRRTEAVWbgATi1AKVWrWq0oS0iZKq8rqbxoq/ZDtJkygvwIFrkTiBxAHxGFx4AQ59BMSxSFw4MF47ysdOiq3EuzO//3hmd73LPcsMQsbOZ2bnrly9ps1fX7hx89btxaXlO/uB2/QNUTRcy/VLXA+EZTqiGJqhJUqeL3SbW+KAN15F/oOW8APTdd6EHU8c2vqJY9ZMQw8jU1VY1tFSiq0yea2ojUzSSEFy5dzluU2owjG4YEATbBDgQIhtC3QI8K5ABhh4aDuEHtp8bJnSL6APC6htIiWQ0NHawP8T7FUSq4P9KGYg1Qa+xcKfj8oVSLMf7Au7YN/ZV/aT/Z0aqydjRLl08MljrfCOFt/eK/z5r8rGZwj1oeoSBUf68ppCqMFzWYuJtXnSElVpxPFb3bOLwot8uveQfWK/sL6P7Jx9wwqd1m/j857If8DoaTneAiNVoYVPA1I4xj3sDXLl6OtJaz/JZ8iPK1zZ9/E2Mcc6qnXZc6E9NcIp8a5TgusQXIfgugTXJbg2wbUJrkZwNYJrEFyD4NYJbp3gtghui+C2CW6b4EoEVyK4MsGVCY4RHFM4nsxvtJapeeXJvMZ+dT65/BIGfkH4bVz1Q6Iq+yrlYY7mGBdbKDJQyIAko9yiFT7KDmwqHY3SgB9VDO19QmXJfYOjZVI36qGUOaXi3JSKC/JrtScqGVpVxc5YVkPNzkROo6rdJH8DfRbuTrsKsTGxojj2NxQqT1B5hXotd5sUZCVdwz0o2jvj2RycGnEe8a5Ee7JRXDzDMpMnltrYz65mnqxm956m1l4mp9k83IcH8AjjP4M12MTxL2IOdXgHZ/Bee6zltbJWidHZmURzF8Yu7fgfjLCQCg==</latexit>

K ` (x, x0 ) ⌘ E [zi` (x)zi` (x0 )]=<latexit sha1_base64="dRe8k7P6/VXC7507Sd2oVEolzP4=">AAAIenichZXNbtNAEMcnLdAQPtrCBYlL2qhtoqbVJlSiQkJqgYpWqao0pTRVnES22SRW7DjYzqeVF+AFOHACiQPiMbhwRuLQR0Aci8SlB8ZrR07ibbGVeHfm9x/P7K53paaqmBYhZ6Gp6WvXb8yEb0Zu3b5zd3Zu/t5rU28ZMj2WdVU38pJoUlVp0GNLsVSabxpU1CSVnkj1547/pE0NU9Ebr6xekxY1sdpQKoosWmgqz/3IlASqqvFusruSiCwL9G1LaUcFTbRqkmTvDIRkoc+IshLvJvzmSqIYFYTI8tOoYCpVTSxLpXR0ddjplNJCMuJHKdv9ku1IhYU1YSE1KCsOqUVtwdCiL7ODOElmxoDEICKotGIVhGZNiXviNUeIWSQ4xhW0Gkq1ZhUj5bkYWSfsigYbKa8RA+/K6vPTuyDAG9BBhhZoQKEBFrZVEMHEuwApINBEWxFstBnYUpifwgAiqG0hRZEQ0VrH/yr2Cp61gX0npsnUMr5FxZ+ByigskZ/kCzkn38lX8otcXBrLZjGcXHr4lFwtbZZn3z04+vtflYZPC2q+6gqFhPTVNVlQgU1Wi4K1NZnFqVJ247f778+PnuSW7GXyifzG+j6SM/INK2y0/8ifD2nuA0ZfYuNNMZIAbXzKEMMxtrE3zFVCn82sAy8fnx9X6Kxv4K1gjjVUi6ynQ+fSCF3Ou7ocrsfhehyuz+H6HK7D4TocrsLhKhyuzuHqHG6bw21zuD0Ot8fhMhwuw+HyHC7P4U453CmHIxyOBDjJm19nLfPmVfLm1fUH51NiX8LQTzl+DVe9TwisH6SamKMyxrkWHmkGSJNLOrk5K3yUHdqCtDNKQ35U4dsHHJXK9g0JLZO6UQ9PmQ1UnL2k4iP2tWoTlfjWoGJ/LCtfsz+R06jqwMtfRp+Ku9NBgNiZWFES9ncCVI5D5QLUC7bbxCDN6AruQc7e6c7m8NRw83B3Jb4n7cTFMyw1eWIFG6/T66lH6+nDjdjWM+80C8NDWIQ4xn8MW7CL438McmgzVApVQ7WZi/BiOBFeddGpkKe5D2NXeOMf9ezVvA==</latexit>

2
b +

2
w Ezi` 1 ⇠GP(0,K ` 1 )⇥(zi` 1 (x))• H6$+'&-0&1%)%$3"'1)"'4&'13,0*%&8'4?=
CMD'(;'.&7.&++$3"'CND'@*,&.$1)/')00.3-$,)%$3"'
COD'J")/?%$1)/'+3/*%$3"'#3.'+0&1$#$1'P /$Q&'R&GS
12/30

Security Classification(zi` 1 (x0 ))⇤NNGP (Neural Network Gaussian Process) (2)• T6&"'P $+'R&GS CF63U>)*/ NIIVL'G&&W'NIMXD'=
2 q
K ` (x, x0 ) = b2 + w K ` 1 (x, x)K ` 1 (x0 , x0 )
2⇡ ⇣
⌘
` 1
` 1
` 1
⇥ sin ✓x,x✓x,x
0 + (⇡
0 ) cos ✓x,x0
!
`
0
K
(x,
x
)
`
1
p
✓x,x0 = cos
K ` (x, x)K ` (x0 , x0 )<latexit sha1_base64="OtJcMBUNis/jw/ks9eudqB2RHqY=">AAAI9HichZVLb9NAEMcnPIN5lMIFiYtF1NIIWm0CCISExKtSUVDVB32gbhPZZtNYtWNju2lay1+AL8ABLoA4ICQOfAUufAEkOMINcQSJCwfGa5uk8bTYSrw78/uPZ3bXu7prmX7A2JfCvv0HDh46XDyiHD12/MTQyeFTi76z4RliwXAsx1vWNV9YZlssBGZgiWXXE5qtW2JJX78T+5c6wvNNp/0g2HLFqq2ttc2maWgBmhrDhVqtzoVljXUvds+XldEbKvfNNVtr6PWqekHlTU8zwtS0Wa9GYZW7ZqRw/7EXhLV6GGvHK1EsL/d3z8fhIpXzlXE2cUV0V5VRlbd8VzNEeFnYEQ9MW/gKt0QzGMNwZpsHLRFo/0I0wjijCHNQxvCV6rhKA2WVG45P+xTumWutoKxwrqREDGTeuFjU1kNUqFkmsuD+MYnCrNjMVs6aSZG91zROltgEk5eab1TSRgnSa8YZ3j8FHB6BAwZsgA0C2hBg2wINfLxXoAIMXLStQog2D1um9AuIQEHtBlICCQ2t6/i/hr2V1NrGfhzTl2oD32Lhz0OlCiPsE3vDfrKP7C37zv7sGiuUMeJctvCpJ1rhNoaenJn//V+Vjc8AWj3VHgod6b1rCqAJ12QtJtbmSktcpZHE72w//Tl/fW4kHGUv2Q+s7wX7wj5ghe3OL+P1rJh7htFH5HgLjMShg08DSjjGIfayXHX0hdIapfn0+J0KR/Y9vE3MsYVqTfYc2Nw1Qpd4V5fgtghui+C2CW6b4DYJbpPgmgTXJLh1glsnuFsEd4vg7hHcPYKrEVyN4JYJbpngHhLcQ4JjBMdynJ7Ob7yWqXnV03lN/Pn51OWXkPkF4bdx1fcILvt5ysUczR1cYqFIP0f6JBnnFq/wfjaz5el4lDK+X9GzR4TKkvuGjpZBXb+HUs7kKp7ZpeJ5+bXaA5X0rHnF/R1Z9TT3B3LqV02n+Rvos3B3ms4RkwMrSsf+ZI6aI6i5HHVX7jYlqEq6iXtQvHcms5mdGkkeya5Ee6pxXDzDKoMnVr6xWJ2oXJqozl4u3bydnmZFOAvnYAzjX4WbMIXjvwBG4X3hc+Fr4VuxU3xefFV8naD7CqnmNOy4iu/+Am4oApU=</latexit>• 2*/%$K/)?&.'@@'$+'34%)$"&8'Y*+%'4?',)%.$-',*/%$0/$1)%$3"+Z

13/30

Security Classification

NN-iWMM - Component 1: ARD-NNGP• NNGP [1] defines Gaussian processes equivalent to a ∞-unit NN• ARD [2] allows to determine the dimensionality automatically
20D space

100D space
n

Y1 …

Y100

１
.
.
.
N

n

DR by
１
ARD-NNGP

X1 …

X20

ARD weights■: selecteddimensions

.
.
.

N

intrinsic structure:
3D manifold

Select 3D subspacecorrectly[1] Lee, J. +: Deep neural networks as gaussian processes, International Conference on Learning Representation 2018 48 478–487 (2018)[2] Mackay, D.: Bayesian Non-Linear Modeling for the Prediction Competition, ASHRAE Transaction100(2) 1053–1062 (1994)
14/30

Security Classification

NN-iWMM - Component 2: ∞-GMM• Can infer the number of clusters unlike finite GMM
infinite-GMM

Finite GMM (K=5)

Identify as a single cluster

15/30

Security Classification

Over-separated

NN-iWMM: Generative process• Incorporate ARD-NNGP and ∞~GMM into a latent variable model∞-GMM

×: to be optimized✔:no need to be optimized
NN-iWMM

ARD-NNGP

16/30

Security Classification# of layers

×# of units✔

# of latent dims✔

# of clusters✔

NPDV: Probabilistic model• The joint distribution of NPDV①②

①︓The likelihood of NN-iWMM ②︓Visualization loss
- Note: ②＝Regularization term to infer the posterior of ①
• Two algorithms based on NPDV- NPDV (MF) : NN-iWMM + Matrix Factorization- NPDV (t-SNE) : NN-iWMM + t-SNE
17/30

Security Classification

NPDV: Training algorithm• Employ variational inference to infer the posterior of NPDV-Maximize the ELBO consisting of 4 terms:• Difficulty: (*) cannot be computed analytically⇒ Approximate (*) using reparameterization trick
18/30

Security Classification

Experiments 1: Simulation study

19/30

Security Classification

Data generation setup• Generate 100-dimensional data through NN transformation• Apply 4 methods including NPDV (MF) to recover the original data
100D space
n

Y1

１
・
・
・
Mammoth in
3-dimensional space

20/30

N

Security Classification… Y100

3D space

Visualization results• NPDV (MF) could recover the original mammoth shape accurately• Some body parts (horn, pow) was found through estimated clusters(a) NPDV (MF)21/30

(b) t-SNE(2008)(c) Trimap(2018)

Security Classification(d) PaCMAP(2021)

Experiment 2: Real data experiments@ 20-newsgroups
20-Newsgroups:
Dataset of ~20,000 documents from 20 newsgroups of USENET(Lang 1995)

22/30

Security Classification

Experimental settings• Dataset# of labels

Input

6

TF-IDF 1,000• Methods

D

Remarks
20 labels are converted into 6 parent labels

Remarkst-SNE (2008)-

PaCMAP (2021)-

VSB-DVM[1] + t-SNE Hyperparamters are tuned by Bayesian optimization
NPDV (t-SNE)# of layers is set to 6

• Performance measure: kNN classification accuracy[1] Yang, X.+.: VSB-DVM: An end-to-end Bayesian nonparametric generalization of deep variational Mixture Model,
23/30

Security Classification

kNN classification accuracy• NPDV shows comparable performance to a well-tuned VSB-DVM• Less tuning time than VSB-DBM (NPDV : 8 hours vs. VSB-DMV: 6 days)

kNN accuracy

0.78
0.76
0.74
0.72
0.7
0.68
k=10

t-SNE

24/30

k=20

PaCMAP

VSB-DVM

Security Classification

k=30

NPDV(t-SNE)

kNN classification accuracy• NPDV shows comparable performance to a well-tuned VSB-DVM• Less tuning time than VSB-DBM (NPDV : 8 hours vs. VSB-DMV: 6 days)

kNN accuracy

0.78
0.76
0.74
0.72
0.7
0.68
k=10

t-SNE

k=20

PaCMAP

VSB-DVM

k=30

NPDV(t-SNE)

Achieve comparable acc. to a well-tuned NN with low tuning cost
25/30

Security Classification

Qualitative comparison @ 20 newsgroups• Coloring provides intuitive understanding for the cluster structure• NPDV shows better cluster separation than (b)(a) NPDV(t-SNE)

26/30(b) VSB-DVM

Security Classification(c) t-SNE(2008)(d) PaCMAP(2021)

Relation between estimated clusters and ground truth• The clusters correlate with ground truth labels strongly⇒ NPDV estimates the plausible clusters without label information
Cross table of estimated clusters & ground truth# of docs

27/30

Security Classification

Discovering clusters in Brown corpus
Brown corpus:
Balanced corpus of 1M words of English in various texts(Kucela and Francis 1967)

28/30

Security Classification

Latent cluster discovery @ Brown corpus• Visualize plausible topics and their relationships through NPDV

29/30

Security Classification

Summary• NN-iWMM utilizes the power of neural clustering models
with much less parameters• NPDV integrates NN-iWMM and a visualization method
to render latent coordinates suitable for visualization• NPDV outperforms existing methods, and shows comparable accuracy
to a well-tuned neural clustering models with much less tuning time.
• NPDV estimates plausible clusters without label information

30/30

Security Classification