Bethe free energy, Kikuchi approximations, and belief propagation algorithms

This is anupdatedandexpandedversionof TR2000-26,but it is still in draft form. Beliefpropagation(BP)wasonlysupposedtowork for tree-likenetworksbutworks surprisinglywell in many applicationsinvolving networkswith loops,includingturbo codes.However, therehasbeenlittle understandingof thealgorithmor thenatureof thesolutionsit findsfor generalgraphs. We show that BP canonly converge to a stationarypoint of an approximatefree energy, known astheBethefreeenergy in statisticalphysics.This resultcharacterizes BP fixed-pointsand makes connectionswith variationalapproachesto approximate inference. More importantly, our analysislets us build on the progressmadein statistical physicssinceBethe’ s approximationwasintroducedin 1935.Kikuchi andothershave shown how to constructmoreaccuratefree energy approximations,of which Bethe’ s approximationis the simplest. Exploiting the insightsfrom our analysis,we derive generalizedbeliefpropagation(GBP)versionsof theseKikuchi approximations. These new messagepassingalgorithmscanbesignificantlymoreaccuratethanordinaryBP, at anadjustableincreasein complexity. We illustratesucha new GBPalgorithmon a grid Markov network andshow thatit givesmuchmoreaccuratemarginalprobabilities thanthosefoundusingordinaryBP. This work may not be copiedor reproducedin whole or in part for any commercialpurpose.Permissionto copy in wholeor in partwithoutpaymentof feeis grantedfor nonprofiteducational andresearchpurposesprovidedthatall such wholeor partialcopiesincludethefollowing: anoticethatsuchcopying is by permissionof MitsubishiElectricResearch Laboratories, Inc.;anacknowledgmentof theauthorsandindividualcontributionsto thework; andall applicableportions of thecopyright notice.Copying, reproduction,or republishingfor any otherpurposeshallrequirealicensewith payment of feeto MitsubishiElectricResearchLaboratories, Inc. All rightsreserved. W`p ¦`fkU ̈e(f¡ ̈WÆYÆ§¡ ̄Yt¡Xc©W(pRYPUX\Æ¿W`cp'¡ÅW`ÀMYPp YP=RU ̈W]Ð9c_YP¡ ̈fMÀÇU ̈fkffkp=f}UXe`\(g'« p=Wt[Ép Y`§¡ n f¥SVf}¡ n fTÀÇUXf}fTf}p=fkU ̈e(\c©p §¡ ̄Yt¡Xc§xu ¡ ̈cd£}Y`¬3 n \9§xcd£}§kÊÑ n cd§UXfk§ ̈ =¬©¡M£ n YPU ̄Y`£w¡ ̈fkU ̈cÒ}f;§VS m Í=Ð9f;Z u¿W`cp'¡X§Y`pRZ _YP«`f;§ £wW(p=p=fk£w¡ ̈cW`pR§Ä[Éc©¡ n ¦tYPUXcdYt¡ ̈cW`p5YP¬ YP=RU ̈W'Y`£ n f;§o¡XWY`==UXW]Ð9c©_Yt¡Xfcp9ÀÇfkU ̈fkpR£wf(Ê H W`UXf¥c©_¿W`U ̈¡XY`p(¡X¬W Z=fk§Xå=gP[ n W'§xfZ=fk£wW9Z9cp=e YP¬e`W(U ̈c©¡ n _ c§ fkè' =c¦tYP¬f}p'¡V¡ ̈WS m W`p Y0¬W W` \e`U ̄YP nÆé j9g Q;O]ê°Ê ç ¡ n f}UÉ§ ̈ R£}£}fk§X§ÀÇ R¬Î£kY`§ ̈fk§c©p5£w¬ RZ9f¥£wW`_= =¡ ̈f}U ¦ c§ ̈cW`p =UXW`a=¬f}_§ÅYPpRZ!_fkZ=c£kYP¬Z=cY`e`W`p5§xcd§ é j=gAN gQ]N;êV§ ̈ =e(e`fk§x¡ ̈cp=e ¡ n fÓ§ ̈ R£}£}fk§X§0W`ÀÉS m W(p ¬W'W( \ie`U ̄YP n §¥c§0p=WP¡0¬c Ã Ê h p \e`U ̄YP n c£kYP¬¿_W Z=f}¬3£}Y`p×a¿f0£wW(p'¦(f}U ̈¡ ̈f;Zc©p'¡XWÅ¡ n cd§VÀÇW`UX_ía5f}ÀÇW`UXf¥Z=W`cp=e cp9ÀÇf}UXf}pR£}fÄ¡ n UXW` Re n Y §x =c©¡XY`a=¬f£w¬ R§¡Xf}UXc©pRe WPÀÎp=W9Z9fk§Acp'¡ ̈W ¬dYPUXe`fkU7pRW Z=fk§ é Q]skê°ÊAÑ n UXW` =e n ¡ n c§ ¡ ̈U ̄YPp5§ÀÇW(U ̈_Yt¡Xc©W(p3gÈW(p=f£}Y`p/YP=R¬ Ã Q ÁÇÿ xÿ Ã Áÿ Ã ÁQ Ã [ n fkU ̈f  ̈Áÿ Ã c§T¡ n f¬©W9£}Y`¬ äxfk¦'cdZ9fkpR£wf;åÀÇW`U pRW Z=f þ ̄g (ÁÇÿ  ̈ÿ! Ã cd§T¡ n f£wW(_RYt¡Xc©a=c¬c¡\ _Yt¡XU ̈c©Ða5f}¡[fkf}p p=W9Z9fk§ÄþYPpRZ "5g=YPpRZ c§ÉY p=W(U ̈_YP¬cÒkYt¡Xc©W(p×£wW(pR§x¡XYPp'¡;Ê$#vp [ n YP¡VÀÇW(¬! Ã'& (*),+.'ÁÇÿ  ̈ÿ! Ã XÁÇÿ Ã /.0 21 3546 % / XÁÇÿ Ã ÁGN Ã 7  ̈ÁÇÿ Ã'& (  ̈Áÿ Ã /.0 21 3 % /  ̈Áÿ Ã Áj Ã [ n fkU ̈f ( Z=f}p=W`¡ ̈fk§ÄYÅp=W`UX_YP¬c " _fkY`pR§ÉYP¬¬Îp=W9Z9fk§Vp=fkc©e n a¿W`UXcp=e p=W9Z9f þ ̄g5f}Ð9£}f}9¡*"5ÊT®Mf}UXf % ÅU ̈f}ÀÇf}U ̄§É¡XW¡ n fÅ_fk§X§XYPe`f0¡ n YP¡Tp=W9Z9f þÉ§xfkpRZ=§M¡ ̈WÓp=W9Z9f9" Y`pRZ 7 7cd§K¡ n f¥a5