On the genus of the star graph Sarmad Abbasi Department of Computer Science , Rutgers University , Piscataway NJ 08855 . Abstract The star graph Sn is a graph with Sn the set of all permutations over f1 ; : : : ; ng as its vertex set ; two vertices ss1 and ss2 are connected if ss1 can be obtained form ss2 by swapping the first element of ss1 with one of the other n ? 1 elements . In this paper we establish the genus of the star graph . We show that the genus , gn of Sn , is exactly equal to n ! -LRB- n ? 4 -RRB- = 6 +1 by establishing a lower bound and inductively giving a drawing on a surface of appropriate genus . 1 Introduction In -LSB- 1 -RSB- Akers , Harel and Krishnamurthy introduced the star graph as a computational network for parallel computing . It was also shown that the star graph has many properties which are desirable for practical networks ; such as low diameter , low degree , symmetry and low fault diameter . They also designed efficient routing algorithms on the star graph . Since then some work has been done to design efficient parallel algorithms for the star graph -LSB- 6 -RSB- . In this paper we are concerned with the genus of this graph . Akers , Harel and Krishnamurthy -LSB- 1 , 2 -RSB- conjectured that the genus of the star graph is n ? 3 . We show that this conjecture is far from the truth : in fact the genus , gn of Sn , is given by gn = n ! -LRB- n ? 4 -RRB- 6 + 1 : 2 Cayley graphs Given ? a group and ae1 ; ae2 ; : : : ; aek a set of generators for ? , the Cayley graph G -LRB- ? ; ae1 ; : : : ; aek -RRB- is obtained by taking ? as the vertex set ; two vertices ss1 ; ss2 are connected if and only if ss1 = aej ss2 for some j . Many well known interconnection networks can be expressed as Cayley graphs . For example , we can realize the n-cube as a Cayley graph by taking ? as the product of n copies of Z2 , that is , ? = nM i = 1 Z2 : B A C 4321 132 312213 123 321231 34212431 14234123 31242134 4213 2143 314241321 432 43121342 2413 1243 D D B 3412 A 1234 4231 C 32412341 13242314 3214 Figure 1 : Star Graphs S3 and S4 The generators of ? are given by -LRB- 1 ; ; ; : : : ; 0 -RRB- ; -LRB- 0 ; 1 ; ; : : : ; 0 -RRB- ; : : : ; -LRB- 0 ; ; ; : : : ; 1 -RRB- . In -LSB- 2 -RSB- it has been suggested that in general Cayley graphs make a good choice for interconnection networks . We want to study Cayley graphs of familiar groups . Therefore , Sn the group of permutations of f1 ; : : : ; ng is a natural choice . We shall restrict our attention to Cayley graphs obtained from a set of transpositions as generators of Sn . Given a set of transpositions T , we can construct a transposition graph GT on n vertices f1 ; : : : ; ng , such that fi ; jg is an edge in GT if and only if -LRB- i ; j -RRB- 2 T . The following theorem , first given by Cayley himself , characterizes when T is a minimal set of generators for Sn . Theorem 2.1 -LRB- Cayley -RRB- A set of transpositions T is a minimal set of generators for Sn if and only if the transposition graph GT is a tree spanning f1 ; : : : ; ng . The star graph Sn is a Cayley graph obtained by taking ? = Sn , the group of all permutations of f1 ; 2 ; : : : ; ng with set of generators Tn = f -LRB- 1 ; 2 -RRB- ; -LRB- 1 ; 3 -RRB- ; : : : ; -LRB- 1 ; n -RRB- g. Thus there is an edge between two permutations ss1 and ss2 , if ss2 can be obtained from ss1 by swapping the first element of ss1 with one of the other n ? 1 elements . The transposition graph GTn can be drawn to look like a star -LRB- This explains the reason for the name \ star graph '' -RRB- . The star graph Sn is n ? 1 regular with n ! vertices and n ! -LRB- n ? 1 -RRB- = 2 edges . Figure 1 shows the star graphs S3 and S4 . We have labeled the vertices by appropriate permutations written in straight line notation -LRB- In straight line notation a permutation ss is written as -LRB- ss -LRB- 1 -RRB- ; ss -LRB- 2 -RRB- ; : : : ; ss -LRB- n -RRB- -RRB- . -RRB- which we will use henceforth . Note that S4 consists of four copies of S3 , where the i-th copy of S3 is obtained by considering vertices with ss -LRB- 4 -RRB- = i . We can generalize this decomposition as follows . Let S in be the subgraph of Sn induced by all the permutations ss with ss -LRB- n -RRB- = i . Then it is easy to see that S in is isomorphic to Sn ? 1 . Hence Sn consists of n copies of Sn ? 1 . In order to understand the genus of the star graph we will have to further decompose Sn into smaller star graphs . For every ss = -LRB- ss -LRB- 1 -RRB- ; : : : ; ss -LRB- n -RRB- -RRB- , we write ss = -LRB- ffss ; ss ; ss ; ffiss -RRB- , where ffss = -LRB- ss -LRB- 1 -RRB- ; ss -LRB- 2 -RRB- -RRB- , ss = ss -LRB- 3 -RRB- , ss = -LRB- ss -LRB- 4 -RRB- ; : : : ; ss -LRB- n ? 1 -RRB- -RRB- , and ffiss = ss -LRB- n -RRB- . Hence we view every permutation ss as illustrated below : ss = -LRB- ffss z -RCB- | -LCB- ss -LRB- 1 -RRB- ; ss -LRB- 2 -RRB- ; ss z -RCB- | -LCB- ss -LRB- 3 -RRB- ; ss z -RCB- | -LCB- ss -LRB- 4 -RRB- ; : : : ; ss -LRB- n ? 1 -RRB- ; ffiss z -RCB- | -LCB- ss -LRB- n -RRB- -RRB- Let Sn -LSB- = ; ffi = i -RSB- denote the graph induced by all the permutations ss such that ss = and ffiss = i ; that is , Sn -LSB- = ; ffi = i -RSB- consists of all the permutations whose last n ? 3 entries are fixed as -LRB- ; i -RRB- . Similarly , we can define Sn -LSB- = -RSB- etc. . For example , S6 -LSB- = -LRB- 2 ; 6 -RRB- -RSB- is the graph induced by all permutations of the form -LRB- ? ; ? ; ? ; 2 ; 6 ; ? -RRB- . It can easily be seen that for any and i , Sn -LSB- = ; ffi = i -RSB- is isomorphic to S3 and Sn -LSB- = -RSB- is isomorphic to S4 . Lastly , we also note that S in = Sn -LSB- ffi = i -RSB- . 3 The genus of the star graph The genus of a graph is the minimal genus of an orientable surface on which the graph can be drawn without any edge crossings . The genus of a graph may be considered as a measure of non-planarity . Hence if we can draw a graph on a plane without any edge crossings then its genus is 0 . Graphs with high genus are \ extremely non-planar . '' For more formal treatment of this subject see -LSB- 3 , 4 -RSB- . In this section our goal is to prove the following theorem : Theorem 3.1 For n > = 3 , the genus of the star graph Sn is given by , gn = n ! -LRB- n ? 4 -RRB- 6 + 1 : Our method is based on that of Beineke and Harary -LSB- 5 -RSB- , who computed the genus of the n-cube . To show the above result we first establish a lower bound for gn which follows from the Euler characteristic formula . Theorem 3.2 Let gn denote the genus of Sn , then for n > = 3 , gn > = n ! -LRB- n ? 4 -RRB- 6 + 1 : Proof : Consider any optimal embedding of Sn in a orientable surface of genus gn . Let Fn be the number of faces of Sn on the surface . Further , let Vn and En denote the number of vertices and edges in Sn . Notice , that for n > = 3 ; the girth1 of Sn is 6 . Therefore , it follows that each face in the embedding has at least 6 edges . Thus , 6Fn < = 2En : -LRB- 1 -RRB- Now , according to Euler characteristic formula we have , 2gn = En ? Fn ? Vn + 2 : Using inequality -LRB- 1 -RRB- we get , 2gn > = 2 = 3En ? Vn + 2 : 1The girth of a graph G is the length the smallest cycle in G. 21343124 1324 23141234 3214 12434213 2413 142341232 143 432134212431 4231 32412341 341243121342 3142 41321432 -LRB- 1 -RRB- -LRB- 2 -RRB- -LRB- 3 -RRB- -LRB- 4 -RRB- A A B B C CD D E E F F G G Figure 2 : The Genus of S4 is 1 Since Sn has n ! vertices and n ! -LRB- n ? 1 -RRB- = 2 edges , a simple calculation gives us the desired result . To obtain the upper bound we have to do a little more work . We start by observing that n ! -LRB- n ? 4 -RRB- = 6 + 1 is a unique solution to the following recursion : fn = -LRB- ; if n = 3 , nfn ? 1 + n ! 6 ? n + 1 ; otherwise -LRB- 2 -RRB- To prove Theorem 3.1 it suffices to show that g3 = and gn < = ngn ? 1 + n ! = 6 ? n + 1 : S3 is a hexagon ; hence g3 = 0 . However , S4 is more interesting . We want to embed S4 in a surface of genus 1 . We think of a torus -LRB- a surface of genus 1 -RRB- as a square whose opposite sides are identified appropriately . Therefore Figure 2 shows that g4 = 1 by drawing it on a square with opposite sides identified . This figure will play an important role in the proof for the upper bound . We observe that in Figure 3 -LRB- a -RRB- every S4 -LSB- ffi = i -RSB- is a face . Also , every S4 -LSB- = i -RSB- is a face -LRB- see Figure 3 -LRB- b -RRB- -RRB- . Figure 4 is obtained by simply sliding each S i4 to align them in a line . Figure 5 shows that we can think of a torus as four spheres with four handles attached between them . Hence our original drawing of Figure 2 , can be used to obtain a drawing of S4 such that each S i4 is drawn on a sphere -LRB- Figure 6 -RRB- . Four handles have been attached between the spheres to complete 1 23 4 3 3 3 -LRB- a -RRB- -LRB- b -RRB- 4 4 1 1 2 Figure 3 : Each S4 -LSB- ffi = i -RSB- and S4 -LSB- = i -RSB- is a face the drawing . Further , every face in Figure 2 is still a face in Figure 6 . For simplicity , the edges between the different S i4 are not drawn . To embed Sn in an orientable surface we will take n embedded copies of Sn ? 1 in a surface of genus gn ? 1 such that Sn -LSB- = ; ffi = i -RSB- is a face for every and i. For every , we will then use four handles as in Figure 6 to connect the rest of the edges in Sn -LSB- = -RSB- . Therefore by attaching a total of n ! = 6 handles we will obtain a drawing of Sn . The next lemma and recursion -LRB- 2 -RRB- , tell us that n ! = 6 handles are sufficient for our purpose . Lemma 3.1 Let M1 ; : : : ; Mn be orientable surfaces , all of genus g . If we add k > = n ? 1 handles between them to make a connected orientable surface M , then the genus of M is ng + k ? n + 1 . Proof : Consider two disconnected surfaces of genus g1 and g2 . If the two surfaces are connected by a handle then the resulting surface has genus g1 + g2 . Further , if we add a handle to a connected surface we increase its genus by one . Now the result follows from noting that if we attach k handles between n surfaces one by one to obtain M , then exactly n ? 1 handles will be used to connect disconnected surfaces . Comparing the above lemma and recursion for n ! -LRB- n ? 4 -RRB- = 6 + 1 ; we notice that if we start with Sn and embed every Sn ? 1 inductively in n surfaces of genus gn ? 1 then we are allowed A AA A B C D B C EFG EFG Figure 4 : Another drawing of S4 321 Figure 5 : The torus shown as four spheres with four handles attached to attach n ! = 6 handles between the n surfaces . Call an edge in Sn old if it connects two vertices in S in for some i. Further , an edge is new if it is not old . Hence all the new edges of Sn connect some vertex in S in to some vertex in Sjn for i < > j. For example , the thin edges in Figure 2 -LRB- a -RRB- are the new edges of S4 . The following lemma gives us a nice drawing of S4 provided we have a nice drawing of its old edges . Lemma 3.2 Assume we have a drawing of S4 which contains the old edges of S4 on some -LRB- not necessarily connected -RRB- surface M . Further , in this drawing S i4 is a face for every i . Then we can obtain a drawing of S4 by attaching four handles to this surface such that , every S4 -LSB- = i -RSB- is a face for every i. Proof : Given a drawing in which every S i4 is a face we can \ inflate '' the surfaces under each S i4 to form spheres over S i4 . Now four handles can be attached to these spheres as shown in Figure 6 to route all the new edges . Since this new Figure is similar to Figure 6 , it can be readily seen that every S4 -LSB- = i -RSB- is a face . Now we are ready to prove our upper bound . The proof is by induction on a slightly stronger statement . Theorem 3.3 For n > 3 there exists a embedding of Sn on a surface of genus n ! -LRB- n ? 4 -RRB- = 6 +1 such that every Sn -LSB- = ; ffi = i -RSB- is a face . 1 2 3 4 Figure 6 : Drawing a S4 on a torus Proof : For n = 4 the result follows from Figure 2 . Assume that n > = 5 ; then by induction we can embed every S in in a surface of genus gn ? 1 such that every Sn -LSB- = ; ffi = i -RSB- is a face . Hence we have a drawing of all the old edges of Sn . Fix a and consider Sn -LSB- = -RSB- which is isomorphic to S4 . The new edges of Sn -LSB- = -RSB- correspond to the thin edges of S4 in Figure 3 -LRB- a -RRB- . Further , for every i , Sn -LSB- = ; ffi = i -RSB- corresponds to S i4 . Since each Sn -LSB- = ; ffi = i -RSB- is a face by Lemma 3.2 we can add all the new edges in Sn -LSB- = -RSB- by attaching only four handles . Notice that every new edge of Sn is a new edge in Sn -LSB- = -RSB- for some unique . Hence , if we repeat this process for all choices of we get a embedding of Sn . Furthermore , in this drawing every S4 -LSB- = i ; = -RSB- is a face . There are ? n4 ? -LRB- n ? 4 -RRB- ! choices for , and each requires 4 handles . We have added 4 ? n4 ? -LRB- n ? 4 -RRB- ! = n ! 6 handles . Hence by Lemma 3.1 the genus of the new surface is ngn ? 1 + n ! = 6 ? n + 1 . Further , from our construction this drawing of Sn has every Sn -LSB- = i ; = -RSB- as a face . To continue the induction on this stronger statement we must obtain a drawing of Sn such that every Sn -LSB- = ; ffi = i -RSB- is a face . Hence by renaming every vertex ss = -LRB- ffss ; ss ; ss ; ffiss -RRB- to -LRB- ffss ; ffiss ; ss ; ss -RRB- we get the desired drawing . Acknowledgments : I would like to thank Daniel Gordon for helping with this project and Andrew Granville for giving me valuable suggestions to improve the presentation . I would also like to thank Rod Canfield for his constant encouragement and guidance . References -LSB- 1 -RSB- S. B. Akers , D. Harel , and B. Krishnamurthy . The star graph : An attractive alternative to the n?cube . In Proceedings of the International Conference on Parallel Processing , pages 393 -LCB- 400 , 1987 . -LSB- 2 -RSB- S. B. Akers and B. Krishnamurthy . A group theoretic model for symmetric interconnection networks . IEEE Trans . Comput. , 38:555 -LCB- 566 , 1989 . -LSB- 3 -RSB- B. Bollobas . Graph Theory . Addison Wesely , Cambridge , Massachusetts , 1987 . -LSB- 4 -RSB- G. Chartrand L. Lesniak Graphs and Digraphs . Wadsworth & Brooks/Cole Mathematics Series , Monterey , California , 1986 . -LSB- 5 -RSB- F. Harary and L. W. Beineke . The genus of the n-cube . Canad . J. Math. , 17 -LRB- 3 -RRB- :494 -LCB- 496 , 1965 . -LSB- 6 -RSB- A. Menn and A. K. Somani . An efficient sorting algorithm for the star graph interconnection network . In Proceedings of the International Conference on Parallel Processing , volume III , pages 1 -LCB- 8 , 1990 .

O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O

cayley graphs
interconnection networks
star graph


On the Strange a Posteriori degeneracy of Normal Mixtures , and Related Reparameterization Theorems Eric S. Ristad Peter N. Yianilos Research Report ? CS-TR-541-96 December , 1996 Abstract This short paper illuminates certain fundamental aspects of the nature of normal -LRB- Gaussian -RRB- mixtures . Thinking of each mixture component as a class , we focus on the corresponding a posteriori class probability functions . It is shown that the relationship between these functions and the mixture 's parameters , is highly degenerate -LCB- and that the precise nature of this degeneracy leads to somewhat unusual and counter-intuitive behavior . Even complete knowledge of a mixture 's a posteriori class behavior , reveals essentially nothing of its absolute nature , i.e. mean locations and covariance norms . Consequently a mixture whose means are located in a small ball anywhere in space , can project arbitrary class structure everywhere in space . The well-known expectation maximization -LRB- EM -RRB- algorithm for Maximum Likelihood -LRB- ML -RRB- optimization may be thought of as a reparameterization of the problem in which the search takes place over the space of sample point weights . Motivated by EM we characterize the expressive power of similar reparameterizations , where the objective is instead to maximize the a posteriori likelihood of a labeled training set . This is relevant to , and a generalization of a common heuristic in machine learning in which one increases the weight of a mistake in order to improve classification accuracy . We prove that EM-style reparameterization is not capable of expressing arbitrary a posteriori behavior , and is therefore incapable of expressing some solutions . However a slightly different reparameterization is presented which is almost always fully expressive -LCB- a fact proven by exploiting the degeneracy described above . ? Both authors are with the Department of Computer Science , Princeton University , 35 Olden Street , Princeton , NJ 08544 . The second author is also with the NEC Research Institute , 4 Independence Way , Princeton , NJ 08540 . The first author is partially supported by Young Investigator Award IRI-0258517 from the National Science Foundation . Email : fristad,pnyg@cs.princeton.edu . 1 Introduction A normal mixture is a finite stochastic combination of multivariate normal -LRB- Gaussian -RRB- densities . That is , a probability density function p on Rd of the form : p -LRB- x -RRB- = k X i = 1 miN?i ; ? i -LRB- x -RRB- where m1 ; : : : ; mk > = with Pki = 1 = 1 , and N?i ; ? i -LRB- x -RRB- denotes the normal density with mean ? i and covariance ? i. Each constituent normal density is referred to as a component of the mixture , and m1 ; : : : ; mk are the mixing coefficients . Normal mixtures have proven useful in several areas including pattern recognition -LSB- 4 -RSB- and speech recognition -LSB- 6 , 2 , 5 -RSB- -LCB- along with vector quantization and many others . Each component of the mixture is thought of as corresponding to some class denoted ! 1 ; : : : ; ! k , and the a posteriori class probabilities p -LRB- ! 1jx -RRB- ; : : : ; p -LRB- ! kjx -RRB- are used to effect classification . This paper considers certain aspects of the relationship between the two faces of a normal mixture , i.e. the mixture itself versus the a posteriori class functions it induces . Section 2 shows that this relationship is not one-to-one and exposes the considerable degeneracy wherein many distinct normal mixtures induce the same class functions . As a positive result of this degeneracy one can search the entire space of class functions without considering all possible mixtures . This is relevant to problems whose solution depends only on the mixture 's second face , i.e. its induced class functions . An example of such a problem is that of maximizing the a posteriori class probability of a labeled training set . Here the objective is to predict the correct labels , not model the observation vectors themselves . Section 3 shows that such search problems may almost always be carried out using reparameterizations we refer to as emphasis methods that are motivated by the simple and intuitive expectation maximization -LRB- EM -RRB- algorithm -LSB- 1 , 3 , 7 -RSB- for unsupervised maximum likelihood -LRB- ML -RRB- parameter estimation . The reparameterized problem can not express an arbitrary mixture but nevertheless , because of degeneracy , can express a mixture that induces optimal class functions with respect to the prediction of training labels . To clarify these problems and their relationship to the notion of emphasis we begin with review . Given an unlabeled training set the EM algorithm iteratively improves -LRB- in the ML sense -RRB- any mixture -LCB- unless it is already locally optimal . The improved means and covariances are given by a simple weighted averaging computation . Given data values s1 ; : : : ; sn and a normal mixture M , one begins by computing p -LRB- ! ijsj ; M -RRB- ; 81 < = i < = k ; 1 < = j < = n. Conceptually the result is a k ? n table of nonnegative values . The rows of this table are then normalized so that their sum is one . Each row then corresponds to a convex combination of the samples . The entries along a row are thought of as weights attributed to the sample . Each improved mean vector ? i is merely the corresponding weighted average -LRB- convex combination -RRB- of the sample vectors . Each improved covariance is the sample convex combination of outer products -LRB- sj ? ? i -RRB- -LRB- sj ? ? i -RRB- t . The improved mixing parameters are obtained by normalizing the vector consisting of the table row weights prior to their normalization . This process is repeated in an elegant cycle , i.e. each table induces a mixture that gives rise to a new table , and so on . We view this table as reparameterizing the problem of searching for an ML mixture . The point is that one might instead have used a direct parameterization consisting of the individual means , covariance matrices , and mixing parameters -LCB- and employed standard constrained optimization methods Decision boundary m 1 m 2 Figure 1 : A even mixture of two normal densities in R2 , each with identity covariance . The two classes are separated by a linear decision boundary . from numerical analysis . We loosely say that such a table driven scheme is an emphasis reparameterization . Only a locally optimal solution is found through EM but it is important to realize that the global optimum is a stationary point of the iteration . This means that it can be expressed via emphasis , i.e. that there exists a table which induces the globally optimal mixture . This is interesting because not all mixtures can be induced from the table . In particular , the induced means must lie within the convex hull of the sample set , and the covariances are similarly constrained . The optimization problem corresponding to the other , a posteriori face of the mixture seeks to maximize the a posteriori likelihood of a labeled training set . That is , to find a model M which maximizes : n Y i = 1 p -LRB- ! -LRB- si -RRB- jsj ; M -RRB- where ! -LRB- si -RRB- is the class label associated with sample sj . This is sometimes called the maximum mutual information -LRB- MMI -RRB- criterion in the speech recognition literature -LSB- 2 -RSB- . We remark that more general forms of this problem may be stated in which the labels are themselves probabilities . Sections 3 and 4 consider the question of whether emphasis-like reparameterizations may be used to attack the more general form of this problem . This is relevant to a common heuristic in machine learning in which one somehow increases the weight of -LRB- emphasizes -RRB- a mistake in order to improve classification accuracy . It is natural to wonder whether such approaches are even capable of expressing the solution . In the case of normal mixtures this paper illuminates the issue considerably . By theorem 2 one can almost always exactly match the a posteriori behavior of an arbitrary mixture by an emphasis method that slightly modifies the EM approach . Section 4 exposes the limitations of EM-style emphasis and shows that it is not in this sense universal . These results depend on a detailed understanding of the degenerate relationship between a normal mixture and its induced class functions . A convenient visualization of these a posteriori class functions , focuses on decision boundaries , i.e. the surfaces along which classification is ambiguous . Imagery like ours in figure 1 , and pages 28 -LCB- 31 of -LSB- 4 -RSB- , suggest an intuitive relationship between mixture component locations , and the resulting a posteriori class structure and decision surfaces . One imagines each mean to be asserting ownership over some volume of space surrounding it . This view is however far from the truth and the a posteriori class structures arising from normal mixtures are far more interesting than these simple examples suggest . Theorem 1 reveals that the a posteriori class behavior of normal mixtures is far stranger and counter-intuitive than is generally understood . It shows that knowledge of a mixture 's a posteriori class structure -LRB- which may be thought of as decision surfaces -RRB- , tells us essentially nothing about the absolute location of the mixture 's means -LCB- or the absolute nature of its covariances . One can easily imagine a normal mixture , and picture its corresponding class structure . Now if one is instead given only this class structure , the theorem says that infinitely many normal mixtures are consistent with it -LCB- and in particular that the means of such a mixture can be located within an arbitrarily small ball , located anywhere in space . Despite the popularity of normal mixtures for classification problems , the precise nature of this highly degenerate relationship between class functions and mixtures does not seem to have been considered in the literature . Part of the reason may be that the simple view in which means dominate some portion of the space around them is exactly the case when all covariances have identical determinant , and uniform mixing coefficients are used . The most common example of this setting consists of nearest-neighbor Euclidean clustering which corresponds is the identity covariance case . In practice , if the determinants are somewhat comparable , and the mixing coefficients nearly uniform , the simple view is almost always valid . But in many cases , such as when estimating mixture parameters , no such constraint is imposed , and the means , covariance matrices , and mixing coefficients are free to assume any values . Here the full range of strange behavior may be expected . 2 A Posteriori Degeneracy Let s1 ; : : : ; sn 2 Rd be a set of observations . Any set of positive weights 1 ; : : : ; n , gives rise in a natural way to a normal density . That is , the normal density having mean : ? = 1 Pni = 1 i n X i = 1 isi -LRB- 1 -RRB- and covariance : ? = 1 Pni = 1 i n X i = 1 i -LRB- si ? ? -RRB- -LRB- si ? ? -RRB- t -LRB- 2 -RRB- which may be thought of as the weighted maximum likelihood -LRB- ML -RRB- model corresponding to the observations . Given k sets of weights , as many normal densities arise . Adding k additional positive mixing weights w1 ; : : : ; wk serves to specify a mixture of these k densities , with the probability of the ith component given by wi = Pki = 1 wi . Thus a total of nk + k weights induce a normal mixture , and we loosely refer to them as emphasis parameters . The well known expectation maximization -LRB- EM -RRB- method may be viewed as a process which accepts as input a normal mixture , and outputs an emphasis parameter set -LCB- from which an improved normal mixture arises . This cycle continues as the algorithm climbs towards a local maximum . The global maximum is a stationary point of this iteration . From our viewpoint , what is interesting here is that it is possible to search for an optimal normal mixture , by searching over the space of values for the nk + k emphasis parameters -LCB- rather than using the canonical parameterization consisting of the unknown mean vectors , covariance matrices , and mixing coefficients . Now the emphasis parameterization above can not express an arbitrary mixture . This is because each induced mean is a convex combination of the observations , and must therefore lie within their convex hull . An immediate corollary to our EM comments above is then that the means of an optimal mixture lie within the convex hull of the observation set . Similarly , not all covariance matrices can be generated by emphasis , and the covariances of an optimal mixture are similarly constrained . Expectation maximization strives to maximize the probability of the observation set . If instead the function to be optimized depends only on the a posteriori behavior of the mixture , it is natural to ask : When can emphasis of an observation set induce a mixture which is class equivalent to an arbitrary one ? Obviously the observation set must satisfy certain orthodoxy conditions such as being of sufficient size and possessing in some sense enough independence . But beyond these issues the fact that many mean vectors and covariance matrices are not expressible by emphasis would seem to present a much larger obstacle . In this section we show that the the relationship between a normal mixture and its class functions is highly degenerate -LCB- so much so that constrained expressiveness of emphasis does immediately rule out an affirmative answer to our question . Examples and discussion follow the main mathematical development . The section ends with a technical convergence-rate result that is needed to establish the next sections ' reparameterization theorem . Definition 1 A finite mixture is a probability function on a measure space X , arising from k conditional probability functions -LRB- components -RRB- as follows : p -LRB- x -RRB- = k X i = 1 p -LRB- x ; ! i -RRB- = k X i = 1 p -LRB- xj!i -RRB- p -LRB- ! i -RRB- where the constants fp -LRB- ! i -RRB- g are referred to as the mixing coefficients . Any such mixture induces k a posteriori class functions : p -LRB- ! ijx -RRB- = p -LRB- xj!i -RRB- p -LRB- ! i -RRB- p -LRB- x -RRB- Two finite mixtures are class-equivalent if they induce the same class functions . The class functions are not independent since they are constrained by : P k p -LRB- ! ijx -RRB- = 1 . We begin with a simple proposition which allows us in what follows , to focus on ratios of conditional probabilities , and ignore constant factors . This will be important since the nonconstant portion of the ratio of two normal densities is a rather simple object . Proposition 1 Let p -LRB- x -RRB- be a k-component mixture , and p0 -LRB- xj ! 1 -RRB- ; : : : ; p0 -LRB- xj!k -RRB- be a collection of strictly positive conditional probability functions . If for some j , there exist constants Ci ; j such that 8i < > j and x 2 X : p -LRB- xj!i -RRB- p -LRB- xj!j -RRB- = Ci ; j ? p0 -LRB- xj!i -RRB- p0 -LRB- xj!j -RRB- -LRB- 3 -RRB- then there exist mixing coefficients p0 -LRB- ! 1 -RRB- ; : : : ; p0 -LRB- ! k -RRB- such that the resulting mixture p0 -LRB- x -RRB- is classequivalent to p -LRB- x -RRB- . proof : We begin by giving a formula for mixing coefficients p0 -LRB- ! 1 -RRB- ; : : : ; p0 -LRB- ! 1 -RRB- , such that : Ci ; j p -LRB- ! i -RRB- p -LRB- ! j -RRB- | -LCB- z -RCB- ki ; j = p0 -LRB- ! i -RRB- p0 -LRB- ! j -RRB- -LRB- 4 -RRB- Relaxing for the moment the restriction Pk ` = 1 p -LRB- ! ` -RRB- = 1 , observe that if p -LRB- ! j -RRB- were 1 , then setting p -LRB- ! i -RRB- = ki ; j ; i < > j , would satisfy the equation . Normalization then yields a solution which does sum to 1 : p0 -LRB- ! j -RRB- = 1 1 + P ` 6 = j k ` ; j p0 -LRB- ! i -RRB- = ki ; j 1 + P ` 6 = j k ` ; j i < > j Multiplying each side of Eq . 4 by the corresponding side of Eq . 3 yields : p -LRB- xj!i -RRB- p -LRB- ! i -RRB- p -LRB- xj!j -RRB- p -LRB- ! j -RRB- = p0 -LRB- xj!i -RRB- p0 -LRB- ! i -RRB- p0 -LRB- xj!j -RRB- p0 -LRB- ! j -RRB- 8i < > j ; x 2 X -LRB- 5 -RRB- The approach above works so long as p -LRB- ! j -RRB- < > 0 . If it is zero , then p may be treated as a k ? 1 component mixture , and the proposition follows by induction . Till now j has been fixed and Eq . 5 is established for i < > j . This equation is however easily extended to any pair i ; ` of indices . Let fi denote p -LRB- xj!i -RRB- p -LRB- ! i -RRB- and f denote p0 -LRB- xj!i -RRB- p0 -LRB- ! i -RRB- . Then : fi f ` = fi fj ? fj f ` = f i f j ? f j f ` = f i f ` Now we may write : p -LRB- ! ijx -RRB- = fi Pk ` = 1 f ` = 1 1 + P ` 6 = i f ` = fi and a corresponding expression for p0 -LRB- ! ijx -RRB- in terms of ff ` g . We have already seen that all ratios fi = f ` = f i = f ` , so P ` 6 = i f ` = fi = P ` 6 = i f ` = f i whence p -LRB- ! ijx -RRB- = p0 -LRB- ! ijx -RRB- and we are done . 2 We now specialize our discussion to mixtures whose components are normal densities . Definition 2 A d-dimensional k-component normal mixture , is a finite mixture of multivariate normal densities : N ? ; ? -LRB- x -RRB- , 1 -LRB- 2ss -RRB- d = 2 ? 1 = 2 ? e ? 12 -LRB- x ?? -RRB- t ?? 1 -LRB- x ?? -RRB- -LRB- 6 -RRB- The mixture 's parameters are then ? = f ? 1 ; : : : ; ? k ; ? 1 ; : : : ; ? k ; p -LRB- ! 1 -RRB- ; : : : ; p -LRB- ! k -RRB- g . The following theorem exposes the large degeneracy in the relationship between normal mixtures , and their induced class functions . Theorem 1 Let p be a d-dimensional normal mixture with k components . For any x 2 Rd and ffl > , there exists a d-dimensional k-component normal mixture p0 , such that : 1 . k ? 0 ? xk < ffl 2 . k ? 0k2 < ffl 3 . p0 and p are class-equivalent where k ? 0k2 refers to the Frobenius/Euclidean matrix norm . proof : Begin by selecting some distinguished component j. By proposition 1 we have only to produce ? 01 ; : : : ; ? 0k and ? 01 ; : : : ; ? 0k such that the ratio conditions of the proposition are satisfied . Since constant factors do not matter , we ignore several portions of the definition of a normal density Eq . 6 . Namely , the leading constant and the constant portion of the exponent once multiplied out . The result is that : N ? ; ? -LRB- x -RRB- / e ? 12 -LRB- xt ?? 1x ? 2 ? t ?? 1x -RRB- The proportional ratio condition of proposition 1 then leads immediately to the following necessary and sufficient conditions : ?? 1 i ? ?? 1 j = ? 0 ? 1 i ? ? 0 ? 1 j ? ti ?? 1 i ? ? tj ?? 1 j = ? 0ti ? 0 ? 1 i ? ? 0tj ? 0 ? 1 j -RRB- 8i < > j -LRB- 7 -RRB- We set ? 0j = x and begin by sketching the rest of the proof . The constraints are satisfied by choosing ? 0 ? 1 j so that each of its eigenvalues is large . Each resulting ? 0 ? 1 i must then be positive definite and will also have large eigenvalues . Both ? 0j and ? 0i then have small norm , satisfying the requirements of the theorem . In this process , it is only ? 0 ? 1 j we are free to choose . Each choice determines the ? 0i , and ? 0i -LRB- since we have already set ? 0j = x -RRB- . In the limit , as we choose ? 0 ? 1 j with increasingly large eigenvalues , k ? 0i ? ? 0jk = k ? 0i ? xk approaches zero whence we can satisfy the theorem 's other condition . Denote by > = -LRB- A -RRB- the largest eigenvalue of positive definite matrix A , and by > = -LRB- A -RRB- the smallest . For matrices A ; B , it then follows easily from the Cauchy-Schwartz inequality that > = -LRB- A + B -RRB- > = > = -LRB- A -RRB- ? > = -LRB- B -RRB- , by writing k -LSB- -LRB- A + B -RRB- + -LRB- ? B -RRB- -RSB- uk < = k -LRB- A + B -RRB- uk + k ? Buk where u denotes any unit length vector . Now the first constraint in Eq . 7 may be written : ? 0 ? 1 i = ? 0 ? 1 j + -LRB- ?? 1 i ? ?? 1 j -RRB- -LRB- 8 -RRB- and we then have : >=-LRB- ? 0 ? 1 i -RRB- > = >=-LRB- ? 0 ? 1 j -RRB- ? >=-LRB- ?? 1 i ? ?? 1 j -RRB- The parenthesized term is constant since we are given both ?? 1 i and ?? 1 j , and by subadditivity of symmetric matrix spectral radii , does not exceed their sum >=-LRB- ?? 1 i -RRB- + >=-LRB- ?? 1 j -RRB- . We can easily choose ? 0 ? 1 j such that >=-LRB- ? 0 ? 1 j -RRB- is arbitrarily large , e.g. c ? I where c is a large positive constant . It follows then that >=-LRB- ? 0 ? 1 i -RRB- will also be arbitrarily large , ensuring that it is positive definite . Next recall that the column norms of a matrix A can not exceed its spectral radius -LRB- operate on each member of the canonical basis -RRB- . In our positive definite setting each is then bounded above by > = -LRB- A -RRB- , so that kAk2 < = p d > = -LRB- A -RRB- . Choosing the smallest eigenvalue of ? 0 ? 1 j and ? 0 ? 1 i to be arbitrarily large , forces >=-LRB- ? 0j -RRB- and >=-LRB- ? 0i -RRB- to be arbitrarily small -LCB- satisfying the theorem 's first condition . We set ? 0j to be equal to x from the statement of the theorem , and the second constraint in Eq . 7 may be rearranged to yield : ? 0i = -LRB- ? 0i ? 0 ? 1 j -RRB- x + -LSB- ? 0i -LRB- ?? 1 i ? i + ?? 1 j ? j -RRB- -RSB- -LRB- 9 -RRB- By our earlier discussion , we may force ? 0i to be arbitrarily close to zero -LCB- forcing the bracketed term in Eq . 9 to approach zero as well . We next show that the first parenthesized term ? 0i ? 0 ? 1 j tends to the identity matrix I as >=-LRB- ? 0 ? 1 j -RRB- ! 1 . This then demonstrates that ? 0i ! x satisfying the theorem 's second condition , and finishes the proof . It is easy to see that ? 0i ? 0 ? 1 j ! I if and only if -LRB- ? 0i ? 0 ? 1 j -RRB- ? 1 = ? 0j ? 0 ? 1 i ! I . Using Eq . 8 this becomes : I + ? 0j -LRB- ?? 1 i ? ?? 1 j -RRB- ! I which is clearly the case since ? 0j ! 0 . 2 To recapitulate , any location for ? 0j , and sufficiently large -LRB- in the > = sense -RRB- ? 0 ? 1 j , will give rise using the proof 's constructions , to values for the other means , covariances , and mixture coefficients , such that a class-equivalent mixture results . The proof goes on to establish that in the limit , everything is confined as required within an ffl-neighborhood . Figure 2 provides a simple one dimensional example of the normal mixture to class function degeneracy . The left pair of Gaussians G1 , G2 both have zero mean . The taller one , G1 corresponds to the first class and has variance 1 = 2 , while G2 has variance 1 and corresponds to the second class . When mixed evenly , the a posteriori probability C1 results . Notice that C1 assumes value 1 = 2 where G1 crosses G2 . The right pair of Gaussians G1 ' , G2 ' have means at 4 and 5 , and variances of 1 = 5 and 1 = 4 respectively . An even mixture of them would result in a class function very different from C1 . But if very little weight -LRB- ss 5:74234 ? 10 ? 5 -RRB- is placed on G1 ' , its mode drops under the graph of G2 ' , and surprisingly , the induced class function is exactly C1 . The parameters of this new mixture follow immediately from the calculations within the proof of theorem 1 . Figure 3 depicts another two class problem , this time in two dimensions . Here various elements of each class are arranged so that they may be perfectly separated by a circular decision boundary . It 0.5 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 C1 G1 G2 G1 ' G2 ' Figure 2 : Two ways to induce a single class function . An illustration of the degeneracy in the relationship between normal mixtures , and their induced class functions . Zero mean normal densities G1 and G2 when mixed evenly induce C1 , the a posteriori probability of G1 . Normal densities G1 ' and G2 ' located in the right portion of the graph do not have zero means , but may be mixed -LRB- in a far from even way -RRB- so that C1 results . Figure 3 : If data points of two colors are imagined to hug the inside and outside of a circular decision boundary , one might think that a Gaussian mixture model capable of separating the classes , would necessarily have component means at the center -LCB- but this is not the case . is clear that we can create such a boundary by modeling each class with a normal density centered at the origin . We may evenly mix these components and set their covariance matrices to be multiples of the identity matrix ; chosen so that the two density 's intersection is the desired circle . This is in some sense the canonical solution but infinitely many others are possible . To begin with , it is not necessary that both covariance matrices be multiples of the identity . It suffices that their difference be such a multiple . It is not necessary that their means be located at the origin . Given a choice of covariance matrices , and the location of one mean , a location for the other may be computed which gives rise to the same boundary . Moreover , by choosing the covariances to be nearly zero , corresponding to highly peaked densities , the two means may be located within an arbitrary ffl-neighborhood anywhere in R2 . This illustrates the degeneracy by focusing only on the decision boundary , but theorem 1 shows that the family of class functions may be matched everywhere . The situation in figure 3 was first contrived in an attempt to disprove our emphasis reparameterization theorem , the subject to which we next turn . The data points are arranged along only part of the circle so that no convex combination of them , can possibly express the origin . That is , the origin is outside of the convex hull of the dataset . At that time we thought that this would prevent such combinations from leading to a mixture capable of inducing the desired circular decision boundary . The revelation of theorem 1 is that the component means of such a mixture can be located anywhere . We have seen that all the ? 0i converge to ? 0j as the latter tends to zero . At the same time , the ? 0i approach ? 0j . The rate of convergence is our next topic . In particular we will show that j ? 0i ? ? 0j j ! quadratically as ? 0j ! 0 . This is formalized in the following proposition which is needed to establish the main result of the next section . We remark that the convergence of each ? 0i to ? 0j is only linear . Proposition 2 k ? 0i ? ? 0jk = O -LRB- > = 2 -LRB- ? 0j -RRB- -RRB- proof : The difference ?? 1 i ? ?? 1 j is constant and is denoted by C. Then : ? 0i ? ? 0j = -LRB- ? 0 ? 1 j + C -RRB- ? 1 ? -LRB- ? 0 ? 1 j -RRB- ? 1 Denoting ? 0 ? 1 j as P for clarity this becomes : -LRB- P + C -RRB- ? 1 ? P ? 1 = -LSB- -LRB- I + CP ? 1 -RRB- P -RSB- ? 1 ? P ? 1 = P ? 1 -LSB- I ? -LRB- I + CP ? 1 -RRB- ? 1 -RSB- -LRB- 10 -RRB- Now clearly I + CP ? 1 ! I with > = -LRB- P ? 1 -RRB- and the eigenvalues of I + CP ? 1 approach unity at this rate . So -LRB- I + CP ? 1 -RRB- ? 1 ! I as well -LCB- also with > = -LRB- P ? 1 -RRB- . Hence kI ? -LRB- I + CP ? 1 -RRB- ? 1k = O -LRB- > = -LRB- P ? 1 -RRB- -RRB- in Eq . 10 . But kP ? 1k = O -LRB- > = -LRB- P ? 1 -RRB- -RRB- as well , so k -LRB- P + C -RRB- ? 1 ? P ? 1k = O -LRB- > = 2 -LRB- P ? 1 -RRB- -RRB- . 2 3 Emphasis Reparameterization The question \ Can emphasis induce a mixture class-equivalent to an arbitrary one ? '' posed at the beginning of the previous section is reasonable since from theorem 1 we know that the means of such a mixture can lie within an arbitrarily small ball located anywhere in space . So confinement to the convex hull of the samples is not necessarily a restriction . Likewise the covariances can be chosen to have arbitrarily small Frobenius norm . The main result of this section is a qualified positive answer to this question . It states that a modified form of emphasis can , for almost all sufficiently large observation sets , induce a mixture which is class equivalent to an arbitrary one . Modifications are needed because even though the means and covariances may be chosen with a great deal of freedom , they are still highly constrained by the conditions of Eq . 7 -LCB- and it is sometimes impossible to generate a mixture which has satisfies these constraints by the form of emphasis employed by EM . The next section expands on the limits of EM-style emphasis . The required modifications consist of a single change to Eq . 2 which becomes : ? = n X i = 1 i -LRB- si ? ? -RRB- -LRB- si ? ? -RRB- t -LRB- 11 -RRB- where the leading normalization has been removed . This allows the scale of the covariance to be adjusted without affecting the mean . Before stating and proving our reparameterization result , a particular matrix is defined whose nonsingularity is a condition of the theorem . Condition 1 Denote by Si the column vector formed by concatenating sisti and si , and let n -LRB- the number of samples -RRB- equal d -LRB- d + 3 -RRB- = 2 . Next form square matrix S by combining these columns . For example , if d = 2 we have : S , B B B B @ s1 ; 1s1 ; 2 s2 ; 1s2 ; 2 s3 ; 1s3 ; 2 s4 ; 1s4 ; 2 s5 ; 1s5 ; 2 s21 ; 1 s22 ; 1 s23 ; 1 s24 ; 1 s25 ; 1 s21 ; 2 s22 ; 2 s23 ; 2 s24 ; 2 s25 ; 2 s1 ; 1 s2 ; 1 s3 ; 1 s4 ; 1 s5 ; 1 s1 ; 2 s2 ; 2 s3 ; 2 s4 ; 2 s5 ; 2 1 C C C C A Our condition is that S be nonsingular . It is not hard to show that this condition is almost always satisfied , a fact formalized by : Proposition 3 Regarding a sample of size d -LRB- d + 3 -RRB- = 2 as a single random variable of dimension d2 -LRB- d + 3 -RRB- = 2 , the set of such vectors which give rise to singular S matrices , has measure zero . proof : Since the matrix consists of monomials of distinct structure , no non-zero linear combination -LRB- over the ring of polynomials -RRB- of its columns or rows vanishes . So its determinant , as a polynomial , is not identically zero . But the roots of a non-zero polynomial form a set of zero measure . 2 Next we simplify our setting , adjust notation , and establish one more proposition before turning to the theorem . Without loss of generality , we may assume that our samples have mean zero . To see this , first let t denote some target mean , where we seek stochastic values such that P isi = t . We may equivalently solve P is0i = t0 , where s0i , si ? E -LSB- S -RSB- and t0 = t?E -LSB- S -RSB- , since P is0i = -LRB- P isi -RRB- ? E -LSB- S -RSB- . The problem of expressing a target covariance is unchanged by coordinate translation , since -LRB- si?t -RRB- = -LRB- s0i ? t0 -RRB- . Notice that this is true even if the are not stochastic . So in what follows , we will drop the prime notation and simply assume that E -LSB- fsig -RSB- = 0 . The mixture we generate will have all of its means located near zero , the mean of the sample set . The distinguished mixture component , identified with index j in the previous section , is located exactly at the origin . The location of the other mean vectors , are thought of as small displacements ffi from the origin . We then write si -LRB- ffi -RRB- to mean si ? ffi , and the S matrix is then parameterized by ffi in the obvious way , and denoted S -LRB- ffi -RRB- . Note that S -LRB- 0 -RRB- is just S as originally defined . Proposition 4 Given nonsingular S , 9c > such that 8ffi satisfying kffik < c , S -LRB- ffi -RRB- is nonsingular . proof : Immediate from the continuity of the ffi parameterization of S. 2 Each value of ffi corresponds to a choice of mean vector , and gives rise to a linear mapping S -LRB- ffi -RRB- . That is , for each choice of ffi , a linear map arises . The proposition tells us that so long as our mixture 's means are kept within distance c of the origin , all associated linear mappings are invertible . The utility of S -LRB- ffi -RRB- is that it allows us to simultaneously express the objectives of obtaining by emphasis a specified mean vector , and covariance matrix . If ? is a nonnegative emphasis vector , ffit is the targeted mean vector , and ? t the desired covariance matrix , then we must solve the system S -LRB- ffit -RRB- ? = ? t : 0 . Here ? t is regarded as a vector which is concatenated -LRB- denoted \ : '' -RRB- with the d-dimensional zero vector . The ? above is nonnegative , but not necessarily stochastic . This may seem to be inconsistent with our definition of the generation of a mean vector by emphasis , because of the normalization it includes . But it is not since P i -LRB- si ? ffi -RRB- = is equivalent to -LRB- 1 = P i -RRB- P i = ffi , which is exactly our definition . Theorem 2 Given any d-dimensional normal mixture M with k components , and s1 ; : : : ; sn 2 Rd such that n > = d -LRB- d + 3 -RRB- = 2 with some subset of size d -LRB- d + 3 -RRB- = 2 satisfying condition 1 , then there exists a k ? n table of nonnegative values f i ; jg , and nonnegative values m1 ; : : : ; mk ? 1 with Pmk < = 1 , such that the normal mixture M generated as described below , is class-equivalent to M . 1 . The mixing parameters of M are m1 ; : : : ; mk ? 1 ; 1 ? Pk ? 1 j = 1 mj . 2 . Each mean ? 0i within M is given by : ? i = 1 Pnj = 1 i ; j n X j = 1 i ; j sj 3 . Each covariance ? 0i within M is given by : ? i = n X j = 1 i ; j -LRB- sj ? ? i -RRB- -LRB- sj ? ? i -RRB- t Also , the number of parameters may be reduced to -LRB- k ? 1 -RRB- d -LRB- d + 3 -RRB- = 2 + d , matching the complexity of the canonical parameterization , by replacing the top - table row , by a single new parameter . proof : Choose some subset of the fsig of size exactly d -LRB- d + 3 -RRB- = 2 that satisfies condition 1 . We disregard the other elements entirely , and therefore simply assume that n = d -LRB- d + 3 -RRB- = 2 . As argued earlier , we may also assume without loss of generality that the fsig have mean zero . Then the distinguished mixture component from theorem 1 , with parameters -LRB- ? 0j ; ? 0j -RRB- , is chosen as follows . Its mean ? 0j is the origin , and its covariance ? 0j is proportional to the average of the element selfouter-products , i.e. : ? 0j = 1 n n X i = 1 sisti This may be thought of as choosing the first row of the table to be 1 = n , and introducing a new scale parameter . The table 's top row elements are no longer parameters , so the total becomes -LRB- k ? 1 -RRB- d -LRB- d + 3 -RRB- = 2 table parameters , plus d ? 1 mixing parameters , plus -LCB- matching the count in the statement of the theorem . As described in the proof of theorem1 , the choice of ? 0j is arbitrary , but ? 0j may need to be scaled down to preserve the positive nature of all matrices . The number of resulting parameters in a direct parameterization -LRB- i.e. consisting of means , covariances , and mixing parameters -RRB- , then matches our count . As ! we know that the ? 0i ! ? 0j = 0 . Our first step is to choose sufficiently small , so that the largest k ? 0ik is smaller than the c of proposition 4 . Next , is possibly reduced further until each covariance matrix ? 0i arising from Eq . 4 is positive . Each ? 0 corresponds to a ffi value in our definition of S -LRB- ffi -RRB- . The reference mean , located at the origin , corresponds to ffi = 0 , and by our construction , arises from weighting the s ` by non-negative values -LCB- in this case uniform ones . Associated with each ? 0i there is also a ? 0i . Recall from our earlier discussion , that our focus is on the equation S -LRB- ? i -RRB- ? = ? i : 0 . Since S -LRB- ffi -RRB- is non-singular for each of the ? 0i , we know that some vector of values satisfies this equation . We have only to show that solutions consisting only of nonnegative values can be found . We will say than a point is representable under some S -LRB- ffi -RRB- , if its inverse image under this mapping consists only of non-negative values . There exists a representable open neighborhood about -LRB- ? 0j : 0 -RRB- under S -LRB- 0 -RRB- , since the inverse image of this point1 is the constant vector ff = n , and consists of strictly positive values . 1The inverse image is a unique point because S -LRB- ffi -RRB- is invertible . Within our bounds on ffi , it uniform-continuously2 parameterizes S -LRB- ffi -RRB- . Hence there exist values c0 and ffl , such that for all ffi with kffik < = c0 , the open ball about -LRB- ? 0j : 0 -RRB- , with radius ffl , is representable under S -LRB- ffi -RRB- . To avoid introducing another symbol , let ffl now denote the maximum such radius .3 Next notice that the space of representable points is convex and includes the origin . So representability of -LRB- ? 0j : 0 -RRB- implies representability for all values . Now if necessary , reduce further so that all the ? 0 are within c0 of the origin . Now let P denote the subspace of nonnegative ? vectors . Let : T , \ kffik < = c0 S -LRB- ffi -RRB- P Subspace T is the portion of the range , representable under any S -LRB- ffi -RRB- such that ffi < = c0 . About -LRB- ? 0j : 0 -RRB- we have seen that there is a representable ball of radius ffl which touches the boundary of T . We now denote this radius by ffl -LRB- -RRB- since we are about to consider its behavior as ! 0 . By our earlier comments , T includes -LRB- ? 0j : 0 -RRB- , and all proportional vectors , in particular as ! 0 . The geometric observation key to our proof is that ffl -LRB- -RRB- can shrink only as fast as itself , since this value represents the shortest distance to some point set , in this case T c . We imagine T to be a tunnel , leading to the origin while constricting , with the boundary of T c forming the tunnel 's wall . Focus now on some ? 0i . While there is a corresponding representable ball about -LRB- ? 0j : 0 -RRB- of radius ffl -LRB- -RRB- , there is no guarantee that ? 0i is within it . As ! 0 , we have seen that the radius of this ball shrinks linearly with . But the distance of ? 0i from ? 0j by proposition 2 shrinks quadraticly with , whence eventually , i.e. for small enough , ? 0i becomes representable . Then is reduced as necessary for for each component of the mixture , so that afterwards , every ? 0 : is representable . 2 Other emphasis theorems are possible . In particular it is much easier to establish a similar result in which the weights may be either positive or negative because then the only requirement is that S -LRB- ffi -RRB- be nonsingular . 4 The Limitations of EM-style Emphasis In the previous section we saw that using a particular form of emphasis reparameterization , the a posteriori behavior of any mixture could be matched . We say that such a sample set and emphasis method is universal . This section is by contrast essentially negative , and begins with an example demonstrating that even in one dimension , EM-style emphasis is not universal . Our example is in R1 , and includes two observation points located at and 1 . The leading normalization factors in Eq . 2,1 amount to enforcing convex combination , so there is only one degree of freedom given our two element observation set . If denotes the weight of the first point , then the second has weight 1 ? . The induced mean is just and the induced variance -LRB- 1 ? -RRB- . Our objective is to generate a two element mixture which is class equivalent to an arbitrary one , so two emphasis parameters 1 and 2 are needed . The two constant differences to be matched -LRB- from Eq . 7 -RRB- are 2and in fact nearly linearly for small ffi . 3A maximum exists because entries in S corresponding to covariance diagonal , are nonnegative , preventing many negative values from being representable . denoted ?? and ?? . That these are independent and unconstrained characteristics an equivalence class follows from the observation that we may , without loss of generality , set one of the target means to zero . The constraints then become : 1 1 -LRB- 1 ? 1 -RRB- ? 1 2 -LRB- 1 ? 2 -RRB- = ?? 1 1 ? 1 ? 1 1 ? 2 = ?? Choose ?? > 0 . From our second constraint we have 1 = 1 ? 1 =[ ? mu + 1 = -LRB- 1 ? 2 -RRB- -RSB- from which it follows that 1 = 2 only when they both are 1 . But in this case ?? = 0 , contradicting our choice . So 1 < > 2 . Now when 2 = 0 , 1 > 0 , so here 2 < 1 . Because they are never equal , and their relationship is continuous , this inequality must hold for any pair satisfying the constraints . But it is easily verified that ????? = 1 = fl1 ? 1 = fl2 whence this quantity is negative . So no pair 1 ; 2 exists which satisfies the constraints given say ?? = 1 and ?? = 2 . We remark that if instead , one is interested in matching the a posteriori class behavior of a mixture at only the given sample points , then the example 's argument does not apply . Indeed it may be verified that this less stringent requirement can be satisfied . This is not entirely surprising since given exactly d -LRB- d + 3 -RRB- = 2 sample points , one has three free parameters , and must match only two . This is interesting , and we conjecture that it is true more generally for d > 1 and k > 2 -LCB- so long as n = d -LRB- d + 3 -RRB- = 2 . In any event , as a result of our arguments at the end of this section , it can not be true for arbitrarily large n . Since we are interested in reparameterizing problems with arbitrarily many samples , we will not consider the matter further in this paper . The example above applies to dimension 1 only , and a particular set of sample points . Moreover , its analytical approach does not generalize easily . We now show that in the general case EM-style emphasis is not universal . Without loss of generality we will assume ? 0j = -LCB- since otherwise , the problem , and any solution thereto , can be translated appropriately . We may also assume the sample points are distinct . Eq . 7 becomes : ?? i = ? 0 ? 1 i ? 0i -LRB- 12 -RRB- where ?? i , ?? 1 i ? i ? ?? 1 j ? j . Our focus is on the simple rearrangement of Eq . 12 : ? 0i ?? i = ? 0i -LRB- 13 -RRB- Independent of the ? 01 ; : : : ; ? 0k we are free to set ? 01 ; : : : ; ? 0k so that the ?? i have arbitrary values . In particular , we will set them so that ?? i = -LRB- CC : : : C -RRB- t where C > 1 is a constant . Now focus on target mixtures such that ?? i , ?? 1 i ? ?? 1 j has value ffI where ! 1 . Then ? 0 ? 1 i = ffI + ? 0 ? 1 j . So independent of the choice of particular ? 1 ; : : : ; ? k , and ? 0j , each ? 0 ? 1 i , where i < > j , will approach ffI . There inverses therefore approach the diagonal matrix 1 = ffI . As ! 1 , we adjust means as necessary so as to maintain ?? i = -LRB- CC : : : C -RRB- t . Then from Eq . 13 it is apparent that ? 0i is very nearly a scaled up copy of the diagonal of ? 0i . Since the off diagonal elements approach zero , it must eventually be the case that k ? 0ik < k ? 0ik . Also , since the sample set is finite , it must eventually be that both are smaller than say 1 = 100th of the smallest distance between sample points . When this happens , the distance from ? 0i to the nearest sample will be at least k ? 0ik . Now the covariance ? 0i is a convex combination of matrices of the form -LRB- s ` ?? 0i -RRB- -LRB- s ` ?? 0i -RRB- t -LCB- and the diagonals are nonnegative . The norm of ? 0i is at least equal to the norm of its diagonal , but the diagonal is just the distance from the sample point to ? 0i . From this it follows that k ? 0ik > = k ? 0ik which presents a contradiction , whence EM-style emphasis is not universal . Our arguments above establish : Theorem 3 In the setting of theorem 2 , altered after the fashion of EM so that each covariance ? 0i is given by : ? i = 1 Pnj = 1 i ; j n X j = 1 i ; j -LRB- sj ? ? i -RRB- -LRB- sj ? ? i -RRB- t there exist target mixtures which may not be generated by emphasis . We have seen that no number of sample points make EM-style emphasis universal . But given enough points , we can certainly identify a particular class equivalence-class , since all functions involved are analytic with a finite number of parameters . So given enough sample points , a reparameterization must be universal in order to match a target distribution at the sample points only . Now we are interested in reparameterizations which apply given an unlimited number of points . Therefore , unlike the modified emphasis reparameterization introduced in the previous section , EM-style reparameterization is simply not expressive enough to safely reparameterize a posteriori optimization problems . 5 Concluding Remarks The a posteriori degeneracy we clarify in section 2 for normal mixtures must be added to the list of remarkable characteristics of Gaussians . We have not considered analogues of theorem 1 for other densities . The reparameterization of section 3 is of mathematical interest , and lends some credibility to approaches which attempt to maximize a posteriori probability by adjusting weights on the available samples -LCB- perhaps according to some error measure . An examination of reparameterization as a primary optimization technique , represents an interesting area for future work . We must however caution that while our proofs are constructive , we have not considered the matter of numerical sensitivity . Indeed , if one attempts to emulate a mixture using means far displaced from their natural locations , the mixing parameters become quite small . In these cases floating point underflow is a virtual certainty . One should not interpret section 4 to say that EM-style emphasis will not work in practice for many problems . It merely exposes its theoretical limitations . Acknowledgments We thank Leonid Gurvits for several helpful discussions -LCB- and in particular for pointing out the simple factorization in proof of proposition 2 , and simplifying our proof of proposition 3 . We also thank Joe Kilian for several helpful discussions which contributed to our understanding of degeneracy and class equivalence . References -LSB- 1 -RSB- L. E. Baum and J. E. Eagon , An inequality with application to statistical estimation for probabalistic functions of a Markov process and to models for ecology , Bull . AMS , 73 -LRB- 1967 -RRB- , pp. 360 -LCB- 363 . -LSB- 2 -RSB- P. F. Brown , Acoustic-phonetic modeling problem in automatic speech recognition , PhD thesis , Carnegie-Mellon University , 1987 . -LSB- 3 -RSB- A. P. Dempster , N. M. Laird , and D. B. Rubin , Maximum-likelihood from incomplete data via the EM algorithm , J. Royal Statistical Society Ser . B -LRB- methodological -RRB- , 39 -LRB- 1977 -RRB- , pp. 1 -LCB- 38 . -LSB- 4 -RSB- R. O. Duda and P. E. Hart , Pattern Classification and Scene Analysis , John Wiley & Sons , Inc. , 1973 . -LSB- 5 -RSB- X. D. Huang , Y. Ariki , and M. A. Jack , Hidden Markov Models for Speech Recognition , Edinburgh University Press , 1990 . -LSB- 6 -RSB- L. R. Rabiner , B. H. Juang , S. E. Levinson , and M. M. Sondhi , Recognition of isolated digits using hidden markov models with continuous mixture densities , AT&T Technical Journal , -LRB- 1985 -RRB- . -LSB- 7 -RSB- R. A. Redner and H. F. Walker , Mixture densities , maximum likelihood , and the EM algorithm , SIAM Review , 26 -LRB- 1984 -RRB- , pp. 195 -LCB- 239 .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O

em
expectation maximization
mmi
normal mixtures

gaussian mixtures
maximize mutual information
supervised learning