Talk:Expectation–maximization algorithm

message: I am not an expert (far from it) but there seems to be an error in the final step, calculation of Sigma(t+1) from Sigma(t). Shouldn't there be a square root over the entire term? Two (not very scientific) arguments: stddev (meaning Sigma) is given in the same units as x. We can see that the calculation of Sigma(t+1) is a weighted average (weighted by T) of the square of delta x (meaning it is in x^2 units). So from the point of view of the dimensions of Sigma and x, there should be a square root. The other argument is that when implementing this algorithm (straight from wikipedia), I got several errors, until adding the square root over the average of delta_x^2.
To summarize: Right now the last lines read Sigma_j(t+1) = <some expression> , but I believe they should read Sigma_j(t+1) = sqrt( <the same expression> ). Can someone who knows what he's doing have a look?  —Preceding unsigned comment added by 212.150.57.142 (talk) 11:18, 18 January 2010 (UTC)
replies: text: \Sigma is the covariance matrix, which is the multivariate extension of the variance, and so is in square units. The code in R (programming language) that generated the picture is in the comments at :File:Em old faithful.gif (though I used a built-in function to calculate the empirical covariance). —3mta3 (talk) 23:04, 18 January 2010 (UTC)
thread_title: Error in Gaussian Mixture Example?

message: I don't think this is strictly correct:
M-Step: re-estimate the distribution parameters to [M]aximize the likelihood of the data, given the expected estimates of the unknown variables 
The issue is "given the expected estimates of the unknown variables." It is not quite that simple in some cases.  For example suppose m were missing and had an expectation with respect to current parameter estimates and observed data.  Note now E(log(m)) != log(E(m)).  Now suppose that such a thing appeared in your log likelihood (a more complicated version appears in my work e.g. E(log(det(I)) != log(det(E(I))).  You have to maximize the left of my equation but that is not the same thing as maximizing the right.  Now in the more popular examples of EM this is never an issue!  Or as I like to say to my advisor E(l(m,pi)| obs, current) = l(E(m| obs, current),pi) which is a really nice feature that essentially says find the expectations for the missing data and plug em in to your log likelihood.  This happens for example in binomial models.  
Why didn't I change it directly?  Because I am actually thumbing through literature to make sure this is true "You have to maximize the left of my equation but that is not the same thing as maximizing the right."  Even though the two quantities arn't equal, they may be maximized at the same time given some constraints on the problem.  So dunno...  —Preceding unsigned comment added by 129.123.249.243 (talk) 00:10, 22 January 2009 (UTC)
replies: text: I agree. I think the problem arises because the expectation can be used in the simple linear case used in most examples, which gives the wrong impression for more advanced uses. –3mta3 (talk) 12:06, 22 March 2009 (UTC)
thread_title:  Not strictly correct 

message: Can someone contribute a very simple toy example to show how EM works? The one currently there (a mixture of Gaussians) is too long and is highly mathematical. My feeling is that very few people are going to wade through it.
My usual comment on such matters: Wikipedia is supposed to be an encyclopedia, not a maths textbook! 99.9% of readers of an article will not be experts, but will be trying to understand a new subject. Therefore articles should concentrate on building readers' insight, and not on being pedantically correct and complete. That comes later. Now, this may come as a bit of a shock to some of you, but most people regard complex mathematical formulae as obstacles to be overcome (or ignored).
"There's nothing so complicated that it can't be explained simply" (or something to that effect) -- A. Einstein
--84.9.77.220 (talk) 23:23, 21 February 2008 (UTC)
replies: text: The Gaussian mixture is the canonical example. I've simplified it somewhat, reducing it to a 2 component mixture, and getting rid of the unnecessary Lagrange multiplier business. –3mta3 (talk) 09:58, 23 March 2009 (UTC)

text: That is a bit complicated... The more simple example is the coin-tossing example. Given a set of observations of Heads-Tails, such as HTHHTTTHHHTTH, find the most likely biasing for the coin, where 0.5 would be a "fair" coin that equally give heads and tails results. To do this, one needs to compute the probability of observing that sequence for a certain biasing of the coin (as a formula), then take the derivative of this to find the local maxima. The local maxima will give the most likely (expectation maximised) result. The probability function for observing a particular sequence is simply the binomial distribution pmf formula, with p as the dependant variable (V = \textstyle {n \choose k}\, p^k (1-p)^{n-k}, where n is number of tosses, k is number of heads (or tails, but choose one)). Differentiate the binomial (\frac{dV}{dP}) to locate the maxima and you are done. I haven't written this up, as it has been a while since I have done EM properly. (I believe the answer is p =  k/n .) You can then follow this up, assuming the given EM bias, with a confidence interval computation, to find out the certainty of the coin being biased (confidence interval), and when you are 99% certain, you then tackle the coin-tosser and demand your bet losses back. User A1 (talk) 10:35, 6 July 2010 (UTC)
replies: text:  Of course, this particular example does not display the iterative nature of many EM methods. User A1 (talk) 10:54, 6 July 2010 (UTC)
replies: text: The other problem with your suggestion is that it acts on a non continuous parameter space (eg brand). this is a slightly different problem. User A1 (talk) 12:33, 6 July 2010 (UTC)
thread_title:  Simple example? 

message: I propose the following "simple example". It might be stupid or wrong, but I just want to start somewhere.
Estimation algorithms like expectation maximization are used when one cannot try all possible combinations to optimize a model. This happens in real-world situations when single tests are too expensive or time-consuming and in numerical optimization procedure when there are too many combinations to check.
A simple example application of Expectation Maximization could be the optimization of a formula to recognize cars that contain drugs based on their outlook (brand, colors, etc). The formula specifies the probability that a car contains drugs based on its outlook (brand, color, year). It is a lot too time-consuming to search all cars to optimize the formula, so one can not search all cars. One could sample cars randomly but this might not be the most effective way to optimize the formula. Instead, based on an initial very simplified or a random formula what cars to search, one could first use the simplified formula to calculate the probability for each car.
* Expectation step: One samples from these values, e.g. rather pick cars with higher probabilities than with lower ones. These cars will then be searched. 
* Maximization step: Based on the results, the parameters of the formula will be adapted to fit as well as possible the results of the searches. 
Maximilianh (talk) 10:20, 6 July 2010 (UTC)
replies: text: That is a bit complicated... The more simple example is the coin-tossing example. Given a set of observations of Heads-Tails, such as HTHHTTTHHHTTH, find the most likely biasing for the coin, where 0.5 would be a "fair" coin that equally give heads and tails results. To do this, one needs to compute the probability of observing that sequence for a certain biasing of the coin (as a formula), then take the derivative of this to find the local maxima. The local maxima will give the most likely (expectation maximised) result. The probability function for observing a particular sequence is simply the binomial distribution pmf formula, with p as the dependant variable (V = \textstyle {n \choose k}\, p^k (1-p)^{n-k}, where n is number of tosses, k is number of heads (or tails, but choose one)). Differentiate the binomial (\frac{dV}{dP}) to locate the maxima and you are done. I haven't written this up, as it has been a while since I have done EM properly. (I believe the answer is p =  k/n .) You can then follow this up, assuming the given EM bias, with a confidence interval computation, to find out the certainty of the coin being biased (confidence interval), and when you are 99% certain, you then tackle the coin-tosser and demand your bet losses back. User A1 (talk) 10:35, 6 July 2010 (UTC)
replies: text:  Of course, this particular example does not display the iterative nature of many EM methods. User A1 (talk) 10:54, 6 July 2010 (UTC)
replies: text: The other problem with your suggestion is that it acts on a non continuous parameter space (eg brand). this is a slightly different problem. User A1 (talk) 12:33, 6 July 2010 (UTC)
thread_title: Proposal

message: There is a serious conflict between the phrase "expected value" as it is used in this article vs how it is defined in the expected value article. Jim Bowery (talk) 18:41, 16 October 2008 (UTC)
replies: text: Given that it is hard to pin down people's motivations for liking or disliking an algorithm, perhaps we should drop the sentence from the article. But, based on what I heard at machine learning conferences, EM was preferred due to convergence. I believe that the paper  triggered the shift from gradient descent (used in neural networks) to EM. That paper doesn't have a nice simple M-step like mixture of Gaussians (see equations 27-29, and following), so the popularity wasn't based on simplicity, I think. -- hike395 07:34, 1 May 2004 (UTC)
replies: text: Well, I agree that it's hard to divine motivation. I dropped the sentence that referred to a reason for popularity. Wile E. Heresiarch 22:45, 1 May 2004 (UTC)

text: You note that EM is only guaranteed to maximize likelihood locally.  This point should be made clearly from the start of the article.  And yes, other methods such as gradient ascent or conjugate gradient will work just as well.  (In fact, EM is closely related to gradient ascent and can be regarded as a "self-scaling" version of gradient ascent (i.e., it picks its own stepsize).  In many instances of EM, like the forward-backward and inside-outside algorithms, the E step is actually computing the gradient of the log probability.) 70.16.28.121 01:48, 13 August 2006 (UTC)
thread_title:  Expected value 

message: Can someone tells me the subscripts next to the expected value (the big E) is suppose to mean? I am not familier with math notation. -confused
:The notation is used from probability theory, where the vertical bar in an expression P(A|B) denotes a conditional probability. In the article, the notation is saying given x (the observations) and the current likelihood expression \theta^{(t)}, what is the expected value in the case of the imposed statistical model. The key link is the conditional distribution link above. t may denote the tth iteration of the EM algorithm used. User A1 (talk) 10:34, 26 June 2009 (UTC)
thread_title:  Notation 

message: On 3 January 2007 someone has removed this animated image : 
I thought the image was quite nice, but maybe it shouldn't be at the top of the image as in 
but more to the bottom. So I thought if someone (with a little bit more knowledge about this topic than me) can write a little piece about the application in question we could move it to the "Examples" section. Skaakt 09:56, 9 February 2007 (UTC)
replies: text: This image doesn't really show anything about expectation maximization other than it yields incremental improvements at each step. It may make sense in the context of an application though. Sancho 06:18, 11 February 2008 (UTC)
thread_title:  Animated picture 

message: The section on alpha-EM should be nixed IMO. It is neither 1) notable (the linked paper is not highly cited; it is mostly self-sited in conference reports) 2) clearly explained (e.g. "α-log likelihood ratio" and "α-divergence" are undefined) 3) of value to a general readership. Given the username who added it (Yasuo2) who also wrote the article on its author, I assume this is a vanity edit.  — Preceding unsigned comment added by 108.75.137.21 (talk) 14:31, 6 May 2012 (UTC)
thread_title:  Eliminate alpha-EM? 

message: Your edit here made nonsense of a grammatical (but long and awkward) sentence.  What was your intent?  Dicklyon (talk) 17:15, 1 July 2012 (UTC)
replies: text: You are misreading the (poorly explained) notation ... X and Z refer to the whole dataset at once, not per case. It seems to be saying that in  this "typical case" the potentially observed variables may be either continuous or discrete, but that only the discrete ones might be missing in this supposedly "typical case". But X and Z sort the potentially observable data into two separate sets, those actually observed, X, and those missing, Z. The "typical case" seesm to correspond to the case of a "mixture model" where an observed value (in X, continuous or discrete) may have arisen from one of a number of populations (labelled by a  discrete value which is either in X if observed, or in Z if missing), in unknown proportions, with each sub-population having its own density or mass function, with its own parameters.... and in this "typical case" the label is always misssing: see mixture model where this instance of EM is discussed. In the general case parts or all of Z would be continuous. As to pseudocode, it seems impossible to do this for the general description of EM being covered in this article as the important steps need to be converted first into an implemention by thinking mathematically and statistically, and applying algebra. If you had a specific application of EM in mind, there might be pseudocode in the corresponding article. Melcombe (talk) 12:17, 21 July 2012 (UTC)
thread_title:  minor grammar corrections? 

message: This article sounds like written by mathematicians for mathematicians. Learning curve is too steep. Since the algorithm is used by widespread computer library like OpenCV, it would be useful to add concrete, practical and simple examples. -- O.C.
The EM solution to the Gaussian mixture model, presented here, is about as simple as it gets. The only simplification that is possible is to use univariate instead of multivariate normal RV, but I suspect you would still have the same objections. Fact of the matter is, the EM algorithm is conceptually difficult. Without at least a year or two of mathematical statistics under one's belt, I don't think it's reasonable to expect to develop a deep intuitive understanding of the EM algorithm in its general form. This is my general reaction to the preponderance of "this math article is too hard" comments about technical topics on wikipedia. An explanation of the EM algorithm (and many other technical topics) *not* in mathematical terms would be so vague that it would be essentially meaningless.  (I'm not an editor, but those are my two cents.) Hawthoreus (talk) 21:33, 22 January 2014 (UTC)
thread_title:  Accessibility 

message: Shouldn't the \tau_j be omitted from the fourth indented equation, section 12.1, since the likelihood is conditional on observing \mathbf{z}? This would make sense since f(x_i;z_i,\theta) = \sum_{j=1}^2 \mathbb{I}(z_i=j) \phi(x_i;\mu_j,\sigma_j), with \phi the normal density function with given parameters \mu_j and \sigma_j. With \tau_j it isn't a true density. Hawthoreus (talk) 21:15, 22 January 2014 (UTC)
thread_title:  Incorrect formula in Gaussian Mixture Example? 

message: The statement "that the derivative of the likelihood is (arbitrarily close to) zero at that point, which in turn means that the point is either a maximum or a saddle point" is flagged with a "citation needed," but this property of smooth functions is, I believe, covered in secondary school calculus classes. I think that the "citation" flag detracts from the discussion for no practical effect, and should be removed.
PCT4671 (talk) 03:01, 18 March 2014 (UTC)
replies: text: An inflection point is where the second derivative is zero.  Where the derivative is zero, it's a min or max or saddle point.  A ref for the contents of that paragraph would be useful, so we could rewrite it more clearly, and back up the "in fact it can be proven that in this particular context ..." whatever it's trying to say.  Dicklyon (talk) 03:10, 18 March 2014 (UTC)
thread_title:  Why is a "citation needed" for the statement that the derivative is zero at a inflection point? 

message: Apparently, in response to a previous comment, the Alternatives section was re-written, but now there are no links to information about the alternatives. 
I would be grateful if the section either hyperlinked the alternatives to other articles in Wikipedia, or provided a citation to find references.
Thanks
Gwis999 (talk) 17:27, 24 January 2019 (UTC)
thread_title:  References for Alternatives section