That Simple Device Already Used by Gauss

From November 1998 until September 1999, Jorma Rissanen and I met on a regular basis. Here I recall some of our stimulating conversations and some of the work that we did together. This work, based almost exclusively on a single page of [12], was left unfinished and has never been published, but it has indirectly had a profound impact on my career. 1 Meet Jorma Rissanen I first met Jorma in November 1998. I had just obtained my Ph.D. in Amsterdam and started a postdoc at Stanford University. These were exciting times: it was at the height of the dot-com boom, and Stanford was right in the middle of it. Since my thesis was all about the MDL Principle, I had suggested that Jorma and I could meet in person during my stay in California. Jorma replied that he would like to. I was delighted, honored but also a bit worried, since I had been forewarned that Jorma was not your “usual” kind of scientist. San Francisco, Category Theory and Statistics I found that, while Jorma was not one for polite small talk, he did like having lots of beer with a small circle of academic friends (for most scientists it seems to be the other way around). He didn’t talk much, but what he said was invariably to the point, direct and frank. During our first meeting, when I told him that I lived in San Francisco since it seemed to me so much nicer than Palo Alto, his immediate reply was “I don’t like San Francisco”. Later that day, we talked about science in general, and I was quite surprised to learn that, in his first years at IBM, Jorma had been a serious student of category theory – he even published on it while he was professor in Sweden [15]. He went on to say that he found category theory to be much easier than statistics, the field to which he had made such major contributions. When I asked him why, he said “because much of statistics is nonsense. It is exceedingly hard to teach yourself nonsense! ”. Vintage Jorma: blunt and sharp at the same time. Some fear him for this, others, like me, find Jorma’s conversation delightfully refreshing. Jorma is, in fact, notorious for his strong opinions about sub-fields of mathematics – when a Ph.D. student once told him that he had spent a lot of time studying complex function theory, Jorma exclaimed (not entirely seriously, I believe) “you’ve wasted your youth! ” Jorma could be as harsh about his own work as he could be about others work – I vividly recall him saying “how could I have been so stupid” when I suggested that the central proof in [13] was much more difficult than was needed. When I told him that I did not quite understand another proof of his, in [11], he told me that he himself only understands it some of the time, “when I have one of my better days”. I am referring to the proof of what is perhaps his most well-known theorem: the central result of [11], which is an extension of the information inequality, but, from a statistical point of view, can also be interpreted as — as Jorma put it, now less modestly but entirely correctly — “a grand Cramer-Rao theorem. ” Soccer and Impounded Cars Our first meeting went quite well, and I was honored that Jorma asked me to visit him again. In the end, I visited him every 3-4 weeks during my year at Stanford at the time, Jorma was still at IBM Almaden, in San Jose, just 40 minutes further down the highway. I usually arrived at 10 in the morning and stayed for the whole day. Between 12 and 2 however, Jorma would often leave to play soccer with a group of friends, something he did three times a week. At the time he was 67, but still very good at it: in his youth Jorma had seriously considered becoming a professional soccer player. In the end, he had decided to pursue his Ph.D. instead – as Jorma told me during one of my visits, he still doesn’t know whether he has made the right choice. When Jorma went out for soccer, I used to have lunch at IBM’s luxurious cafeteria, paid for by Jorma’s card. On one occasion, my car had broken down, and Jorma told me that Nemo, one of his soccer friends, could probably arrange a good new car for me for as little as $ 150: Nemo was a car dealer who took over impounded cars from the police if they hadn’t been picked up for more than a year. I was fascinated: a brilliant scientist who always speaks his mind, played soccer at world-class level and counts impounded car dealers among his friends. During my visits, Jorma and I also did some work together. Rather than actually working on a joint publication, we were both pursuing related but distinct ideas, always discussing our latest progress during our meetings. So what did we work on? 2 Prediction is Coding I learned about the MDL Principle from the monograph “Stochastic Complexity in Statistical Inquiry” [12], the “little green book,” in which Jorma so eloquently puts forward the main ideas underlying the MDL Principle. In Chapter 2, Jorma notes that different research communities have a different understanding of the concept of a “model”. In statistics, a model is usually a family of probability distributions, for example, the Gaussian or normal family. In other fields such as pattern recognition and machine learning, a model is usually a family of deterministic hypotheses or predictors. For example, we may try to find a relationship between a variable X, taking values in some set X , and a variable Y , taking values in Y, by considering a “model” F , which is really a family of deterministic functions f : X → Y. f is then fit to the data (x1, y1), . , (xn, yn), using, for example, the least squares criterion; concrete examples are given below. Jorma claims that, from an MDL perspective, there is no real distinction between the probabilistic and the deterministic type of model: they may both be viewed as defining codes or equivalently, description methods for the data. Statistical inference should then proceed by selecting the model (set of description methods) that leads to the shortest codelength of the data. A Simple Device Already Used by Gauss How, then, should we associate models with codes? For probabilistic models this is obvious: the Kraft inequality tells us that for every probability distribution P , there exists a uniquely decodable code such that, for all outcomes x, the codelength of x is (essentially) equal to − log p(x), p being the mass function of P . The information inequality indicates that this particular code is the only reasonable code that one may want to associate with P [3, Chapter 3]. But what about deterministic predictors f? According to Jorma, we should map them to codes as follows. We first map each f to a conditional distribution pf , defined in such a way that − log pf (y | x) is an affine (linear with constant offset) function of the loss L(y, f(x)) that f makes when predicting y given x. We should then use the code with lengths − log pf . His remarks are worth quoting in full [[12], page 18; mathematical notation slightly adjusted, material between square brackets and emphasis added by myself]: . The two views however can be reconciled by the simple device used already by Gauss for the distributions bearing his name. In fact, for any desired distance measure L(yi, ŷi), [and any predictor f under consideration] define a density function pf (yi | xi) = Ke−L(yi,f(xi)), (1) where K is so chosen that pf becomes a proper density function over the range of yi. Taking the product of these, pf (y | x) = ∏n i=1 pf (yi | xi), over the observed data set, gives the desired conditional density function for sequences [. ] [For example] let the data consist of a binary sequence y = y1, . , yn. With some predictor ŷi as a function of the past observations, let L(yi, ŷi) = 0 if the prediction is correct, i.e., if yi = ŷi, else let L(yi, ŷi) = 1. The desired criterion for the goodness of the predictors is the number of mispredictions in the observed sequence. Picking in (1) the base of the exponential as 2, we get P (1 | ŷi) = K or K/2, depending on what the predicted symbol is. In either case, P (0 | ŷi) = 1−P (1 | ŷi), which makesK = 2/3. With this the number of mistaken predictions made differs only by a constant from the quantity − ∑ i log p(yi | ŷi), which is seen to be an expression in terms of probabilistic model [and corresponds to the codelength of the data according to a particular uniquely decodable code] . As another example, with L(yi, ŷi) = (yi − ŷi), (1) defines a normal density function with mean ŷi and variance 1/2. The induced normal density function for the data pf (y | x), as defined by its negative logarithm, is − log pf (y | x) = n ∑