Segmentation problems

We introduce and study a novel genre of optimization problems, which we callsegmentation problems . Our motivation, in part, is the development of a framework for evaluating certain data mining and clustering operations in terms of their utility in decision-making. For any classical optimization problem, the corresponding segmentation problem seeks to partition a set of cost vectors into several segments, so that the overall cost is optimized. This framework contains a number of standard combinatorial clustering problems as special cases, and many segmentation problems turn out to beMAXSNP-complete even when the corresponding “un-segmented” version is easy to solve. We develop approximation algorithms for two natural and interesting problems in this class — the HYPERCUBE SEGMENTATION PROBLEM and theCATALOG SEGMENTATION PROBLEM — and present a general greedy scheme, which can be specialized to approximate a large class of segmentation problems. Finally, we indicate some connections to local search, game theory, and sensitivity analysis in line ar programming. 1 Motivation: a microeconomic view of data mining Data mining is about extracting interesting patterns from raw data. There is some agreement in the literature on what qualifies as a “pattern” (association rules and correlation s [1, 7, 14] as well as clustering of the data points [8], are som e common classes of patterns sought), but only disjointed dis cussion of what “interesting” means. Most work on data Department of Computer Science, Cornell University, Ithac a NY 14853. Email: kleinber@cs.cornell. edu. Supported in part by an Alfred P. Sloan Research Fellowship and by NSF Faculty Early Career D velopment Award CCR-9701399. This work was performed in part whil e v siting the IBM Almaden Research Center. yComputer Science Division, Soda Hall, UC Berkeley, CA 94720 . christos@cs.berkeley. edu. Research performed in part while vis iting the IBM Almaden Research Center, and supported by NSF grants CCR-96 26361 and IRI-9712131. z IBM Almaden Research Center, 650 Harry Road, San Jose CA 9512 0. pragh@almaden.ibm. com mining studies how patterns are to be extracted automatically, presumably for subsequent human evaluation of the extent in which they are interesting. Automatically focusi ng on the “interesting” patterns has received very limited for mal treatment. Patterns are often deemed “interesting” on the basis of theirconfidence and support [1], information content[24], andunexpectedness [17, 23]. The more promising concept ofactionability— the ability of the pattern to suggest concrete and profitable action by the decision-makers [18, 21, 23] — has not been defined rigorously or elaborated on in the data mining literature. We want to develop a theory of the value of extracted patterns. We believe that the question can only be addressed in a microeconomicframework. A pattern in the data is interesting only to the extent in which it can be used in the decision-making process of the enterprise to increase util ity.1 Any enterprise faces an optimization problem, which can generally be stated as max x2D f(x); whereD is the domain of all possible decisions (production plans, marketing strategies, etc.), and f(x) is theutility or valueof decisionx 2 D. Such optimization problems are the object of study in optimization and microeconomics. 2 The feasible regionD and the objectivef(x) are both comparably complex components of the problem —and classical optimization theory often treats them in a unified way via Lagrange multipliers and penalty functions [3]. Howeve r, from our point of view there is a major difference between the two: the feasible region D is basically endogenous to the enterprise, while the objective function f(x) is a function that reflects the enterprise’s interaction with a multitude of other agents in the market (customers, suppliers, employee s, 1To quote [6],“merely finding the patterns is not enough. You must be able to respond to the patterns, to act on them, ultimately tu rning the data into information, the information into action, and the acti on into value.” 2There is such an optimization problem associated with virtu ally every enterprise; however in real life such problems are so involved and complex, that often nobody knows exactly their detailed formulation . The decisionmakers of the enterprise base their decisions on a very rough , approximate, and heuristic understanding of the nature and behavior of th e underlying optimization problem. The fact that the details of the optim ization problem being solved are nebulous and unknown to the decision-maker s does not make the problem less real —or its mathematical study less us eful. In fact, economic theory during this century has flourished on models such as these, in which the precise nature of the functions involved is esse ntially unknowable; the mathematical insights derived from the abstract p roblem are still valuable as heuristic guides in decision-making. competitors, the rest of the world). That is, at a first approx imation the objective function can be rewritten as f(x) =Xi2C fi(x); whereC is a set of agents or other factors influencing the utility of the enterprise. We shall be calling elements of C “customers. ” We shall be deliberately vague on what they are. There are two different possible views here: On a concrete level, we can think of them as profiles of customers and other relevant agents, about whom we have gathered relevant information by a first stage of data mining; it is this fir t stage that our point of view seeks to influence and direct. A more abstract, but potentially equally useful, point of vie w s that, alternatively, we can also think of the elements of C as rows of the raw table being mined —customers, transactions, shipments, and so on. What makes this relevant to data mining is the following crucial assumption: we assume that the contribution of customeri to the utilityof the enterprise under decision x, fi(x), is in fact a complicated function of the data we have on customeri. Let yi denote the data we have on customer i; then fi(x) is justg(x; yi), some fixed function of the decision and the data. Hence, our problem is to max x2DXi2C g(x; yi): The conventional practice in studying such problems is to replace Pi2C g(x; yi) by g(x; ŷ), where ŷ is someaggregatevalue3 (aggregate demand of a product, aggregate consumer utility function, etc.). Such aggregation is understood to be inaccurate, resulting in suboptimal decisio ns, because ofnon-linearities(non-zero second partial derivatives) in the functiong(x; yi). Aggregation has been tolerated in traditional microeconomics because (1) the computa tional requirements otherwise would be enormous, and (2) it is difficult to obtain the datayi. The point in data mining, in our view, is thatwe now have the computational power and wealth of data necessary to attack the unaggregated optimization problem, to study the intricate ways in which corre lations and clusters in the data affect the enterprise’s opt imal