Approximation Lasso Methods for Language Modeling
Lasso is a regularization method for parameter estimation in linear models. It optimizes the model parameters with respect to a loss function subject to model complexities. This paper explores the use of lasso for statistical language modeling for text input. Owing to the very large number of parameters, directly optimizing the penalized lasso loss function is impossible. Therefore, we investigate two approximation methods, the boosted lasso (BLasso) and the forward stagewise linear regression (FSLR). Both methods, when used with the exponential loss function, bear strong resemblance to the boosting algorithm which has been used as a discriminative training method for language modeling. Evaluations on the task of Japanese text input show that BLasso is able to produce the best approximation to the lasso solution, and leads to a significant improvement, in terms of character error rate, over boosting and the traditional maximum likelihood estimation.

title: A rate of convergence result for a universal D-semifaithful code
abstract: The problem of optimal rate universal coding is considered in the context of rate-distortion theory. A D-semifaithful universal coding scheme for discrete memoryless sources is given. The main result is a refined covering lemma based on the random coding argument and the method of types. The average codelength of the code is shown to approach its lower bound, the rate-distortion function, at a rate O(n/sup -1/log n), and this is conjectured to be optimal based on a result of A.J. Pilc (1968). Issues of constructiveness and universality are addressed.

title: The Minimum Description Length Principle in Coding and Modeling
abstract: We review the principles of minimum description length and stochastic complexity as used in data compression and statistical modeling. Stochastic complexity is formulated as the solution to optimum universal coding problems extending Shannon's basic source coding theorem. The normalized maximized likelihood, mixture, and predictive codings are each shown to achieve the stochastic complexity to within asymptotically vanishing terms. We assess the performance of the minimum description length criterion both from the vantage point of quality of data compression and accuracy of statistical inference. Context tree modeling, density estimation, and model selection in Gaussian linear regression serve as examples.

title: Wavelet thresholding via MDL for natural images
abstract: We study the application of Rissanen's (1989) principle of minimum description length (MDL) to the problem of wavelet denoising and compression for natural images. After making a connection between thresholding and model selection, we derive an MDL criterion based on a Laplacian model for noiseless wavelet coefficients. We find that this approach leads to an adaptive thresholding rule. While achieving mean-squared-error performance comparable with other popular thresholding schemes, the MDL procedure tends to keep far fewer coefficients. From this property, we demonstrate that our method is an excellent tool for simultaneous denoising and compression. We make this claim precise by analyzing MDL thresholding in two optimality frameworks; one in which we measure rate and distortion based on quantized coefficients and one in which we do not quantize, but instead record rate simply as the number of nonzero coefficients.

title: Adaptive wavelet thresholding for image denoising and compression
abstract: The first part of this paper proposes an adaptive, data-driven threshold for image denoising via wavelet soft-thresholding. The threshold is derived in a Bayesian framework, and the prior used on the wavelet coefficients is the generalized Gaussian distribution (GGD) widely used in image processing applications. The proposed threshold is simple and closed-form, and it is adaptive to each subband because it depends on data-driven estimates of the parameters. Experimental results show that the proposed method, called BayesShrink, is typically within 5% of the MSE of the best soft-thresholding benchmark with the image assumed known. It also outperforms SureShrink (Donoho and Johnstone 1994, 1995; Donoho 1995) most of the time. The second part of the paper attempts to further validate claims that lossy compression can be used for denoising. The BayesShrink threshold can aid in the parameter selection of a coder designed with the intention of denoising, and thus achieving simultaneous denoising and compression. Specifically, the zero-zone in the quantization step of compression is analogous to the threshold value in the thresholding function. The remaining coder design parameters are chosen based on a criterion derived from Rissanen's minimum description length (MDL) principle. Experiments show that this compression method does indeed remove noise significantly, especially for large noise power. However, it introduces quantization noise and should be used only if bitrate were an additional concern to denoising.

title: Wavelet thresholding for multiple noisy image copies
abstract: This correspondence addresses the recovery of an image from its multiple noisy copies. The standard method is to compute the weighted average of these copies. Since the wavelet thresholding technique has been shown to effectively denoise a single noisy copy, we consider in this paper combining the two operations of averaging and thresholding. Because thresholding is a nonlinear technique, averaging then thresholding or thresholding then averaging produce different estimators. By modeling the signal wavelet coefficients as Laplacian distributed and the noise as Gaussian, our investigation finds the optimal ordering to depend on the number of available copies and on the signal-to-noise ratio. We then propose thresholds that are nearly optimal under the assumed model for each ordering. With the optimal and near-optimal thresholds, the two methods yield similar performance, and both show considerable improvement over merely averaging.

title: Curve-based representation of moving object trajectories
abstract: In recent years, many emerging database applications deal with continuously moving data objects - each data object moves continuously and frequently reports its current location, moving direction, and speed to the database server. A database server for these applications keeps track of the trajectories of individual moving objects and processes queries referring to the past or future trajectories. Related techniques view a moving object trajectory as a sequence of connected line segments. However, most natural moving objects, such as airplanes, vessels, and vehicles, draw a smooth trajectory with no angles. This paper presents our curve-based trajectory representation models. The presented results show that the curve-based models provide much more accurate trajectories than the line-based models when we have the same amount of data (same number of reported points). In other words, the curve-based models require a smaller amount of data while providing the same accuracy in trajectory representation.

title: Community-based service location
abstract: lthough both computing and communications capabilities are improving exponentially, there has been a more rapid increase in available bandwidth than in computing power over recent years . Naturally enough, while the network-related improvements create new possibilities for applications, the improvements and especially their differential, render obsolete traditional ways of thinking about telecommunications and computing. There is an ongoing debate occurring among two main factions within the telecommunications industry [1, 4]. On one side are traditional telephone companies, who advocate intelligent networks with increased intelligence embedded within the network and controlled by the telecommunications providers. The intelligence would be reflected in features such as for caller identification or call forwarding. On the other side are the proponents of (what may be called) stupid networks, who take their inspiration from the Internet. In this view, as bandwidth becomes plentiful, intelligence will propagate to the edges of the network and the network itself will provide no more than bit transport. The overwhelming advantage of stupid networks is that, like the Internet, they naturally support heterogeneity and extensibility. End users can choose whichever applications they like and invoke whichever services they prefer without requiring consistent changes throughout a large network. Consider telephone directories. White pages and yellow pages are essential for making a telecommunications system practical, but are classical centralized functionalities. Internet portals are a close analogue of telephone directories, and provide a one-size-fits-all solution to users’ information needs. However, just as advances in computing are driving manufacturing from mass production to mass customization, advances in communication are driving information access from large portals toward personal contacts. Large portals won’t be eliminated, but will increasingly be supplemented and superseded by personalized sources of information as people increasingly want to receive information and advice from those whom they know and trust. Accordingly, in this article we consider the problem of service location. We study an approach that places the intelligence on the endpoints, enabling the users to locate desirable services based on trustworthy, personalized recommendations of their peers. The task is not only to locate a particular service, but also to locate a service that is rated highly by one’s friends and associ-

title: A retrieval technique for high-dimensional data and partially specified queries
abstract: While the persistent data of many advanced database applications, such as OLAP and scientific studies, are characterized by very high dimensionality, typical queries posed on these data appeal to a small number of relevant dimensions. Unfortunately, the multi-dimensional access methods designed for high-dimensional data perform rather poorly for these partially specified queries. The retrieval technique proposed in this paper uses a combination of two complementary measures to support efficient processing of partial queries over high-dimensional data. First, an elaborate storage organization, called the inverted space, allows the system administrator to control the size of individual indexes in order to avoid the negative impact of extremely high data dimensionality on the retrieval performance. Second, a new indexing structure, which is designed to support the inverted-space storage organization, enables efficient query processing in projected spaces with moderate dimensionality. This indexing mechanism is a general-purpose point access method that effectively attacks the limitations of KDB-trees in spaces with many dimensions, while preserving the simplicity and relatively good performance of the later structure in low-dimensional spaces. The analytical and experimental results show that the new indexing scheme outperforms two other variants of KDB-trees investigated in the paper for both fully and partially specified queries.

title: Implementing KDB-trees to support high-dimensional data
abstract: The problem of retrieving large volumes of high dimensional data is an important and timely issue in the area of database management. The guiding idea of the paper is to develop a general-purpose point access method that attacks the limitations of KDB-trees in high-dimensional spaces, while preserving their relatively good performance in low-dimensional situations. The proposed structure, called high-dimensional KDB-tree, eliminates downward propagation of splits associated with the original KDB-tree structure, which results in low storage utilization and rapid deterioration of the retrieval performance. Additional improvements in the storage and retrieval performance are achieved by removing certain redundant information from the interior nodes. Experimental results show that, in high-dimensional spaces, the proposed structure outperforms the original KDB-trees by a significant margin, while incurring no loss of performance in low-dimensional spaces. The structure also outperforms two other variants of KDB-trees investigated in the paper.

title: Processing partially specified queries over high-dimensional databases
abstract: In recent years, high-dimensional database applications deal with multidimensional ad hoc queries that refer to an arbitrary number of arbitrarily (unpredictably) chosen dimensions of high-dimensional data. This paper thoroughly and systematically investigates possible secondary-storage based solutions to the problems of processing multidimensional ad hoc query in transactional or semi-transactional environments. Then a complementary solution called the indexed and transposed access method (ITAM) is proposed. This method is based on two complementary measures - a multidimensional access method and a proposed access method called the Opus-path. The performance of multidimensional access methods deteriorates rapidly as the ratio of query dimensionality to data dimensionality decreases. On the other hand, the Opus-path shows retrieval performance that is actually better when the ratio is low.