approximate maximum entropy joint feature inference consistent with arbitrary lower order probability constraints application to statistical classification we propose a new learning method for discrete space statistical classifiers similar to chow and liu 1968 and cheeseman 1983 we cast classification inference within the more general framework of estimating the joint probability mass function p m f for the feature vector class label pair cheeseman s proposal to build the maximum entropy me joint p m f consistent with general lower order probability constraints is in principle powerful allowing general dependencies between features however enormous learning complexity has severely limited the use of this approach alternative models such as bayesian networks bns require explicit determination of conditional independencies these may be difficult to assess given limited data here we propose an approximate me method which like previous methods incorporates general constraints while retaining quite tractable learning the new method restricts joint p m f support during learning to a small subset of the full feature space classification gains are realized over dependence trees tree augmented naive bayes networks bns trained by the kutato algorithm and multilayer perceptrons extensions to more general inference problems are indicated we also propose a novel exact inference method when there are several missing features

statistical mechanics and phase transitions in clustering a new approach to clustering based on statistical physics is presented the problem is formulated as fuzzy clustering and the association probability distribution is obtained by maximizing the entropy at a given average variance the corresponding lagrange multiplier is related to the temperature and motivates a deterministic annealing process where the free energy is minimized at each temperature critical temperatures are derived for phase transitions when existing clusters split it is a hierarchical clustering estimating the most probable cluster parameters at various average variances

advances in residual vector quantization a review advances in residual vector quantization rvq are surveyed definitions of joint encoder optimality and joint decoder optimality are discussed design techniques for rvqs with large numbers of stages and generally different encoder and decoder codebooks are elaborated and extended fixed rate rvqs and variable rate rvqs that employ entropy coding are examined predictive and finite state rvqs designed and integrated into neural network based source coding structures are revisited successive approximation rvqs that achieve embedded and refinable coding are reviewed a new type of successive approximation rvq that varies the instantaneous block rate by using different numbers of stages on different blocks is introduced and applied to image waveforms and a scalar version of the new residual quantizer is applied to image subbands in an embedded wavelet transform coding system

a survey of the theory of source coding with a fidelity criterion the purpose of this paper is threefold 1 to acquaint the reader with the types of problems that have been considered in the area of source coding with a fidelity criterion 2 to survey results that have been obtained on these problems and 3 to outline future research trends in the area

optimal quantizer design for noisy channels an approach to combined source channel coding we present an analysis of the zero memory quantization of memoryless sources when the quantizer output is to be encoded and transmitted across a noisy channel necessary conditions for the joint optimization of the quantizer and the encoder decoder pair are presented and an iterative algorithm for obtaining a locally optimum system is developed the performance of this locally optimal system obtained for the class of generalized gaussian distributions and the binary symmetric channel is compared against the optimum performance theoretically attainable using rate distortion theoretic arguments as well as against the performance of lloyd max quantizers encoded using the natural binary code and the folded binary code it is shown that this optimal design could result in substantial performance improvements the performance improvements are more noticeable at high bit rates and for broad tailed densities

nonlinear parameter estimation via the genetic algorithm a modified genetic algorithm is used to solve the parameter identification problem for linear and nonlinear iir digital filters under suitable hypotheses the estimation error is shown to converge in probability to zero the scheme is also applied to feedforward and recurrent neural networks

constrained clustering as an optimization method a deterministic annealing approach to clustering is derived on the basis of the principle of maximum entropy this approach is independent of the initial state and produces natural hierarchical clustering solutions by going through a sequence of phase transitions it is modified for a larger class of optimization problems by adding constraints to the free energy the concept of constrained clustering is explained and three examples are are given in which it is used to introduce deterministic annealing the previous clustering method is improved by adding cluster mass variables and a total mass constraint the traveling salesman problem is reformulated as constrained clustering yielding the elastic net en approach to the problem more insight is gained by identifying a second lagrange multiplier that is related to the tour length and can also be used to control the annealing process the open path constraint formulation is shown to relate to dimensionality reduction by self organization in unsupervised learning a similar annealing procedure is applicable in this case as well

least squares quantization in pcm it has long been realized that in pulse code modulation pcm with a given ensemble of signals to handle the quantum values should be spaced more closely in the voltage regions where the signal amplitude is more likely to fall it has been shown by panter and dite that in the limit as the number of quanta becomes infinite the asymptotic fractional density of quanta per unit voltage should vary as the one third power of the probability density per unit voltage of signal amplitudes in this paper the corresponding result for any finite number of quanta is derived that is necessary conditions are found that the quanta and associated quantization intervals of an optimum finite quantization scheme must satisfy the optimization criterion used is that the average quantization noise power be a minimum it is shown that the result obtained here goes over into the panter and dite result as the number of quanta become large the optimum quautization schemes for 2 b quanta b 1 2 cdots 7 are given numerically for gaussian and for laplacian distribution of signal amplitudes

a stochastic self organizing map for proximity data we derive an efficient algorithm for topographic mapping of proximity data tmp which can be seen as an extension of kohonen s self organizing map to arbitrary distance measures the tmp cost function is derived in a baysian framework of folded markov chains for the description of autoencoders it incorporates the data by a dissimilarity matrix and the topographic neighborhood by a matrix of transition probabilities from the principle of maximum entropy a nonfactorizing gibbs distribution is obtained which is approximated in a mean field fashion this allows for maximum likelihood estimation using an expectation maximization algorithm in analogy to the transition from topographic vector quantization to the self organizing map we suggest an approximation to tmp that is computationally more efficient in order to prevent convergence to local minima an annealing scheme in the temperature parameter is introduced for which the critical temperature of the first phase transition is calculated in terms o

information cut for clustering using a gradient descent approach we introduce a new graph cut for clustering which we call the information cut it is derived using parzen windowing to estimate an information theoretic distance measure between probability density functions we propose to optimize the information cut using a gradient descent based approach our algorithm has several advantages compared to many other graph based methods in terms of determining an appropriate affinity measure computational complexity memory requirements and coping with different data scales we show that our method may produce clustering and image segmentation results comparable or better than the state of the art graph based methods

unsupervised texture classification using vector quantization and deterministic relaxation neural network this paper describes the use of a neural network architecture for classifying textured images in an unsupervised manner using image specific constraints the texture features are extracted by using two dimensional 2 d gabor filters arranged as a set of wavelet bases the classification model comprises feature quantization partition and competition processes the feature quantization process uses a vector quantizer to quantize the features into codevectors where the probability of grouping the vectors is modeled as gibbs distribution a set of label constraints for each pixel in the image are provided by the partition and competition processes an energy function corresponding to the a posteriori probability is derived from these processes and a neural network is used to represent this energy function the state of the network and the codevectors of the vector quantizer are iteratively adjusted using a deterministic relaxation procedure until a stable state is reached the final equilibrium state of the vector quantizer gives a classification of the textured image a cluster validity measure based on modified hubert index is used to determine the optimal number of texture classes in the image

the design of joint source and channel trellis waveform coders the generalized lloyd algorithm is applied to the design of joint source and channel trellis waveform coders to encode discrete time continuous amplitude stationary and ergodic sources operating over discrete memoryless noisy channels experimental results are provided for independent and autoregressive gaussian sources binary symmetric channels and absolute error and squared error distortion measures performance of the joint codes is compared with the tandem combination of a trellis source code and a trellis channel code on the independent gaussian source using the squared error distortion measure operating over an additive white gaussian noise channel it is observed that the jointly optimized codes achieve performance close to or better than that of separately optimized tandem codes of the same constraint length performance improvement via a predictive joint source and channel trellis code is demonstrated for the autoregressive gaussian source using the squared error distortion measure

deterministic annealing gustafson kessel fuzzy clustering algorithm abstract the gustafson and kessel gk fuzzy clustering algorithm proposed by gustafson and kessel in 1979 was the first important extension to the fuzzy c means fcm algorithm up to now the gk algorithm had become one of the most commonly used fuzzy clustering algorithms where the mahalanobis distance is used as a dissimilarity measure to provide more effectiveness and robustness than the fcm algorithm recently chaomurilige et al 2015 proposed a theoretical analysis on the parameter selection for the gk algorithm in which they indicated that the parameter of the fuzziness index heavily influences the performance of the gk algorithm in this paper we propose a novel gk fuzzy clustering algorithm based on the deterministic annealing approach for decreasing the effect of parameters we first consider maximizing the shannon s entropy of membership functions to the gk objective function and then use deterministic annealing to adjust the annealing parameter we also mathematically provide a theoretical initialization lower bound for the annealing parameter of the proposed deterministic annealing gk da gk algorithm comparisons between the da gk algorithm and other methods are made the computational complexity of the proposed method is also provided experimental results and comparisons actually verify theoretical results and also indicate the superiority and effectiveness of the proposed da gk algorithm

a fast search algorithm for vector quantization using mean pyramids of codewords one of the most serious problems for vector quantization especially for high dimensional vectors is the high computational complexity of searching for the closest codeword in the codebook design and encoding phases although quantizing high dimensional vectors rather than low dimensional vectors results in better performance the computation time needed for vector quantization grows exponentially with the vector dimension this makes high dimensional vectors unsuitable for vector quantization to overcome this problem a fast search algorithm under the assumption that the distortion is measured by the squared euclidean distance is proposed using the mean pyramids of codewords the algorithm ran reject many codewords that are impossible matches and hence save a great deal of computation time the algorithm is efficient for high dimensional codeword searches experimental results confirm the effectiveness of the proposed method

equivariant alperin robinsonʼs conjecture reduces to almost simple k groups in a recent paper gabriel navarro and pham huu tiep show that the so called alperin weight conjecture can be verified via the classification of the finite simple groups provided any simple group fulfills a very precise list of conditions our purpose here is to show that the equivariant refinement of the alperin s conjecture for blocks formulated by geoffrey robinson in the eighties can be reduced to checking the same statement on any central k extension of any finite almost simple group or of any finite simple group up to verifying an almost necessary condition in appendix a we develop some old arguments that we need in the proof

stability based validation of clustering solutions data clustering describes a set of frequently employed techniques in exploratory data analysis to extract natural group structure in data such groupings need to be validated to separate the signal in the data from spurious structure in this context finding an appropriate number of clusters is a particularly important model selection question we introduce a measure of cluster stability to assess the validity of a cluster model this stability measure quantifies the reproducibility of clustering solutions on a second sample and it can be interpreted as a classification risk with regard to class labels produced by a clustering algorithm the preferred number of clusters is determined by minimizing this classification risk as a function of the number of clusters convincing results are achieved on simulated as well as gene expression data sets comparisons to other methods demonstrate the competitive performance of our method and its suitability as a general validation tool for clustering solutions in real world problems

resolution based complexity control for gaussian mixture models in the domain of unsupervised learning mixtures of gaussians have become a popular tool for statistical modeling for this class of generative models we present a complexity control scheme which provides an effective means for avoiding the problem of overfitting usually encountered with unconstrained mixtures of gaussians in high dimensions according to some prespecified level of resolution as implied by a fixed variance noise model the scheme provides an automatic selection of the dimensionalities of some local signal subspaces by maximum likelihood estimation together with a resolution based control scheme for adjusting the number of mixture components we arrive at an incremental model refinement procedure within a common deterministic annealing framework which enables an efficient exploration of the model space the advantages of the resolution based framework are illustrated by experimental results on synthetic and high dimensional real world data

stochastic relaxation gibbs distributions and the bayesian restoration of images we make an analogy between images and statistical mechanics systems pixel gray levels and the presence and orientation of edges are viewed as states of atoms or molecules in a lattice like physical system the assignment of an energy function in the physical system determines its gibbs distribution because of the gibbs distribution markov random field mrf equivalence this assignment also determines an mrf image model the energy function is a more convenient and natural mechanism for embodying picture attributes than are the local characteristics of the mrf for a range of degradation mechanisms including blurring nonlinear deformations and multiplicative or additive noise the posterior distribution is an mrf with a structure akin to the image model by the analogy the posterior distribution defines another imaginary physical system gradual temperature reduction in the physical system isolates low energy states annealing or what is the same thing the most probable states under the gibbs distribution the analogous operation under the posterior distribution yields the maximum a posteriori map estimate of the image given the degraded observations the result is a highly parallel relaxation algorithm for map estimation we establish convergence properties of the algorithm and we experiment with some simple pictures for which good restorations are obtained at low signal to noise ratios

general statistical inference for discrete and mixed spaces by an approximate application of the maximum entropy principle we propose a method for learning a general statistical inference engine operating on discrete and mixed discrete continuous feature spaces such a model allows inference on any of the discrete features given values for the remaining features applications are e g to medical diagnosis with multiple possible diseases fault diagnosis information retrieval and imputation in databases bayesian networks bns are versatile tools that possess this inference capability however bns require explicit specification of conditional independencies which may be difficult to assess given limited data alternatively cheeseman 1983 proposed finding the maximum entropy me joint probability mass function pmf consistent with arbitrary lower order probability constraints this approach is in principle powerful and does not require explicit expression of conditional independence however until now the huge learning complexity has severely limited the use of this approach here we propose an approximate me method which also encodes arbitrary low order constraints but while retaining quite tractable learning our method uses a restriction of joint pmf support during learning to a subset of the feature space results on the university of california irvine repository reveal performance gains over several bn approaches and over multilayer perceptrons

vector quantization by deterministic annealing a deterministic annealing approach is suggested to search for the optimal vector quantizer given a set of training data the problem is reformulated within a probabilistic framework no prior knowledge is assumed on the source density and the principle of maximum entropy is used to obtain the association probabilities at a given average distortion the corresponding lagrange multiplier is inversely related to the temperature and is used to control the annealing process in this process as the temperature is lowered the system undergoes a sequence of phase transitions when existing clusters split naturally without use of heuristics the resulting codebook is independent of the codebook used to initialize the iterations

trellis coded quantization designed for noisy channels trellis coded quantization tcq of memoryless sources is developed for transmission over a binary symmetric channel the optimized tcq coder can achieve essentially the same performance as ayanoglu and gray s 1987 unconstrained trellis coding optimized for the binary symmetric channel but with a much lower implementation complexity for transmission rates above 1 b sample in most cases the optimized tcq coder also provides larger signal to noise ratio than farvardin and vaishampayan s 1991 channel optimized vector quantization algorithms are developed for the joint design of trellis coded quantization modulation tcq tcm the jointly designed tcq tcm system outperforms the straightforward cascade of separately designed tcq and tcm systems the improvement is most significant at low channel signal to noise ratio for a first order gauss markov source the predictive tcq tcm performance can exceed that of optimum pulse amplitude modulation

stochastic organization of output codes in multiclass learning problems the best known decomposition schemes of multiclass learning problems are one per class coding opc and error correcting output coding ecoc both methods perform a prior decomposition that is before training of the classifier takes place the impact of output codes on the inferred decision rules can be experienced only after learning therefore we present a novel algorithm for the code design of multiclass learning problems this algorithm applies a maximum likelihood objective function in conjunction with the expectation maximization em algorithm minimizing the augmented objective function yields the optimal decomposition of the multiclass learning problem in two class problems experimental results show the potential gain of the optimized output codes over opc or ecoc methods

on the performance and complexity of channel optimized vector quantizers the performance and complexity of channel optimized vector quantizers are studied for the gauss markov source observations on the geometric structure of these quantizers are made which have an important implication on the encoding complexity for the squared error distortion measure it is shown that an operation equivalent to a euclidean distance measurement with respect to an appropriately defined set of points used to identify the encoding regions can be used to perform the encoding this implies that the encoding complexity is proportional to the number of encoding regions it is then demonstrated that for very noisy channels and a heavily correlated source when the codebook size is large the number of encoding regions is considerably smaller than the codebook size implying a reduction in encoding complexity

a sequence based extension of mean field annealing using the forward backward algorithm application to image segmentation optimization based on mean field theory including mean field annealing mfa is widely used for discrete optimization label assignment problems defined on the pixel sites of an image one formulation of mfa is via maximum entropy where one seeks the joint distribution over the random assignments subject to an average level of cost mfa is obtained by assuming the assignments at each pixel are statistically independent given the observed image alternatively we make the less restrictive assumption of independent row labelings the independence assumption means that at each step mfa optimizes over only one pixel whereas our method jointly optimizes over an entire row i e our method is less greedy in principle an mfa extension could be developed that explicitly re estimates the row labeling distributions but such an approach is in practice infeasible even so we can indirectly implement this re estimation by re estimating quantities that determine the row labeling distributions these quantities are the a posteriori site probabilities re estimable via the well known forward backward f b algorithm thus our algorithm which descends in the me lagrangian free energy consists of iterative application of f b to the image rows columns at convergence maximum a posteriori site labeling is performed our method was applied to segmentation of both real and synthetic noise corrupted images it achieved lower markov random field model potentials and better segmentations compared with other methods and in high noise standard mfa

tensor product basis approximations for volterra filters the paper studies approximations for a class of nonlinear filters known as volterra filters although the volterra filter provides a relatively simple and general representation for nonlinear filtering it is often highly overparameterized due to the large number of parameters the utility of the volterra filter is limited the overparameterization problem is addressed in the paper using a tensor product basis approximation tpba in many cases a volterra filter may be well approximated using the tpba with far fewer parameters hence the tpba offers considerable advantages over the original volterra filter in terms of both implementation and estimation complexity furthermore the tpba provides useful insight into the filter response the paper studies the crucial issue of choosing the approximation basis several methods for designing an appropriate approximation basis and error bounds on the resulting mean square output approximation error are derived certain methods are known to be nearly optimal

adaptive volterra filters with evolutionary quadratic kernels using a combination scheme for memory control this paper proposes a new paradigm for adaptive volterra filtering using a time variant size of the quadratic kernel memory in order to optimally identify any unknown transversal second order nonlinear system to this end competing volterra structures of different sizes are employed in a hierarchical combination scheme so as to find the best configuration of the second order kernel memory using the already known diagonal coordinate representation the length and number of required quadratic kernel diagonals can be concurrently estimated by monitoring the combination performance subsequently the memory size of the involved models is dynamically increased or decreased following a set of intuitive rules since this automatic memory adaptation is performed along with the coefficient updates an efficient volterra filter is realized offering great flexibility and minimizing the risk of under or overmodeling any given quadratic nonlinearity besides the straightforward scheme a simplified version is presented greatly reducing the algorithmic demands this efficient version is based on a virtualization of the competing volterra filters by jointly using common coefficients and hence exhibits a computational complexity suitable for practical implementations the robust estimation performance of the approach is demonstrated by various examples for a nonlinear acoustic echo cancellation scenario involving stationary noise real speech signals and realistic volterra kernels

the multitone channel the maximum bit rate of multitone qam quadrature amplitude modulation over a general linear channel is found first the overall bit rate for an awgn channel with a two level transfer function is maximized using a multitone qam system the power distribution between the tones and the number of bits symbol per tone is optimized for a given symbol error rate extending these results to the general channel it is shown that the optimum power division for multitone signals is similar to the water pouring solution of information theory furthermore multitone qam performance is about 9 db worse than the channel capacity independent of the channel characteristics the multitone results throughout are compared to those of an equivalent single tone linearly equalized system the comparison shows that the multitone system is useful for some channels e g those with deep nulls in the transfer function the maximum bit error rate over a twisted pair channel which is performance dominated by near end crosstalk next is also found

a mapping approach to rate distortion computation and analysis in rate distortion theory results are often derived and stated in terms of the optimizing density over the reproduction space in this paper the problem is reformulated in terms of the optimal mapping from the unit interval with lebesgue measure that would induce the desired reproduction probability density this results in optimality conditions that are random relatives of the known lloyd 1982 optimality conditions for deterministic quantizers the validity of the mapping approach is assured by fundamental isomorphism theorems for measure spaces we show that for the squared error distortion the optimal reproduction random variable is purely discrete at supercritical distortion where the shannon 1948 lower bound is not tight the gaussian source is thus the only source that produces continuous reproduction variables for the entire range of positive rate to analyze the evolution of the optimal reproduction distribution we use the mapping formulation and establish an analogy to statistical mechanics the solutions are given by the distribution at isothermal statistical equilibrium and are parameterized by the temperature in direct correspondence to the parametric solution of the variational equations in rate distortion theory the analysis of an annealing process shows how the number of symbols grows as the system undergoes phase transitions thus an algorithm based on the mapping approach often needs but a few variables to find the exact solution while the blahut 1972 algorithm would only approach it at the limit of infinite resolution finally a quick deterministic annealing algorithm to generate the rate distortion curve is suggested the resulting curve is exact as long as continuous phase transitions in the process are accurately followed

a reduction theorem for the alperin weight conjecture the alperin weight conjecture was stated by j alperin in 1986 and constitutes one of the main problems in the representation theory of finite groups in this paper we provide a reduction to simple groups and as an application we prove the conjecture in several cases

entropy constrained tree structured vector quantizer design current methods for the design of pruned or unbalanced tree structured vector quantizers such as the generalized breiman friedman olshen stone gbfos algorithm proposed in 1980 are effective but suffer from several shortcomings we identify and clarify issues of suboptimality including greedy growing the suboptimal encoding rule and the need for time sharing between quantizers to achieve arbitrary rates we then present the leaf optimal tree design lotd method which with a modest increase in design complexity alters and reoptimizes tree structures obtained from conventional procedures there are two main advantages over existing methods first the optimal entropy constrained nearest neighbor rule is used for encoding at the leaves second explicit quantizer solutions are obtained at all rates without recourse to time sharing we show that performance improvement is theoretically guaranteed simulation results for image coding demonstrate that close to 1 db reduction of distortion for a given rate can be achieved by this technique relative to the gbfos method

on the reduction of alperin s conjecture to the quasi simple groups abstract we show that the refinement of alperin s conjecture proposed in l puig 2009 10 chap 16 can be proved by checking that this refinement holds on any central k extension of a finite group h containing a normal simple group s with trivial centralizer in h and p cyclic quotient h s this paper improves our result in l puig 2009 10 theorem 16 45 and repairs some bad arguments there

linear codes interpolation from noisy patterns by means of a vector quantization process abstract an algorithm inferring a boolean linear code from noisy patterns received by a noisy channel under the assumption of uniform occurrence distribution over the codewords and an upper bound to the amount of data are presented a vector quantizer is designed from the noisy patterns choosing the obtained codebook as code approximation it is shown both theoretically and experimentally that when the data are affected by independent random errors this strategy requires a small number of patterns to obtain a good identification with high probability of the code from the noisy data

an analysis of the elastic net approach to the traveling salesman problem this paper analyzes the elastic net approach durbin and willshaw 1987 to the traveling salesman problem of finding the shortest path through a set of cities the elastic net approach jointly minimizes the length of an arbitrary path in the plane and the distance between the path points and the cities the tradeoff between these two requirements is controlled by a scale parameter k a global minimum is found for large k and is then tracked to a small value in this paper we show that 1 in the small k limit the elastic path passes arbitrarily close to all the cities but that only one path point is attracted to each city 2 in the large k limit the net lies at the center of the set of cities and 3 at a critical value of k the energy function bifurcates we also show that this method can be interpreted in terms of extremizing a probability distribution controlled by k the minimum at a given k corresponds to the maximum a posteriori map bayesian estimate of the tour under a natural statistical interpretation the analysis presented in this paper gives us a better understanding of the behavior of the elastic net allows us to better choose the parameters for the optimization and suggests how to extend the underlying ideas to other domains

deterministic annealing based optimization for zero delay source channel coding in networks this paper studies the problem of global optimization of zero delay source channel codes that map between the source space and the channel space under a given transmission power constraint and for the mean square error distortion particularly we focus on two well known network settings the wyner ziv setting where only a decoder has access to side information and the distributed setting where independent encoders transmit over independent channels to a central decoder prior work derived the necessary conditions for optimality of the encoder and decoder mappings along with a greedy optimization algorithm that imposes these conditions iteratively in conjunction with the heuristic noisy channel relaxation method to mitigate poor local minima while noisy channel relaxation is arguably effective in simple settings it fails to provide accurate global optimization in more complicated settings considered in this paper we propose a powerful nonconvex optimization method based on the concept of deterministic annealing which is derived from information theoretic principles and was successfully employed in several problems including vector quantization classification and regression we present comparative numerical results that show strict superiority of the proposed method over greedy optimization methods as well as prior approaches in literature

genetic algorithm based identification of nonlinear systems by sparse volterra filters a parsimonious parameterization scheme is proposed to model the sparse volterra filter so that the number of volterra kernels to be estimated is greatly reduced representing the volterra filter using a linear vector equation the genetic algorithm is applied to search the significant terms among all possible candidate vectors as the significant terms are detected the associated volterra kernels are estimated using the least square error method the problem to be solved is in essence the application of the genetic algorithm to combinatorial optimization an operator called forced mutation is proposed along with the genetic algorithm to overcome the difficulties usually encountered when applying the genetic algorithm to combinatorial optimization

transmission of analog signals using multicarrier modulation a combined source channel coding approach we have proposed and analyzed a combined source channel coding scheme using multicarrier modulation by changing the power and modulation of subchannels carrying different bits of the quantized signal the channel induced distortion can be minimized we derive an algorithm for the allocation of power among subchannels analog sources quantized by scalar and vector quantizers are considered for a gaussian source scalar quantized to 8 bit multicarrier transmission yields a 13 db improvement of signal to distortion ratio over a single carrier system quantizers having a smaller number of bits yield smaller improvements we consider both full search and binary tree search vector quantizers using a natural binary coding scheme for a vector quantizer using 2 b sample multicarrier transmission yields an improvement in signal to distortion ratio that lies between 0 3 and 0 8 db depending on the quantizer design

on finite block length quantization distortion we investigate the upper and lower bounds on the quantization distortions for independent and identically distributed continuous sources in the finite block length regime we derive a lower bound on the quantization distortion as well as an upper bound on the quantization distortion based on random quantization codebooks moreover we apply the obtained bounds to the continuous gaussian source for the gaussian source we propose a computationally tractable method to numerically compute the upper and lower bounds for both bounded and unbounded quantization codebooks numerical results show that the gap between the upper and lower bounds is small for the block length of several hundreds and the upper and lower bounds are tighter than those proposed in the existing works

gradient algorithms for designing predictive vector quantizers a predictive vector quantizer pvq is a vector extension of a predictive quantizer it consists of two parts a conventional memoryless vector quantizer vq and a vector predictor two gradient algorithms for designing a pvq are developed in this paper the steepest descent sd algorithm and the stochastic gradient sg algorithm both have the property of improving the quantizer and the predictor in the sense of minimizing the distortion as measured by the average mean squared error the differences between the two design approaches are the period and the step size used in each iteration to update the codebook and predictor the sg algorithm updates once for each input training vector and uses a small step size while the sd updates only once for a long period possibly one pass over the entire training sequence and uses a relatively large step size code designs and tests are simulated for both gauss markov sources and for sampled speech waveforms and the results are compared to codes designed using techniques that attempt to optimize only the quantizer for the predictor and not vice versa

vector quantization with transmission energy allocation for time varying channels this work is concerned with the problem of designing robust vector quantizer vq based communication systems for operation over time varying gaussian channels transmission energy allocation to vq codeword bits according to their error sensitivities is a powerful tool for improving robustness to channel noise the power of this technique can be further enhanced by appropriately combining it with index assignment methods we pose the corresponding joint optimization problem and suggest a simple iterative algorithm for finding a locally optimal solution the susceptibility of the solution to poor local minima is significantly reduced by an enhanced version of the algorithm which invokes the method of noisy channel relaxation whereby the vq system is optimized while gradually decreasing the assumed level of channel noise in a series of experiments the resulting combined technique is shown to outperform standard pseudo gray coding by up to 3 5 db and to exhibit graceful degradation at mismatched channel conditions finally we extend these ideas to the case where both the transmitter and the receiver have information on the current state of a time varying channel the proposed method is based on switched encoding and adaptive decoding experimental results show that the proposed system achieves close to optimal performance

predictive vector quantizer design using deterministic annealing a new approach is proposed for predictive vector quantizer pvq design which is inherently probabilistic and is based on ideas from information theory and analogies to statistical physics the approach effectively resolves three longstanding fundamental shortcomings of standard pvq design the first complication is due to the pvq prediction loop which has a detrimental impact on the convergence and the stability of the design procedure the second shortcoming is due to the piecewise constant nature of the quantizer function which makes it difficult to optimize the predictor with respect to the overall reconstruction error finally a shortcoming inherited from standard vq design is the tendency of the design algorithm to terminate at a locally rather than the globally optimal solution we propose a new pvq design approach that embeds our previous asymptotic closed loop acl approach within a deterministic annealing da framework the overall da acl method profits from its two main components in a complementary way acl is used to overcome the first difficulty and offers the means for stable quantizer design as it provides an open loop design platform yet allows the pvq design algorithm to asymptotically converge to optimization of the closed loop performance objective da simultaneously mitigates or eliminates the remaining design shortcomings its probabilistic framework replaces hard quantization with a differentiable expected cost function that can be jointly optimized for the predictor and quantizer parameters and its annealing schedule allows the avoidance of many poor local optima substantial performance gains over traditional methods have been achieved in the simulations

a study of vector quantization for noisy channels several issues related to vector quantization for noisy channels are discussed an algorithm based on simulated annealing is developed for assigning binary codewords to the vector quantizer code vectors it is shown that this algorithm could result in dramatic performance improvements as compared to randomly selected codewords a modification of the simulated annealing algorithm for binary codeword assignment is developed for the case where the bits in the codeword are subjected to unequal error probabilities resulting from unequal levels of error protection an algorithm for the design of an optimal vector quantizer for a noisy channel is briefly discussed and its robustness under channel mismatch conditions is studied numerical results for a stationary first order gauss markov source and a binary symmetric channel are provided it is concluded that the channel optimized vector quantizer design algorithm if used carefully can result in a fairly robust system with no additional delay the case in which the communication channel is nonstationary as in mobile radio channels is studied and some preliminary ideas for quantizer design are presented

pseudo gray coding a pseudo gray code is an assignment of n bit binary indexes to 2 points in a euclidean space so that the hamming distance between two points corresponds closely to the euclidean distance pseudo gray coding provides a redundancy free error protection scheme for vector quantization vq of analog signals when the binary indexes are used as channel symbols on a discrete memoryless channel and the points are signal codevectors binary indexes are assigned to codevectors in a way that reduces the average quantization distortion introduced in the reproduced source vectors when a transmitted index is corrupted by channel noise a globally optimal solution to this problem is generally intractable due to an inherently large computational complexity a locally optimal solution the binary switching algorithm is introduced based on the objective of minimizing a useful upper bound on the average system distortion the algorithm yields a significant reduction in average distortion and converges in reasonable running times the sue of pseudo gray coding is motivated by the increasing need for low bit rate vq based encoding systems that operate on noisy channels such as in mobile radio speech communications

a unified approach to tree structured and multistage vector quantization for noisy channels the large encoding complexity and sensitivity to channel errors of vector quantization vq are discussed the performance of two low complexity vqs the tree structured vq tsvq and the multistage vq msvq when used over noisy channels are analyzed an algorithm is developed for the design of channel matched tsvq cm tsvq and channel matched msvq cm msvq under the squared error criterion extensive numerical results are given for the correlation coefficient 0 9 comparisons with the ordinary tsvq and msvq designed for the noiseless channel show substantial improvements when the channel is very noisy the cm msvq which can be regarded as a block structured combined source channel coding scheme is compared with a block structured tandem source channel coding scheme with the same block length as the cm msvq for the gauss markov source the cm msvq outperforms the tandem scheme in all cases that the authors have considered it is demonstrated that the cm msvq is fairly robust to channel mismatch

a sparse interpolated scheme for implementing adaptive volterra filters in most practical applications the major drawback for using adaptive volterra filters is the large number of coefficients to cope with several research works discussing strategies to reduce the computational burden of these structures have been presented in the open literature for such a common approach has been the use of some type of sparseness in volterra filter kernels in this work a sparse interpolated approach with the interpolation having the purpose of recreating in an approximate way the elements disregarded for obtaining sparse kernels is presented and discussed thus for the adaptive sparse interpolated volterra filter coefficient update expressions considering both least mean square lms and normalized lms nlms algorithms are derived by using a constrained approach in general the proposed strategy outperforms other sparse schemes in terms of the tradeoff between computational complexity and mean square error mse performance as shown through numerical simulations

hierarchical unsupervised learning with growing via phase transitions we address unsupervised learning subject to structural constraints with particular emphasis placed on clustering with an imposed decision tree structure most known methods are greedy optimizing one node of the tree at a time to minimize a local cost by constrast we develop a joint optimization method derived based on information theoretic principles and closely related to known methods in statistical physics the approach is inspired by the deterministic annealing algorithm for unstructured data clustering which was based on maximum entropy inference the new approach is founded on the principle of minimum cross entropy using informative priors to approximate the unstructured clustering solution while imposing the structural constraint the resulting method incorporates supervised learning principles applied in an unsupervised problem setting in our approach the tree grows by a sequence of bifurcations that occur while optimizing an effective free energy cost at decreasing temperature scales thus estimates of the tree size and structure are naturally obtained at each temperature in the process examples demonstrate considerable improvement over known methods

design of discrete coefficient fir filters by a fast entropy directed deterministic annealing algorithm we present an entropy directed deterministic annealing optimization algorithm and show its applicability to the problem of designing digital filters with discrete coefficients each implemented as a sum of signed power of two terms and additional general hardware constraints the algorithm is based on analogies from statistical mechanics and is related to the well known mean field annealing algorithm it utilizes estimates of conditional entropy to prune the problem during the optimization thereby reducing the computational time by 30 to 50 in conjunction with a scheme to compute the value of the objective function as a sequence of updates this approach leads to a very fast algorithm as an application example demonstrating the potential of the new method we consider the design of digital filters with discrete coefficients consisting of a minimum number of signed power of two terms

bayesian motion estimation and interpolation in interlaced video sequences motion estimation in video is a very important tool for such applications as interpolation motion compensation and segmentation however in interlaced video it is often desirable to perform interpolation prior to the motion estimation in this correspondence a novel technique is presented that rather than minimizing the interpolation error in each of the frames used for motion estimation seeks to make these errors equal the new method is shown to perform deinterlacing better than several other approaches also based on bayesian estimation