map estimation via agreement on trees message passing and linear programming we develop and analyze methods for computing provably optimal maximum a posteriori probability map configurations for a subclass of markov random fields defined on graphs with cycles by decomposing the original distribution into a convex combination of tree structured distributions we obtain an upper bound on the optimal value of the original problem i e the log probability of the map assignment in terms of the combined optimal values of the tree problems we prove that this upper bound is tight if and only if all the tree distributions share an optimal configuration in common an important implication is that any such shared configuration must also be a map configuration for the original distribution next we develop two approaches to attempting to obtain tight upper bounds a a tree relaxed linear program lp which is derived from the lagrangian dual of the upper bounds and b a tree reweighted max product message passing algorithm that is related to but distinct from the max product algorithm in this way we establish a connection between a certain lp relaxation of the mode finding problem and a reweighted form of the max product min sum message passing algorithm

a general formula for compound channel capacity a general formula for the capacity of arbitrary compound channels with the receiver channel state information is obtained using the information density approach no assumptions of ergodicity stationarity or information stability are made and the channel state set is arbitrary a direct constructive proof is given to prove achievability we generalize feinstein lemma to the compound channel setting and to prove converse we generalize verdu han lemma to the same compound setting a notion of a uniform compound channel is introduced and the general formula is shown to reduce to the familiar sup inf expression for such channels as a by product the arbitrary varying channel capacity is established under maximum error probability and deterministic coding conditions are established under which the worst case and compound channel capacities are equal so that the full channel state information at the transmitter brings in no advantage the compound inf information rate plays a prominent role in the general formula its properties are studied and a link between the information unstable and information stable regimes of a compound channel is established the results are extended to include varepsilon capacity of compound channels sufficient and necessary conditions for the strong converse to hold are given

the compound channel capacity of a class of finite state channels a transmitter and receiver need to be designed to guarantee reliable communication on any channel belonging to a given family of finite state channels defined over common finite input output and state alphabets both the transmitter and receiver are assumed to be ignorant of the channel over which transmission is carried out and also ignorant of its initial state for this scenario we derive an expression for the highest achievable rate as a special case we derive the compound channel capacity of a class of gilbert elliott channels

structured prediction dual extragradient and bregman projections we present a simple and scalable algorithm for maximum margin estimation of structured output models including an important class of markov networks and combinatorial models we formulate the estimation problem as a convex concave saddle point problem that allows us to use simple projection methods based on the dual extragradient algorithm nesterov 2003 the projection step can be solved using dynamic programming or combinatorial algorithms for min cost convex flow depending on the structure of the problem we show that this approach provides a memory efficient alternative to formulations based on reductions to a quadratic program qp we analyze the convergence of the method and present experiments on two very different structured prediction tasks 3d image segmentation and word alignment illustrating the favorable scaling properties of our algorithm

capacity of the gaussian arbitrarily varying channel the gaussian arbitrarily varying channel with input constraint gamma and state constraint lambda admits input sequences x x sub 1 x sub n of real numbers with sigma x sub i sup 2

communication over individual channels a communication problem in considered where no mathematical model is specified for the channel the achievable rates are determined as a function of the channel input and output sequences known a posteriori without assuming any a priori relation between them for discrete channels the empirical mutual information between the input and output sequences is shown to be achievable while for continuous channels the achievable rate is based on the empirical correlation between the sequences a rate adaptive scheme employing feedback which achieves these rates asymptotically with a guaranteed reliability without prior knowledge of the channel behavior is presented

on the decoding of convolutional codes on an unknown channel an algorithm is proposed for universal decoding of convolutional trellis codes employed over unknown channels on discrete memoryless channels and at rates below the channel s computational cutoff rate for a uniform input distribution the algorithm achieves an asymptotic complexity performance tradeoff similar to the tradeoff achieved by the viterbi 1979 algorithm but with the benefit that the algorithm s implementation does not require knowledge of the channel law the algorithm is also applicable to channels with memory and in particular to intersymbol interference isi channels to channels with nonlinear isi and even to general finite state channels

fast approximate energy minimization via graph cuts many tasks in computer vision involve assigning a label such as disparity to every pixel a common constraint is that the labels should vary smoothly almost everywhere while preserving sharp discontinuities that may exist e g at object boundaries these tasks are naturally stated in terms of energy minimization the authors consider a wide class of energies with various smoothness constraints global minimization of these energy functions is np hard even in the simplest discontinuity preserving case therefore our focus is on efficient approximation algorithms we present two algorithms based on graph cuts that efficiently find a local minimum with respect to two types of large moves namely expansion moves and swap moves these moves can simultaneously change the labels of arbitrarily large sets of pixels in contrast many standard algorithms including simulated annealing use small moves where only one pixel changes its label at a time our expansion algorithm finds a labeling within a known factor of the global minimum while our swap algorithm handles more general energy functions both of these algorithms allow important cases of discontinuity preserving energies we experimentally demonstrate the effectiveness of our approach for image restoration stereo and motion on real data with ground truth we achieve 98 percent accuracy

on the capacity of the carbon copy onto dirty paper channel the carbon copy onto dirty paper ccdp channel is the compound writing on dirty paper channel in which the channel output is obtained as the sum of the channel input white gaussian noise and a gaussian state sequence randomly selected among a set possible realizations the transmitter has non causal knowledge of the set of possible state sequences but does not know which sequence is selected to produce the channel output we study the capacity of the ccdp channel for two scenarios 1 the state sequences are independent and identically distributed and 2 the state sequences are scaled versions of the same sequence in the first scenario we show that a combination of superposition coding time sharing and gel fand pinsker binning is sufficient to approach the capacity to within 3 bits per channel use for any number of possible state realizations in the second scenario we derive capacity to within 4 bits per channel use for the case of two possible state sequences this result is extended to the ccdp channel with any number of possible state sequences under certain conditions on the scaling parameters which we denote as strong fading regime we conclude by providing some remarks on the capacity of the ccdp channel in which the state sequences have any jointly gaussian distribution

on the statistical analysis of dirty pictures may 7th 1986 professor a f m smith in the chair summary a continuous two dimensional region is partitioned into a fine rectangular array of sites or pixels each pixel having a particular colour belonging to a prescribed finite set the true colouring of the region is unknown but associated with each pixel there is a possibly multivariate record which conveys imperfect information about its colour according to a known statistical model the aim is to reconstruct the true scene with the additional knowledge that pixels close together tend to have the same or similar colours in this paper it is assumed that the local characteristics of the true scene can be represented by a nondegenerate markov random field such information can be combined with the records by bayes theorem and the true scene can be estimated according to standard criteria however the computational burden is enormous and the reconstruction may reflect undesirable largescale properties of the random field thus a simple iterative method of reconstruction is proposed which does not depend on these large scale characteristics the method is illustrated by computer simulations in which the original scene is not directly related to the assumed random field some complications including parameter estimation are discussed potential applications are mentioned briefly

on the algorithmic implementation of multiclass kernel based vector machines in this paper we describe the algorithmic implementation of multiclass kernel based vector machines our starting point is a generalized notion of the margin to multiclass problems using this notion we cast multiclass categorization problems as a constrained optimization problem with a quadratic objective function unlike most of previous approaches which typically decompose a multiclass problem into multiple independent binary classification tasks our notion of margin yields a direct method for training multiclass predictors by using the dual of the optimization problem we are able to incorporate kernels with a compact set of constraints and decompose the dual problem into multiple optimization problems of reduced size we describe an efficient fixed point algorithm for solving the reduced optimization problems and prove its convergence we then discuss technical details that yield significant running time improvements for large datasets finally we describe various experiments with our approach comparing it to previously studied kernel based methods our experiments indicate that for multiclass problems we attain state of the art accuracy

a linear programming approach to max sum problem a review the max sum labeling problem defined as maximizing a sum of binary i e pairwise functions of discrete variables is a general np hard optimization problem with many applications such as computing the map configuration of a markov random field we review a not widely known approach to the problem developed by ukrainian researchers schlesinger et al in 1976 and show how it contributes to recent results most importantly those on the convex combination of trees and tree reweighted max product in particular we review schlesinger et al s upper bound on the max sum criterion its minimization by equivalent transformations its relation to the constraint satisfaction problem the fact that this minimization is dual to a linear programming relaxation of the original problem and the three kinds of consistency necessary for optimality of the upper bound we revisit problems with boolean variables and supermodular problems we describe two algorithms for decreasing the upper bound we present an example application for structural image analysis

the theory and practice of bayesian image labeling integrating disparate sources of information has been recognized as one of the keys to the success of general purpose vision systems image clues such as shading texture stereo disparities and image flows provide uncertain local and incomplete information about the three dimensional scene spatial a priori knowledge plays the role of filling in missing information and smoothing out noise this thesis proposes a solution to the longstanding open problem of visual integration it reports a framework based on bayesian probability theory for computing an intermediate representation of the scene from disparate sources of information the computation is formulated as a labeling problem local visual observations for each image entity are reported as label likelihoods they are combined consistently and coherently on hierarchically structured label trees with a new computationally simple procedure the pooled label likelihoods are fused with the a priori spatial knowledge encoded as markov random fields mrf s the a posteriori distribution of the labelings are thus derived in a bayesian formalism a new inference method highest confidence first hcf estimation is used to infer a unique labeling from the a posteriori distribution unlike previous inference methods based on the mrf formalism hcf is computationally efficient and predictable while meeting the principles of graceful degradation and least commitment the results of the inference process are consistent with both observable evidence and a priori knowledge the effectiveness of the approach is demonstrated with experiments on two image analysis problems intensity edge detection and surface reconstruction for edge detection likelihood outputs from a set of local edge operators are integrated with a priori knowledge represented as an mrf probability distribution for surface reconstruction intensity information is integrated with sparse depth measurements and a priori knowledge coupled mrf s provide a unified treatment of surface reconstruction and segmentation and an extension of hcf implements a solution method experiments using real image and depth data yield robust results the framework can also be generalized to higher level vision problems as well as to other domains

universal communication part ii channels with memory consider communication over a channel whose probabilistic model is completely unknown vector wise and is not assumed to be stationary communication over such channels is challenging because knowing the past does not indicate anything about the future the existence of reliable feedback and common randomness is assumed in a previous paper it was shown that the shannon capacity cannot be attained in general if the channel is not known an alternative notion of capacity was defined as the maximum rate of reliable communication by any block coding system used over consecutive blocks this rate was shown to be achievable for the modulo additive channel with an individual unknown noise sequence and not achievable for some channels with memory in this paper this capacity is shown to be achievable for general channel models possibly including memory as long as this memory fades with time in other words there exists a system with feedback and common randomness that without knowledge of the channel asymptotically performs as well as any block code which may be designed knowing the channel for channels in which memory does not fade with time a weaker type of capacity is shown to be achievable

polynomial time approximation algorithms for the ising model the paper presents a randomised algorithm which evaluates the partition function of an arbitrary ferromagnetic ising system to any specified degree of accuracy the running time of the algorithm in

finite state channels with time invariant deterministic feedback we consider capacity of discrete time channels with feedback for the general case where the feedback is a time invariant deterministic function of the output samples under the assumption that the channel states take values in a finite alphabet we find a sequence of achievable rates and a sequence of upper bounds on the capacity the achievable rates and the upper bounds are computable for any n and the limits of the sequences exist we show that when the probability of the initial state is positive for all the channel states then the capacity is the limit of the achievable rate sequence we further show that when the channel is stationary indecomposable and has no intersymbol interference isi its capacity is given by the limit of the maximum of the normalized directed information between the input xn and the output yn i e c limnrarrinfin 1 n max i xnrarryn where the maximization is taken over the causal conditioning probability q xnparzn 1 defined in this paper the main idea for obtaining the results is to add causality into gallager s results on finite state channels the capacity results are used to show that the source channel separation theorem holds for time invariant determinist feedback and if the state of the channel is known both at the encoder and the decoder then feedback does not increase capacity

capacity region of the finite state multiple access channel with and without feedback the capacity region of the finite state multiple access channel fs mac with feedback that may be an arbitrary time invariant function of the channel output samples is considered we characterize both an inner and an outer bound for this region using massey s directed information these bounds are shown to coincide and hence yield the capacity region of indecomposable fs macs without feedback and of stationary and indecomposable fs macs with feedback where the state process is not affected by the inputs though multiletter in general our results yield explicit conclusions when applied to specific scenarios of interest for example our results allow us to do the following 1 identify a large class of fs macs that includes the additive mod2 noise mac where the noise may have memory for which feedback does not enlarge the capacity region 2 deduce that for a general fs mac with states that are not affected by the input if the capacity region without feedback is zero then so is the capacity region with feedback 3 deduce that the capacity region of a mac that can be decomposed into a multiplexer concatenated by a point to point channel with without or with partial feedback the capacity region is given by sigmam rm les c where c is the capacity of the point to point channel and m indexes the encoders moreover we show that for this family of channels source channel coding separation holds

capacity theorems for discrete finite state broadcast channels with feedback and unidirectional receiver cooperation in this paper we consider the discrete time varying broadcast channel bc with memory under the assumption that the channel states belong to a set of finite cardinality we study the achievable rates in several scenarios of feedback and full unidirectional receiver cooperation in particular we focus on two scenarios the first scenario is the general finite state broadcast channel fsbc where both receivers send feedback to the transmitter while one receiver also sends its channel output to the second receiver the second scenario is the degraded fsbc where only the strong receiver sends feedback to the transmitter using a superposition codebook construction we derive the capacity regions for both scenarios combining elements from these two basic results we obtain the capacity regions for a number of additional broadcast scenarios with feedback and unidirectional receiver cooperation

feedback capacity of the compound channel in this work we find the capacity of a compound finite state channel fsc with time invariant deterministic feedback we consider the use of fixed length block codes over the compound channel our achievability result includes a proof of the existence of a universal decoder for the family of fscs with feedback as a consequence of our capacity result we show that feedback does not increase the capacity of the compound gilbert elliot channel additionally we show that for a stationary and uniformly ergodic markovian channel if the compound channel capacity is zero without feedback then it is zero with feedback finally we use our result on the fsc to show that the feedback capacity of the memoryless compound channel is given by infthetas maxqx i x y thetas

binary intersymbol interference channels gallager codes density evolution and code performance bounds we study the limits of performance of gallager codes low density parity check ldpc codes over binary linear intersymbol interference isi channels with additive white gaussian noise awgn using the graph representations of the channel the code and the sum product message passing detector decoder we prove two error concentration theorems our proofs expand on previous work by handling complications introduced by the channel memory we circumvent these problems by considering not just linear gallager codes but also their cosets and by distinguishing between different types of message flow neighborhoods depending on the actual transmitted symbols we compute the noise tolerance threshold using a suitably developed density evolution algorithm and verify by simulation that the thresholds represent accurate predictions of the performance of the iterative sum product algorithm for finite but large block lengths we also demonstrate that for high rates the thresholds are very close to the theoretical limit of performance for gallager codes over isi channels if c denotes the capacity of a binary isi channel and if c sub i i d denotes the maximal achievable mutual information rate when the channel inputs are independent and identically distributed i i d binary random variables c sub i i d spl les c we prove that the maximum information rate achievable by the sum product decoder of a gallager coset code is upper bounded by c sub i i d the last topic investigated is the performance limit of the decoder if the trellis portion of the sum product algorithm is executed only once this demonstrates the potential for trading off the computational requirements and the performance of the decoder

discriminative learning of max sum classifiers the max sum classifier predicts n tuple of labels from n tuple of observable variables by maximizing a sum of quality functions defined over neighbouring pairs of labels and observable variables predicting labels as map assignments of a random markov field is a particular example of the max sum classifier learning parameters of the max sum classifier is a challenging problem because even computing the response of such classifier is np complete in general estimating parameters using the maximum likelihood approach is feasible only for a subclass of max sum classifiers with an acyclic structure of neighbouring pairs recently the discriminative methods represented by the perceptron and the support vector machines originally designed for binary linear classifiers have been extended for learning some subclasses of the max sum classifier besides the max sum classifiers with the acyclic neighbouring structure it has been shown that the discriminative learning is possible even with arbitrary neighbouring structure provided the quality functions fulfill some additional constraints in this article we extend the discriminative approach to other three classes of max sum classifiers with an arbitrary neighbourhood structure we derive learning algorithms for two subclasses of max sum classifiers whose response can be computed in polynomial time i the max sum classifiers with supermodular quality functions and ii the max sum classifiers whose response can be computed exactly by a linear programming relaxation moreover we show that the learning problem can be approximately solved even for a general max sum classifier

an information theoretic and game theoretic study of timing channels this paper focuses on jammed timing channels pure delay jammers with a maximum delay constraint an average delay constraint or a maximum buffer size constraint are explored for continuous time or discrete time packet waveforms fluid waveform approximations of each of these classes of waveforms are employed to aid in analysis channel capacity is defined and an information theoretic game based on mutual information rate is studied min max optimal jammers and max min optimal input processes are sought bounds on the min max and max min mutual information rates are described and numerical examples are given for maximum delay constrained mdc jammers with continuous time packet waveforms saddle point input and jammer strategies are identified the capacity of the maximum delay constrained jamming channel with continuous time packet waveforms is shown to equal the mutual information rate of the saddle point for mdc jammers with discrete time packet waveforms saddle point strategies are shown to exist jammers which have quantized batch departures at regular intervals are shown to perform well input processes with batches at regular intervals perform well for mdc or maximum buffer size constrained jammers

a coding theorem for a class of stationary channels with feedback a coding theorem is proved for a class of stationary channels with feedback in which the output yn f xn m n zn m n is the function of the current and past m symbols from the channel input xn and the stationary ergodic channel noise zn in particular it is shown that the feedback capacity is equal to limnrarr infin supp x n y n 1 1 n i xn rarr yn where i xn rarr yn sigmai 1 n i xi yi yi 1 denotes the massey directed information from the channel input to the output and the supremum is taken over all causally conditioned distributions p xn yn 1 pii 1 n p xi xi 1 yi 1 the main ideas of the proof are a classical application of the shannon strategy for coding with side information and a new elementary coding technique for the given channel model without feedback which is in a sense dual to gallager s lossy coding of stationary ergodic sources a similar approach gives a simple alternative proof of coding theorems for finite state channels by yang kavcic tatikonda chen berger and permuter weissman goldsmith

capacity and coding for the gilbert elliott channels the gilbert elliott channel a varying binary symmetric channel with crossover probabilities determined by a binary state markov process is treated in general such a channel has a memory that depends on the transition probabilities between the states a method of calculating the capacity of this channel is introduced and applied to several examples and the question of coding is addressed in the conventional usage of varying channels a code suitable for memoryless channels is used in conjunction with an interleaver with the decoder considering the deinterleaved symbol stream as the output of a derived memoryless channel the transmission rate is limited by the capacity of this memoryless channel which is often considerably less than the capacity of the original channel a decision feedback decoding algorithm that completely recovers this capacity loss is introduced it is shown that the performance of a system incorporating such an algorithm is determined by an equivalent genie aided channel the capacity of which equals that of the original channel the calculated random coding exponent of the genie aided channel indicates a considerable increase in the cutoff rate over that of the conventionally derived memoryless channel

lower bounds and approximations for the information rate of the isi channel we consider the discrete time intersymbol interference isi channel model with additive gaussian noise and fixed independent identically distributed inputs in this setting we investigate the expression put forth by shamai and laroia as a conjectured lower bound for the input output mutual information after application of a minimum mean square error decision feedback equalizer receiver a low signal to noise ratio snr expansion is used to prove that the conjectured bound does not hold under general conditions and to characterize inputs for which it is particularly ill suited one such input is used to construct a counterexample indicating that the shamai laroia expression does not always bound even the achievable rate of the channel thus excluding a natural relaxation of the original conjectured bound however this relaxed bound is then shown to hold for any finite entropy input and isi channel when the snr is sufficiently high we derive two conditions under which the relaxed bound holds involving compound channel capacity and quasiconvexity arguments finally new simple bounds for the achievable rate are proven and compared with other known bounds information estimation relations and estimation theoretic bounds play a key role in establishing our results