design of capacity approaching irregular low density parity check codes we design low density parity check ldpc codes that perform at rates extremely close to the shannon capacity the codes are built from highly irregular bipartite graphs with carefully chosen degree patterns on both sides our theoretical analysis of the codes is based on the work of richardson and urbanke see ibid vol 47 no 2 p 599 618 2000 assuming that the underlying communication channel is symmetric we prove that the probability densities at the message nodes of the graph possess a certain symmetry using this symmetry property we then show that under the assumption of no cycles the message densities always converge as the number of iterations tends to infinity furthermore we prove a stability condition which implies an upper bound on the fraction of errors that a belief propagation decoder can correct when applied to a code induced from a bipartite graph with a given degree distribution our codes are found by optimizing the degree structure of the underlying graphs we develop several strategies to perform this optimization we also present some simulation results for the codes found which show that the performance of the codes is very close to the asymptotic theoretical bounds

faulty successive cancellation decoding of polar codes for the binary erasure channel in this paper the faulty successive cancellation decoding of polar codes for the binary erasure channel is studied to this end a simple erasure based fault model is introduced to represent errors in the decoder and it is shown that under this model polarization does not happen meaning that fully reliable communication is not possible at any rate furthermore a lower bound on the frame error rate of polar codes under faulty successive cancellation decoding is provided which is then used along with a well known upper bound in order to choose a blocklength that minimizes the erasure probability under faulty decoding finally an unequal error protection scheme that can re enable asymptotically erasure free transmission at a small rate loss and by protecting only a constant fraction of the decoder is proposed the same scheme is also shown to significantly improve the finite length performance of the faulty successive cancellation decoder by protecting as little as 1 5 of the decoder

efficient erasure correcting codes we introduce a simple erasure recovery algorithm for codes derived from cascades of sparse bipartite graphs and analyze the algorithm by analyzing a corresponding discrete time random process as a result we obtain a simple criterion involving the fractions of nodes of different degrees on both sides of the graph which is necessary and sufficient for the decoding process to finish successfully with high probability by carefully designing these graphs we can construct for any given rate r and any given real number spl epsiv a family of linear codes of rate r which can be encoded in time proportional to ln 1 spl epsiv times their block length n furthermore a codeword can be recovered with high probability from a portion of its entries of length 1 spl epsiv rn or more the recovery algorithm also runs in time proportional to n ln 1 spl epsiv our algorithms have been implemented and work well in practice various implementation issues are discussed

reliable computation by formulas in the presence of noise it is shown that if formulas are used to compute boolean functions in the presence of randomly occurring failures then 1 there is a limit strictly less than 1 2 to the failure probability per gate that can be tolerated and 2 formulas that tolerate failures must be deeper and therefore compute more slowly than those that do not the heart of the proof is an information theoretic argument that deals with computation and errors in very general terms the strength of this argument is that it applies with equal ease no matter what types of gate are available its weaknesses is that it does not seem to predict quantitatively the limiting value of the failure probability or the ratio by which computation proceeds more slowly in the presence of failures

weight distribution of low density parity check codes we derive the average weight distribution function and its asymptotic growth rate for low density parity check ldpc code ensembles we show that the growth rate of the minimum distance of ldpc codes depends only on the degree distribution pair it turns out that capacity achieving sequences of standard unstructured ldpc codes under iterative decoding over the binary erasure channel bec known to date have sublinearly growing minimum distance in the block length

practical perfect sampling using composite bounding chains the dirichlet multinomial model a discrete data augmentation scheme together with two different parameterizations yields two gibbs samplers for sampling from the posterior distribution of the hyperparameters of the dirichlet multinomial hierarchical model under a default prior distribution the finite state space nature of this data augmentation permits us to construct two perfect samplers using bounding chains that take advantage of monotonicity and anti monotonicity in the target posterior distribution but both are impractically slow we demonstrate that a composite algorithm that strategically alternates between the two samplers updates can be substantially faster than either individually the speed gains come because the composite algorithm takes a divide and conquer approach in which one update quickly shrinks the bounding set for the augmented data and the other update immediately coalesces on the parameter once the augmented data bounding set is a singleton we theoretically bound the expected time until coalescence for the composite algorithm and show via simulation that the theoretical bounds can be close to actual performance copyright 2013 oxford university press

expander codes using expander graphs we construct a new family of asymptotically good linear error correcting codes these codes have linear time sequential decoding algorithms and logarithmic time parallel decoding algorithms that use a linear number of processors we present both randomized and explicit constructions of these codes experimental results demonstrate the good performance of the randomly chosen codes

generalized statistical mechanics connection with thermodynamics the authors show the manner through which the recent generalization by tsallis 1988 of the boltzmann gibbs statistics becomes consistent with a generalized thermodynamics preserving the legendre transformation framework of standard thermodynamics in addition to that they generalize the shannon additivity

analysis of the m1 model well posedness and diffusion asymptotics abstract this paper is devoted to the analysis of the m 1 model which arises in radiative transfer theory the derivation of the model is based on the entropy minimization principle which leads to a hyperbolic system of balance laws with relaxation in the multi dimensional case we establish the existence uniqueness of a globally defined smooth solution under a suitable smallness condition on the initial data in the one dimensional case we show that the smallness condition does not depend on the particles mean free path so that we can also rigorously justify the consistency of the model with the diffusion asymptotics the result extends the analysis of coulombel et al j f coulombel f golse t goudon diffusion approximation and entropy based moment closure for kinetic equations asymptotic analysis 45 2005 1 39 to the case where the entropy functional accounts for relaxation towards the planckian state which is physically more relevant

analysis of sum product decoding of low density parity check codes using a gaussian approximation density evolution is an algorithm for computing the capacity of low density parity check ldpc codes under message passing decoding for memoryless binary input continuous output additive white gaussian noise awgn channels and sum product decoders we use a gaussian approximation for message densities under density evolution to simplify the analysis of the decoding algorithm we convert the infinite dimensional problem of iteratively calculating message densities which is needed to find the exact threshold to a one dimensional problem of updating the means of the gaussian densities this simplification not only allows us to calculate the threshold quickly and to understand the behavior of the decoder better but also makes it easier to design good irregular ldpc codes for awgn channels for various regular ldpc codes we have examined thresholds can be estimated within 0 1 db of the exact value for rates between 0 5 and 0 9 codes designed using the gaussian approximation perform within 0 02 db of the best performing codes found so far by using density evolution when the maximum variable degree is 10 we show that by using the gaussian approximation we can visualize the sum product decoding algorithm we also show that the optimization of degree distributions can be understood and done graphically using the visualization

constructing free energy approximations and generalized belief propagation algorithms important inference problems in statistical physics computer vision error correcting coding theory and artificial intelligence can all be reformulated as the computation of marginal probabilities on factor graphs the belief propagation bp algorithm is an efficient way to solve these problems that is exact when the factor graph is a tree but only approximate when the factor graph has cycles we show that bp fixed points correspond to the stationary points of the bethe approximation of the free energy for a factor graph we explain how to obtain region based free energy approximations that improve the bethe approximation and corresponding generalized belief propagation gbp algorithms we emphasize the conditions a free energy approximation must satisfy in order to be a valid or maxent normal approximation we describe the relationship between four different methods that can be used to generate valid approximations the bethe method the junction graph method the cluster variation method and the region graph method finally we explain how to tell whether a region based approximation and its corresponding gbp algorithm is likely to be accurate and describe empirical results showing that gbp can significantly outperform bp

using linear programming to decode binary linear codes a new method is given for performing approximate maximum likelihood ml decoding of an arbitrary binary linear code based on observations received from any discrete memoryless symmetric channel the decoding algorithm is based on a linear programming lp relaxation that is defined by a factor graph or parity check representation of the code the resulting lp decoder generalizes our previous work on turbo like codes a precise combinatorial characterization of when the lp decoder succeeds is provided based on pseudocodewords associated with the factor graph our definition of a pseudocodeword unifies other such notions known for iterative algorithms including stopping sets irreducible closed walks trellis cycles deviation sets and graph covers the fractional distance d sub frac of a code is introduced which is a lower bound on the classical distance it is shown that the efficient lp decoder will correct up to spl lceil d sub frac 2 spl rceil 1 errors and that there are codes with d sub frac spl omega n sup 1 spl epsi an efficient algorithm to compute the fractional distance is presented experimental evidence shows a similar performance on low density parity check ldpc codes between lp decoding and the min sum and sum product algorithms methods for tightening the lp relaxation to improve performance are also provided

computing linear transformations with unreliable components we consider the problem of computing a binary linear transformation when all circuit components are unreliable two models of unreliable components are considered probabilistic errors and permanent errors we introduce the encoded technique that ensures that the error probability of the computation of the linear transformation is kept bounded below a small constant independent of the size of the linear transformation even when all logic gates in the computation are noisy by deriving a lower bound we show that in some cases the computational complexity of the encoded technique achieves the optimal scaling in error probability further we examine the gain in energy efficiency from the use of a voltage scaling scheme where gate energy is reduced by lowering the supply voltage we use a gate energy reliability model to show that tuning gate energy appropriately at different stages of the computation dynamic voltage scaling in conjunction with encoded can lead to orders of magnitude energy savings over the classical uncoded approach finally we also examine the problem of computing a linear transformation when noiseless decoders can be used providing upper and lower bounds to the problem

signal codes convolutional lattice codes the coded modulation scheme proposed in this paper has a simple construction an integer sequence representing the information is convolved with a fixed continuous valued finite impulse response fir filter to generate the codeword a lattice point due to power constraints the code construction includes a shaping mechanism inspired by precoding techniques such as the tomlinson harashima filter we naturally term these codes convolutional lattice codes or alternatively signal codes due to the signal processing interpretation of the code construction surprisingly properly chosen short fir filters can generate good codes with large minimal distance decoding can be done efficiently by sequential decoding or for better performance by bidirectional sequential decoding error analysis and simulation results indicate that for the additive white gaussian noise awgn channel convolutional lattice codes with computationally reasonable decoders can achieve low error rate close to the channel capacity

performance of belief propagation for decoding ldpc codes in the presence of channel estimation error in this paper we investigate the performance of the belief propagation bp algorithm for decoding low density parity check codes over the additive white gaussian noise channel when there is an incorrect estimate of the channel signal to noise ratio snr referred to as snr mismatch at the decoder at the extremes for over and underestimation of snr the performance of bp tends to that of min sum algorithm and the channel bit error rate respectively our results for regular codes indicate that the sensitivity to mismatch increases by increasing the variable node degree and by decreasing the check node degree the effect of variable node degree however appears to be more profound such that at a given rate the codes with the smallest variable and check degrees are more robust against snr mismatch for irregular codes by comparing the thresholds of a few ensembles we demonstrate that the ensemble which performs better in the absence of mismatch can perform worse in the presence of it to obtain our asymptotic results we propose a computationally efficient method based on the gaussian approximation of density evolution in the presence of snr mismatch we also show that the asymptotic results are consistent with simulation results for codes with finite block lengths

design of length compatible polar codes based on the reduction of polarizing matrices length compatible polar codes are a class of polar codes which can support a wide range of lengths with a single pair of encoder and decoder in this paper we propose a method to construct length compatible polar codes by employing the reduction of the 2n 2n polarizing matrix proposed by arikan the conditions under which a reduced matrix becomes a polarizing matrix supporting a polar code of a given length are first analyzed based on these conditions length compatible polar codes are constructed in a suboptimal way by codeword puncturing and information refreezing processes they have low encoding and decoding complexity since they can be encoded and decoded in a similar way as a polar code of length 2n numerical results show that length compatible polar codes designed by the proposed method provide a performance gain of about 1 0 5 0 db over those obtained by random puncturing when successive cancellation decoding is employed

euclidean and space time block codes relationship optimality performance analysis revisited an equivalent model for a multiple input multiple output communication system with space time block codes stbcs is proposed based on a revealed connection between stbcs and euclidean codes examples of distance spectra signal constellations and signal coordinate diagrams of euclidean codes equivalent to simplest orthogonal stbcs are given a new asymptotic upper bound for the symbol error rate ser of stbcs based on the distance spectra of the equivalent euclidean codes is derived and new general design criteria for signal constellations of the optimal code are proposed some bounds relating distance properties dimensionality and cardinality of stbcs with constituent signals of equal energy are given and new signal constellations with cardinalities of 8 and 16 for alamouti s code are designed a general methodology for performance analysis of stbcs is revisited as an example of the application of this methodology an exact evaluation of the ser of an orthogonal stbc is given namely a new expression for the ser of alamouti s code with binary phase shift keying signals is derived

the art of signaling fifty years of coding theory in 1948 shannon developed fundamental limits on the efficiency of communication over noisy channels the coding theorem asserts that there are block codes with code rates arbitrarily close to channel capacity and probabilities of error arbitrarily close to zero fifty years later codes for the gaussian channel have been discovered that come close to these fundamental limits there is now a substantial algebraic theory of error correcting codes with as many connections to mathematics as to engineering practice and the last 20 years have seen the construction of algebraic geometry codes that can be encoded and decoded in polynomial time and that beat the gilbert varshamov bound given the size of coding theory as a subject this review is of necessity a personal perspective and the focus is reliable communication and not source coding or cryptography the emphasis is on connecting coding theories for hamming and euclidean space and on future challenges specifically in data networking wireless communication and quantum information theory

stochastic belief propagation a low complexity alternative to the sum product algorithm the belief propagation bp or sum product algorithm is a widely used message passing method for computing marginal distributions in graphical models at the core of the bp message updates when applied to a graphical model involving discrete variables with pairwise interactions lies a matrix vector product with complexity that is quadratic in the state dimension d and requires transmission of a d 1 dimensional vector of real numbers messages to its neighbors since various applications involve very large state dimensions such computation and communication complexities can be prohibitively complex in this paper we propose a low complexity variant of bp referred to as stochastic belief propagation sbp as suggested by the name it is an adaptively randomized version of the bp message updates in which each node passes randomly chosen information to each of its neighbors the sbp message updates reduce the computational complexity per iteration from quadratic to linear in d without assuming any particular structure of the potentials and also reduce the communication complexity significantly requiring only log2d bits transmission per edge moreover we establish a number of theoretical guarantees for the performance of sbp showing that it converges almost surely to the bp fixed point for any tree structured graph and for any graph with cycles satisfying a contractivity condition in addition for these graphical models we provide nonasymptotic upper bounds on the convergence rate showing that the l norm of the error vector decays no slower than o 1 t with the number of iterations t on trees and the normalized mean squared error decays as o 1 t for general graphs this analysis also supported by experimental results shows that sbp can provably yield reductions in computational and communication complexities for various classes of graphical models

signal propagation and noisy circuits the information carried by a signal decays when the signal is corrupted by random noise this occurs when a message is transmitted over a noisy channel as well as when a noisy component performs computation we first study this signal decay in the context of communication and obtain a tight bound on the rate at which information decreases as a signal crosses a noisy channel we then use this information theoretic result to obtain depth lower bounds in the noisy circuit model of computation defined by von neumann in this model each component fails produces 1 instead of 0 or vice versa independently with a fixed probability and yet the output of the circuit is required to be correct with high probability von neumann showed how to construct circuits in this model that reliably compute a function and are no more than a constant factor deeper than noiseless circuits for the function we provide a lower bound on the multiplicative increase in circuit depth necessary for reliable computation and an upper bound on the maximum level of noise at which reliable computation is possible

possible generalization of boltzmann gibbs statistics with the use of a quantity normally scaled in multifractals a generalized form is postulated for entropy namelys q k 1 i 1 w p i q q 1 whereq characterizes the generalization andp i are the probabilities associated withw microscopic configurations w the main properties associated with this entropy are established particularly those corresponding to the microcanonical and canonical ensembles the boltzmann gibbs statistics is recovered as theq 1 limit

information theoretical approach to radiative transfer abstract the maximum entropy formalism is used to obtain the radiation and matter distribution functions for radiative systems is steady nonequilibrium states under the gray approximation the radiation distribution function is expanded in a smallness parameter which vanishes at equilibrium in the first near equilibrium approximation we derive the results of near equilibrium diffusion theory this may be regarded as an analogue to the kinetic theoretical result according to which in the first enskog approximation the fourier heat conduction equation is obtained the theory is also developed up to the second order leading to results which apply to situations further away from equilibrium than those corresponding to near equilibrium diffusion theory a simple application is analyzed

an analysis of the block error probability performance of iterative decoding asymptotic iterative decoding performance is analyzed for several classes of iteratively decodable codes when the block length of the codes n and the number of iterations i go to infinity three classes of codes are considered these are gallager s regular low density parity check ldpc codes tanner s generalized ldpc gldpc codes and the turbo codes due to berrou et al it is proved that there exist codes in these classes and iterative decoding algorithms for these codes for which not only the bit error probability p sub b but also the block frame error probability p sub b goes to zero as n and i go to infinity

information theory and the spectrum of isotropic turbulence the problem of determining the spectrum of isotropic turbulence can be thought of as one of finding the most appropriate joint probability distribution for the flow taken as a whole from the point of view of information theory what one means by the most appropriate distribution is clearly defined and easily justified it is the probability distribution that maximises the information theory entropy subject to whatever constraints one can impose on the flow in this work the relevant constraints are taken to be the reynolds number and energy dissipation rate of the flow energy balance on average at every point in wavenumber space and adherence to the navier stokes equations using these constraints it is shown that the maximum entropy formalism leads to a pair of coupled equations describing the distribution of energy in the turbulent spectrum and the correlations between the amplitudes of velocity components with nearly identical wavenumbers although solutions to these equations are not presented it develops that if a power law solution exists it can only be the kolmogorov law e k varies as k 53 in arriving at this result a useful concept is that of the turbulent temperature defined as the reciprocal of the derivative of the entropy with respect to the local energy dissipation rate this quantity plays a role directly analogous to the thermodynamic temperature governing the rate of energy exchange between different wavenumbers it is found that within the spectrum s inertial subrange the turbulent temperature is virtually constant with only a minute temperature gradient required to drive the energy cascade

gallager b ldpc decoder with transient and permanent errors this paper studies the performance of a noisy gallager b decoder for regular ldpc codes we assume that the noisy decoder is subject to both transient processor errors and permanent memory errors we permit different error rates at different functional components in addition for the sake of generality we allow asymmetry in the permanent error rates of component outputs and thus we model error propagation in the decoder via a suitable asymmetric channel we then develop a density evolution type analysis on this asymmetric channel the recursive expression for the bit error probability is derived as a function of the code parameters node degrees codeword weight transmission error rate and the error rates of the permanent and the transient errors based on this analysis we then derive the residual error of the gallager b decoder for the regime where the transmission error rate and the processing error rates are small in this regime we further observe that the residual error rate can be well approximated by a suitable combination of the transient error rate and the permanent error rate at variable nodes provided that the check node degree is large enough based on this insight we then propose and analyze a scheme for detecting permanent errors and correcting detected residual errors the scheme exploits the parity check equations of the code and reuses the existing hardware to locate permanent errors in memory blocks performance analysis and simulation results show that with high probability the detection scheme discovers correct locations of permanent memory errors while with low probability it mislabels the functional memory as being defective the proposed error detection and correction scheme can be implemented in circuit and is useful in combating failures arising from aging

recursive consistent estimation with bounded noise estimation problems with bounded uniformly distributed noise arise naturally in reconstruction problems from over complete linear expansions with subtractive dithered quantization we present a simple recursive algorithm for such bounded noise estimation problems the mean square error mse of the algorithm is almost o 1 n sup 2 where n is the number of samples this rate is faster than the o 1 n mse obtained by standard recursive least squares estimation and is optimal to within a constant factor

tight bounds for ldpc and ldgm codes under map decoding a new method for analyzing low density parity check ldpc codes and low density generator matrix ldgm codes under bit maximum a posteriori probability map decoding is introduced the method is based on a rigorous approach to spin glasses developed by francesco guerra it allows one to construct lower bounds on the entropy of the transmitted message conditional to the received one based on heuristic statistical mechanics calculations we conjecture such bounds to be tight the result holds for standard irregular ensembles when used over binary input output symmetric bios channels the method is first developed for tanner graph ensembles with poisson left degree distribution it is then generalized to multi poisson graphs and by a completion procedure to arbitrary degree distribution

a more accurate one dimensional analysis and design of irregular ldpc codes we introduce a new one dimensional 1 d analysis of low density parity check ldpc codes on additive white gaussian noise channels which is significantly more accurate than similar 1 d methods our method assumes a gaussian distribution in message passing decoding only for messages from variable nodes to check nodes compared to existing work which makes a gaussian assumption both for messages from check nodes and from variable nodes our method offers a significantly more accurate estimate of convergence behavior and threshold of convergence similar to previous work the problem of designing irregular ldpc codes reduces to a linear programming problem however our method allows irregular code design in a wider range of rates without any limit on the maximum variable node degree we use our method to design irregular ldpc codes with rates greater than 1 4 that perform within a few hundredths of a decibel from the shannon limit the designed codes perform almost as well as codes designed by density evolution

density evolution and functional threshold for the noisy min sum decoder this paper investigates the behavior of the min sum decoder running on noisy devices our aim is to evaluate the robustness of the decoder to computation noise caused by the faulty logic in the processing units this type of noise represents a new source of errors that may occur during the decoding process to this end we first introduce probabilistic models for the arithmetic and logic units of the finite precision min sum decoder and then carry out the density evolution analysis of the noisy min sum decoder we show that in some particular cases the noise introduced by the device can help the min sum decoder to escape from fixed points attractors and may actually result in an increased correction capacity with respect to the noiseless decoder we also point out a specific threshold phenomenon referred to as functional threshold which accurately describes the convergence behavior of noisy decoders the behavior of the noisy ms is demonstrated in the asymptotic limit of the code length through a noisy version of density evolution and is also verified in the finite length case by monte carlo simulations

correctness of local probability propagation in graphical models with loops graphical models such as bayesian networks and markov networks represent joint distributions over a set of variables by means of a graph when the graph is singly connected local propagation rules of the sort proposed by pearl 1988 are guaranteed to converge to the correct posterior probabilities recently a number of researchers have empirically demonstrated good performance of these same local propagation schemes on graphs with loops but a theoretical understanding of this performance has yet to be achieved for graphical models with a single loop we derive an analytical relationship between the probabilities computed using local propagation and the correct marginals using this relationship we show a category of graphical models with loops for which local propagation gives rise to provably optimal maximum a posteriori assignments although the computed marginals will be incorrect we also show how nodes can use local information in the messages they receive in order to correct their computed marginals we discuss how these results can be extended to graphical models with multiple loops and show simulation results suggesting that some properties of propagation on single loop graphs may hold for a larger class of graphs specifically we discuss the implication of our results for understanding a class of recently proposed error correcting codes known as turbo codes

a new class of multiple rate codes based on block markov superposition transmission the hadamard transform ht over the binary field provides a natural way to implement multiple rate codes referred to as ht coset codes where the code length n 2 p is fixed but the code dimension k can be varied from 1 to n 1 by adjusting the set of frozen bits the ht coset codes including reed muller rm codes and polar codes as typical examples can share the same fundamental encoder decoder architecture with the implementation complexity of order o n log n however to guarantee that all codes with designated rates perform well ht coset coding usually requires a sufficiently large code length which in turn causes difficulties in the determination of which bits are better for being frozen in this paper we propose to transmit short ht coset codes in the so called block markov superposition transmission bmst manner at the transmitter signals are spatially coupled via superposition resulting in long codes at the receiver these coupled signals are recovered by a sliding window iterative soft successive cancellation decoding algorithm most importantly the performance around or below the bit error rate ber of 10 5 can be predicted by a simple genie aided lower bound both these bounds and simulation results show that the bmst of short ht coset codes performs well within one db away from the corresponding shannon limits in a wide range of code rates

polar coded modulation a framework is proposed that allows for a joint description and optimization of both binary polar coding and 2m ary digital pulse amplitude modulation pam schemes for the latter the multilevel coding mlc approach as well as bit interleaved coded modulation bicm are considered the conceptual equivalence of polar coding and multilevel coding is covered in detail based on an alternative characterization of the channel polarization phenomenon rules for the optimum choice of the labeling in coded modulation schemes employing polar codes are developed simulation results regarding the error performance of the proposed schemes on the awgn channel are included

belief propagation algorithms on noisy hardware the wide recognition that emerging nano devices will be inherently unreliable motivates the evaluation of information processing algorithms running on noisy hardware as well as the design of robust schemes for reliable performance against hardware errors of varied characteristics in this paper we investigate the performance of a popular statistical inference algorithm belief propagation bp on probabilistic graphical models implemented on noisy hardware and we propose two robust implementations of the bp algorithm targeting different computation noise distributions we assume that the bp messages are subject to zero mean transient additive computation noise we focus on graphical models satisfying the contraction mapping condition that guarantees the convergence of the noise free bp we first upper bound the distances between the noisy bp messages and the fixed point of noise free bp as a function of the iteration number next we propose two implementations of bp namely censoring bp and averaging bp that are robust to computation noise censoring bp rejects incorrect computations to keep the algorithm on the right track to convergence while averaging bp takes the average of the messages in all iterations up to date to mitigate the effects of computation noise censoring bp works effectively when with high probability the computation noise is exactly zero and averaging bp although having a slightly larger overhead works effectively for general zero mean computation noise distributions sufficient conditions on the convergence of censoring bp and averaging bp are derived simulations on the ising model demonstrate that the two proposed implementations successfully converge to the fixed point achieved by noise free bp additionally we apply averaging bp to a bp based image denoising algorithm and as a bp decoder for ldpc codes in the image denoising application averaging bp successfully denoises an image even when nominal bp fails to do so in the presence of computation noise in the bp ldpc decoder application the power of averaging bp is manifested by the reduction in the residual error rates compared with the nominal bp decoder

perfect samplers for mixtures of distributions summary we consider the construction of perfect samplers for posterior distributions associated with mixtures of exponential families and conjugate priors starting with a perfect slice sampler in the spirit of mira and co workers the methods rely on a marginalization akin to rao blackwellization and illustrate the duality principle of diebolt and robert a first approximation embeds the finite support distribution on the latent variables within a continuous support distribution that is easier to simulate by slice sampling but we later demonstrate that the approximation can be very poor we conclude by showing that an alternative perfect sampler based on a single backward chain can be constructed this alternative can handle much larger sample sizes than the slice sampler first proposed

on implementation of min sum algorithm and its modifications for decoding low density parity check ldpc codes the effects of clipping and quantization on the performance of the min sum algorithm for the decoding of low density parity check ldpc codes at short and intermediate block lengths are studied it is shown that in many cases only four quantization bits suffice to obtain close to ideal performance over a wide range of signal to noise ratios moreover we propose modifications to the min sum algorithm that improve the performance by a few tenths of a decibel with just a small increase in decoding complexity a quantized version of these modified algorithms is also studied it is shown that when optimized modified quantized min sum algorithms perform very close to and in some cases even slightly outperform the ideal belief propagation algorithm at observed error rates

on the construction of polar codes for channels with moderate input alphabet sizes current deterministic algorithms for the construction of polar codes can only be argued to be practical for channels with small input alphabet sizes in this paper we show that any construction algorithm for channels with moderate input alphabet size which follows the paradigm of degrading after each polarization step will inherently be impractical with respect to a certain hard underlying channel this result also sheds light on why the construction of low density parity check codes using density evolution is impractical for channels with moderate sized input alphabets

approximating the α permanent summary the standard matrix permanent is the solution to a number of combinatorial and graph theoretic problems and the weighted permanent is the density function for a class of cox processes called boson processes the exact computation of the ordinary permanent is known to be p complete and the same appears to be the case for the permanent for most values of at present the lack of a satisfactory algorithm for approximating the permanent is a formidable obstacle to the use of boson processes in applied work this paper proposes an importance sampling estimator using nonuniform random permutations generated in a cycle format empirical investigation reveals that the estimator works well for the sorts of matrices that arise in point process applications involving up to a few hundred points we conclude with a numerical illustration of the bayes estimate of the intensity function of a boson point process which is a ratio of permanents

perfect simulation of conditionally specified models we discuss how the ideas of producing perfect simulations based on coupling from the past for finite state space models naturally extend to multivariate distributions with infinite or uncountable state spaces such as auto gamma auto poisson and autonegative binomial models using gibbs sampling in combination with sandwiching methods originally introduced for perfect simulation of point processes

bayesian density estimation and inference using mixtures abstract we describe and illustrate bayesian inference in models for density estimation using mixtures of dirichlet processes these models provide natural settings for density estimation and are exemplified by special cases where data are modeled as a sample from mixtures of normal distributions efficient simulation methods are used to approximate various prior posterior and predictive distributions this allows for direct inference on a variety of practical issues including problems of local versus global smoothing uncertainty about density estimates assessment of modality and the inference on the numbers of components also convergence results are established for a general class of normal mixture models

mismatched decoding revisited general alphabets channels with memory and the wide band limit the mismatch capacity of a channel is the highest rate at which reliable communication is possible over the channel with a given possibly suboptimal decoding rule this quantity has been studied extensively for single letter decoding rules over discrete memoryless channels dmcs here we extend the study to memoryless channels with general alphabets and to channels with memory with possibly non single letter decoding rules we also study the wide band limit and in particular the mismatch capacity per unit cost and the achievable rates on an additive noise spread spectrum system with single letter decoding and binary signaling

the extraction and complexity limits of graphical models for linear codes two broad classes of graphical modeling problems for codes can be identified in the literature constructive and extractive problems the former class of problems concern the construction of a graphical model in order to define a new code the latter class of problems concern the extraction of a graphical model for a fixed given code the design of a new low density parity check code for some given criteria e g target block length and code rate is an example of a constructive problem the determination of a graphical model for a classical linear block code that implies a decoding algorithm with desired performance and complexity characteristics is an example of an extractive problem this work focuses on extractive graphical model problems and aims to lay out some of the foundations of the theory of such problems for linear codes the primary focus of this work is a study of the space of all graphical models for a fixed given code the tradeoff between cyclic topology and complexity in this space is characterized via the introduction of a new bound the forest inducing cut set bound fi csb the proposed bound provides a more precise characterization of this tradeoff than that which can be obtained using existing tools e g the csb and can be viewed as a generalization of the square root bound for tail biting trellises to graphical models with arbitrary cyclic topologies searching the space of graphical models for a given code is then enabled by introducing a set of basic graphical model transformation operations that are shown to span this space finally heuristics for extracting novel graphical models for linear block codes using these transformations are investigated

performance of ldpc codes under faulty iterative decoding departing from traditional communication theory where decoding algorithms are assumed to perform without error a system where noise perturbs both computational devices and communication channels is considered here this paper studies limits in processing noisy signals with noisy circuits by investigating the effect of noise on standard iterative decoders for low density parity check ldpc codes concentration of decoding performance around its average is shown to hold when noise is introduced into message passing and local computation density evolution equations for simple faulty iterative decoders are derived in one model computing nonlinear estimation thresholds shows that performance degrades smoothly as decoder noise increases but arbitrarily small probability of error is not achievable probability of error may be driven to zero in another system model the decoding threshold again decreases smoothly with decoder noise as an application of the methods developed an achievability result for reliable memory systems constructed from unreliable components is provided

expander graph arguments for message passing algorithms we show how expander based arguments may be used to prove that message passing algorithms can correct a linear number of erroneous messages the implication of this result is that when the block length is sufficiently large once a message passing algorithm has corrected a sufficiently large fraction of the errors it will eventually correct all errors this result is then combined with known results on the ability of message passing algorithms to reduce the number of errors to an arbitrarily small fraction for relatively high transmission rates the results hold for various message passing algorithms including gallager s hard decision and soft decision with clipping decoding algorithms our results assume low density parity check ldpc codes based on an irregular bipartite graph

channel combining and splitting for cutoff rate improvement the cutoff rate r0 w of a discrete memoryless channel dmc w is often used as a figure of merit alongside the channel capacity c w if a channel w is split into two possibly correlated subchannels w1 w2 the capacity function always satisfies c w1 c w2 lesc w while there are examples for which r0 w1 r0 w2 r0 w the fact that cutoff rate can be created by channel splitting was noticed by massey in his study of an optical modulation system this paper gives a general framework for achieving similar gains in the cutoff rate of arbitrary dmcs by methods of channel combining and splitting the emphasis is on simple schemes that can be implemented in practice we give several examples that achieve significant gains in cutoff rate at little extra system complexity theoretically as the complexity grows without bound the proposed framework is capable of boosting the cutoff rate of a channel to arbitrarily close to its capacity in a sense made precise in the paper apart from massey s work the methods studied here have elements in common with forney s concatenated coding idea a method by pinsker for cutoff rate improvement and certain coded modulation techniques namely ungerboeck s set partitioning idea and imai hirakawa multilevel coding these connections are discussed in the paper

convergence analysis of reweighted sum product algorithms markov random fields are designed to represent structured dependencies among large collections of random variables and are well suited to capture the structure of real world signals many fundamental tasks in signal processing e g smoothing denoising segmentation etc require efficient methods for computing approximate marginal probabilities over subsets of nodes in the graph the marginalization problem though solvable in linear time for graphs without cycles is computationally intractable for general graphs with cycles this intractability motivates the use of approximate ldquomessage passingrdquo algorithms this paper studies the convergence and stability properties of the family of reweighted sum product algorithms a generalization of the widely used sum product or belief propagation algorithm in which messages are adjusted with graph dependent weights for pairwise markov random fields we derive various conditions that are sufficient to ensure convergence and also provide bounds on the geometric convergence rates when specialized to the ordinary sum product algorithm these results provide strengthening of previous analyses we prove that some of our conditions are necessary and sufficient for subclasses of homogeneous models but not for general models the experimental simulations on various classes of graphs validate our theoretical results

lower bounds for the complexity of reliable boolean circuits with noisy gates proves that the reliable computation of any boolean function with sensitivity s requires spl omega s log s gates if the gates fail independently with a fixed positive probability this theorem was stated by dobrushin and ortyukov 1977 but their proof was found by pippenger stamoulis and tsitsiklis 1991 to contain some errors

acceleration of stochastic approximation by averaging a new recursive algorithm of stochastic approximation type with the averaging of trajectories is investigated convergence with probability one is proved for a variety of classical optimization and identification problems it is also demonstrated for these problems that the proposed algorithm achieves the highest possible rate of convergence

endcoding complexity versus minimum distance a bound on the minimum distance of a binary error correcting code is established given constraints on the computational time space complexity of its encoder where the encoder is modeled as a branching program the bound obtained asserts that if the encoder uses linear time and sublinear memory in the most general sense then the minimum distance of the code cannot grow linearly with the block length when the rate is nonvanishing that is the minimum relative distance of the code tends to zero in such a setting the setting is general enough to include nonserially concatenated turbo like codes and various generalizations our argument is based on branching program techniques introduced by ajtai the case of constant depth and or circuit encoders with unbounded fanins are also considered

sufficient conditions for convergence of the sum product algorithm novel conditions are derived that guarantee convergence of the sum product algorithm also known as loopy belief propagation or simply belief propagation bp to a unique fixed point irrespective of the initial messages for parallel synchronous updates the computational complexity of the conditions is polynomial in the number of variables in contrast with previously existing conditions our results are directly applicable to arbitrary factor graphs with discrete variables and are shown to be valid also in the case of factors containing zeros under some additional conditions the conditions are compared with existing ones numerically and if possible analytically for binary variables with pairwise interactions sufficient conditions are derived that take into account local evidence i e single variable factors and the type of pair interactions attractive or repulsive it is shown empirically that this bound outperforms existing bounds

capacity achieving ensembles for the binary erasure channel with bounded complexity we present two sequences of ensembles of nonsystematic irregular repeat accumulate ira codes which asymptotically as their block length tends to infinity achieve capacity on the binary erasure channel bec with bounded complexity per information bit this is in contrast to all previous constructions of capacity achieving sequences of ensembles whose complexity grows at least like the log of the inverse of the gap in rate to capacity the new bounded complexity result is achieved by puncturing bits and allowing in this way a sufficient number of state nodes in the tanner graph representing the codes we derive an information theoretic lower bound on the decoding complexity of randomly punctured codes on graphs the bound holds for every memoryless binary input output symmetric mbios channel and is refined for the binary erasure channel

channel polarization a method for constructing capacity achieving codes for symmetric binary input memoryless channels a method is proposed called channel polarization to construct code sequences that achieve the symmetric capacity i w of any given binary input discrete memoryless channel b dmc w the symmetric capacity is the highest rate achievable subject to using the input letters of the channel with equal probability channel polarization refers to the fact that it is possible to synthesize out of n independent copies of a given b dmc w a second set of n binary input channels wn i 1 les i les n such that as n becomes large the fraction of indices i for which i wn i is near 1 approaches i w and the fraction for which i wn i is near 0 approaches 1 i w the polarized channels wn i are well conditioned for channel coding one need only send data at rate 1 through those with capacity near 1 and at rate 0 through the remaining codes constructed on the basis of this idea are called polar codes the paper proves that given any b dmc w with i w 0 and any target rate r i w there exists a sequence of polar codes cfrn nges1 such that cfrn has block length n 2n rate ges r and probability of block error under successive cancellation decoding bounded as pe n r les o n 1 4 independently of the code rate this performance is achievable by encoders and decoders with complexity o n logn for each

reduced complexity decoding of ldpc codes various log likelihood ratio based belief propagation llr bp decoding algorithms and their reduced complexity derivatives for low density parity check ldpc codes are presented numerically accurate representations of the check node update computation used in llr bp decoding are described furthermore approximate representations of the decoding computations are shown to achieve a reduction in complexity by simplifying the check node update or symbol node update or both in particular two main approaches for simplified check node updates are presented that are based on the so called min sum approximation coupled with either a normalization term or an additive offset term density evolution is used to analyze the performance of these decoding algorithms to determine the optimum values of the key parameters and to evaluate finite quantization effects simulation results show that these reduced complexity decoding algorithms for ldpc codes achieve a performance very close to that of the bp algorithm the unified treatment of decoding techniques for ldpc codes presented here provides flexibility in selecting the appropriate scheme from performance latency computational complexity and memory requirement perspectives

improved low density parity check codes using irregular graphs we construct new families of error correcting codes based on gallager s 1973 low density parity check codes we improve on gallager s results by introducing irregular parity check matrices and a new rigorous analysis of hard decision decoding of these codes we also provide efficient methods for finding good irregular structures for such decoding algorithms our rigorous analysis based on martingales our methodology for constructing good irregular codes and the demonstration that irregular structure improves performance constitute key points of our contribution we also consider irregular codes under belief propagation we report the results of experiments testing the efficacy of irregular codes on both binary symmetric and gaussian channels for example using belief propagation for rate 1 4 codes on 16000 bits over a binary symmetric channel previous low density parity check codes can correct up to approximately 16 errors while our codes correct over 17 in some cases our results come very close to reported results for turbo codes suggesting that variations of irregular low density parity check codes may be able to match or beat turbo code performance