design of capacity approaching irregular low density parity check codes we design low density parity check ldpc codes that perform at rates extremely close to the shannon capacity the codes are built from highly irregular bipartite graphs with carefully chosen degree patterns on both sides our theoretical analysis of the codes is based on the work of richardson and urbanke see ibid vol 47 no 2 p 599 618 2000 assuming that the underlying communication channel is symmetric we prove that the probability densities at the message nodes of the graph possess a certain symmetry using this symmetry property we then show that under the assumption of no cycles the message densities always converge as the number of iterations tends to infinity furthermore we prove a stability condition which implies an upper bound on the fraction of errors that a belief propagation decoder can correct when applied to a code induced from a bipartite graph with a given degree distribution our codes are found by optimizing the degree structure of the underlying graphs we develop several strategies to perform this optimization we also present some simulation results for the codes found which show that the performance of the codes is very close to the asymptotic theoretical bounds

coding schemes for user cooperation in low power regimes we consider the design of cooperation coding schemes for a two user multiaccess channel mac in particular we consider two block markov coding schemes namely the multiplexed coding and the superposition coding although the fully multiplexed fmp coding outperforms the superposition coding in theory it is difficult to implement with practical error correction codes we therefore introduce a partially multiplexed pmp coding scheme for code rate r 0 5 and propose a simplified superposition coding scheme the outage analysis shows that these cooperation schemes provide significant gains over the non cooperative mac in the low power regime when the bandwidth w rarr infin we then propose a pmp code design using irregular repeat accumulate ira codes the design method can be extended to build the low rate pmp codes using irregular repeat zigzag hadamard irzh codes a new class of capacity achieving low rate codes simulation results demonstrate that the block markov multiplexed coding using pmp ira codes outperforms the practical superposition coding by 2 db at 10 2 fer and it provides a 10 db gain over the noncooperative mac the multiplex coding using pmp irzh codes offers 1 db gain over the superposition coding and a 9 5 db gain over noncooperative mac at 10 2 fer moreover the pmp irzh coding performs only 0 5 db away from the outage capacity of multiplexed coding in the low power regime

phase transitions in simple learning the authors investigate learning in the simplest type of a layered neural network the one layer perceptron the learning process is treated as a statistical dynamical problem quantities one is interested in include the relaxation time the learning time and the capacity and how they depend on noise and constraints on the weights the relaxation time is calculated as a function of the noise level and the number p of associations to be learned they consider three different cases for input patterns that are random and uncorrelated in the first where the connection weights are constrained to satisfy n 1 sigma i omega i2 s2 there is a critical value of p n separating regimes of perfect and imperfect learning at zero noise in contrast the second model unconstrained learning exhibits a different kind of transition at p n and noise plays no role in the third model where the constraint is imposed only on the thermal fluctuations there is a line of phase transitions terminating at p n and zero noise they have also considered learning with correlated input patterns the most important difference is the emergence of a second relaxation time which the authors interpret as the time it takes to learn a prototype of the patterns

hyperbolic cone manifolds short geodesics and schwarzian derivatives given a geometrically finite hyperbolic cone manifold with the cone singularity sufficiently short we construct a one parameter family of cone manifolds decreasing the cone angle to zero we also control the geometry of this one parameter family via the schwarzian derivative of the projective boundary and the length of closed geodesics

a statistical mechanical analysis of a bayesian inference scheme for an unrealizable rule within a bayesian framework we consider a system that learns from examples in particular using a statistical mechanical formalism we calculate the evidence and two performance measures namely the generalization error and the consistency measure for a linear perceptron trained and tested on a set of examples generated by a nonlinear teacher the teacher is said to be unrealizable because the student can never model it without error in fact our model allows us to interpolate between the known linear case and an unrealizable nonlinear case a comparison of the hyperparameters which maximize the evidence with those that optimize the performance measures reveals that when the student and teacher are fundamentally mismatched the evidence procedure is a misleading guide to optimizing the performance measures considered

finite size effects in bayesian model selection and generalization we show that in supervised learning from a supplied data set bayesian model selection based on the evidence does not optimize generalization performance even for a learnable linear problem this is demonstrated by examining the finite size effects in hyperparameter assignment from the evidence procedure and the resultant generalization performance our approach demonstrates the weakness of average case and asymptotic analyses using simulations we corroborate our analytic results and examine an alternative model selection criterion namely cross validation this numerical study shows that the cross validation hyperparameter estimates correlate more strongly than those of the evidence with optimal performance however we show that for a sufficiently large input dimension the evidence procedure could provide a reliable alternative to the more computationally expensive cross validation

low density parity check codes based on finite geometries a rediscovery and new results this paper presents a geometric approach to the construction of low density parity check ldpc codes four classes of ldpc codes are constructed based on the lines and points of euclidean and projective geometries over finite fields codes of these four classes have good minimum distances and their tanner 1981 graphs have girth 6 finite geometry ldpc codes can be decoded in various ways ranging from low to high decoding complexity and from reasonably good to very good performance they perform very well with iterative decoding furthermore they can be put in either cyclic or quasi cyclic form consequently their encoding can be achieved in linear time and implemented with simple feedback shift registers this advantage is not shared by other ldpc codes in general and is important in practice finite geometry ldpc codes can be extended and shortened in various ways to obtain other good ldpc codes several techniques of extension and shortening are presented long extended finite geometry ldpc codes have been constructed and they achieve a performance only a few tenths of a decibel away from the shannon theoretical limit with iterative decoding

finite size effects in learning and generalization in linear perceptrons most properties of learning and generalization in linear perceptrons can be derived from the average response function g we present a method for calculating g using only simple matrix identities and partial differential equations using this method we first rederive the known result for g in the thermodynamic limit of perceptrons of infinite size n which has previously been calculated using replica and diagrammatic methods we also show explicitly that the response function is self averaging in the thermodynamic limit extensions of our method to more general learning scenarios with anisotropic teacher space priors input distributions and weight decay terms are discussed finally finite size effects are considered by calculating the o 1 n correction to g we verify the result by computer simulations and discuss the consequences for generalization and learning dynamics in linear perceptrons of finite size

exit functions of hadamard components in repeat zigzag hadamard rzh codes with parallel decoding the extrinsic information transfer exit functions of hadamard codes in the context of repeat zigzag hadamard rzh codes with parallel decoding are investigated exit functions over both the binary erasure channel bec and the binary input additive white gaussian noise biawgn channel are derived the application of these exit functions in the design of low rate capacity approaching irregular rzh irzh codes is also considered using the exit functions the bit error rate ber for a given code profile can be easily estimated and the differential evolution de technique can be employed to find the optimal degree profile for given design parameters the exit functions derived in this communication can serve as an effective tool for designing low rate irzh codes with parallel decoding in both bec and biawgn channels

bilayer low density parity check codes for decode and forward in relay channels this paper describes an efficient implementation of binning for decode and forward df in relay channels using low density parity check ldpc codes bilayer ldpc codes are devised to approach the theoretically promised rate of the df relaying strategy by incorporating relay generated parity bits in specially designed bilayer graphical code structures while conventional ldpc codes are sensitively tuned to operate efficiently at a certain channel parameter the proposed bilayer ldpc codes are capable of working at two different channel parameters and two different rates that at the relay and at the destination to analyze the performance of bilayer ldpc codes bilayer density evolution is devised as an extension of the standard density evolution algorithm based on bilayer density evolution a design methodology is developed for the bilayer codes in which the degree distribution is iteratively improved using linear programming further in order to approach to the theoretical df rate for a wide range of channel parameters this paper proposes two different forms of bilayer codes the bilayer expurgated and bilayer lengthened codes it is demonstrated that the rate of a properly designed bilayer ldpc code can closely approach the theoretical df limit finally it is shown that a generalized version of the proposed bilayer code construction is applicable to relay networks with multiple relays

efficient erasure correcting codes we introduce a simple erasure recovery algorithm for codes derived from cascades of sparse bipartite graphs and analyze the algorithm by analyzing a corresponding discrete time random process as a result we obtain a simple criterion involving the fractions of nodes of different degrees on both sides of the graph which is necessary and sufficient for the decoding process to finish successfully with high probability by carefully designing these graphs we can construct for any given rate r and any given real number spl epsiv a family of linear codes of rate r which can be encoded in time proportional to ln 1 spl epsiv times their block length n furthermore a codeword can be recovered with high probability from a portion of its entries of length 1 spl epsiv rn or more the recovery algorithm also runs in time proportional to n ln 1 spl epsiv our algorithms have been implemented and work well in practice various implementation issues are discussed

soft output decoding of rotationally invariant codes over channels with phase noise we consider rotationally invariant ri trellis coded modulations tcms transmitted over channels affected by phase noise to describe the main ideas of this paper we first concentrate as a case study on the simplest ri scheme namely the differentially encoded m ary phase shift keying m psk signal for this problem we use the framework based on factor graphs fgs and the sum product algorithm spa to derive the exact maximum a posteriori map symbol detection algorithm by analyzing its properties we demonstrate that it can be implemented by a forward backward estimator of the phase probability density function followed by a symbol by symbol completion to produce the a posteriori probabilities of the information symbols to practically implement the forward backward phase estimator we propose a couple of schemes with different complexity the resulting algorithms exhibit an excellent performance and in one case only a limited complexity increases with respect to the algorithm that perfectly knows the channel phase the properties of the optimal decoder and the proposed practical decoding schemes are then extended to the case of a generic ri code the proposed soft output algorithms can also be used in iterative decoding schemes for concatenated codes employing ri inner components among them in the numerical results we consider repeat accumulate ra codes and other serially concatenated schemes recently proposed in the technical literature

extrinsic information transfer functions model and erasure channel properties extrinsic information transfer exit charts are a tool for predicting the convergence behavior of iterative processors for a variety of communication problems a model is introduced that applies to decoding problems including the iterative decoding of parallel concatenated turbo codes serially concatenated codes low density parity check ldpc codes and repeat accumulate ra codes exit functions are defined using the model and several properties of such functions are proved for erasure channels one property expresses the area under an exit function in terms of a conditional entropy a useful consequence of this result is that the design of capacity approaching codes reduces to a curve fitting problem for all the aforementioned codes a second property relates the exit function of a code to its helleseth klove levenshtein information functions and thereby to the support weights of its subcodes the relation is via a refinement of information functions called split information functions and via a refinement of support weights called split support weights split information functions are used to prove a third property that relates the exit function of a linear code to the exit function of its dual

a recursive approach to low complexity codes a method is described for constructing long error correcting codes from one or more shorter error correcting codes referred to as subcodes and a bipartite graph a graph is shown which specifies carefully chosen subsets of the digits of the new codes that must be codewords in one of the shorter subcodes lower bounds to the rate and the minimum distance of the new code are derived in terms of the parameters of the graph and the subeodes both the encoders and decoders proposed are shown to take advantage of the code s explicit decomposition into subcodes to decompose and simplify the associated computational processes bounds on the performance of two specific decoding algorithms are established and the asymptotic growth of the complexity of decoding for two types of codes and decoders is analyzed the proposed decoders are able to make effective use of probabilistic information supplied by the channel receiver e g reliability information without greatly increasing the number of computations required it is shown that choosing a transmission order for the digits that is appropriate for the graph and the subcodes can give the code excellent burst error correction abilities the construction principles

maximum sum rate of repeat accumulate interleave division system by fixed point analysis a multi user repeat accumulate interleave division raid system is considered for a multiple access channel mac with binary inputs equal power and symbol synchronization in the system a regular repeat accumulate ra code serially concatenated with block spreading is employed for each user at the receiver multi user message passing decoding is performed on a single factor graph over the mac with additive white gaussian noise awgn a fixed point analysis is developed to obtain the optimal code rate and the spreading length that give the maximum sum rate for an arbitrary small decoding error rate

optimal puncturing ratios and energy allocation for multiple parallel concatenated codes we propose a systematic design framework for optimal low complexity punctured multiple parallel concatenated codes mpccs based on minimizing the convergence threshold using extrinsic information transfer exit charts as the convergence threshold is related to the area between the two exit curves the corresponding optimization problem is equivalent to a curve fitting problem the exit curves are determined by the respective exit functions of the constituents which can be conveniently shaped through the use of random puncturing and unequal energy allocations across parallel coding streams the design task is therefore to find the optimal combination of constituents puncturing ratios and energy allocation for matching the exit curves a search over all rate one convolutional codes of memory length four or less is performed identifying 98 classes of codes with unique exit functions out of a total of 310 codes low complexity mpccs with up to four constituents are found where the convergence thresholds are observed to be within 0 1 db or less of the fundamental minimum signal to noise ratio snr corresponding to the binary phase shift keying bpsk capacity for code rates 1 3 les r les 7 8 further allowing for unequal energy allocation the convergence thresholds for lower code rates are similarly improved

efficient encoding of low density parity check codes low density parity check ldpc codes can be considered serious competitors to turbo codes in terms of performance and complexity and they are based on a similar philosophy constrained random code ensembles and iterative decoding algorithms we consider the encoding problem for ldpc codes more generally we consider the encoding problem for codes specified by sparse parity check matrices we show how to exploit the sparseness of the parity check matrix to obtain efficient encoders for the 3 6 regular ldpc code for example the complexity of encoding is essentially quadratic in the block length however we show that the associated coefficient can be made quite small so that encoding codes even of length n spl sime 100000 is still quite practical more importantly we show that optimized codes actually admit linear time encoding

design of irregular ldpc codes with optimized performance complexity tradeoff the optimal performance complexity tradeoff for error correcting codes at rates strictly below the shannon limit is a central question in coding theory this paper proposes a numerical approach for the minimization of decoding complexity for long block length irregular low density parity check ldpc codes the proposed design methodology is applicable to any binary input memoryless symmetric channel and any iterative message passing decoding algorithm with a parallel update schedule a key feature of the proposed optimization method is a new complexity measure that incorporates both the number of operations required to carry out a single decoding iteration and the number of iterations required for convergence this paper shows that the proposed complexity measure can be accurately estimated from a density evolution and extrinsic information transfer chart analysis of the code a sufficient condition is presented for convexity of the complexity measure in the variable edge degree distribution when it is not satisfied numerical experiments nevertheless suggest that the local minimum is unique the results presented herein show that when the decoding complexity is constrained the complexity optimized codes significantly outperform threshold optimized codes at long block lengths within the ensemble of irregular codes

achieving near capacity performance on multiple antenna channels with a simple concatenation scheme this paper proposes a capacity approaching yet simple scheme for multi input multiple output mimo channels the proposed scheme is based on a concatenation of a mixture of short memory length convolutional codes or repetition codes and a short and simple rate 1 linear block code followed by either 1 dimensional 1 d anti gray or gray mapping of quadrature phase shift keying qpsk modulation by interpreting the rate 1 code and the 1 d mapping as a multi d mapping performed over multiple transmit antennas the error performance is analyzed in two regions in the error floor region a tight union bound and the corresponding design criterion on the asymptotic performance are derived the bound provides a useful tool to predict the error performance at relatively low bit error rate ber values based on the obtained design criterion an optimal rate 1 code for each 1 d mapping is then constructed to achieve the best asymptotic performance in the turbo pinch off region by using extrinsic information transfer exit charts the most suitable mixed codes are selected for both symmetric and asymmetric antenna configurations it is demonstrated that the simple concatenation scheme can achieve a near capacity performance over the mimo channels furthermore its error performance is shown to be comparable to that obtained by using well designed irregular ldpc and ra codes and therefore the proposed scheme significantly outperforms a scheme employing a parallel concatenated turbo code simulation results in various cases are provided to verify the analysis

reliability based coded modulation with low density parity check codes in this letter we consider the interleaver design in bit interleaved coded modulation bicm with low density parity check ldpc codes the design paradigm is to provide more coding protection through iterative decoding to bits that are less protected by modulation and are thus less reliable at the output of the demodulator the design is carried out by an ad hoc search algorithm over the column permutations of the parity check matrix our simulations show that the proposed reliability based coded modulation scheme can improve the error rate performance of conventional bicm schemes based on regular ldpc codes by a few tenths of a decibel with no added complexity

classification of unique mappings for 8psk based on bit wise distance spectra the performance of bit interleaved coded modulation bicm with or without iterative decoding id is significantly influenced by the mapping of bits to the symbol constellation our main objective in this paper is to develop a systematic design approach for bicm id schemes ensuring the best possible performance with iterative decoding although useful mappings for bicm id have been found based on various search strategies no attempt has been made to systematically enumerate and classify all unique mappers for a given constellation as the basis for a systematic enumeration and classification we define the average bit wise distance spectrum for a mapping from bits to symbols different bit wise distance spectra are derived assuming no prior information or full prior information respectively the bit wise distance spectra determine corresponding bit wise error probability and bit wise mutual information the latter allows us to use the classification of mappings with unique bit wise distance spectra to also classify mappings with unique extremal points in the corresponding extrinsic information transfer exit curves as an example of our approach we classify 8psk mappings into 86 classes of unique mappings according to bit wise distance spectra the classification can be used to significantly reduce the complexity of the search for suitable mappers for bicm id for 8psk and a given encoder only 86 different mappings need to be investigated as examples of the systematic design approach the best 8psk mappings for minimizing the convergence threshold are found for concatenation with the rate 1 2 5 7 8 and 133 171 8 convolutional codes and the rate 1 2 umts turbo code with identical constituent convolutional codes 15 13 8

analysis of sum product decoding of low density parity check codes using a gaussian approximation density evolution is an algorithm for computing the capacity of low density parity check ldpc codes under message passing decoding for memoryless binary input continuous output additive white gaussian noise awgn channels and sum product decoders we use a gaussian approximation for message densities under density evolution to simplify the analysis of the decoding algorithm we convert the infinite dimensional problem of iteratively calculating message densities which is needed to find the exact threshold to a one dimensional problem of updating the means of the gaussian densities this simplification not only allows us to calculate the threshold quickly and to understand the behavior of the decoder better but also makes it easier to design good irregular ldpc codes for awgn channels for various regular ldpc codes we have examined thresholds can be estimated within 0 1 db of the exact value for rates between 0 5 and 0 9 codes designed using the gaussian approximation perform within 0 02 db of the best performing codes found so far by using density evolution when the maximum variable degree is 10 we show that by using the gaussian approximation we can visualize the sum product decoding algorithm we also show that the optimization of degree distributions can be understood and done graphically using the visualization

finite size effects and optimal test set size in linear perceptrons fluctuations in the test error are important in the learning theory of finite dimensional systems as they represent how well the test error matches the average test error by explicitly finding the variance of the test error due to randomness present in both the data set and algorithm for a linear perceptron of dimension n we are able to address such questions as the optimal test set size where exact results were not tractable a good approximation is given to the variance we find that the optimal test set size possesses a phase transition between linear and 2 3 power law scaling in the system size n

grassmannian packings from operator reed muller codes this paper introduces multidimensional generalizations of binary reed muller codes where the codewords are projection operators and the corresponding subspaces are widely separated with respect to the chordal distance on grassmannian space parameters of these grassmannian packings are derived and a low complexity decoding algorithm is developed by modifying standard decoding algorithms for binary reed muller codes the subspaces are associated with projection operators determined by pauli matrices appearing in the theory of quantum error correction and this connection with quantum stabilizer codes may be of independent interest the grassmannian packings constructed here find application in noncoherent wireless communication with multiple antennas where separation with respect to the chordal distance on grassmannian space guarantees closeness to the channel capacity it is shown that the capacity of the noncoherent multiple input multiple output mimo channel at both low and moderate signal to noise ratio snr under the constraint that only isotropically distributed unitary matrices are used for information transmission is closely approximated by these packings

dehn filling volume and the jones polynomial given a hyperbolic 3 manifold with torus boundary we bound the change in volume under a dehn filling where all slopes have length at least 2 this result is applied to give explicit diagrammatic bounds on the volumes of many knots and links as well as their dehn fillings and branched covers finally we use this result to bound the volumes of knots in terms of the coefficients of their jones polynomials

short non binary low density parity check codes for phase noise channels this paper considers the design of short non binary low density parity check ldpc codes over finite fields of order m for channels with phase noise in particular m ary differential phase shift keying dpsk modulated code symbols are transmitted over an additive white gaussian noise awgn channel with the wiener phase noise at the receiver side non coherent detection takes place with the help of a multi symbol detection algorithm followed by a non binary decoding step both the detector and the decoder operate on a joint factor graph as a benchmark finite length bounds and information rate expressions are computed and compared with the codeword error rate cer performance as well as the iterative threshold of the obtained codes as a result performance within 1 2 db from finite length bounds is obtained down to a cer of 10 3

exit chart based power allocation for iterative frequency domain mimo detector transmission power allocation in single carrier multiple input multiple output mimo systems with iterative frequency domain fd soft cancellation sc minimum mean squared error mmse equalization is considered a novel framework for transmission power minimization subject to equalizer convergence constraints referred as convergence constrained power allocation ccpa method is proposed based on extrinsic information transfer exit chart analysis the proposed method decouples the spatial interference between the streams using singular value decomposition svd and minimizes the transmission power while achieving the target mutual information for each stream after iterations at the receiver we show that the transmission power optimization can be formulated as a convex optimization problem three ccpa methods one approximately optimal and other two heuristic methods inspired by the lagrange duality are derived the numerical results demonstrate that the proposed scheme outperforms the existing linear precoding schemes moreover the proposed heuristic schemes can achieve performance close with that of the approximately optimal method in terms of the equalizer convergence properties as well as transmission power

lower bounds on volumes of hyperbolic haken 3 manifolds we prove a volume inequality for 3 manifolds having c o metrics bent along a surface and satisfying certain curvature conditions the result makes use of perelman s work on the ricci flow and geometrization of closed 3 manifolds corollaries include a new proof of a conjecture of bonahon about volumes of convex cores of kleinian groups improved volume estimates for certain haken hyperbolic 3 manifolds and a lower bound on the minimal volume of orientable hyperbolic 3 manifolds an appendix compares estimates of volumes of hyperbolic 3 manifolds drilled along a closed embedded geodesic with experimental data

performance of weighted nonbinary repeat accumulate codes over gf q with q ary orthogonal modulation an approximate upper bound is derived to the maximum likelihood decoding threshold of weighted nonbinary repeat accumulate wnra codes over gf q with q ary orthogonal modulation and coherent detection over the additive white gaussian noise channel as with binary ra codes wnra codes show the potential for achieving the channel capacity with decreasing code rate for large enough q

low rate repeat zigzag hadamard codes in this paper we propose a new class of low rate error correction codes called repeat zigzag hadamard rzh codes featuring simple encoder and decoder structures and flexible coding rate rzh codes are serially concatenated turbo like codes where the outer code is a repetition code and the inner code is a punctured zigzag hadamard zh code by analyzing the code structure of rzh codes we prove that both systematic and nonsystematic rzh codes are good codes in the sense that for an rzh code ensemble there exists a positive number gamma0 such that for any binary input memoryless channel whose bhattacharyya noise parameter is less than the average block error probability of maximum likelihood ml decoding approaches zero two decoding algorithms serial and parallel decoders for rzh codes are proposed we then employ the extrinsic information transfer exit chart technique to design irregular rzh codes results show that the optimized irregular rzh codes exhibit a performance that is very close to capacity in the low rate regime

upper bounds to error probabilities of coded systems beyond the cutoff rate a family of upper bounds to error probabilities of coded systems was recently proposed by d divsalar see ieee communication theory workshop 1999 jpl tmo prog rep 42 139 1999 these bounds are valid for transmission over the additive white gaussian noise channel and require only the knowledge of the weight spectrum of the code words after illustrating these bounds we extend them to fading channels contrary to the union bound our bounds maintain their effectiveness below the signal to noise ratio snr at which the cutoff rate of the channel equals the rate of the code some applications are shown first we derive upper bounds to the minimum snr necessary to achieve zero error probability as the code block length increases to infinity next we use our bounds to predict the performance of turbo codes and low density parity check codes

convergence constrained multiuser transmitter receiver optimization in single carrier fdma convergence constrained power allocation ccpa in single carrier multiuser mu single input multiple output simo systems with turbo equalization is considered in this paper in order to exploit the full benefit of the iterative receiver its convergence properties need to be taken into account also at the transmitter side the proposed scheme can guarantee that the desired quality of service qos is achieved after a sufficient number of iterations we propose two different successive convex approximations for solving the non convex power minimization problem subject to user specific qos constraints the results of an extrinsic information transfer exit chart analysis demonstrate that the proposed ccpa scheme can achieve the design objective numerical results show that the proposed schemes can achieve superior performance in terms of power consumption as compared to linear receivers with and without precoding as well as to the iterative receiver without precoding

design of repeat accumulate codes for iterative detection and decoding extrinsic information transfer exit charts are used to design systematic and nonsystematic repeat accumulate ra codes for iterative detection and decoding the convergence problems of nonsystematic ra codes are solved by introducing a biregular or doped layer of check nodes as examples such nonsystematic codes are designed for multi input multi output mimo fading channels and are shown to operate close to capacity

designing a good low rate sparse graph code this paper deals with the design of low rate sparse graph codes having a linear minimum distance d min in the blocklength n its main contributions are a a necessary condition on a general family of sparse graph codes with linear d min b a justification of having degree 1 bits in the low rate code structure c a new efficient ensemble of low rate sparse graph codes with bits of degree 1 designed so that the necessary condition a is satisfied

finite length analysis of low density parity check codes on the binary erasure channel in this paper we are concerned with the finite length analysis of low density parity check ldpc codes when used over the binary erasure channel bec the main result is an expression for the exact average bit and block erasure probability for a given regular ensemble of ldpc codes when decoded iteratively we also give expressions for upper bounds on the average bit and block erasure probability for regular ldpc ensembles and the standard random ensemble under maximum likelihood ml decoding finally we present what we consider to be the most important open problems in this area

coding theorems for turbo code ensembles this paper is devoted to a shannon theoretic study of turbo codes we prove that ensembles of parallel and serial turbo codes are good in the following sense for a turbo code ensemble defined by a fixed set of component codes subject only to mild necessary restrictions there exists a positive number spl gamma sub 0 such that for any binary input memoryless channel whose bhattacharyya noise parameter is less than spl gamma sub 0 the average maximum likelihood ml decoder block error probability approaches zero at least as fast as n sup spl beta where spl beta is the interleaver gain exponent defined by benedetto et al in 1996

design of irregular weighted nonbinary repeat accumulate codes over gf q with q ary orthogonal modulation using a gaussian approximation using a gaussian approximation to the distribution of the message vectors under density evolution we design irregular weighted nonbinary repeat accumulate codes over gf q with q ary orthogonal modulation the resulting codes achieve a frame error rate of 10 1 within 0 56 to 0 91 db of channel capacity under the awgn channel

statistical mechanical approach to image processing the basic frameworks and techniques of the bayesian approach to image restoration are reviewed from the statistical mechanical point of view first a few basic notions in digital image processing are explained to convince the reader that statistical mechanics has a close formal similarity to this problem second the basic formulation of the statistical estimation from the observed degraded image by using the bayes formula is demonstrated the relationship between bayesian statistics and statistical mechanics is also listed particularly it is explained that some correlation inequalities on the nishimori line of the random spin model also play an important role in bayesian image restoration third the framework of bayesian image restoration for binary images by means of the ising model is reviewed some practical algorithms for binary image restoration are given by employing the mean field and the bethe approximations finally bayesian image restoration for a grey level image using the gaussian model is reviewed and the gaussian model is extended to a more practical probabilistic model by introducing the line state to treat the effects of edges the line state is also extended to quantized values

generalized low density parity check codes based on hadamard constraints in this paper we consider the design and analysis of generalized low density parity check gldpc codes in awgn channels the gldpc codes are specified by a bipartite tanner graph as with standard ldpc codes but with the single parity check constraints replaced by general coding constraints in particular we consider imposing hadamard code constraints at the check nodes for a low rate approach termed ldpc hadamard codes we introduce a low complexity message passing based iterative soft input soft output siso decoding algorithm which employs the a posteriori probability app fast hadamard transform fht for decoding the hadamard check codes at each decoding iteration the achievable capacity with the gldpc codes is then discussed a modified ldpc hadamard code graph is also proposed we then optimize the ldpc hadamard code ensemble using a low complexity optimization method based on approximating the density evolution by a one dimensional dynamic system represented by an extrinsic mutual information transfer exit chart simulation results show that the optimized ldpc hadamard codes offer better performance in the low rate region than low rate turbo hadamard codes but also enjoy a fast convergence rate a rate 0 003 ldpc hadamard code with large block length can achieve a bit error rate ber performance of 10 5 at 1 44 db which is only 0 15 db away from the ultimate shannon limit 1 592 db and 0 24 db better than the best performing low rate turbo hadamard codes

concatenated zigzag hadamard codes in this correspondence we introduce a new class of low rate error correction codes called zigzag hadamard zh codes and their concatenation schemes each member of this class of codes is specified by a highly structured zigzag graph with each segment being a hadamard codeword the zh codes enjoy extremely simple encoding and very low complexity soft input soft output siso decoding based on a posteriori probability app fast hadamard transform fht technique we present an asymptotic performance analysis of the proposed concatenated zh codes using the extrinsic mutual information transfer exit chart for infinite length codes we also provide a union bound analysis of the error performance for finite length codes furthermore the concatenated zh codes are shown to be a good class of codes in the low rate region specifically a rate 0 0107 concatenated code with three zh components and an interleaver size of 65536 can achieve the bit error rate ber performance of 10 sup 5 at 1 15db which is only 0 44 db away from the ultimate shannon limit the proposed concatenated zh codes offer similar performance as another class of low rate codes the turbo hadamard codes and better performance than superorthogonal turbo codes with much lower encoding and decoding complexities

the performance of serial turbo codes does not concentrate minimum distances and maximum likelihood error probabilities of serial turbo codes with uniform interleaver are analyzed it is shown that for a fraction of interleavers approaching one as the block length grows large the minimum distance of serial turbo codes grows as a positive power of their block length while their error probability decreases exponentially fast in some positive power of their block length on sufficiently good memoryless channels such a typical code behavior contrasts the performance of the average serial turbo code whose error probability is dominated by an asymptotically negligible fraction of poorly performing interleavers and decays only as a negative power of the block length the analysis proposed in this paper relies on precise bounds of the minimum distance of the typical serial turbo code whose scaling law is shown to depend both on the free distance of its outer constituent encoder which determines the exponent of its sub linear growth in the block length and on the effective free distance of its inner constituent encoder the latter is defined as the smallest weight of codewords obtained when the input word of the inner encoder has weight two and appears as a linear scaling factor for the minimum distance of the typical serial turbo code hence despite the lack of concentration of the maximum likelihood error probability around its expected value the main design parameters suggested by the average code analysis turn out to characterize also the performance of the typical serial turbo code by showing for the first time that the typical serial turbo code s minimum distance scales linearly in the effective free distance of the inner constituent encoder the presented results generalize and improve upon the probabilistic bounds of kahale and urbanke as well as the deterministic upper bound of bazzi mahdian and spielman where only the dependence on the outer encoder s free distance was proved

iterative decoding of binary block and convolutional codes iterative decoding of two dimensional systematic convolutional codes has been termed turbo de coding using log likelihood algebra we show that any decoder can be used which accepts soft inputs including a priori values and delivers soft outputs that can be split into three terms the soft channel and a priori inputs and the extrinsic value the extrinsic value is used as an a priori value for the next iteration decoding algorithms in the log likelihood domain are given not only for convolutional codes but also for any linear binary systematic block code the iteration is controlled by a stop criterion derived from cross entropy which results in a minimal number of iterations optimal and suboptimal decoders with reduced complexity are presented simulation results show that very simple component codes are sufficient block codes are appropriate for high rates and convolutional codes for lower rates less than 2 3 any combination of block and convolutional component codes is possible several interleaving techniques are described at a bit error rate ber of 10 sup 4 the performance is slightly above or around the bounds given by the cutoff rate for reasonably simple block convolutional component codes interleaver sizes less than 1000 and for three to six iterations

a more accurate one dimensional analysis and design of irregular ldpc codes we introduce a new one dimensional 1 d analysis of low density parity check ldpc codes on additive white gaussian noise channels which is significantly more accurate than similar 1 d methods our method assumes a gaussian distribution in message passing decoding only for messages from variable nodes to check nodes compared to existing work which makes a gaussian assumption both for messages from check nodes and from variable nodes our method offers a significantly more accurate estimate of convergence behavior and threshold of convergence similar to previous work the problem of designing irregular ldpc codes reduces to a linear programming problem however our method allows irregular code design in a wider range of rates without any limit on the maximum variable node degree we use our method to design irregular ldpc codes with rates greater than 1 4 that perform within a few hundredths of a decibel from the shannon limit the designed codes perform almost as well as codes designed by density evolution

on the information rate and repeat accumulate code design for phase noise channels we investigate the information rate of channels affected by phase noise aiming at predicting the ultimate performance limits in this scenario moreover a closed form upper bound is also derived for phase shift keying psk modulations finally we consider the design of nonsystematic irregular repeat accumulate ra codes for this channel trying to give new insights on the codes to be employed for such an application

word hyperbolic dehn surgery in the late 1970 s thurston dramatically changed the nature of 3 manifold theory with the introduction of his geometrisation conjecture and by proving it in the case of haken 3 manifolds 23 the conjecture for general closed orientable 3 manifolds remains perhaps the most important unsolved problem in the subject a weaker form of the conjecture 19 deals with the fundamental group of a closed orientable 3 manifold it proposes that either it contains z z as a subgroup or it is word hyperbolic in the sense of gromov 11 word hyperbolic groups are precisely those groups which satisfy a linear isoperimetric inequality they are of fundamental importance in geometric group theory and have very many useful properties

raptor codes in the low snr regime in this paper we revisit the design of raptor codes for binary input additive white gaussian noise channels where we are interested in very low signal to noise ratios snrs a linear programming degree distribution optimization problem is defined for raptor codes in the low snr regime through several approximations we also provide an exact expression for the polynomial representation of the degree distribution with infinite maximum output node degree in the low snr regime which enables us to calculate the exact value of the fractions of output nodes of small degrees a more practical degree distribution design is also proposed for raptor codes in the low snr regime where we include the rate efficiency and the decoding complexity in the optimization problem and an upper bound on the maximum rate efficiency is derived for given design parameters simulation results show that the raptor code with the designed degree distributions can approach rate efficiencies larger than 0 95 in the low snr regime

optimal power rate allocation and code selection for iterative joint detection of coded random cdma iterative interference cancellation of coded code division multiple access cdma using random spreading with linear cancellation is analyzed if users are grouped into power classes and shannon bound approaching codes are used a geometric power distribution achieves the additive white gaussian noise awgn channel shannon bound as the numbers of classes becomes large the optimal distribution of the size of these classes is shown to be uniform if users are grouped into different rate classes with equal powers among equal rate users the shannon bound for awgn channels can be achieved with an arbitrary distribution of the classes sizes provided that the size of the largest rate class obeys the mild condition that its ratio of size to processing gain is much smaller than the inverse of the signal to noise ratio snr the case of equal powers and equal rates among all users is addressed as a worst case scenario it is argued that simple repetition codes provide for a larger achievable capacity than stronger codes it is shown that this capacity monotonically increases as the rate of the code decreases a density evolution analysis is used to show that the achievable rates exceed those of a minimum mean square error filter applied to the uncoded signals this lower bound is tight for small ratios of bit energy to noise power and otherwise the iterative cancellation receiver provides an appreciably larger capacity relating to recent result from the application of statistical mechanics it is shown that the repetition coded system with iterative cancellation achieves the performance of an equivalent optimal joint detector for uncoded transmission

a fresh look at coding for q ary symmetric channels this paper studies coding schemes for the q ary symmetric channel based on binary low density parity check ldpc codes that work for any alphabet size q 2m m n thus complementing some recently proposed packet based schemes requiring large q first theoretical optimality of a simple layered scheme is shown then a practical coding scheme based on a simple modification of standard binary ldpc decoding is proposed the decoder is derived from first principles and using a factor graph representation of a front end that maps q ary symbols to groups of m bits connected to a binary code the front end can be processed with a complexity that is linear in m log2q an extrinsic information transfer chart analysis is carried out and used for code optimization finally it is shown how the same decoder structure can also be applied to a larger class of q ary channels

exit functions for binary input memoryless symmetric channels use of extrinsic information transfer exit functions characterizing the amplification of mutual information between the input and output of the maximum a posteriori map decoder significantly facilitates analysis of iterative coding schemes previously exit functions derived for binary erasure channels becs were used as an approximation for other channels here we improve on this approach by introducing more accurate methods to construct exit functions for binary input memoryless symmetric bms channels by defining an alternative pseudo map decoder coinciding with the map decoder over bec we provide an expression for the exit functions of block codes over bec furthermore we draw a connection between the exit function over bec and the exit function over the bms channel under certain conditions this is used for deriving accurate or approximate expressions of exit functions over bms channels in certain scenarios

the nishimori line and bayesian statistics the nishimori line is a line or hypersurface in the parameter space of systems with quenched disorder where simple expressions of the averages of physical quantities over the quenched random variables are obtained it has been playing an important role in the theoretical studies of the random frustrated systems since its discovery in around 1980 in this paper an interpretation of the nishimori line from the viewpoint of statistical information processing is developed our main aim is the reconstruction of the whole theory of the nishimori line from the viewpoint of bayesian statistics or almost equivalently from the viewpoint of the theory of error correcting codes as a byproduct of the interpretation counterparts of the nishimori line in models without gauge invariance are given we also discussed the issues on the finite temperature decoding of error correcting codes and clarify the role of gauge invariance in this topic

on the density of geometrically finite kleinian groups the density conjecture of bers sullivan and thurston predicts that each complete hyperbolic 3 manifold m with finitely generated fundamental group is an algebraic limit of geometrically finite hyperbolic 3 manifolds we prove that the conjecture obtains for each complete hyperbolic 3manifold with no cusps and incompressible ends

a close to capacity dirty paper coding scheme the writing on dirty paper channel model offers an information theoretic framework for precoding techniques for canceling arbitrary interference known at the transmitter it indicates that lossless precoding is theoretically possible at any signal to noise ratio snr and thus dirty paper coding may serve as a basic building block in both single user and multiuser communication systems we design an end to end coding realization of a system materializing a significant portion of the promised gains we employ multidimensional quantization based on trellis shaping at the transmitter coset decoding is implemented at the receiver using virtual bits combined with iterative decoding of capacity approaching codes we achieve an improvement of 2db over the best scalar quantization scheme code design is done using the exit chart technique

generalization in a linear perceptron in the presence of noise the authors study the evolution of the generalization ability of a simple linear perceptron with n inputs which learns to imitate a teacher perceptron the system is trained on p alpha n example inputs drawn from some distribution and the generalization ability is measured by the average agreement with the teacher on test examples drawn from the same distribution the dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at alpha 1 when there are no errors static noise in the training examples if the examples are produced by an erroneous teacher overfitting is observed i e the generalization error starts to increase after a finite time of training it is shown that a weight decay of the same size as the variance of the noise errors on the teacher improves on the generalization and suppresses the overfitting the generalization error as a function of time is calculated numerically for various values of the parameters finally dynamic noise in the training is considered white noise on the input corresponds on average to a weight decay and can thus improve generalization whereas white noise on the weights or the output degrades generalization generalization is particularly sensitive to noise on the weights for alpha 1 where it makes the error constantly increase with time but this effect is also shown to be damped by a weight decay weight noise and output noise acts similarly above the transition at alpha 1

simple map decoding of first order reed muller and hamming codes a maximum a posteriori map probability decoder of a block code minimizes the probability of error for each transmitted symbol separately the standard way of implementing map decoding of a linear code is the bahl cocke jelinek raviv bcjr algorithm which is based on a trellis representation of the code the complexity of the bcjr algorithm for the first order reed muller rm 1 codes and hamming codes is proportional to n sup 2 where n is the code s length in this correspondence we present new map decoding algorithms for binary and nonbinary rm 1 and hamming codes the proposed algorithms have complexities proportional to q sup 2 n log sub q n where q is the alphabet size in particular for the binary codes this yields complexity of order n log n

concatenated tree codes a low complexity high performance approach this paper is concerned with a family of concatenated tree ct codes ct codes are special low density parity check ldpc codes consisting of several trees with large spans they can also be regarded as special turbo codes with hybrid recursive nonrecursive parts and multiple constituent codes ct codes are decodable by the belief propagation algorithm they combine many advantages of ldpc and turbo codes such as low decoding cost fast convergence speed and good performance

bayesian model choice asymptotics and exact calculations abstract model determination is a fundamental data analytic task here we consider the problem of choosing amongst a finite with loss of generality we assume two set of models after briefly reviewing classical and bayesian model choice strategies we present a general predictive density which includes all proposed bayesian approaches we are aware of using laplace approximations we can conveniently assess and compare asymptotic behavior of these approaches concern regarding the accuracy of these approximation for small to moderate sample sizes encourages the use of monte carlo techniques to carry out exact calculations a data set fit with nested non linear models enables comparison between proposals and between exact and asymptotic values

design of low density parity check codes for modulation and detection a coding and modulation technique is studied where the coded bits of an irregular low density parity check ldpc code are passed directly to a modulator at the receiver the variable nodes of the ldpc decoder graph are connected to detector nodes and iterative decoding is accomplished by viewing the variable and detector nodes as one decoder the code is optimized by performing a curve fitting on extrinsic information transfer charts design examples are given for additive white gaussian noise channels as well as multiple input multiple output mimo fading channels where the receiver but not the transmitter knows the channel for the mimo channels the technique operates within 1 25 db of capacity for various antenna configurations and thereby outperforms a scheme employing a parallel concatenated turbo code by wide margins when there are more transmit than receive antennas

capacity achieving ensembles for the binary erasure channel with bounded complexity we present two sequences of ensembles of nonsystematic irregular repeat accumulate ira codes which asymptotically as their block length tends to infinity achieve capacity on the binary erasure channel bec with bounded complexity per information bit this is in contrast to all previous constructions of capacity achieving sequences of ensembles whose complexity grows at least like the log of the inverse of the gap in rate to capacity the new bounded complexity result is achieved by puncturing bits and allowing in this way a sufficient number of state nodes in the tanner graph representing the codes we derive an information theoretic lower bound on the decoding complexity of randomly punctured codes on graphs the bound holds for every memoryless binary input output symmetric mbios channel and is refined for the binary erasure channel

cusp shapes under cone deformation a horospherical torus about a cusp of a hyperbolic manifold inherits a euclidean similarity structure called a cusp shape we bound the change in cusp shape when the hyperbolic structure of the manifold is deformed via cone deformation preserving the cusp the bounds are in terms of the change in structure in a neighborhood of the singular locus alone we then apply this result to provide information on the cusp shape of many hyperbolic knots given only a diagram of the knot specifically we show there is a universal constant c such that if a knot admits a prime twist reduced diagram with at least c crossings per twist region then the length of the second shortest curve on the cusp torus is bounded the bound is linear in the number of twist regions of the diagram

unveiling turbo codes some results on parallel concatenated coding schemes a parallel concatenated coding scheme consists of two simple constituent systematic encoders linked by an interleaver the input bits to the first encoder are scrambled by the interleaver before entering the second encoder the codeword of the parallel concatenated code consists of the input bits to the first encoder followed by the parity check bits of both encoders this construction can be generalized to any number of constituent codes parallel concatenated schemes employing two convolutional codes as constituent codes in connection with an iterative decoding algorithm of complexity comparable to that of the constituent codes have been previously shown to yield remarkable coding gains close to theoretical limits they have been named and are known as turbo codes we propose a method to evaluate an upper bound to the bit error probability of a parallel concatenated coding scheme averaged over all interleavers of a given length the analytical bounding technique is then used to shed some light on some crucial questions which have been floating around in the communications community since the proposal of turbo codes

improved low density parity check codes using irregular graphs we construct new families of error correcting codes based on gallager s 1973 low density parity check codes we improve on gallager s results by introducing irregular parity check matrices and a new rigorous analysis of hard decision decoding of these codes we also provide efficient methods for finding good irregular structures for such decoding algorithms our rigorous analysis based on martingales our methodology for constructing good irregular codes and the demonstration that irregular structure improves performance constitute key points of our contribution we also consider irregular codes under belief propagation we report the results of experiments testing the efficacy of irregular codes on both binary symmetric and gaussian channels for example using belief propagation for rate 1 4 codes on 16000 bits over a binary symmetric channel previous low density parity check codes can correct up to approximately 16 errors while our codes correct over 17 in some cases our results come very close to reported results for turbo codes suggesting that variations of irregular low density parity check codes may be able to match or beat turbo code performance

linear model selection by cross validation abstract we consider the problem of selecting a model having the best predictive ability among a class of linear models the popular leave one out cross validation method which is asymptotically equivalent to many other model selection methods such as the akaike information criterion aic the c p and the bootstrap is asymptotically inconsistent in the sense that the probability of selecting the model with the best predictive ability does not converge to 1 as the total number of observations n we show that the inconsistency of the leave one out cross validation can be rectified by using a leave n v out cross validation with n v the number of observations reserved for validation satisfying n v n 1 as n this is a somewhat shocking discovery because nv n 1 is totally opposite to the popular leave one out recipe in cross validation motivations justifications and discussions of some practical aspects of the use of the leave n v out cross validation method are provided and results