the use of mixed logit models to reflect heterogeneity in capture recapture studies summary we examine issues in estimating population size n with capture recapture models when there is variable catchability among subjects we focus on a logistic normal mixed model for which the logit of the probability of capture is an additive function of a random subject and a fixed sampling occasion parameter when the probability of capture is small or the degree of heterogeneity is large the log likelihood surface is relatively flat and it is difficult to obtain much information about n we also discuss a latent class model and a log linear model that account for heterogeneity and show that the log linear model has greater scope models assuming homogeneity provide much narrower intervals for n but are usually highly overly optimistic the actual coverage probability being much lower than the nominal level

the matrix equation log xy log x log y abstract let x y be n n complex matrices such that x y xy have no eigenvalues on r and log xy log x log y we prove that if n 2 or if n 3 and x y are simultaneously triangularizable then x y commute in both cases we reduce the problem to a result in complex analysis

analysis of capture recapture data with a rasch type model allowing for conditional dependence and multidimensionality summary in this article we show that if subjects are assumed to be homogeneous within a finite set of latent classes the basic restrictions of the rasch model conditional independence and unidimensionality can be relaxed in a flexible way by simply adding appropriate columns to a basic design matrix when discrete covariates are available so that subjects may be classified into strata we show how a joint modeling approach can achieve greater parsimony parameter estimates may be obtained by maximizing the conditional likelihood given the total number of captures with a combined use of the em and fisher scoring algorithms we also discuss a technique for obtaining confidence intervals for the size of the population under study based on the profile likelihood

matrix completion with column manipulation near optimal sample robustness rank tradeoffs this paper considers the problem of matrix completion when some number of the columns are completely and arbitrarily corrupted potentially by a malicious adversary it is well known that standard algorithms for matrix completion can return arbitrarily poor results if even a single column is corrupted one direct application comes from robust collaborative filtering here some number of users are so called manipulators who try to skew the predictions of the algorithm by calibrating their inputs to the system in this paper we develop an efficient algorithm for this problem based on a combination of a trimming procedure and a convex program that minimizes the nuclear norm and the ell 1 2 norm our theoretical results show that given a vanishing fraction of observed entries it is nevertheless possible to complete the underlying matrix even when the number of corrupted columns grows significantly our results hold without any assumptions on the locations or values of the observed entries of the manipulated columns moreover we show by an information theoretic argument that our guarantees are nearly optimal in terms of the fraction of sampled entries on the authentic columns the fraction of corrupted columns and the rank of the underlying matrix our results therefore sharply characterize the tradeoffs between sample robustness and rank in matrix completion

a nonparametric multidimensional latent class irt model in a bayesian framework we propose a nonparametric item response theory model for dichotomously scored items in a bayesian framework the model is based on a latent class lc formulation and it is multidimensional with dimensions corresponding to a partition of the items in homogenous groups that are specified on the basis of inequality constraints among the conditional success probabilities given the latent class moreover an innovative system of prior distributions is proposed following the encompassing approach in which the largest model is the unconstrained lc model a reversible jump type algorithm is described for sampling from the joint posterior distribution of the model parameters of the encompassing model by suitably post processing its output we then make inference on the number of dimensions i e number of groups of items measuring the same latent trait and we cluster items according to the dimensions when unidimensionality is violated the approach is illustrated by two examples on simulated data and two applications based on educational and quality of life data

phase transitions for high dimensional clustering and related problems consider a two class clustering problem where we observe x i ell i mu z i z i stackrel mathit i i d sim n 0 i p 1 leq i leq n the feature vector mu in r p is unknown but is presumably sparse the class labels ell i in 1 1 are also unknown and the main interest is to estimate them we are interested in the statistical limits in the two dimensional phase space calibrating the rarity and strengths of useful features we find the precise demarcation for the region of impossibility and region of possibility in the former useful features are too rare weak for successful clustering in the latter useful features are strong enough to allow successful clustering the results are extended to the case of colored noise using le cam s idea on comparison of experiments we also extend the study on statistical limits for clustering to that for signal recovery and that for global testing we compare the statistical limits for three problems and expose some interesting insight we propose classical pca and important features pca if pca for clustering for a threshold t 0 if pca clusters by applying classical pca to all columns of x with an l 2 norm larger than t we also propose two aggregation methods for any parameter in the region of possibility some of these methods yield successful clustering we discover a phase transition for if pca for any threshold t 0 let xi t be the first left singular vector of the post selection data matrix the phase space partitions into two different regions in one region there is a t such that cos xi t ell rightarrow 1 and if pca yields successful clustering in the other cos xi t ell leq c 0 0 our results require delicate analysis especially on post selection random matrix theory and on lower bound arguments

optimal detection of sparse principal components in high dimension we perform a finite sample analysis of the detection levels for sparse principal components of a high dimensional covariance matrix our minimax optimal test is based on a sparse eigenvalue statistic alas computing this test is known to be np complete in general and we describe a computationally efficient alternative test using convex relaxations our relaxation is also proved to detect sparse principal components at near optimal detection levels and performs very well on simulated datasets ams 2000 subject classifications primary 62h25 secondary 62f04 90c22

sample size determination within the scope of conditional maximum likelihood estimation with special focus on testing the rasch model this paper refers to the exponential family of probability distributions and the conditional maximum likelihood cml theory it is concerned with the determination of the sample size for three groups of tests of linear hypotheses known as the fundamental trinity of wald score and likelihood ratio tests the main practical purpose refers to the special case of tests of the class of rasch models the theoretical background is discussed and the formal framework for sample size calculations is provided given a predetermined deviation from the model to be tested and the probabilities of the errors of the first and second kinds

a nonparametric approach for assessing latent trait unidimensionality assuming a nonparametric family of item response theory models a theory based procedure for testing the hypothesis of unidimensionality of the latent space is proposed the asymptotic distribution of the test statistic is derived assuming unidimensionality thereby establishing an asymptotically valid statistical test of the unidimensionality of the latent trait based upon a new notion of dimensionality the test is shown to have asymptotic power 1 a 6300 trial monte carlo study using published item parameter estimates of widely used standardized tests indicates conservative adherence to the nominal level of significance and statistical power averaging 81 out of 100 rejections for examinee sample sizes and psychological test lengths often incurred in practice

geometry moments and conditional independence trees with hidden variables we study the geometry of the parameter space for bayesian directed graphical models with hidden variables that have a tree structure and where all the nodes are binary we show that the conditional independence statements implicit in such models can be expressed in terms of polynomial relationships among the central moments this algebraic structure will enable us to identify the inequality constraints on the space of the manifest variables that are induced by the conditional independence assumptions as well as determine the degree of unidentifiability of the parameters associated with the hidden variables by understanding the geometry of the sample space under this class of models we shall propose and discuss simple diagnostic methods 1 introduction graphical models have proved to be a powerful tool for building bayesian models to analyze multivariate problems where all variables are observed e g spiegelhalter dawid lauritzen and cowell 1993 in particular it is possible to estimate all the conditional probabilities that parameterize such models by using a conjugate analysis however when all the data on certain variables in an explanatory model are missing conjugacy usually disappears estimates of these conditional probabilities become highly dependent on one another and they often cannot be determined from data no matter how extensive that data is see e g settimi and smith 1998 in this paper we propose a geometrical approach to analyze such difficulties we first observe that conditional independence assumptions implicit in directed graphical models induce some constraints on the model space that can be expressed as polynomial equations among the central moments we then exploit such an algebraic structure to explore the geometry and the irregularities of the parameter space for bayesian directed graphical models with hidden variables defined over a set of binary variables and such that the conditional independence assumptions are represented via a directed tree understanding the geometry and the singularities of the parameter spaces will enable us to investigate practical statistical issues such as parameter identifiability model dimension and diagnostic methods in the statistical analyses of problems with missing data it has been common practice either to use various methods of approximation to calculate the posterior probabilities of the model parameters see e g spiegelhalter and

statistical algorithms and a lower bound for detecting planted cliques we introduce a framework for proving lower bounds on computational problems over distributions against algorithms that can be implemented using access to a statistical query oracle for such algorithms access to the input distribution is limited to obtaining an estimate of the expectation of any given function on a sample drawn randomly from the input distribution rather than directly accessing samples most natural algorithms of interest in theory and in practice for example moments based methods local search standard iterative methods for convex optimization mcmc and simulated annealing can be implemented in this framework our framework is based on and generalizes the statistical query model in learning theory kearns 1998 our main application is a nearly optimal lower bound on the complexity of any statistical query algorithm for detecting planted bipartite clique distributions or planted dense subgraph distributions when the planted clique has size o n1 2 for any constant 0 the assumed hardness of variants of these problems has been used to prove hardness of several other problems and as a guarantee for security in cryptographic applications our lower bounds provide concrete evidence of hardness thus supporting these assumptions

loglinear multidimensional irt models for polytomously scored items a loglinear irt model is proposed that relates polytomously scored item responses to a multidimensional latent space the analyst may specify a response function for each response indicating which latent abilities are necessary to arrive at that response each item may have a different number of response categories so that free response items are more easily analyzed conditional maximum likelihood estimates are derived and the models may be tested generally or against alternative loglinear irt models

expanding the rasch model to a general model having more than one dimension the well known rasch model is generalized to a multicomponent model so that observations of component events are not needed to apply the model it is shown that the generalized model has retained the property of the specific objectivity of the rasch model for a restricted variant of the model maximum likelihood estimates of its parameters and a statistical test of the model are given the results of an application to a mathematics test involving six components are described

statistical computational tradeoffs in planted problems and submatrix localization with a growing number of clusters and submatrices we consider two closely related problems planted clustering and submatrix localization in the planted clustering problem a random graph is generated based on an underlying cluster structure of the nodes the task is to recover these clusters given the graph the submatrix localization problem concerns locating hidden submatrices with elevated means inside a large real valued random matrix of particular interest is the setting where the number of clusters submatrices is allowed to grow unbounded with the problem size these formulations cover several classical models such as planted clique planted densest subgraph planted partition planted coloring and the stochastic block model which are widely used for studying community detection graph clustering and bi clustering for both problems we show that the space of the model parameters cluster submatrix size edge probabilities and the mean of the submatrices can be partitioned into four disjoint regions corresponding to decreasing statistical and computational complexities 1 the impossible regime where all algorithms fail 2 the hard regime where the computationally expensive maximum likelihood estimator mle succeeds 3 the easy regime where the polynomial time convexified mle succeeds 4 the simple regime where a local counting thresholding procedure succeeds moreover we show that each of these algorithms provably fails in the harder regimes our results establish the minimax recovery limits which are tight up to universal constants and hold even with a growing number of clusters submatrices and provide order wise stronger performance guarantees for polynomial time algorithms than previously known our study demonstrates the tradeoffs between statistical and computational considerations and suggests that the minimax limits may not be achievable by polynomial time algorithms

reliability of test scores in nonparametric item response theory three methods for estimating reliability are studied within the context of nonparametric item response theory two were proposed originally by mokken 1971 and a third is developed in this paper using a monte carlo strategy these three estimation methods are compared with four classical lower bounds to reliability finally recommendations are given concerning the use of these estimation methods

a note on the computation of the second order derivatives of the elementary symmetric functions in the rasch model jansen 1984 gave explicit formulas for the computation of the second order derivatives of the elementary symmetric functions but they are only applicable to those pairs of items which have unequal parameters it is shown here that for equal parameters similar explicit formulas do exist too facilitating the application of the newton raphson procedure to estimate the parameters in the rasch model and related models according to the conditional maximum likelihood principle

marginal maximum likelihood estimation for the one parameter logistic model two algorithms are described for marginal maximum likelihood estimation for the one parameter logistic model the more efficient of the two algorithms is extended to estimation for the linear logistic model numerical examples of both procedures are presented

likelihood inference on the underlying structure of irt models the assumptions underlying item response theory irt models may be expressed as a set of equality and inequality constraints on the parameters of a latent class model it is well known that the same assumptions imply that the parameters of the manifest distribution have to satisfy a more complicated set of inequality constraints which however are necessary but not sufficient in this paper we describe how the theory for likelihood based inference under equality and inequality constraints may be used to test the underlying assumptions of irt models it turns out that the analysis based directly on the latent structure is simpler and more flexible in particular we indicate how several interesting extensions of the rasch model may be obtained by partial relaxation of the basic constraints an application to a data set provided by educational testing service is used to illustrate the approach

a class of multidimensional irt models for testing unidimensionality and clustering items we illustrate a class of multidimensional item response theory models in which the items are allowed to have different discriminating power and the latent traits are represented through a vector having a discrete distribution we also show how the hypothesis of unidimensionality may be tested against a specific bidimensional alternative by using a likelihood ratio statistic between two nested models in this class for this aim we also derive an asymptotically equivalent wald test statistic which is faster to compute moreover we propose a hierarchical clustering algorithm which can be used when the dimensionality of the latent structure is completely unknown for dividing items into groups referred to different latent traits the approach is illustrated through a simulation study and an application to a dataset collected within the national assessment of educational progress 1996

association marginal modeling of multivariate categorical responses a maximum likelihood approach abstract generalized log linear models can be used to describe the association structure and or the marginal distributions of multivariate categorical responses we simultaneously model the association structure and marginal distributions using association marginal am models which are specially formulated generalized log linear models that combine two models an association a model which describes the association among all the responses and a marginal m model which describes the marginal distributions of the responses because the model s composite link function is not required to be invertible a large class of models can be entertained and model specification is typically straightforward we propose a mixed freedom constraint parameterization that exploits the special structure of an am model using this parameterization maximum likelihood fitting is straightforward and typically feasible for large sparse tables when a parsimonious association model is used the size of the fitting proble

information limits for recovering a hidden community we study the problem of recovering a hidden community of cardinality k from an n times n symmetric data matrix a where for distinct indices i j a ij sim p if i j both belong to the community and a ij sim q otherwise for two known probability distributions p and q depending on n if p mathrm bern p and q mathrm bern q with p q it reduces to the problem of finding a densely connected k subgraph planted in a large erdos renyi graph if p mathcal n mu 1 and q mathcal n 0 1 with mu 0 it corresponds to the problem of locating a k times k principal submatrix of elevated means in a large gaussian random matrix we focus on two types of asymptotic recovery guarantees as n to infty 1 weak recovery expected number of classification errors is o k and 2 exact recovery probability of classifying all indices correctly converges to one under mild assumptions on p and q and allowing the community size to scale sublinearly with n we derive a set of sufficient conditions and a set of necessary conditions for recovery which are asymptotically tight with sharp constants the results hold in particular for the gaussian case and for the case of bounded log likelihood ratio including the bernoulli case whenever p q and 1 p 1 q are bounded away from zero and infinity previous work has shown that if weak recovery is achievable then exact recovery is achievable in linear additional time by a simple voting procedure we provide a converse showing the condition for the voting procedure to succeed is almost necessary for exact recovery

test theory and psychometrika the past twenty five years on the occasion ofpsychometrika s fiftieth anniversary the past twenty five years developments in mental test theory are reviewed with special emphasis on the topics receiving attention in the pages of this journal analogous reviews forpsychometrika s first quarter century were given by gulliksen and guilford in 1961 much of the recent progress in test theory and in other branches of psychometrics as well has been made by treating the problems in this field as being essentially ones of statistical inference it is concluded that a research in test theory is in a healthy state and b psychometrika is an important source of information about that research

semiparametric estimation in the rasch model and related exponential response models including a simple latent class model for item analysis abstract the rasch model for item analysis is an important member of the class of exponential response models in which the number of nuisance parameters increases with the number of subjects leading to the failure of the usual likelihood methodology both conditional likelihood methods and mixture model techniques have been used to circumvent these problems in this article we show that these seemingly unrelated analyses are in fact closely linked to each other despite dramatic structural differences between the classes of models implied by each approach we show that the finite mixture model for j dichotomous items having t latent classes gives the same estimates of item parameters as conditional likelihood on a set whose probability approaches one if t j 1 2 unconditional maximum likelihood estimators for the finite mixture model can be viewed as keifer wolfowitz estimators for the random effects version of the rasch model latent class versions of the model are especially attractive when t is

tensor svd statistical and computational limits in this paper we propose a general framework for tensor singular value decomposition tensor singular value decomposition svd which focuses on the methodology and theory for extracting the hidden low rank structure from high dimensional tensor data comprehensive results are developed on both the statistical and computational limits for tensor svd this problem exhibits three different phases according to the signal to noise ratio snr in particular with strong snr we show that the classical higher order orthogonal iteration achieves the minimax optimal rate of convergence in estimation with weak snr the information theoretical lower bound implies that it is impossible to have consistent estimation in general with moderate snr we show that the non convex maximum likelihood estimation provides optimal solution but with np hard computational cost moreover under the hardness hypothesis of hypergraphic planted clique detection there are no polynomial time algorithms performing consistently in general

exact tests for the rasch model via sequential importance sampling rasch proposed an exact conditional inference approach to testing his model but never implemented it because it involves the calculation of a complicated probability this paper furthers rasch s approach by 1 providing an efficient monte carlo methodology for accurately approximating the required probability and 2 illustrating the usefulness of rasch s approach for several important testing problems through simulation studies our monte carlo methodology is shown to compare favorably to other monte carlo methods proposed for this problem in two respects it is considerably faster and it provides more reliable estimates of the monte carlo standard error

a sample coverage approach to multiple system estimation with application to census undercount abstract the concept of sample coverage used in animal abundance estimation is modified to evaluate the undercount of a census lack of independence between the census and its postenumeration survey leads to correlation bias for the standard estimator of population size an additional recapture sample besides the census and postenumeration survey can be used to estimate the correlation bias due to two types of dependences this work expresses the correlation bias for the three sample model as a function of expected sample coverage and measures of dependence between lists a nonparametric population size estimator that incorporates the correlation bias is proposed a simulation study investigates the performance of the proposed procedure data from a 1988 dress rehearsal study for the 1990 census conducted by the u s bureau of the census are used to illustrate the proposed three system estimation procedure and to compare the resulting estimates with those given by darroch fienberg glonek and junker

phase transitions for detecting latent geometry in random graphs random graphs with latent geometric structure are popular models of social and biological networks with applications ranging from network user profiling to circuit design these graphs are also of purely theoretical interest within computer science probability and statistics a fundamental initial question regarding these models is when are these random graphs affected by their latent geometry and when are they indistinguishable from simpler models without latent structure such as the erd s renyi graph mathcal g n p we address this question for two of the most well studied models of random graphs with latent geometry the random intersection and random geometric graph our results are as follows a the random intersection graph is defined by sampling n random sets s 1 s 2 ldots s n by including each element of a set of size d in each s i independently with probability delta and including the edge i j if s i cap s j ne emptyset we prove that the random intersection graph converges in total variation to an erd s renyi graph if d tilde omega n 3 and does not if d o n 3 for both dense and sparse edge densities p this resolves an open problem in fill et al random struct algorithms 16 2 156 176 2000 rybarczyk random struct algorithms 38 1 2 205 234 2011 and kim et al random struct algorithms 52 4 662 679 2018 the same result was obtained simultaneously and independently by bubeck et al when random intersection graphs lose geometry manuscript 2019 b we strengthen the preceding argument to show that the matrix of random intersection sizes s i cap s j converges in total variation to a symmetric matrix with independent poisson entries this yields the first total variation convergence result for tau random intersection graphs where the edge i j is included if s i cap s j ge tau more precisely our results imply that if p is bounded away from 1 then the tau random intersection graph with edge density p converges to mathcal g n p if d omega tau 3 n 3 c the random geometric graph on mathbb s d 1 is defined by sampling x 1 x 2 dots x n uniformly at random from mathbb s d 1 and including the edge i j if vert x i x j vert 2 le tau a result of bubeck et al random struct algorithms 49 503 532 2016 showed that this model converges to mathcal g n p in total variation where p is chosen so that the models have matching edge densities as long as d omega n 3 it was conjectured in bubeck et al 2016 that this threshold decreases drastically for p small we make the first progress towards this conjecture by showing convergence if d tilde omega left min pn 3 p 2 n 7 2 right our proofs are a hybrid of combinatorial arguments direct couplings and applications of information inequalities previous upper bounds on the total variation distance between random graphs with latent geometry and mathcal g n p have typically not been both combinatorial and information theoretic while this interplay is essential to the sharpness of our bounds

computational and statistical boundaries for submatrix localization in a large noisy matrix we study in this paper computational and statistical boundaries for submatrix localization given one observation of one or multiple nonoverlapping signal submatrix of magnitude lambda and size k m times k n embedded in a large noise matrix of size m times n the goal is to optimal identify the support of the signal submatrix computationally and statistically two transition thresholds for the signal to noise ratio lambda sigma are established in terms of m n k m and k n the first threshold sf snr c corresponds to the computational boundary we introduce a new linear time spectral algorithm that identifies the submatrix with high probability when the signal strength is above the threshold sf snr c below this threshold it is shown that no polynomial time algorithm can succeed in identifying the submatrix under the hidden clique hypothesis the second threshold sf snr s captures the statistical boundary below which no method can succeed in localization with probability going to one in the minimax sense the exhaustive search method successfully finds the submatrix above this threshold in marked contrast to submatrix detection and sparse pca the results show an interesting phenomenon that sf snr c is always significantly larger than sf snr s under the sub gaussian error model which implies an essential gap between statistical optimality and computational efficiency for submatrix localization

on commuting exponentials in low dimensions abstract we deal with square matrices a b of dimension d 2 or 3 over the complex field such that ab ba we introduce the relations g t exp ta b exp ta exp b and g t exp ta b exp ta exp b exp b exp ta in dimension 2 we characterize the a b couples satisfying gt for any t n in dimension 2 or 3 we show that if g t is satisfied for any t n then a and b are simultaneously triangularizable in this manner we do not need the 2i congruence free postulate anymore which has been supposed by researchers since 1954

loglinear rasch model tests existing statistical tests for the fit of the rasch model have been criticized because they are only sensitive to specific violations of its assumptions contingency table methods using loglinear models have been used to test various psychometric models in this paper the assumptions of the rasch model are discussed and the rasch model is reformulated as a quasi independence model the model is a quasi loglinear model for the incomplete subgroup score item 1 item 2 itemk contingency table using ordinary contingency table methods the rasch model can be tested generally or against less restrictive quasi loglinear models to investigate specific violations of its assumptions

eca high dimensional elliptical component analysis in non gaussian distributions abstractwe present a robust alternative to principal component analysis pca called elliptical component analysis eca for analyzing high dimensional elliptically distributed data eca estimates the eigenspace of the covariance matrix of the elliptical data to cope with heavy tailed elliptical distributions a multivariate rank statistic is exploited at the model level we consider two settings either that the leading eigenvectors of the covariance matrix are nonsparse or that they are sparse methodologically we propose eca procedures for both nonsparse and sparse settings theoretically we provide both nonasymptotic and asymptotic analyses quantifying the theoretical performances of eca in the nonsparse setting we show that eca s performance is highly related to the effective rank of the covariance matrix in the sparse setting the results are twofold i we show that the sparse eca estimator based on a combinatoric program attains the optimal rate of convergence ii based on some recent d

residualanalysis in the polytomous rasch model residuals for check of model fit in the polytomous rasch model are examined comparisons are made between using counts for all response pattern and using item totals for score groups for the construction of the residuals comparisons are also for the residuals based on score group totals made between using as basis the item totals or using the estimated item parameters the developed methods are illustrated by two examples one from a psychiatric rating scale one from a danish welfare study

computational and statistical tradeoffs via convex relaxation modern massive datasets create a fundamental problem at the intersection of the computational and statistical sciences how to provide guarantees on the quality of statistical inference given bounds on computational resources such as time or space our approach to this problem is to define a notion of algorithmic weakening in which a hierarchy of algorithms is ordered by both computational efficiency and statistical efficiency allowing the growing strength of the data at scale to be traded off against the need for sophisticated processing we illustrate this approach in the setting of denoising problems using convex relaxation as the core inferential tool hierarchies of convex relaxations have been widely used in theoretical computer science to yield tractable approximation algorithms to many computationally intractable tasks in the current paper we show how to endow such hierarchies with a statistical characterization and thereby obtain concrete tradeoffs relating algorithmic runtime to amount of data

two applications of the theory of primary matrix functions among all the ways one might define f a for a square complex matrix a and a given function f c c the notion of a primary matrix function is perhaps the most useful and natural using only basic conceptual properties of primary matrix functions we consider whether f a can have rank one and whether for a given b there is a unique a such that f a b

caracterisation spectrale de la forme de jordan abstract soit a m n c ou m n c designe l ensemble des matrices n n a coefficients complexes nous montrons qu on peut completement caracteriser la forme de jordan de a en examinant le polynome caracteristique de ta x pour tous les t c et tous les x m n c ceci nous permet de donner une demonstration plus elementaire d un theoreme de baribeau et ransford sur les transformations holomorphes de m n c qui preservent le spectre denote by m n c the set of complex n n matrices and let a m n c we give a variational purely spectral characterization of the jordan form of a by examining the characteristic polynomial of the perturbed matrices ta x for t c and x m n c this allows us to give a more elementary proof of a theorem of baribeau and ransford on spectrum preserving holomorphic maps on m n c

a multiple record systems estimation method that takes observed and unobserved heterogeneity into account summary we present a model to estimate the size of an unknown population from a number of lists that applies when the assumptions of a homogeneity of capture probabilities of individuals and b marginal independence of lists are violated this situation typically occurs in epidemiological studies where the heterogeneity of individuals is severe and researchers cannot control the independence between sources of ascertainment we discuss the situation when categorical covariates are available and the interest is not only in the total undercount but also in the undercount within each stratum resulting from the cross classification of the covariates we also present several techniques for determining confidence intervals of the undercount within each stratum using the profile log likelihood thereby extending the work of cormack 1992 biometrics48 567 576

conditional association and unidimensionality in monotone latent variable models latent variable models represent the joint distribution of observable variables in terms of a simple structure involving unobserved or latent variables usually assuming the conditional independence of the observable variables given the latent variables these models play an important role in educational measurement and psychometrics in sociology and in population genetics and are implicit in some work on systems reliability we study a broad class of latent variable models namely the monotone unidimensional models in which the latent variable is a scalar the observable variables are conditionally independent given the latent variable and the conditional distribution of the observables given the latent variable is stochastically increasing in the latent variable all models in this class imply a new strong form of positive dependence among the observable variables namely conditional positive association this positive dependence condition may be used to test whether any model in this class can provide an adequate fit to observed data various applications generalizations and a numerical example are discussed

a class of regression models for multivariate categorical responses summary when data composed of several categorical responses together with categorical or continuous predictors are observed it is often useful to relate the response probabilities to the predictors via a generalised linear model with a composite link function this paper discusses a class of link functions that lie between the two extremes of the multivariate logistic transform of mccullagh nelder 1989 and the log linear decomposition of contingency table analysis the models derived from these link functions are shown to inherit various desirable properties of both the multivariate logistic regression models and the log linear regression models a computational scheme for implementing these models is derived and they are demonstrated to be computationally more tractable than the multivariate logistic regression models their application is illustrated in a numerical example

testing the conditional independence and monotonicity assumptions of item response theory when item characteristic curves are nondecreasing functions of a latent variable the conditional or local independence of item responses given the latent variable implies nonnegative conditional covariances between all monotone increasing functions of a set of item responses given any function of the remaining item responses this general result provides a basis for testing the conditional independence assumption without first specifying a parametric form for the nondecreasing item characteristic curves the proposed tests are simple have known asymptotic null distributions and possess certain optimal properties in an example the conditional independence hypothesis is rejected for all possible forms of monotone item characteristic curves

positive quadrant dependence and marginal modeling in two way tables with ordered margins for a collection of two way tables where subjects are cross classified according to the same pair of ordinal categorical variables conditionally on the value of one or more discrete explanatory variables we propose a general approach to likelihood inference that combines marginal modeling with fitting and testing of inequality constraints such as those implied by the assumption that one marginal distribution is stochastically larger than the other positive dependence and stronger positive dependence the approach is based on parameterizing bivariate conditional distributions with global logits and global log odds ratios and we provide a general framework for handling models defined by equality and inequality constraints on these parameters in this way such models as marginal homogeneity proportional odds among row or columns margins and plackett distribution may be treated together with various models defined by inequality constraints on the same parameters such as for instance those implied by

on the bayesian estimation of a closed population size in the presence of heterogeneity and model uncertainty we consider the estimation of the size of a closed population often of interest for wild animal populations using a capture recapture study the estimate of the total population size can be very sensitive to the choice of model used to fit to the data we consider a bayesian approach in which we consider all eight plausible models initially described by otis et al 1978 wildlife monographs 62 1 135 within a single framework including models containing an individual heterogeneity component we show how we are able to obtain a model averaged estimate of the total population incorporating both parameter and model uncertainty to illustrate the methodology we initially perform a simulation study and analyze two datasets where the population size is known before considering a real example relating to a population of dolphins off northeast scotland

on the limitation of spectral methods from the gaussian hidden clique problem to rank one perturbations of gaussian tensors we consider the following detection problem given a realization of a symmetric matrix x of dimension n distinguish between the hypothesis that all upper triangular variables are independent and identically distributed i i d gaussians variables with mean 0 and variance 1 and the hypothesis where x is the sum of such matrix and an independent rank one perturbation this setup applies to the situation where under the alternative there is a planted principal submatrix b of size l for which all upper triangular variables are i i d gaussians with mean 1 and variance 1 whereas all other upper triangular elements of x not in b are i i d gaussians variables with mean 0 and variance 1 we refer to this as the gaussian hidden clique problem when l 1 epsilon sqrt n epsilon 0 it is possible to solve this detection problem with probability 1 o n 1 by computing the spectrum of x and considering the largest eigenvalue of x we prove that this condition is tight in the following sense when l no algorithm that examines only the eigenvalues of x can detect the existence of a hidden gaussian clique with error probability vanishing as n to infty we prove this result as an immediate consequence of a more general result on rank one perturbations of k dimensional gaussian tensors in this context we establish a lower bound on the critical signal to noise ratio below which a rank one signal cannot be detected

some open problems in matrix theory arising in linear systems and control control theory has long provided a rich source of motivation for developments in matrix theory accordingly we discuss some open problems in matrix theory arising from theoretical and practical issues in linear systems theory and feedback control the problems discussed include robust stability matrix exponent induced norms stabilizability and pole assignability and nonstandard matrix equations a substantial number of references are included to acquaint matrix theorists with problems and trends in this application area feedback control theory has long provided a rich source of motivation for developments in matrix theory the purpose of this paper is to discuss several open problems in matrix theory that arise from theoretical and practical issues in feedback control theory and the associated area of linear systems theory many of these problems are remarkably simple to state are of intense interest in control theory and applications and yet remain unsolved besides leading to the resolution of these problems it is hoped that this paper will help to stimulate increased interaction between matrix and control theorists accord

feeling the bern adaptive estimators for bernoulli probabilities of pairwise comparisons we study methods for aggregating pairwise comparison data among a collection of n items with the goal of estimating the outcome probabilities for future comparisons working within a flexible model that only imposes a form of strong stochastic transitivity we introduce an adaptivity index which compares the risk of our estimator to that of an oracle over appropriate sub models where the oracle knows the specific sub model in the ground truth in addition to measuring the usual worst case risk of an estimator this adaptivity index also captures the extent to which the estimator adapts to instance specific difficulty relative to an oracle estimator first we propose a three step estimator termed count randomize least squares and show that it has adaptivity index upper bounded by sqrt n up to logarithmic factors we then show that conditional on the planted clique hypothesis no computationally efficient estimator can achieve an adaptivity index smaller than sqrt n second we show that a regularized least squares estimator can achieve a poly logarithmic adaptivity index thereby demonstrating a sqrt n gap between optimal and computationally achievable adaptivity finally we prove that the standard least squares estimator which is known to be optimally adaptive in several closely related problems fails to adapt in the context of estimating pairwise probabilities

recovering a single community with side information we study the effect of the quality and quantity of side information on the recovery of a hidden community of size k o n in a graph of size n side information for each node in the graph is modeled by a random vector in which either the vector dimension or the llr of each component with respect to node labels is independent of n these two models represent the variation in quality and quantity of side information under maximum likelihood detection we calculate tight necessary and sufficient conditions for exact recovery of the labels we demonstrate how side information needs to evolve with n in terms of either its quantity or quality to improve the exact recovery threshold a similar set of results are obtained for weak recovery under belief propagation tight necessary and sufficient conditions for weak recovery are calculated when the llrs are constant and sufficient conditions when the llrs vary with n moreover we design and analyze a local voting procedure using side information that can achieve exact recovery when applied after belief propagation

statistical and computational trade offs in estimation of sparse principal components in recent years sparse principal component analysis has emerged as an extremely popular dimension reduction technique for high dimensional data the theoretical challenge in the simplest case is to estimate the leading eigenvector of a population covariance matrix under the assumption that this eigenvector is sparse an impressive range of estimators have been proposed some of these are fast to compute while others are known to achieve the minimax optimal rate over certain gaussian or sub gaussian classes in this paper we show that under a widely believed assumption from computational complexity theory there is a fundamental trade off between statistical and computational performance in this problem more precisely working with new larger classes satisfying a restricted covariance concentration condition we show that there is an effective sample size regime in which no randomised polynomial time algorithm can achieve the minimax optimal rate we also study the theoretical performance of a polynomial time variant of the well known semidefinite relaxation estimator revealing a subtle interplay between statistical and computational efficiency

local homogeneity in latent trait models a characterization of the homogeneous monotone irt model the stochastic subject formulation of latent trait models contends that within a given subject the event of obtaining a certain response pattern may be probabilistic ordinary latent trait models do not imply that these within subject probabilities are identical to the conditional probabilities specified by the model the latter condition is called local homogeneity it is shown that local homgeneity is equivalent to subpopulation invariance of the model in case of the monotone irt model local homogeneity implies absence of item bias absence of item specific traits and the possibility to join overlapping subtests the following characterization theorem is proved the homogeneous monotone irt model holds for a finite or countable item pool if and only if the pool is experimentally independent and pairwise nonnegative association holds in every positive subpopulation

how hard is it to approximate the best nash equilibrium the quest for a polynomial time approximation scheme ptas for nash equilibrium in a two player game which emerged as a major open question in algorithmic game theory seeks to circumvent the ppad completeness of finding an exact nash equilibrium by finding an approximate equilibrium the closely related problem of finding an equilibrium maximizing a certain objective such as social welfare was shown to be np hard gilboa and zemel games econom behav 1 1989 pp 80 93 however this np hardness is unlikely to extend to approximate equilibria since the latter admits a quasi polynomial time algorithm lipton markakis and mehta in proceedings of the 4th acm conference on electronic commerce acm new york 2003 pp 36 41 we show that this optimization problem namely finding in a two player game an approximate equilibrium achieving a large social welfare is unlikely to have a polynomial time algorithm one interpretation of our results is that a ptas for nash equilibrium if it exists should not extend to a ptas for finding the best nash equilibrium technically our result is a reduction from the notoriously difficult problem in modern combinatorics of finding a planted but hidden clique in a random graph g n 1 2 our reduction starts from an instance with planted clique size o log n for comparison the currently known algorithms are effective only for a much larger clique size omega sqrt n

nuclear norm minimization for the planted clique and biclique problems we consider the problems of finding a maximum clique in a graph and finding a maximum edge biclique in a bipartite graph both problems are np hard we write both problems as matrix rank minimization and then relax them using the nuclear norm this technique which may be regarded as a generalization of compressive sensing has recently been shown to be an effective way to solve rank optimization problems in the special case that the input graph has a planted clique or biclique i e a single large clique or biclique plus diversionary edges our algorithm successfully provides an exact solution to the original instance for each problem we provide two analyses of when our algorithm succeeds in the first analysis the diversionary edges are placed by an adversary in the second they are placed at random in the case of random edges for the planted clique problem we obtain the same bound as alon krivelevich and sudakov as well as feige and krauthgamer but we use different techniques

how should we assess the fit of rasch type models approximating the power of goodness of fit statistics in categorical data analysis we investigate the performance of three statistics r 1 r 2 glas in psychometrika 53 525 546 1988 and m 2 maydeu olivares joe in j am stat assoc 100 1009 1020 2005 psychometrika 71 713 732 2006 to assess the overall fit of a one parameter logistic model 1pl estimated by marginal maximum likelihood ml r 1 and r 2 were specifically designed to target specific assumptions of rasch models whereas m 2 is a general purpose test statistic we report asymptotic power rates under some interesting violations of model assumptions different item discrimination presence of guessing and multidimensionality as well as empirical rejection rates for correctly specified models and some misspecified models all three statistics were found to be more powerful than pearson s x 2 against two and three parameter logistic alternatives 2pl and 3pl and against multidimensional 1pl models the results suggest that there is no clear advantage in using goodness of fit statistics specifically designed for rasch type models to test these models when marginal ml estimation is used

sample size determination for rasch model tests this paper is concerned with supplementing statistical tests for the rasch model so that additionally to the probability of the error of the first kind type i probability the probability of the error of the second kind type ii probability can be controlled at a predetermined level by basing the test on the appropriate number of observations an approach to determining a practically meaningful extent of model deviation is proposed and the approximate distribution of the wald test is derived under the extent of model deviation of interest

loglinear rasch models for the analysis of stability and change loglinear unidimensional and multidimensional rasch models are considered for the analysis of repeated observations of polytomous indicators with ordered response categories reparameterizations and parameter restrictions are provided which facilitate specification of a variety of hypotheses about latent processes of change models of purely quantitative change in latent traits are proposed as well as models including structural change a conditional likelihood ratio test is presented for the comparison of unidimensional and multiple scales rasch models in the context of longitudinal research this renders possible the statistical test of homogeneity of change against subject specific change in latent traits applications to two empirical data sets illustrate the use of the models

locally dependent latent trait model and the dutch identity revisited in this paper we propose a class of locally dependent latent trait models for responses to psychological and educational tests typically item response models treat an individual s multiple response to stimuli as conditional independent given the individual s latent trait in this paper instead the focus is on models based on a family of conditional distributions or kernel that describes joint multiple item responses as a function of student latent trait not assuming conditional independence specifically we examine a hybrid kernel which comprises a component for one way item response functions and a component for conditional associations between items given latent traits the class of models allows the extension of item response theory to cover some new and innovative applications in psychological and educational research an em algorithm for marginal maximum likelihood of the hybrid kernel model is proposed furthermore we delineate the relationship of the class of locally dependent models and the log linear model by revisiting the dutch identity holland 1990

on combinatorial testing problems we study a class of hypothesis testing problems in which upon observing the realization of an n dimensional gaussian vector one has to decide whether the vector was drawn from a standard normal distribution or alternatively whether there is a subset of the components belonging to a certain given class of sets whose elements have been contaminated that is have a mean different from zero we establish some general conditions under which testing is possible and others under which testing is hopeless with a small risk the combinatorial and geometric structure of the class of sets is shown to play a crucial role the bounds are illustrated on various examples

log linear models for capture recapture summary log linear models are developed for capture recapture experiments and their advantages and disadvantages discussed ways in which they can be extended sometimes with only partial success to open populations subpopulations trap dependence and long chains of recapture periods are presented the use of residual patterns and analysis of subsets of data to identify behavioural patterns and acceptable models is emphasised and illustrated with two examples 1 capture recapture the biological problem with which we are concerned is that of estimating the parameters of a free living population of unknown size from a sequence of s samples in each of which any individual either is or is not observed the models in this paper require that each animal be marked naturally or artificially in such a way that we can observe the numbers of animals which have any particular pattern of being or not being observed through the sequence of samples there are t 2s 1 different observable patterns which we shall refer to as capture histories although there is no requirement for physical capture and denote each pattern by a sequence of l s and 2 s representing respectively seen or not seen in a particular sample a general pattern of capture history will be denoted a with ra the number of individuals observed to have this pattern thus for example with s 3 samples and a 121 r 2 is the number of animals seen in the first and third samples but not in the second the pattern with all 2 s is by definition unobservable in general the population will be subject to demographic change by birth or immigration and by death or emigration the simplest assumption for the response of an animal to the observer one which underlies the classical petersen schnabel or jolly seber estimators is that every animal alive in the population at a sampling time has the same probability of being observed this probability may vary from one sample to another or be constant over all or a number of samples if this assumption fails it may do so for either or both of two reasons the first is that of trap dependence the probability of an individual being observed in a sample depending on its past history of capture the second is that of heterogeneity between individuals who because of differences in their behaviour or in where they live inherently have different probabilities of being observed in any sample the basic idea of heterogeneity refers to properties of individuals which remain constant over time but individual animals may also have time varying behaviour the aspect most commonly considered being temporary departure of an individual from the population various excellent texts describe theoretical and practical aspects of capture recapture notably the comprehensive one by seber 1982 the introduction by begon 1979 the

volume ratio sparsity and minimaxity under unitarily invariant norms this paper studies non asymptotic minimax estimation of high dimensional matrices and provides tight minimax rates for a large collection of loss functions in a variety of problems via information theoretic methods based on the convex geometry of finite dimensional banach spaces we first develop a volume ratio approach for determining minimax estimation rates of unconstrained mean matrices under all unitarily invariant norm losses which turn out to only depend on the norm of identity matrix in addition we establish the minimax rates for estimating normal mean matrices with submatrix sparsity where the sparsity constraint introduces an additional term in the rate which in contrast to the unconstrained case is determined by the smoothness lipschitz constant of the norm this method is also applicable to the low rank matrix completion problem and extends well beyond the additive noise model in particular it yields tight rates in covariance matrix estimation and poisson rate matrix estimation problems for all unitarily invariant norms

some improved diagnostics for failure of the rasch model although several goodness of fit tests have been developed for the rasch model for dichotomous items most of them are of a global asymptotic and confirmatory type this paper based on ideas from a recent thesis by van den wollenberg offers some suggestions for local small sample and exploratory techniques difficulty plots for person groups scoring right and wrong on a specific item a slope test per item based on a binomial distribution per score group and a unidimensionality check based on an extended hypergeometric distribution per score group

conjunctive item response theory kernels conjunctive item response models are introduced such that a sufficient statistics for latent traits are not necessarily additive in item scores b items are not necessarily locally independent and c existing compensatory additive item response models including the binomial rasch logistic and general locally independent model are special cases simple estimates and hypothesis tests for conjunctive models are introduced and evaluated as well conjunctive models are also identified with cognitive models that assume the existence of several individually necessary component processes for a global ability it is concluded that conjunctive models and methods may show promise for constructing improved tests and uncovering conjunctive cognitive structure it is also concluded that conjunctive item response theory may help to clarify the relationships between local dependence multidimensionality and item response function form

simple capture recapture models permitting unequal catchability and variable sampling effort we consider two capture recapture models that imply that the logit of the probability of capture is an additive function of an animal catchability parameter and a parameter reflecting the sampling effort the models are special cases of the rasch model and satisfy the property of quasi symmetry one model is log linear and the other is a latent class model for the log linear model point and interval estimates of the population size are easily obtained using standard software such as glim

two new test statistics for the rasch model it is shown that the presently available statistical tests for the rasch model are insensitive to violation of the unidimensionality axiom two new test statistics are presented the first one q 1 is sensitive to the same effects as the presently available statistics but has some desirable properties of a nonstatistical nature the second statistic q 2 is sensitive to violation of local stochastic independence and unidimensionality and thus fills an existing gap

a class of latent markov models for capture recapture data allowing for time heterogeneity and behavior effects we propose an extension of the latent class model for the analysis of capture recapture data which allows us to take into account the effect of a capture on the behavior of a subject with respect to future captures the approach is based on the assumption that the variable indexing the latent class of a subject follows a markov chain with transition probabilities depending on the previous capture history several constraints are allowed on these transition probabilities and on the parameters of the conditional distribution of the capture configuration given the latent process we also allow for the presence of discrete explanatory variables which may affect the parameters of the latent process to estimate the resulting models we rely on the conditional maximum likelihood approach and for this aim we outline an em algorithm we also give some simple rules for point and interval estimation of the population size the approach is illustrated by applying it to two data sets concerning small mammal populations

the derivation of some tests for the rasch model from the multinomial distribution the present paper is concerned with testing the fit of the rasch model it is shown that this can be achieved by constructing functions of the data on which model tests can be based that have power against specific model violations it is shown that the asymptotic distribution of these tests can be derived by using the theoretical framework of testing model fit in general multinomial and product multinomial models the model tests are presented in two versions one that can be used in the context of marginal maximum likelihood estimation and one that can be applied in the context of conditional maximum likelihood estimation

canonical forms for complex matrix congruence and congruence abstract canonical forms for congruence and congruence of square complex matrices were given by horn and sergeichuk in linear algebra appl 389 2004 347 353 based on sergeichuk s paper math ussr izvestiya 31 3 1988 481 501 which employed the theory of representations of quivers with involution we use standard methods of matrix analysis to prove directly that these forms are canonical our proof provides explicit algorithms to compute all the blocks and parameters in the canonical forms we use these forms to derive canonical pairs for simultaneous congruence of pairs of complex symmetric and skew symmetric matrices as well as canonical forms for simultaneous congruence of pairs of complex hermitian matrices

on the approximation of the quadratic exponential distribution in a latent variable context summary following cox wermuth 1994 2002 we show that the distribution of a set of binary observable variables induced by a certain discrete latent variable model may be approximated by a quadratic exponential distribution this discrete latent variable model is equivalent to the latent class version of the two parameter logistic model of birnbaum 1968 which may be seen as a generalized version of the rasch model rasch 1960 1961 on the basis of this result we develop an approximate maximum likelihood estimator of the item parameters of the two parameter logistic model which is very simply implemented the proposed approach is illustrated through an example based on a dataset on educational assessment

an exact method for partitioning dichotomous items within the framework of the monotone homogeneity model the monotone homogeneity model mhm also known as the unidimensional monotone latent variable model is a nonparametric irt formulation that provides the underpinning for partitioning a collection of dichotomous items to form scales ellis psychometrika 79 303 316 2014 doi 10 1007 s11336 013 9341 5 has recently derived inequalities that are implied by the mhm yet require only the bivariate inter item correlations in this paper we incorporate these inequalities within a mathematical programming formulation for partitioning a set of dichotomous scale items the objective criterion of the partitioning model is to produce clusters of maximum cardinality the formulation is a binary integer linear program that can be solved exactly using commercial mathematical programming software however we have also developed a standalone branch and bound algorithm that produces globally optimal solutions simulation results and a numerical example are provided to demonstrate the proposed method

pairwise residuals and diagnostic tests for misspecified dependence structures in models for binary longitudinal data maximum likelihood estimation of models for binary longitudinal data is typically inconsistent if the dependence structure is misspecified unfortunately diagnostics specifically designed for detecting such misspecifications are scant we develop residuals and diagnostic tests based on comparing observed and expected frequencies of response patterns over time in the presence of arbitrary time varying and time invariant covariates to overcome the sparseness problem we use lower order marginal tables such as two way tables for pairs of time points aggregated over covariate patterns our proposed pairwise concordance residuals are valuable for exploratory diagnostics and for constructing both generic tests for misspecified dependence structure as well as targeted adjacent pair concordance tests for excess serial dependence the proposed methods are straightforward to implement and work well for general situations regardless of the number of time points and the number and types of covariates

two remarks on matrix exponentials it is shown that for two square matrices a and b with algebraic elements eaeb ebea if and only if ab ba moreover the in some sense best possible perturbation inequality ea eb 2 ea 2 e a b 2 1 for a normal matrix a is proven

information recovery from pairwise measurements this paper is concerned with jointly recovering n node variables left x i right 1 leq i leq n from a collection of pairwise difference measurements imagine we acquire a few observations taking the form of x i x j the observation pattern is represented by a measurement graph mathcal g with an edge set mathcal e such that x i x j is observed if and only if i j in mathcal e to account for noisy measurements in a general manner we model the data acquisition process by a set of channels with given input output transition measures employing information theoretic tools applied to channel decoding problems we develop a unified framework to characterize the fundamental recovery criterion which accommodates general graph structures alphabet sizes and channel transition measures in particular our results isolate a family of minimum channel divergence measures to characterize the degree of measurement corruption which together with the size of the minimum cut of mathcal g dictates the feasibility of exact information recovery for various homogeneous graphs the recovery condition depends almost only on the edge sparsity of the measurement graph irrespective of other graphical metrics alternatively the minimum sample complexity required for these graphs scales like n log n mathsf hel 1 2 min for certain information metric mathsf hel 1 2 min defined in the main text as long as the alphabet size is not super polynomial in n we apply our general theory to three concrete applications including the stochastic block model the random corruption model and the haplotype assembly problem our theory leads to orderwise tight recovery conditions for all these scenarios

marginal regression models for the analysis of positive association of ordinal response variables given a set of discrete response variables some of which are ordinal and an arbitrary set of discrete explanatory variables we propose a simple matrix formulation for parameterising the saturated model as in glonek 1996 this is such that within a hierarchical structure marginal logits and log odds ratios of various possible types together with the remaining log linear interactions of high order may be modelled by equality and inequality constraints inequality constraints are particularly relevant for specifying models of positive association efficient algorithms are provided for computing maximum likelihood estimates under such constraints the asymptotic distribution of the likelihood ratio test is derived and an extension of the usual analysis of deviance is outlined which incorporates inequality constraints

interval estimation for mark recapture studies of closed populations textbooks continue to recommend the use of an asymptotic normal distribution to provide an interval estimate for the unknown size n of a closed population studied by a mark recapture experiment or multiple record system a likelihood interval approach is proposed and its implementation demonstrated for a range of models for such studies including all main effect and interaction models for incomplete contingency tables

latent trait models and dichotomization of graded responses this paper discusses thecompatibility of the polychotomous rasch model with dichotomization of the response continuum it is argued that in the case of graded responses the response categories presented to the subject are essentially an arbitrary polychotomization of the response continuum ranging for example from total rejection or disagreement to total acceptance or agreement of an item or statement because of this arbitrariness the measurement outcome should be independent of the specific polychotomization applied for example presenting a specific multicategory response format should not affect the measurement outcome when such is the case the original polychotomous model is called compatible with dichotomization

computational barriers in minimax submatrix detection this paper studies the minimax detection of a small submatrix of elevated mean in a large matrix contaminated by additive gaussian noise to investigate the tradeoff between statistical performance and computational cost from a complexity theoretic perspective we consider a sequence of discretized models which are asymptotically equivalent to the gaussian model under the hypothesis that the planted clique detection problem cannot be solved in randomized polynomial time when the clique size is of smaller order than the square root of the graph size the following phase transition phenomenon is established when the size of the large matrix p to infty if the submatrix size k theta p alpha for any alpha in 0 2 3 computational complexity constraints can incur a severe penalty on the statistical performance in the sense that any randomized polynomial time test is minimax suboptimal by a polynomial factor in p if k theta p alpha for any alpha in 2 3 1 minimax optimal detection can be attained within constant factors in linear time using schatten norm loss as a representative example we show that the hardness of attaining the minimax estimation rate can crucially depend on the loss function implications on the hardness of support recovery are also obtained

a latent trait model for dichotomous choice data the parella model is a probabilistic parallelogram model that can be used for the measurement of latent attitudes or latent preferences the data analyzed are the dichotomous responses of persons to stimuli with a one zero indicating agreement disagreement with the content of the stimulus the model provides a unidimensional representation of persons and items the response probabilities are a function of the distance between person and stimulus the smaller the distance the larger the probability that a person will agree with the content of the stimulus an estimation procedure based on expectation maximization and marginal maximum likelihood is developed and the quality of the resulting parameter estimates evaluated

incoherence optimal matrix completion this paper considers the matrix completion problem we show that it is not necessary to assume joint incoherence which is a standard but unintuitive and restrictive condition that is imposed by previous studies this leads to a sample complexity bound that is orderwise optimal with respect to the incoherence parameter as well as to the rank r and the matrix dimension n up to a log factor as a consequence we improve the sample complexity of recovering a semidefinite matrix from o nr 2 log 2 n to o nr log 2 n and the highest allowable rank from theta sqrt n log n to theta n log 2 n the key step in proof is to obtain new bounds in terms of the ell infty 2 norm defined as the maximum of the row and column norms of a matrix to illustrate the applicability of our techniques we discuss extensions to singular value decomposition projection structured matrix completion and semisupervised clustering for which we provide orderwise improvements over existing results finally we turn to the closely related problem of low rank plus sparse matrix decomposition we show that the joint incoherence condition is unavoidable here for polynomial time algorithms conditioned on the planted clique conjecture this means it is intractable in general to separate a rank omega sqrt n positive semidefinite matrix and a sparse matrix interestingly our results show that the standard and joint incoherence conditions are associated respectively with the information statistical and computational aspects of the matrix decomposition problem