Investigation into the use of confidence indicators with calibration.
Date: 2005-12-31
Institutions: Eurostat
Subjects: official statistics, statistical method

S

E

I

D

U

T

S

D

N

A

S

R

E

P

A

P

G

N

I

K

R

O

W

ISSN 1725-4825

N
O

I
T
I

D
E

5
0
0
2

Investigation into the use
of confidence indicators
with calibration

E U R O P E A N
C O M M I S S I O N

 
 
 
 
Europe Direct is a service to help you find answers to your questions about the European Union

New freephone number:
00 800 6 7 8 9 10 11

A great deal of additional information on the European Union is available on the Internet.
It can be accessed through the Europa server (http://europa.eu.int).

Luxembourg: Office for Official Publications of the European Communities, 2005

ISSN 1725-4825
ISBN 92-79-01320-3

© European Communities, 2005

Investigation into the Use of  
Confidence Indicators with Calibration 

Paper prepared for: 

Workshop on Frontiers in Benchmarking Techniques and  

Their Application to Official Statistics 

OECD / EUROSTAT 

Luxembourg,  7 & 8 April 2005 

Subject Areas: 

Benchmarking techniques and data quality issues; 
Applications of benchmarking techniques in official statistics. 

Authors: 

Gerard Keogh (gerard.keogh@cso.ie) 
Dave Jennings (dave.jennings@cso.ie) 

Institution: 

Central Statistics Office (Statistical Methods and Development section). 

Abstract: 

Using a standard input-output table with known marginal totals and independent (and 

unbalanced) estimates in individual cells as a basis, this paper investigates methods by 

which the cell estimates can be calibrated to the marginal totals. The paper also 

investigates the use of confidence indicators for each cell estimate which allow a user 

the ability to ensure that the amendments required to calibrate the data are more 

concentrated in the cells in which s/he has smaller confidence. 

 
 
 
 
 
 
 
1.  Introduction 

The techniques of calibration can be extended to areas of statistical non-stochastic 

estimation. In calibration, a set of input data is adjusted to ensure that one or more 

data constraints holds. In an estimation situation when, say, a proportion of the input 

data is known to be accurate and the remainder is of poor quality, the techniques of 

calibration can be used to assist in the estimation of the poorer quality items (provided 

of course that a set of data constraints applies). 

What we will look at is a situation where we wish to estimate a set of constrained 

parameters from an initial set of unconstrained estimates of the parameters. In 

particular we will look at supply and use tables, specifically an Industry by Product 

use table, for which first estimates of the cells are available, but the marginal totals of 

the cells estimates do not agree with the known marginals, which are available from 

another source.   

In other words if Z is the array of initial estimates and T the known marginal totals, 

we will use Z to estimate an array θ whose marginals agree with T. In doing this we 

will attempt to take the statistician’s confidence in the individual estimates in Z into 

account. Frequently, the Z estimates will have come from very disparate sources and 

the statistician’s judgement and experience will often allow him to categorise the 

estimates into some reliability classes. 

Firstly θ is estimated using standard calibration techniques and, subsequently, 

estimates are provided using the Expectation-Maximation algorithm. 

2.  Problem Background and Initial Analysis 

For the purposes of the analysis we are using a preliminary estimate of an industry by 

product use table with nine rows and columns. The data is in €million and, for our 

purposes, represents an aggregation of a much more detailed table with 55 rows and 

columns. We have an accompanying table of confidence indicators supplied by the 

statistician in the area. These indicators are in the range 0.1 (least confidence) to 0.9 

(greatest confidence). We deliberately excluded an indicator of 1 for absolute 

certainty, on the basis that if a cell estimate was known to be ‘correct’ then, for the 

purposes of estimation, it could simply be removed from the table and added back 

2

 
later. Finally we have the given known marginal totals, in €million, with which our 

final estimates must agree. The data is shown in Tables 1 and 2. 

Table 1  Initial Estimates and Row and Column Control Totals 

Initial Estimates 

€m 

Control 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total 

Totals 

Products 

Industries 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

1.5 

25.9

2.1 

3.6 

0.0 

51.4

2.8 

1.8 

6.0

7.9

1.9 

0.1 

0.1 

1.4 

11.2

278.4 

9.1 

48.3

0.3

0.5 

3.8 

7.8

15.5

58.0 

0.3 

41.9

6.6

15.2 

1.0 

18.2

0.6 

20.9

133.6 

2.0 

33.8

0.0 

0.0

0.0 

0.0 

1.1

0.1

0.5

11.0

3.2

10.4

37.4

6.3

1.3

0.0

0.3

5.3

0.6

15.0

4.3

14.9

0.9

0.2

1.4

9.0

2.7

9.9

4.0

9.5

24.6

64.0 

98.7 

266.8

332.9 

270.7 

816.5 1,190.7 

1,168.3 

118.7

137.7 

161.7 

215.6

366.7 

332.4 

102.2

190.3 

321.2 

145.6

367.2 

446.9 

16.5

204.1

223.9 

209.5 

42.0 

41.7

245.0  22.1  178.3

96.2

147.4

142.9

0.0

915.6 

915.7 

Total 

47.6  173.5

735.6  43.7  343.3

166.4

188.7

196.1

1,894.1 3,789.0 

Control 
Totals 

71.2  175.0 722.1  54.0  349.3

221.9

208.2

204.9

1,918.5

3,925.1 

All the data being used, i.e. the initial estimates and the control totals, relate to 2002. 

In other words there is an assumption that the initial estimates represent reasonable 

and unbiased estimates of the true values of their respective cells. If, for instance, 

some of the initial estimates are derived from cell data for a previous year, then we are 

assuming that the user has updated them to the current year. This could be done, for 

example,  by the use of suitable estimates of price and volume changes.  

Table 2  User Supplied Confidence Indicators 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

0.3 

0.3 

0.3 

0.4 

0.5 

0.6 

0.2 

0.7 

0.9 

0.6 

0.1 

0.8 

0.7 

0.4 

0.8 

0.5 

0.6 

0.9 

0.7 

0.7 

0.7 

0.8 

0.4 

0.4 

0.5 

0.1 

0.9 

0.2 

0.4 

0.2 

0.1 

0.3 

0.3 

0.4 

0.5 

0.8 

0.6

0.5

0.6

0.1

0.7

0.3

0.4

0.5

0.7

0.5

0.6

0.1

0.6

0.4

0.3

0.3

0.6

0.7

0.1

0.7

0.7

0.6

0.5

0.3

0.4

0.8

0.9

0.4

0.4

0.8

0.3

0.5

0.3

0.4

0.5

0.8

0.5

0.5

0.2

0.5

0.3

0.2

0.4

0.2

0.9

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

3

 
 
 
 
 
 
 
 
 
 
 
Tabular Analysis 

In any tabular analysis it is instructive to glean insight into the structure in a table. A 

good starting point is to treat the data as if they were counts and the cell entries a 

multinomial sample. Denoting the observed cell entry in row i and column j by zi,j the 

simplest structure for a two-way table is the multiplicative independence model for 

the rows and columns of the table 

and on taking logs this gives the ‘well-known’ loglinear independence model 

zi,j = TiTj  /  T 

log(zi,j) = µ + αi + βj

with µ = –log(T). Using this model on the data in Table 1, the deviance (i.e. the lack 

of fit) is 2,448 and the likelihood ratio statistic is χ2

59(111) = 0.001. Since this is 

highly significant, the independence model is not appropriate and so there is an 

interaction effect in the table. 

The interaction observed is a specific type referred to as ‘mobility’. This is usually 

observed in social mobility tables that relate, say, a father’s social status to that of his 

sons. Table 1 is an input-output table that associates products with industries. This 

means that the main diagonal can be expected to dominate. In a number of 

rows/columns this tends to be the case, but there are some large deviations. In any 

event, the so-called ‘quasi-perfect’ mobility model in multiplicative form is  

ži,j =  TiTj   when i ≠  j; 

     =  zi,j    when i = j, 

where ži,j  is the expected cell estimate under this model. Note that this model states 

that the estimated diagonal elements are equal to their observed counterparts. To fit 

the model the diagonal elements are removed, i.e. made structural zeros, and the 

independence model re-fit. The deviance is reduced to 2,027 and the likelihood ratio 

statistic is χ2

51(77) = 0.02. While this is still significant at the 5% level, it is a 

considerable improvement supporting the belief that some form of imperfect 

‘mobility’ explains the structure of the table. A key attribute of any calibration 

procedure is that the underlying structural relationships are preserved. The methods 

adopted in this paper attempt to meet this requirement. 

4

 
The Problem 

Our aim, given this data, is to arrive at ‘final’ estimates which (a) fulfil the control 

constraints; (b) are ‘close’ to the initial estimates, that is retain the underlying tabular 

structure; and (c) have the higher confidence cells changing less than the lower 

confidence ones. This is set out more clearly below. 

3.  Methodology 

The Loss Function 

For ease of notation we will represent the initial estimates as a vector Z of length 81 

(i.e. the table cells in row major order). The control totals will be represented as a 

vector T of length 18 (row totals first then column totals). Also the confidence 

indicators (P) and final estimates (θ) will be in vector form. 

Firstly, let R be a matrix such that R`Z gives the marginal totals of Z. R is an 81 * 18 

(the number of marginals) matrix, with each column containing 1 in the positions of 

the cells contributing to its marginal and 0 elsewhere. Then our aim is to find θ such 

that  

(cid:131)  R`θ = T 

(cid:131)  L(Z,θ,P) is minimised (where L is a suitable loss function). 

We looked at four different candidates for L but decided only to present results for 

two of them in this paper. The four we looked at were :  

(cid:131)  L1 = (θ-Z)`diag(P)(θ-Z), the (weighted) squared absolute loss; 

(cid:131)  L2 = (θ-Z)`diag(P/Z)(θ-Z), the (weighted) ‘chi-square’ loss; 
(cid:131)  L3 = (θ-Z)`diag(P/Z2)(θ-Z), the (weighted) squared relative loss; 
(cid:131)  L4 = ΣPi((θi/Zi)log(θi/Zi) – θi/Zi + 1), the (weighted) ‘entropy’ loss. 

We decided to go with loss functions L1 and L4. L1 can be useful to cover situations 

where additive adjustments are considered more relevant than relative or 

proportionate ones (e.g. where negative cell values can arise or where zero initial 

values are not necessarily to remain unchanged). L4 does not give very different 

results from L2 and L3 but we chose it because it is the more common one (in its 

unweighted form) used in calibration. 

For L1 we computed the solution directly as: 

θ = Z + diag(1/P) R (R` diag(1/P) R)-1 (T - R` Z) 

5

 
which is the solution to the minimisation of L1 + λ` (T – R` Z), subject to R` Z = T, 

with λ being a Lagrange multiplier. 

For L4, the solution to the minimisation of L4 + λ` (T – R` Z) is Õ = Z # exp(Z # R λ 

/ P). We first solved the control equation, R` Õ - T = 0, for λ and then substituted this 

in  

to get our adjusted cell values. The symbols # and / represent elementwise 

θ = Z # exp(Z # R λ / P) 

multiplication and division respectively. 

The EM and IPF approach 

Returning to the tabular layout, it is assumed for simplicity that the unknown 

complete data  are a multinomial sample of size T falling into I*J cells. The complete 

data are made up of the manifest (i.e. observed) data zij classified according to the I*J 

cells and row and column control totals Ti and Tj that arise from latent data θij whose 

distribution is unknown. The EM (Expectation-Maximisation) algorithm consists of 

computing the cell probabilities from the current estimates of the complete data and 

using these to estimate the distribution of the latent data. In this particular case the 

EM algorithm is in fact better known as IPF (Iterative Proportional Fitting). 

Starting with θij

(0) = zij, the observed data, IPF for the unknown θij is based on the 

simple update row (and column replacing i with j ) rule 

θij 

(t+1) =  [1+( (Ti -θi● 

(t))/ θi● 

(t))] θij 

(t)

until convergence is achieved. This rule simply takes the difference between the 

control total for row i and corresponding current row total estimate and uses it to 

update the current cell estimates. 

To introduce weighting into this scheme a straightforward Bayesian approach is 

adopted. Specifically, the weights are wij = 1/pij where pij are the entries in the 

confidence matrix. These weights of course are not probabilities but are adjusted 

using the simple IPF rule so that each row and column adds to 1. These adjusted 

weights are also denoted by wij. The weighted IPF (row) update rule is given by 

 θij 

(t+1) =  [1+( (Ti -θi● 

(t))*{ wij /Σ wij θij 

(t))})] θij 

(t)

6

 
where the summation is over the columns j, and now θij

(t) is the ‘conditional’ 

likelihood with wij being prior probabilities of confidence in the corresponding cell. 

The factor wij θij (t)/Σ wij θij (t) is then simply the posterior updating weight. 

The results given in Section 5 relate to the weighted IPF calibration only. 

Using the above updating scheme fits the saturated model to the data. This model may 

be too detailed to describe the data and as mentioned above the ‘quasi-perfect’ 

mobility model may also be of interest. In this situation the update scheme separates 

the diagonal of the observed data (and confidences) from the remainder and applies 

the independence model to the off-diagonal elements and the saturated model to the 

diagonal. Therefore the update rules for the off-diagonal elements is: 

which of course is independent of zij. 

θij 

 =  wij Ti Tj /(Σ wij Ti Tj)      

These will generally give slightly biased estimates of the marginals. This is corrected 

by re-fitting the true marginals with simple IPF under the saturated model. This is 

referred to as weighted mobility calibration in Section 3. 

Finally, the confidence matrix is represented in the weighted IPF model by a prior 

weight that multiplies the current estimated (i.e. likelihood) proportions. Therefore it 

is possible to replace the confidence matrix by simulated distribution of imputed 

weights. In this case the weight matrix entries are chosen from the Gamma 

distribution   

pij

♦ = (1/pij) * Gamma(pij)      

These Gamma weights then multiply the cell proportions (having an assumed 

underlying Poisson distribution) giving in effect conjugate Gamma posterior weights. 

This data augmentation procedure is simulated 100 times to give estimated 

coefficients of variation (CVs) for the weighted IPF calibration results.   

4.  Loss Function Results 

Additive Calibration 

To get a basis from which to assess the effect of using the confidence indicators we 

first estimated θ using ‘unweighted’ loss functions (i.e. putting all p = 1). For the 

additive (L1) scenario the resulting changes to each cell are shown in Table 3. We just 

7

 
show the actual changes since relative changes are not relevant in this situation. The 

total row and columns changes are also given to indicate where in the table change is 

‘needed most’. 

In some sense this represents the fairest or most democratic way of assigning the 

required changes among the various cells. It takes no account of (initial) cell size nor 

does it listen to any arguments regarding the relative accuracy or quality of the initial 

cell estimates.. 

Table 3  Additive Calibration with No Confidence Indicators 
              Changes to Cells 

                   €m 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total

4.8 

2.3

0.7 

3.3 

2.8

8.3

-6.0 

-8.4

-10.1 

-7.4 

-7.9

-2.4

-1.5 

-4.0

-5.7 

-3.0 

-3.5

3.6 

1.2

-0.5 

2.1 

1.7

-2.9 

-5.3

-7.0 

-4.3 

-4.8

2.0

7.2

0.7

4.3

-6.4

-2.0

3.2

3.2

-7.6

-3.2

2.0

4.9

34.7

-5.9

-62.2

-1.5

-22.4

3.7

24.0

-3.3

-4.5

-2.8

-34.3

15.5 

13.0

11.4 

14.0 

13.5

19.0

15.0

13.8

15.6

130.9

9.8 

7.3

5.7 

8.3 

7.8

13.3

-0.7 

-3.1

-4.8 

-2.1 

-2.6

1.0 

-1.5

-3.2 

-0.5 

-1.0

2.9

4.5

9.3

-1.1

0.5

8.2

-2.3

-0.7

9.9

79.7

-0.6

-14.4

1.0

0.1

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total 

23.6 

1.5

-13.5 

10.3 

6.0

55.5

19.5

8.8

24.4

136.1

To assist in later comparisons, Table 4 gives a simple analysis of the absolute changes 

in Table 3 classified by the cells’ user supplied confidence indicator.  This backs up 

the democratic argument as, not surprisingly, there are no significant differences 

among the (unused) user confidence indicators. 

Table 4  Additive Calibration with No Confidence Indicators 
              Analysis of Absolute Change 

Confidence 
Indicator 

N 

Obs  N  Mean  Std Dev  Minimum  Maximum 

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

6 

6 

6 

6 

12  12 

13  13 

13  13 

9 

9 

10  10 

3.9 

5.6 

9.2 

6.9 

4.7 

4.8 

3.7 

2.6

5.9

6.2

3.0

2.2

4.3

3.1

1.7

0.6

1.5

0.7

2.1

2.3

0.7

8.4

15.6

19.0

11.4

8.3

15.5

10.1

8

 
 
 
 
 
 
Confidence 
Indicator 

N 

Obs  N  Mean  Std Dev  Minimum  Maximum 

0.8 

0.9 

7 

5 

7 

5 

3.3 

1.4 

4.5

1.0

0.5

0.5

13.0

3.2

When we now introduce the user supplied confidence indicators the situation changes 

somewhat as seen in Table 5. The changes now reflect, to a large extent, the level of 

confidence which the user assigned to each cell. The ideal situation we would like to 

see is something like: if Pi > Pj then |θi – Zi| < |θj – Zj|, or if Pi < Pj then |θi – Zi| > |θj – 

Zj|,  In fact this happens for 2,197 of the 2,876 cell comparisons for which Pi did not 

equal Pj. Given the constraints imposed by the control totals this is probably a 

reasonable result.  

Table 5  Additive Calibration with Confidence Indicators 
              Changes to Cells 

                  €m 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total

5.6 

3.4

0.4 

4.8 

0.9

5.9

-7.2 

-18.0

-5.0 

-7.2 

-6.7

-1.5

-2.7 

-0.6

-3.1 

-7.7 

-3.3

3.4 

2.5

0.0 

6.5 

2.1

-2.5 

-2.2

-6.5 

-6.6 

-3.4

4.4

4.4

0.0

8.8

-4.2

-2.3

1.0

2.0

-7.6

-2.1

1.7

3.0

34.7

-4.7

-5.0

-62.2

-22.4

2.4

24.0

-4.1

-4.2

-4.8

-34.3

8.8 

17.5 

0.2 

0.4 

7.0

7.7

0.9

0.8

9.7 

15.1 

13.7

21.7

14.9

14.6

25.3

130.9

4.3 

6.9 

5.8

15.9

-12.2 

-1.1 

-2.0

-1.1 

-0.4 

-1.1

2.3

2.3

6.8

-0.8

-0.5

6.6

-1.5

-0.6

8.3

79.7

-0.2

-14.4

0.2

0.1

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total 

23.6 

1.5

-13.5 

10.3 

6.0

55.5

19.5

8.8

24.4

136.1

However, it is more realistic to compare the confidence influenced results with the 

earlier ones in which no confidence indicators were used. Table 6 gives an analysis of 

the difference between the two situations. A positive difference indicates an increase 

in absolute change from the ‘no confidence’ to the ‘confidence influenced’ situation. 

The way to read Table 6 is as follows. The first line relates to those cells which were 

given a confidence level of 0.1. There were six such cells and 4.8 was the mean 

difference between the absolute change required for calibration without confidence 

indicators  and that required for calibration with the confidence indicators. Thus on 

average these cells were changed more when the confidence indicators were used. 

9

 
 
 
 
 
 
 
This is a desired result. At the other end of the confidence levels, 0.9, the opposite 

happened in that cells were changed on average by a smaller amount (-0.8).  

Table 6  Additive Calibration 
              Difference in Absolute Changes Arising from ‘with’ and ‘without’ Confidence Indicators 

N 

Conf 

Obs  N  Mean  Std Dev  Minimum Maximum

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

0.8 

0.9 

6 

6 

6 

6 

12  12 

13  13 

13  13 

9 

9 

10  10 

7 

5 

7 

5 

4.8 

4.5 

1.2 

-1.3 

-0.9 

-1.8 

-1.2 

-1.6 

-0.8 

3.3 

3.8 

1.0 

1.0 

0.9 

2.2 

1.8 

2.3 

0.8 

0.5

-0.4

-0.3

-3.1

-2.5

-6.7

-5.0

-6.0

-2.1

9.6

9.8

2.7

-0.0

0.8

1.1

1.3

-0.1

-0.0

Overall, the mean column in Table 6 indicates that the introduction of the confidence 

indicators did have the anticipated effect. However there were some deviations, for 

example one cell whose confidence indicator was 0.7 was in fact changed by more 

(1.3) when the confidence indicators were applied.  

Minimum Entropy Calibration 

Again, for the L4 scenario we first estimated θ using ‘unweighted’ loss functions (p = 

1) and the resulting percentage changes to each cell are in Table 7. The total row and 

column percentage changes are also given to indicate where the greatest relative 

changes must take place. 

10

 
 
Table 7  ‘Min Ent’ Calibration with No Confidence Indicators 
               Percentage Changes to Cells  

       % 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total

5.2 

63.3

4.6 

14.2 

14.6

.

0.5

68.0

54.2

. 

-14.4

-0.1 

2.7 

-2.5

-0.0

-4.5

8.5 

15.4 

0.1 

-2.5 

6.8 

0.5 

0.6

7.9

2.5

3.6

0.3

0.2

5.5

2.1

4.4

3.2

11.7 

2.3 

16.8

55.9

-1.3

28.8 

3.5 

11.3

.

. 

. 

0.1

4.2

0.6

2.4 

0.1 

0.1 

2.7 

0.8 

. 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

55.8 

-15.5 -31.0 

36.1 

-4.1

34.5

12.1

Total 

49.6 

-0.9

-1.8 

23.6 

1.7

33.4

10.3

0.1

1.4

0.3

2.8

4.2

6.4

0.2

0.2

-20.6

-18.7

2.0

1.0

-6.4

19.8

1.4

-16.1

3.7 102.3

3.6

3.1

5.2

4.5

24.8

-7.3

.

1.3

1.9

17.4

-9.4

68.8

21.7

-6.4

0.0

3.6

This method is effectively minimising the entropy in the table and so is, to a large 

extent,  assigning the necessary changes on a relative basis with the larger cells being 

chaged more than the smaller ones. As with the additive situation earlier, we give, in 

Table 8, a simple analysis of the absolute percentage changes in Table 7 classified by 

the cells’ user supplied confidence indicator.  This shows, using the earlier analogy, 

that the percentage changes in this case are democratic in that there is no clear 

relationship between them and the cells’ confidence indicator. In Table 8 the cells 

which had an initial and final value of zero, are included as zero percent change. 

Table 8  ‘Min Ent’ Calibration with No Confidence Indicators 
              Analysis of Absolute Percentage Change 

Confidence 
Indicator 

N 

Obs  N  Mean  Std Dev  Minimum  Maximum 

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

0.8 

0.9 

6 

6 

6 

6 

12  12 

13  13 

13  13 

9 

9 

10  10 

7 

5 

7 

5 

4.9 

24.4 

9.3 

5.9 

11.3 

10.2 

5.7 

7.0 

22.9 

5.4

38.5

15.7

6.8

19.6

20.5

10.5

12.9

21.5

0.0

0.8

0.0

0.1

0.0

0.0

0.0

0.1

0.0

14.4

102.3

55.9

24.8

68.0

63.3

34.5

36.1

55.8

When we introduce the confidence indicators the situation changes significantly, see 

Table 9. The percentage changes follow fairly well, and inversely, the size of the 

confidence indicator. The wished for situations are those in which  Pi > Pj  and  |(θi – 

11

 
 
 
 
 
 
Zi)/ Zi | ≤ |(θj – Zj)/ Zj|,  or  Pi < Pj  and  |(θi – Zi)/ Zi | ≥ |(θj – Zj)/ Zj|. This occurred for 

1,737 of the 2,876 cell comparisons for which Pi did not equal Pj. This is not as good a 

result as for the additive calibration where 2,179 of the comparisons were in the 

desired direction.  

Table 9  ‘Min Ent’ Calibration with Confidence Indicators 
              Percentage Changes to Cells  

      % 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total

11.3 

58.5

3.3 

38.1 

12.5

.

0.6

69.8

54.2

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

0.3

0.2

37.0

1.9

7.0

0.1

1.4

0.3

5.1

4.8

. 

-22.3

-0.1 

3.1 

1.0

8.4

6.0 

38.5 

0.1 

34.9 

14.4

-1.4

-1.4 

0.7 

4.7

7.0 

0.3 

0.2 

2.8 

3.5 

. 

-0.2

0.0

1.0

2.4

.

5.9 

2.9 

15.4

69.3

24.3 

4.0 

15.2

. 

. 

0.2

7.7

0.6

10.0

0.2

55.3 

-6.0

-25.6 

17.8 

-5.6

25.2

11.5

Total 

49.6 

-0.9

-1.8 

23.6 

1.7

33.4

10.3

0.4

-19.1

-18.7

1.7

2.0

-6.2

18.1

2.5

-17.1

3.9

5.3

4.8

4.8

4.5

98.6

26.8

-7.5

.

1.3

1.9

17.4

-9.4

68.8

21.7

-6.4

0.0

3.6

To get a more valid measure of the effect of the confidence indicators, Table 10 gives 

an analysis of the difference between the with and without confidence indicators 

results. A positive difference indicates an increase in absolute percentage change from 

the ‘no confidence’ to the ‘confidence influenced’ situation. 

Table 10  ‘Min Ent’ Calibration 
                Difference in Absolute % Changes Arising from ‘with’ and ‘without’ Confidence 
Indicators 

N 

Conf 

Obs  N  Mean  Std Dev  Minimum Maximum

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

0.8 

0.9 

6 

6 

6 

6 

12  12 

13  13 

13  13 

9 

9 

10  10 

7 

5 

7 

5 

13.2 

7.7 

2.5 

0.4 

0.1 

-0.7 

-1.1 

-3.4 

-3.2 

13.7 

12.4 

4.1 

2.7 

1.8 

1.7 

3.1 

6.7 

4.2 

0.0

-3.7

-1.4

-5.8

-4.5

-4.8

-9.3

-18.3

-9.5

31.5

23.9

13.4

3.9

2.3

0.5

1.5

0.0

0.0

Overall the table shows that we got a desired result with the percentage change 

decreasing with increasing confidence (except for a minor blip between 0.8 and 0.9). 

12

 
 
 
 
 
In particular, all the twelve cells with confidence indicators of 0.8 and 0.9 showed a 

decrease in change, except for the zero cells which did not change. On the other hand 

the six  cells with confidence 0.1 showed an increase in change. As with the additive 

situation, see Table 6, there were a few cases where the introduction of the confidence 

indicators did not have the anticipated effect.  

5.  EM/IPF Results 

Weighted IPF Calibration 

The weighted IPF update rule was next applied to give weighted IPF calibration 

estimates. The resulting percentage changes are computed and displayed in Table 11 

along with total row and column percentage changes, given to indicate where in the 

table change is ‘needed most’. It is clear from the table that this weighted IPF 

procedure gives very large percentage changes that are due in part to small initial 

values in those cells. In addition the largest changes occur where the corresponding 

row and column changes are greatest indicating table structure is preserved.  

Table 11  Weighted IPF Calibration  
                Percent Changes to Cells 

          % 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total 

185.7  54.7 

33.6 

111.6

40.1

59.9

0.0

81.6

42.2

54.2 

0.0  -50.1 

-12.1 

-6.7 -13.9

-1.4

-4.1 -14.3

-13.0

-18.7 

58.1 

5.5 

-2.4 

13.0

-0.6

58.8

6.1

3.1

63.7  17.2 

7.3 

89.8

63.9

19.7

19.3

38.0

-3.1

11.5

36.4 

5.0 

-13.8 

2.5

-5.4

10.1

4.1

-1.5

-12.2

70.6  31.2 

38.9 

57.2

55.6

83.1

84.7

84.6

210.3  33.6 

15.7 

27.2

23.0

57.3

40.8

38.7

1.9 

17.4 

-9.4 

68.8 

21.7 

-6.4 

0 

3.6 

71.6

19.8

-7.4

0.0

1.3

3.0

0.1

4.5

0.0 

0.0 

0.0 

41.3 

6.1 

-10.7 

0.0

3.1

-2.7

-8.8

8.2

13.0

4.3

5.9

Total 

49.6 

-0.9 

-1.8 

23.6

1.7

33.4

10.3

Looking the Table 12 the weighted IPF results are compared against their ‘without’ 

weighting counterparts. That is, the analysis shows the percentage improvement or 

otherwise of weighted IPF absolute percent changes against their straightforward 

counterparts where no weights are applied. It is clear from the figures that where 

confidence is low the IPF ‘without’ weighting is more appropriate. But where 

confidence is high the weighting method gives better results or smaller changes. This 

13

 
 
  
        
 
 
 
is in line with expectations as it indicates that cells with higher confidence values are 

subject to a positive improvement and therefore smaller changes under the weighting 

scheme.  

Table 12  Weighted IPF Calibration  
 Improvement Analysis of Absolute                                                                                          
Percent Changes over IPF (without weighting) 

Confidence 
Indicator 

N 

Obs  N  Mean  Std Dev  Minimum  Maximum 

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

0.8 

0.9 

6 

6 

6  184.7 
6 

40.1 

12  12 

13  13 

13  13 

9 

9 

10  10 

7 

5 

7 

5 

1.3 

11.7 

2.3 

-9.3 

-19.0 

-46.7 

-10.9 

66.9

59.1

32.5

67.7

52.9

61.8

36.3

30.9

21.9

130.3

-27.0

-67.5

-62.4

-57.1

-52.3

-52.0

-88.9

-43.5

273.7

140.3

42.9

219.0

114.3

133.3

44.5

7.8

2.9

Weighted Mobility Calibration 

The weighted ‘quasi-perfect’ mobility update rule is applied to Table 1 with weights 

derived from the ‘confidence’ Table 2. The percentage changes are computed and 

displayed in Table 13 along with total row and columns changes given to indicate 

where in the table change is ‘needed most’. It is clear from the table that this weighted 

IPF procedure gives very large percentage changes in virtually all cells. The diagonal 

is reduced  and size of the off diagonal elements indicates that this model is not 

appropriate for the data. This of course is in line with expectations as the deviance of 

the model was quite large. Given the poor estimates there is no value in looking at the 

smoothness of the confidence indicators. 

14

 
 
(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Table 13  Weighted Mobility Calibration  
                 Percent Changes to Cells 

% 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

Total 

-15.5 

-91.6 

280.2 

-46.4

-24.1 2472.9

0.0 2302.8

110.7 

54.2 

0.0 

30.1 

642.5 

39.7

80.1 1016.5 2577.2

793.7

-49.4 

-18.7 

529.7 

-13.1 

-20.6 

27.2

-43.5

601.1

248.9

60.0

-4.9 

1.9 

2216.5 

856.9  2062.3 

16.1

442.2

3.7

828.1

267.2

-32.4 

17.4 

3041.7 

-45.1 

-36.8  1025.9

-42.2

-18.9

-24.5

1.9

5.1 

-9.4 

27.5 

-56.0 

64.4 

130.3

-10.5

-55.0

199.4

186.5

126.8 

68.8 

2094.2 

-45.4 

-63.2 

112.3

-11.1

199.3

-6.0

122.4

95.7 

21.7 

0.0 

0.0 

0.0 

-25.0 

63.8 

20.0 

0.0

3.4

Total 

49.6 

-0.9 

-1.8 

23.6

Weighted IPF Calibration CVs 

404.6

67.5

204.5

-58.5

-35.5 

-6.4 

3.6

1.7

-9.6

33.4

-23.0

-20.4

10.3

4.5

0.0 

1.3 

0 

3.6 

The data augmentation technique outlined in Section 3 was applied to Table 1 and the 

confidences given in Table 2. The actual CVs produced are given in Table 14 and it is 

clear there is quite wide variation across the table but once again most of the larger 

values are due to small values occurring in that cell in Table 1. Excluding these the 

range for the CVs is between 0 and 50% with an average of about 33%. Thus on 

average it may be expected that a calibrated cell value of 1.0 would lie between 0.33 

and 1.66 in about 95% of samples of confidence values. 

Table 14  Weighted IPF Calibration 
                Estimated CVs of Cell Estimates 

       % 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

(8) 

(9) 

100.8 

19.2 

64.4 

46.7

266.9

29.0 

60.0 

20.4 

6.3 

83.4 

51.9 

26.4 

17.6 

55.4

62.3

21.3

55.4

34.1

52.0 

46.7 

101.0

16.0 

8.6 

55.2

54.8 

37.6 

0.0 

44.9 

102.9 

132.6 

. 

. 

. 

.

12.3 

15.1 

4.5 

12.8

0.0

57.3

17.9

65.1

45.4

85.7

125.1

30.8

37.3

74.1

85.9

16.9

25.9

83.0

29.2

59.8

5.4

0.0

27.1 

55.2

14.4

38.1

29.2

8.6 

2.8 

8.0 

7.2 

67.7

25.1 

37.8

12.0 

30.6

4.9 

4.3

. 

62.4

11.9

27.5

19.2

45.1

18.0

59.6

5.6

Turning to the distribution of CV by confidence level, Table 15, the mean variation is 

not level across the confidence values. Clustering of the values tends to show that 

there is a higher variation across the middle of the confidence range from 0.3 to 0.6. 

15

 
 
        
 
 
 
 
 
 
 
 
 
 
This then suggests that the range of confidence levels from 0.1 to 0.9 are not 

distinctive enough and should be replaced by say 3 levels 0.15, 0.45 and 0.8 

representing complete uncertainty about the data, some knowledge and a high level of 

certainty respectively. These gradations of confidence of course are specific to this 

scheme. However, the data augmentation imputations reflect how variation in the 

confidence impacts on the calibration. It is therefore quite general and so the 

conclusion about the gradations can also be accepted in general.   

Table 15  Weighted IPF Calibration  
                Distribution of CVs by Confidence value  

Confidence 
Indicator 

N 

Obs  N  Mean  Std Dev  Minimum  Maximum 

0.1 

0.2 

0.3 

0.4 

0.5 

0.6 

0.7 

0.8 

0.9 

6 

6 

6 

6 

12  12 

13  13 

13  13 

9 

9 

10  10 

7 

5 

7 

5 

31.5 

40.4 

56.7 

30.1 

49.0 

67.8 

36.8 

30.8 

9.3 

21.1

49.0

28.9

20.6

71.1

39.8

29.3

22.9

5.2

0.0

2.8

7.2

0.0

8.0

11.9

5.6

4.3

4.5

55.4

132.6

101.0

62.3

266.9

125.1

83.4

59.8

15.1

6.  Conclusion 

We have shown that calibration, even in the presence of overlapping and interrelated 

constraints, can take account of external user supplied confidence indicators. However 

we have not been able to ensure the ideal situation in which every high-confidence 

cell was changed by a smaller amount than every low-confidence one. 

The methods we outlined can be of particular use in the provision of early or 

preliminary estimates of statistics when (a) only provisional or uncertain estimates are 

available for intermediate statistics, and (b) more accurate estimates are available for 

some overall constraining data. In such cases, the user can improve, through 

calibration, the intermediate estimates and in doing so can also take account of their 

confidence in the original estimates, based on knowledge or experience. 

Further investigation could profitably be done in the area of finding ways of getting a 

more structured relationship between the confidence indicator and the size of the 

16

 
 
change in each cell. For example, if one cell has confidence twice of another then the 

“expected” change to the former would be half that to the latter. 

Other, perhaps more difficult, investigations could be done into methodologies that 

look at all possible calibrations to a given set of constraints and select the one which 

best meets the requirement that: if Pi > Pj then |θi – Zi| < |θj – Zj|, or if Pi < Pj then |θi – 

Zi| > |θj – Zj|. This stochastic analysis could also examine the impact of sampling 

effects in the given marginal totals as well as the weights to arrive at an ideal 

selection. A bayesian framework that examines the posterior distribution of the cell 

estimates, with say Markov Chain Monte Carlo methods, might provide a good setting 

for further study of weighing in the context of balancing a table to given marginal 

totals.    

17