Sample-based calibration of multiple 

surveys

Jean D. Opsomer, Westat

Weijia Ren, Westat

John Foster, NOAA Fisheries

The views presented are those of the author(s) and do not represent the views of any Government 

Agency/Department

FCSM 2021 • Virtual Meeting• November 2
Outline

1. Introduction: recreational fishing surveys

2. Sample-based calibration

3. Application

4. Conclusions

2
Marine Recreational Information Program (MRIP)

◮ NOAA Fisheries is responsible for managing marine fisheries under

the Magnuson-Stevens Act

◮ MRIP produces estimates of marine recreational catch in US waters

◮ input into stock assessment models, used to set annual catch limits

◮ In MRIP, multiple surveys are combined to create estimates

◮ Access Point Angler Intercept Survey (APAIS)

◮ Fishing Effort Survey (FES)

◮ (others)

◮ APAIS: stratified multi-stage sample of fishing trips, collecting

detailed data on trip and catch characteristics

◮ FES: stratified sample of general population households, collecting

data on number of fishing trips over past 2 months

3
MRIP Estimation

◮ Combination of APAIS and FES:

◮ survey weights in APAIS are calibrated to FES-obtained estimates of

number of trips by state and wave

◮ other adjustments for undercoverage of respective frames

◮ NOAA Fisheries provides public-use datasets with trip-level data and

calibrated weights

◮ Variance estimation: current method uses linearization based on

APAIS design

◮ does not account for calibration to FES

4
Sample-based Calibration

◮ Calibration reduces the variance of survey estimators, so it is

generally beneficial to account for it in variance estimation

◮ In particular, variance of estimated control totals is zero (for

population-based controls)

◮ But: sample-based calibration equalizes estimates between surveys,

may not reduce variance

◮ Important to account for variance contributions from both surveys

into final variance estimates

◮ We describe methods to incorporate calibration into replicate

variance estimation, when calibration totals are themselves random

5
2. Methodology: Primary Survey (APAIS)

◮ Sample s, weights wi

◮ Population total ty =

P

U

yi estimated by tby =

P

s wi yi

◮ e.g. tby = estimated total catch of striped bass by private boat in GA

during May-June 2019

◮ Replication variance estimator

Vb(tby ) = A

X

R

r=1



tb(r )

y − tby

2

with tb

(r )

y =

P

s w

(r )

i

yi

◮ Replicate weights w

(r )

i

,r = 1, . . . , R and constant A determined by

replication method

⇒ Balanced Repeated Replication (BRR) with Fay’s adjustment

⇒ R = 160

6
Calibration Survey (FES)

◮ Sample sC , weights wCi

◮ Estimator btCx =

P

sC

wCixi of length H, to be used as controls

◮ e.g. tbCx,h = estimated number of angler trips by private boat in GA

during May-June 2019

◮ Estimator VbC (btCx ) of H × H variance-covariance matrix Var(btCx )

7
Calibration of Primary Survey: Regression Estimation

◮ Regression estimator with calibration vector btCx

tby,reg = tby + (btCx − btx )

T βb =

X

s

w

∗

i

yi

◮ Define ei = yi − β

T

U xi

, then asymptotic variance

AVar(tby,reg) = Var(tbe ) (variance with fixed controls)

+β

T

UVar(btCx )βU (effect of random controls)

◮ For fixed btCx , Var(tbe ) consistently estimated by

Vb(tby,reg) = A

X

R

r=1



tb(r )

y,reg − tby,reg2

with

tb(r )

y,reg = tb(r )y + (btCx − bt

(r )

x

)

T βb

(r )

=

X

s

w

∗(r )

i

yi

(“apply calibration to each replicate”)

8
Approaches to Estimate AVar(tby,reg)

1. Direct plug-in: Vb(tbeb) + βb

T

Vb(btCx )βb

2. Opsomer and Erciulescu (2021): when replicates are available for

both surveys, create replicated control totals bt

(r )

Cx to calibrate

primary survey replicates (originally proposed by Kott (2005))

3. Fuller (1998): compute eigen-decomposition of Vb(btCx ) and perturb

controls of primary survey replicates

9
Implementing Opsomer and Erciulescu (2021) Method

◮ Applicable when control survey has replicates for variance estimation

◮ Estimate vector btCx =

P

sC

wCixi

◮ Replicate variance-covariance matrix estimator

VbC (btCx ) = AC

X

RC

r=1



bt

(r )

Cx − btCx bt

(r )

Cx − btCxT

with bt

(r )

Cx =

P

sC

w

(r )

Ci xi

◮ Replicate weights w

(r )

Ci ,r = 1, . . . , RC and constant AC determined by

control survey replication method

◮ Assume RC = R

10
Implementing Opsomer and Erciulescu (2021) Method (2)

◮ Adjust control totals in replicates of primary survey, based on

replicates from control survey

tb(r )

y,reg = tb(r )y + (btCx + ar(bt

(r )

Cx − btCx ) − bt

(r )

x

)

T βb

(r )

= tb(r )

y + (btCx − bt

(r )

x

)

T βb

(r )

+ ar(bt

(r )

Cx − btCx )

T βb

(r )

◮ Set ar =

p

AC /A, then

Vb(tby,reg) = A

X

R

r=1



tb(r )

y,reg − tby,reg2

consistent for

AVar(tby,reg) = Var(tbe ) + β

T

UVar(btCx )βU

11
Implementing Fuller (1998) Method

◮ Assume H = R for now

◮ Compute eigen-decomposition of Vb(btCx )

Vb(btCx ) = X

H

h=1

λhqhq

T

h =

X

H

h=1

δhδ

T

h

◮ Adjust control totals in replicates of primary survey

tb(r )

y,reg = tb(r )y + (btCx + arδr − bt

(r )

x

)

T βb

(r )

= tb(r )

y + (btCx − bt

(r )

x

)

T βb

(r )

+ arδrβb

(r )

◮ Set ar = 1/

√

A, then

Vb(tby,reg) = A

X

R

r=1



tb(r )

y,reg − tby,reg2

consistent for

AVar(tby,reg) = Var(tbe ) + β

T

UVar(btCx )βU

12
Implementing Fuller (1998) Method (2)

◮ What if the numbers of control totals and replicates differ?

◮ If H ≤ R:

ar =



√1

A

r = 1, . . . , H

0 r = H + 1, . . . , R

◮ If H > R: use δh corresponding to R largest eigenvalues, which

assumes that

XR

r=1

δhδ

T

h ≈ Vb(btCx )

(low-rank approximation)

◮ Works for calibration by regression, post-stratification and raking

13
3. Application to APAIS Calibration

◮ 2019 APAIS and FES datasets

◮ Post-stratify APAIS weights to match FES estimated trip totals for

16 states, 2 modes (shore, private boat), 6 waves

⇒ 160 control estimates (R = H)

14
3. Application to APAIS Calibration (2)

◮ Distribution of CVs over 160 calibration domains

Mean Min. 1st Qu. Median 3rd Qu. Max.

FES 0.203 0.075 0.169 0.194 0.225 0.702

Trips APAIS (new) 0.203 0.075 0.168 0.193 0.225 0.700

APAIS (old) 0.246 0.070 0.156 0.219 0.294 0.791

Red APAIS (new) 0.620 0.201 0.462 0.654 0.746 1.097

Snapper APAIS (old) 0.607 0.184 0.453 0.541 0.877 1.000

15
3. Application to APAIS Calibration (3)

◮ Investigate scenario when H > R

◮ 172 FES controls: estimated trip totals for 17 “states,” 2 modes

(shore, private boat), 6 waves

◮ Use δh of 160 largest eigenvalues of Vb(btCx )

16
3. Application to APAIS Calibration (4)

◮ Distribution of CVs over 172 calibration domains

Mean Min. 1st Qu. Median 3rd Qu. Max.

FES 0.208 0.100 0.171 0.196 0.233 0.702

Trips APAIS (new) 0.207 0.100 0.171 0.195 0.233 0.700

APAIS (old) 0.247 0.081 0.159 0.219 0.291 0.791

Red APAIS (new) 0.631 0.199 0.498 0.633 0.791 1.073

Snapper APAIS (old) 0.607 0.184 0.453 0.541 0.877 1.000

◮ Distribution of CVs over 160 calibration domains

Mean Min. 1st Qu. Median 3rd Qu. Max.

FES 0.203 0.075 0.169 0.194 0.225 0.702

Trips APAIS (new) 0.203 0.075 0.168 0.193 0.225 0.700

APAIS (old) 0.246 0.070 0.156 0.219 0.294 0.791

Red APAIS (new) 0.620 0.201 0.462 0.654 0.746 1.097

Snapper APAIS (old) 0.607 0.184 0.453 0.541 0.877 1.000

17
4. Conclusions

◮ Sample-based calibration can be very useful in practice, e.g.

◮ organization conducts multiple surveys and wishes to report

consistent estimates

◮ following changes in survey methodology, survey results are no longer

comparable with previous surveys and need to be adjusted

◮ fixed controls are not available

◮ multi-phase samples

◮ Important to reflect calibration to sample-based controls in measures

of precision

◮ Can be accomplished easily within replication methods for primary

survey

Contact: JeanOpsomer@westat.com

18