Program Refinements to Optimize Model Impact and Scalability Based on Evidence
Phase: NA
Conditions: Hiv
Sponsor: City University of New York, School of Public Health

Program Refinements to Optimize Model Impact and Scalability Based on Evidence (PROMISE) 
NCT03628287 
Date: July 27, 2020 
Study Protocol and Statistical Analysis Plan 
 
 
 

Study Protocol and Statistical Analysis Plan 
The full protocol has been previously published.1 Excerpts are provided below. 
Eligibility criteria (for clients and sites) 
PWH eligible for the trial analysis of CCR intervention effects include those newly enrolled in the CCP or 
CCR and having unsuppressed VL (HIV RNA ≥200 copies/mL) as of their latest test in the year prior to 
enrolment or having no VL test result reported to surveillance in the year prior to enrolment (presumed 
out of care).10 To allow 4 months of VL outcome observation per enrollee prior to the start of the next 
phase of CCR rollout, the trial-eligible enrolment window for each 9-month implementation period is 
restricted to the first 5 months. The trial excludes newly awarded (CCP-naïve) agencies and includes only 
the 17 re-awarded agencies, which could be assigned to continue CCP delivery uninterrupted or begin 
CCR delivery in the initial implementation phase. 
Outcome measurement 
To assess the clinical benefit of the programmatic revisions distinguishing the CCR from the CCP, we will 
analyze client-level, surveillance-based laboratory test data.10 The outcome, timely VS (TVS), is defined 
as VL <200 copies/mL on the last VL test reported to the NYC HIV surveillance registry in the 4 months 
following CCP/CCR enrolment (TVS=1). We have chosen to dichotomize VL data for statistical analysis 
using the cut-off value of 200 copies/mL, in accordance with the CDC definition of VS.24 Consistent with 
our prior CCP work,8 10 11 18 those without any VL measure during follow-up will be considered not to 
have achieved VS (TVS=0), given their lack of documented clinical monitoring since their last 
unsuppressed VL. The 4-month follow-up period aligns with US Department of Health and Human 
Services HIV guidelines, which reinforce the standard practice of VL monitoring every 3 to 4 months, or 
more often when adherence difficulties are apparent.25 For PWH starting ART or a modified ART 
regimen, the guidelines recommend VL monitoring every 4 to 8 weeks until VS is reached, and state that 
‘individuals who are adherent to [ART] and do not harbor resistance mutations to the component drugs 
can generally achieve viral suppression 8 to 24 weeks after ART initiation.’25 Recent publications also 
support the applications of shorter-term measures of VS; researchers at NYC’s Health Department have 
proposed adding a 3-month VS indicator for tracking national progress on the HIV care continuum,26 and 
a San Francisco study of a vulnerable population of newly diagnosed patients referred for rapid ART 
initiation found that the median time from start of ART to VS was 41 days.27 Our TVS measure takes into 
account both the timing of routine VL monitoring in as-yet-unsuppressed PWH and current expectations 
for VS achievement in a context of effective ART and universal/immediate treatment policies.25 28 29 
Timeline 
Figure 1 illustrates the three 9-month periods used in the stepped-wedge design: Period 0, with CCP at 
all 17 agencies and no CCR; Period 1, representing CCR implementation only at sites randomized to an 
early start (and thus encompassing the months of simultaneous operation of the CCP and the CCR); and 
Period 2, representing CCR implementation at all 17 sites. 

 
Figure 1: Stepped-wedge design with three implementation periods. CCP, Care Coordination Program. 
Recruitment 
Beyond standard contract startup deliverables based on early program enrolment milestones, no 
specific incentives have been used to encourage recruitment. Analyses will include all eligible 
enrolments in CCP/CCR services at any of the 17 study sites. 
Assignment of interventions 
Randomization 
Though the unit of analysis for TVS is the individual, the unit of randomization is the Care Coordination 
provider agency (i.e., cluster). Cluster randomization serves to minimize crossover between intervention 
conditions and avert the logistical and ethical dilemmas posed by client-level randomisation.30–32  
Agencies were matched and randomized within pairs (including one case in which two smaller agencies 
were matched to a larger one). Matching accounted for characteristics plausibly related to the TVS 
outcome: agency type, primary location/borough and program size (measured via a combination of CCP 
caseload at the time of re-award and award amount). While randomization could not feasibly be 
stratified by each of these variables, the lead analyst suggested pairs maximizing similarity on these 
variables. Pairings were finalized with input from other team members knowledgeable about the 
programs/agencies involved. The lead analyst used a random number generator in Excel to determine 
agency assignments within pairs, and assignments were communicated as contract conditions in the 
notifications of awards. 
Blinding 
Blinding was not feasible for this study. Assignments were transparent to implementing agencies, study 
team members and interested stakeholders, since contracts are publicly available information. 
Data collection, management and analysis 
Data collection 
As with prior studies of CCP effectiveness,8 10–12 18 the outcome measure for clients in both study arms 
will be derived from the NYC HIV surveillance registry (‘the Registry’), a population-based data source of 
electronically reported longitudinal laboratory (VL, CD4) records on all diagnosed NYC PWH.33 34 Use of 
the Registry allows near 100% ascertainment of VS for PWH in NYC HIV medical care, regardless of 

specific NYC medical provider, and for periods extending before and after program enrolment or 
discontinuation. 
Each client’s CCP/CCR enrolment agency and start date are determined from a database of contractually 
required Ryan White Part A provider reporting to the Health Department, the Electronic System for 
HIV/AIDS Reporting and Evaluation (eSHARE). These program reporting-based measures are available 
(non-missing) for all CCP/CCR clients and all implementing agencies. Program data collection forms are 
located on the NYC Health Department website (https://www1.nyc.gov/site/doh/health/health-
topics/aids-hiv-care-coord-tools.page). 
Data management and quality assurance 
All data for the trial are entered as part of established, legally or contractually required reporting, and 
are protected according to CDC physical and electronic data security and confidentiality policies.35 
Health Department staff clean and freeze surveillance data sets on a quarterly basis, and conduct 
matches of program to surveillance data semi-annually, with human review of each near-match by two 
independent analysts and a separate ‘tie-breaker’ when the analysts’ determinations differ. Details on 
the deterministic matching algorithm have been previously described.36 Through the match, participants 
are assigned a unique record number used in merging surveillance and programmatic data for analytical 
data sets, which are stripped of all personal identifiers prior to analysis and stored on the most secured 
drives on the Health Department network. eSHARE data quality is checked by Health Department 
analysts at the time of each monthly extraction. For purposes of payment, provider agencies also review 
draft extracts and fill in any missing enrolment and services data monthly. 
Statistical analysis for the matched-pairs stepped-wedge trial 
Analysis overview and rationale. 
We will apply an innovative, fully conditional analysis that, in addition to allowing for arbitrary period 
effects, allows for arbitrary within-pair site differences. The analysis plan is based on the exact, 
conditional distribution theory of non-central multiple hypergeometric distributions and their 
convolutions,37 which will enable us to estimate and test the effect of the revised intervention as a single 
parameter defined below. The conventional statistical analysis proposed for cross-sectional stepped-
wedge designs (i.e., with independent samples of clients enrolled at each step)38 assumes a mixed model 
with random cluster effects and fixed period effects, but this is not appropriate for our matched-pairs 
stepped-wedge trial. For one, the matching of pairs is under the investigators’ control and so should be 
conditioned on. Second, the generalized linear mixed model has limitations, such as a gratuitous and 
unverifiable assumption of normal distribution for random effects and poor variance estimation 
performance in small samples (of clusters), even with robust variance estimation, such that jackknifing 
must be used. However, the following exact analysis avoids those problems by conditioning out the 
nuisance parameters. 
Analysis approach and assumptions. 
As shown in table 3, for each pair of sites, we will produce two 2×3 tables (one table per site in pair), 
cross-classifying the number of TVS and non-TVS outcomes in Period 0 (with original CCP but no CCR 
implementation), Period 1 (with CCR only at sites assigned to an early start) and Period 2 (with CCR at all 
sites). For identification purposes, we refer to ‘Site 1’ within a matched pair as the site randomized to 

switch in Period 1 (early start) and ‘Site 2’ as the site randomized to switch in Period 2 (delayed start). 
We begin by assuming the following logistic regression model for the three binomial outcomes: the logit 
of the probability of TVS for a given site, period and intervention (CCR versus original CCP) equals an 
intercept representing an arbitrary, pair-specific log odds on TVS for Site 2 in the pair, plus an arbitrary 
log OR (LOR) for Site 1 versus Site 2 in the pair (allowing for imperfectly matched sites), plus two 
arbitrary pair-specific LORs for Period 1 and Period 2 effects relative to Period 0, plus one structural LOR 
of interest, the global intervention effect (non-existent in Period 0, applicable to Site 1 in Period 1, and 
applicable to both sites in Period 2). The exponent of this last parameter is the target of statistical 
inference, namely, the OR for TVS versus non-TVS comparing the CCR to the CCP. A key assumption is 
that any site effects apply in each period and any period effects apply to each site, independent of the 
intervention effect (i.e., that there are no site-by-intervention or period-by-intervention interactions). 
This assumption will be tested and the model elaborated if needed. Note that under the key 
assumption, the constant site and period effects are allowed to vary arbitrarily from one matched pair 
to the next. 
Table 3: Illustration of 2x3 tables cross-classifying TVS and non-TVS outcomes by period. 
Site 1 in pair i (adopts CCR in Period 1)
Period 0 Period 1 Period 2 Total
TVS
Xi10
Xi11
Xi12
Xi1+
No TVS Ni10– Xi10 Ni11– Xi11 Ni12– Xi12 Ni1+– Xi1+
Total
Ni10
Ni11
Ni12
Ni1+
Site 2 in pair i (adopts CCR in Period 2) TVS
Xi20
Xi21
Xi22
Xi2+
No TVS Ni20– Xi20 Ni21– Xi21 Ni22– Xi22 Ni2+– Xi2+
Total
Ni20
Ni21
Ni22
Ni2+
Pair i totals
TVS
Xi+0
Xi+1
Xi+2
Xi++
No TVS Ni+0– Xi+0 Ni+1– Xi+1 Ni+2– Xi+2 Ni++– Xi++
Total
Ni+0
Ni+1
Ni+2
Ni++
 
Light grey cells represent the two 2×3 tables in site pair i. Dark grey cells represent the margins 
upon which the analysis will condition, whereas white cells represent the margins calculated by 
summing or subtracting other fixed margins. 
Estimating the CCR intervention effect. 
Next, by conditioning on the marginal totals within each site (numbers of eligible clients enrolled in each 
period and total numbers of TVS and non-TVS outcomes for each site), the joint distribution of the 
numbers of TVS outcomes for Site 1 by period becomes a non-central multiple hypergeometric 
distribution with only three parameters (the period LORs and the intervention LOR); that is, the 
conditional distribution does not depend on the nuisance site parameters. By further conditioning on 

the sum of TVS outcomes across the two sites in each period, the fully conditional joint distribution 
depends on only one parameter, the intervention effect; that is, the fully conditional distribution 
depends neither on the nuisance site effects nor on the nuisance period effects. In fact, the sufficient 
statistic for the intervention LOR in the fully conditional likelihood function is simply the number of TVS 
outcomes from Site 1 in Period 1. It is then straightforward to calculate the marginal distribution of this 
outcome as a function of the intervention effect. Therefore, we will calculate that distribution for each 
of the 8 matched pairs (including the case of two programs jointly matched to a third) and convolute 
those distributions to obtain the sampling distribution of the sum of sufficient statistics. Once we obtain 
the fully conditional sampling distribution of the sufficient statistic as described above, we will report 
the conditional maximum likelihood estimate of the intervention LOR with an exact, test-based 95% CI. 
The test of the null hypothesis at the two-tailed 0.05 significance level will be based on the exact two-
tailed p value (using the point probability definition),37 and will form the primary outcome analysis. In 
sensitivity analyses, we will also report the Wald, Score and Likelihood Ratio test results, which should 
be close to each other, given client numbers per site per period and the level of TVS from baseline CCP 
data. 
 
 
 

Citations 
1 Irvine MK, Levin B, Robertson MM, et al. PROMISE (Program Refinements to Optimize Model Impact 
and Scalability based on Evidence): a cluster-randomized, stepped-wedge trial assessing effectiveness of 
the revised versus original Ryan White Part A HIV Care Coordination Program for patients with barriers 
to treatment in the USA. BMJ Open. 2020;10(7):e034624. 
 
From published protocol:  
 
8 Irvine MK, Chamberlin SA, Robbins RS, et al. Improvements in HIV care engagement and viral load 
suppression following enrollment in a comprehensive HIV care coordination program. Clin Infect Dis 
2015;60:298–310. 
 
10 Robertson MM, Waldron L, Robbins RS, et al. Using registry data to construct a comparison group for 
programmatic effectiveness evaluation: the New York City HIV Care Coordination Program. Am J Epidemiol 
2018;187:1980–9. 
 
11 Nash D, Robertson MM, Penrose K, et al. Short-term effectiveness of HIV care coordination among persons 
with recent HIV diagnosis or history of poor HIV outcomes. PLoS One 2018;13:e0204017. 
 
12 Robertson MM, Penrose K, Irvine MK, et al. Impact of an HIV care coordination program on durable viral 
suppression. J Acquir Immune Defic Syndr 2019;80:46–55. 
 
18 Irvine MK, Chamberlin SA, Robbins RS, et al. Come as you are: improving care engagement and viral load 
suppression among HIV Care Coordination clients with lower mental health functioning, unstable housing, and 
hard drug use. AIDS Behav 2017;21:1572–9. 
 
24 HIV treatment as prevention. Available: https://www.cdc.gov/hiv/risk/art/ index. html [Accessed 4 Feb 2020]. 
 
25 Panel on antiretroviral guidelines for adults and adolescents. Guidelines for the use of antiretroviral agents 
in adults and adolescents with HIV, table 3 (Pages C-2 to C-4) and Pages C-6, C-7, E-1 and E-2. Department 
of Health and Human Services 2019. 
 
26 Xia Q, Coeytaux K, Braunstein SL, et al. Proposing a new indicator for the National Human 
Immunodeficiency Virus/AIDS Strategy: Percentage of newly diagnosed persons achieving viral suppression 
within 3 months of diagnosis. J Infect Dis 2019;219:851–5. 
 
27 Coffey S, Bacchetti P, Sachdev D, et al. Rapid antiretroviral therapy: high virologic suppression rates with 
immediate antiretroviral therapy initiation in a vulnerable urban clinic population. AIDS 2019;33:825–32. 
 
28 Recommendation to expand antiretroviral therapy to all persons living with HIV - frequently asked questions 
(FAQ) for healthcare providers, 2011. Available: https://www1. nyc. gov/ assets/ doh/downloads/ pdf/ ah/ nyc- 
hivart-faq- provider. pdf [Accessed 5 Feb 2020]. 
 
29 WHO. Guideline on when to start antiretroviral therapy and on pre-exposure prophylaxis for HIV 2015. 
 
30 Featherstone K, Donovan JL. Random allocation or allocation at random? Patients' perspectives of 
participation in a randomised controlled trial. BMJ 1998;317:1177–80. 
 
31 Robinson EJ, Kerr C, Stevens A, et al. Lay conceptions of the ethical and scientific justifications for random 
allocation in clinical trials. Soc Sci Med 2004;58:811–24. 
 
32 Heard K, O’Toole E, Naimpally R, et al. Real world challenges to randomization and their solutions. Boston, 
MA: Abdul Latif Jameel Poverty Action Lab, 2017. 
 

33 New York City Department of Health HIV Epidemiology Group. 25 years of HIV in New York City: lessons 
from surveillance. J Urban Health 2001;78:669–78. 
 
34 Centers for Disease Control and Prevention (CDC). Implementation of named HIV reporting--New York City, 
2001. MMWR Morb Mortal Wkly Rep 2004;52:1248–52. 
 
35 CDC. Data security and confidentiality guidelines for HIV, viral hepatitis, sexually transmitted disease, and 
tuberculosis programs: standards to facilitate sharing and use of surveillance data for public health action. 
Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, 
2011. 
 
36 Drobnik A, Pinchoff J, Bushnell G, et al. Matching HIV, tuberculosis, viral hepatitis, and sexually transmitted 
diseases surveillance data, 2000-2010: identification of infectious disease syndemics in New York City. J Public 
Health Manag Pract 2014;20:506–12. 
 
37 Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. 3rd edn. John Wiley & Sons, 
2013. 
 
38 Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin 
Trials 2007;28:182–91.