Protocol for a Mixed Methods and Multi-site Assessment of the Implementation Process and Outcomes of a New Community-based Frailty Programme
Conditions: Frailty, Frail Elderly Syndrome, Frailty Syndrome
Sponsor: Geriatric Education and Research Institute

Title: 
Protocol for a mixed methods and multi-site 
assessment of the implementation process and 
outcomes of a new community-based frailty 
programme 
 
Document Date:  
March 17, 2021 
 
NCT Number:  
To be provided  

Protocol for a mixed methods and multi-site assessment of the 
1 
implementation process and outcomes of a new community-based 
2 
frailty programme 
3 
 
4 
Woan Shin Tan1,2, Ze Ling Nai1, Hwee Teng Robyn Tan1,3, Sean Nicholas1, Robin Choo1, 
5 
Mimaika Luluina Ginting1, Edward Tan1, Poh Hoon June Teng1, Wee Shiong Lim1,4, Chek Hooi 
6 
Wong1,5, Yew Yoong Ding1,4, the Geriatric Services Hub Programme Group 
7 
 
8 
1 
Geriatric Education & Research Institute, Singapore, Singapore 
9 
2 
Health Services & Outcomes Research Department, National Healthcare Group, 
10 
Singapore, Singapore 
11 
3 
Social Service Research Centre, National University of Singapore, Singapore, Singapore 
12 
4 
Department of Geriatric Medicine, Institute of Geriatrics & Active Ageing, Tock Seng 
13 
Hospital, Singapore, Singapore 
14 
5 
Department of Geriatric Medicine, Khoo Teck Puat Hospital, Singapore, Singapore 
15 
  
16 
Correspondence to: Tan Woan Shin, Geriatric Education & Research Institute 2 Yishun Central 2, 
17 
Singapore 768024 Tel: (65) 68078018.  Email: tan.woan.shin@geri.com.sg
18 

Abstract 
19 
Background: Frailty is increasing in prevalence internationally with population ageing. Frailty 
20 
can be managed or even reversed through community-based interventions delivered by a 
21 
multidisciplinary team of professionals, but to varying degrees of effectiveness. However, many 
22 
of these care models’ implementation insights are contextual and may not be applicable in different 
23 
cultural contexts. The Geriatric Service Hub (GSH) is a novel frailty care programme in Singapore, 
24 
that includes key components of frailty care such as comprehensive geriatric assessments, care 
25 
coordination and the assembly of a multidisciplinary team. This study aims to gain insights on the 
26 
factors influencing the implementation approaches adopted by five participating sites, and the 
27 
effectiveness of the programme. 
28 
Methods: We will adopt a mixed methods approach that includes a qualitative evaluation among 
29 
key stakeholders and participants in the programme, through in-depth interviews and focus group 
30 
discussions. The main topics covered include factors that affected the development and 
31 
implementation of each programme, operations and other contextual factors that influenced 
32 
implementation outcomes. The quantitative evaluation (1) monitors each programme’s care 
33 
process through quality indicators, (2) a multiple-time point survey study to compare programme 
34 
participants’ pre- and post- outcomes on patient engagement (collaboRATE and 13-item Patient 
35 
Activation Measure; PAM), healthcare experiences (Consumer Assessment of Healthcare 
36 
Providers and System Clinician and Group Survey Version 3.0; CG-CAHPS), health status and 
37 
quality of life (Barthel Index of Activities of Daily Living, fall counts, the EuroQol questionnaire 
38 
and the Control, Autonomy, Self-realisation and Pleasure scale; CASP-19), impact on caregivers 
39 
(Zarit Burden Interview) and societal costs (Client Service Receipt Inventory). (3) A 
40 

retrospective cohort design to assess healthcare and cost utilisation between participants of the 
41 
programme and a propensity score matched comparator group.  
42 
Discussion: The GSH sites share a common goal to increase the accessibility of essential 
43 
services to frail older adults and provide comprehensive care. The results of this evaluation study 
44 
will provide valuable evidence to the impact and effectiveness of the GSH and inform the design 
45 
of similar programmes targeting frail older adults.  
46 
 
47 
Trial Registration: To be provided 
48 
 
49 
Keywords: Frailty care, comprehensive geriatric assessment, care coordination, multidisciplinary 
50 
team care 
51 
 
52 
List of Abbreviations used in manuscript:  
53 
 
54 
UK 
United Kingdom 
CGAs 
Comprehensive Geriatric Assessments 
GSH 
Geriatric Services Hub 
RHS 
Regional Healthcare Systems 
CFS 
Clinical Frailty Scale 
GPs 
General Practitioners 
NTFGH 
Ng Teng Fong General Hospital 
PAM-13 
13-item Patient Activation Measure 
CG-CAHPS Consumer Assessment of Healthcare Providers and System Clinician & 
Group Survey Version 3.0 
ADL 
Activities of Daily Living 
CASP-19 
19-item Quality of Life Scale 
ZBI 
Zarit Burden Interview 
CSRI 
Client Service Receipt Inventory 
NRIC 
National Registration Identity Card 
GLM 
Generalised Linear Models 
IRRs 
Incident-rate Ratios 
 
 
55 

Background 
56 
Frailty denotes a state of increased vulnerability due to age-associated decline in function 
57 
and reserve such that the ability to cope with day-to-day or acute stressors is compromised (1). As 
58 
a result, frail older adults are usually more susceptible to adverse outcomes including disability, 
59 
hospitalisation, and mortality (2). A representative population study in the United States reported 
60 
a frailty prevalence of 15% amongst community-dwelling older adults (3). Meanwhile, a study of 
61 
older adults conducted based on the United Kingdom (UK) biobank reported 39% as prefrail and 
62 
4% as frail.  Similar trends are observed in Singapore, an Asian island-state of 5.7 million (4), 
63 
which is also one of the most rapidly ageing countries in the world (5). The reported prevalence 
64 
of frailty in Singapore ranged from 5.7% to 6.2% with corresponding 37% to 46% for pre-frailty 
65 
(6-8) among community-dwelling older adults, depending on the population studied and 
66 
identification tool used (9).  
67 
Frailty can be managed and even reversed (10). With almost 40% of our population assessed 
68 
as frail or in danger of becoming frail, there is a strong need to equip the Singapore healthcare 
69 
system to identify and manage frailty (11). In order to accomplish this whilst addressing the 
70 
complex care needs of frail older adults, care has shifted from a disease-specific to a 
71 
comprehensive approach (12). Comprehensive care for frail older adults usually comprises several 
72 
components, including comprehensive geriatric assessments (CGAs), multidisciplinary teams, 
73 
integrated care plans and a variety of services to cater to the healthcare and social needs of these 
74 
frail older adults (13). Screening for frailty can identify individuals who are most likely to benefit 
75 
from a CGA and targeted interventions (14). In 2017, England became the first in the world to 
76 
mandate the assessment of frailty in adults aged 65 years old and older. The widespread 
77 

deployment of the electronic frailty index, which automatically grades frailty using data available 
78 
in the primary care electronic medical records (15), supported this national effort.  
79 
Innovative care models involving a combination of the above components of care have 
80 
emerged to grow the capacity for more comprehensive care of frail older adults, (16) and to 
81 
improve health and social outcomes (17). Greater integration of care among healthcare 
82 
professionals of different disciplines and across settings for frail older adults has reduced 
83 
unnecessary hospitalisations (18) and supported the maintenance of functional mobility over a 12-
84 
month period (19). A systematic review of integrated or coordinated care found that 
85 
multicomponent care models were more likely to increase patient satisfaction, perceived quality 
86 
of care and patients’ accessibility to care (20). However, there was inconsistent and limited 
87 
evidence on the impact of such models of care on health care costs and outcomes of frail older 
88 
adults (20-22). Taken together, this suggests that more evidence is needed to determine the 
89 
effectiveness of such comprehensive and integrated care models for frail older adults.  
90 
Integrated or coordinated care usually comprised partnerships or collaborations between at 
91 
least two healthcare service providers, such as hospitals and primary care service providers (23). 
92 
Partners in these care models collaborate by sharing expertise through training, whereby members 
93 
of the geriatric expert teams train the primary care staff and nurses to build capabilities in 
94 
comprehensive care of frail older adults (24, 25). Other care models have also found that 
95 
collaborators conduct comprehensive healthcare and needs screening to identify relevant health 
96 
and social needs, and in turn, follow up patients through home visits (26) or by directing them to 
97 
suitable care services (27). Most of the care models also include shared discussions about the 
98 
patients through the platform of multidisciplinary team meetings. In a review of 28 integrated care 
99 
programmes (28), provider commitment and trusting relationships were found to be foundational 
100 

to effective collaborations, communication and knowledge sharing among multidisciplinary 
101 
teams. Successful programme implementation was dependent on the quality of leadership and the 
102 
leaders’ efforts to instil a shared vision and to create an organisational culture that supports practice 
103 
changes and joint governance. 
104 
The purpose of this paper is to describe the protocol for a mixed method, multi-site 
105 
evaluation of the Geriatric Services Hub (GSH), a programme for frail older adults in Singapore, 
106 
to gain insights for improving the projects as they are being implemented and the effects of the 
107 
programme. The GSH is a novel intervention in the local context and comprises core components 
108 
of multicomponent frailty care programmes including CGAs and individually-tailored 
109 
multifactorial intervention delivered by a multidisciplinary team (29). The transferability of 
110 
implementation insights and outcomes from prior frailty programmes which are largely derived 
111 
from Western studies cannot be assumed, since the eventual results of implementation are 
112 
dependent on the context and culture that the health system is situated within. Current evidence on 
113 
the effectiveness of these interventions programme is moreover mixed (17, 22), with little insight 
114 
into whether the programme logic or the delivery process needs to be adjusted. Better 
115 
understanding of the conditions of implementation would support practitioners and policymakers 
116 
to consider how to ensure complex interventions achieve their intended impact. Hence, the 
117 
specificity of cultural context in the unique healthcare system of Singapore and the lack of 
118 
understanding of stakeholder perceptions warrant a comprehensive evaluation of the GSH model. 
119 
In the next segment, we will briefly describe the GSH programme and its position in Singapore’s 
120 
healthcare system.  
121 
Geriatric Services Hubs 
122 

The Singapore Healthcare System consists of three regional healthcare systems (RHS) in the 
123 
central, eastern, and western regions of Singapore. These RHS were established to coordinate and 
124 
organise healthcare service providers, integrate care across providers, and manage population 
125 
health for their respective regions. Hence, each RHS consists of a network led by a major public 
126 
hospital in collaboration with other healthcare providers such as primary care, day rehabilitation, 
127 
and community hospitals within the same geographical region (30). Each RHS is provided with 
128 
funding to implement programmes to deliver comprehensive care beyond the hospital to the 
129 
community. However, fragmentation in healthcare delivery continues to exist between and within 
130 
each RHS. There are other forms of commonly utilised care services that are privately owned and 
131 
not under the jurisdiction of the RHS, which include daycare, day rehabilitation centres, private 
132 
clinics and other variations of healthcare and allied health services. Therefore, although 
133 
geographically located within the same region, differences in governance and financing structures 
134 
between the RHS funded services and privately-owned facilities result in insufficient information 
135 
transfer, in capabilities and capacities between healthcare providers, and acute hospital-centricity.  
136 
Currently, as part of a national effort to engage and support older adults in Singapore, 
137 
community volunteers are screening for frailty amongst older adults. Community nurses have also 
138 
started to perform rapid and targeted geriatric and frailty assessments at neighbour-based nursing 
139 
posts (31). However,  the link between frailty assessments and the wider network of community 
140 
care providers needs to be strengthened. To address this gap, the Singapore Ministry of Health 
141 
provided funding to test the effectiveness of a new programme for frail adults aged 65 years old 
142 
and older – the GSH –, which will be piloted by five sites. While there is no national consensus on 
143 
the tools for identifying or grading frailty, the Ministry has specified the use of the Clinical Frailty 
144 
Scale (CFS) (32) to support the enrollment of older adults living with mild, moderate or severe 
145 

frailty into the GSH programme. This study protocol will focus on the evaluation of these five 
146 
pilot sites. Due to geographical locations, the five pilot sites fall under the jurisdiction of two 
147 
different RHS in Singapore. Each site focuses on its existing strengths and resources to ride on or 
148 
build new partners within their RHS to provide community-anchored referral-gated geriatric care. 
149 
Through these partnerships, the GSH functions as a network of providers led by a core team of 
150 
acute hospital-based healthcare professionals (33). Each site acts as a consolidation point in the 
151 
community by actively receiving referrals from various services providers. All GSH sites share 
152 
the following goals: 
153 
• To provide early identification of frailty and offering comprehensive and coordinated care 
154 
in the community through collaborative working arrangements with partners, including 
155 
polyclinics, general practitioners (GPs), and community health and social service providers 
156 
• To provide CGAs to identify needs of frail older adults and to establish a care plan for each 
157 
individual patient 
158 
• To increase frail older adults access to essential services and transit across shared primary 
159 
care and other community-based providers more seamlessly   
160 
• To provide core services of geriatric assessments, nursing support, therapy service and case 
161 
management/care coordination 
162 
• To build capabilities by providing training to the primary care staff in identifying, caring 
163 
and managing frail older adults.  
164 
Despite these common goals, each GSH was designed differently to best harness their 
165 
existing strengths and resources. As a result, each GSH site focuses on a different mode of 
166 
operation. Table 1 provides an overview of the different programmes based on their funded 
167 
components.  
168 

Table 1. Differences in care models 
169 
Study Site 
Programme Description 
Population 
Targeted 
Primary 
Referral Source 
Main 
Setting and 
delivery 
Programme 
Lead(s) 
Alexandra 
Hospital 
Geriatrician assesses older adults 
for frailty and manages patients 
in the primary care setting 
Age: 65 + 
CFS: 5-7 
General Practice 
(Public), social service 
agencies 
General 
Practice 
(Public) 
Geriatrician 
Changi 
General 
Hospital 
Geriatrician and community 
nurse supports primary care 
clinicians to assess and manage 
patients in the primary care 
setting 
Age: 65 + 
CFS: 5-7 
Hospital Emergency 
Department 
General 
Practice 
(Private, 
Public) 
Geriatrician 
Ng Teng 
Fong 
General 
Hospital 
Geriatrician builds competency of 
primary care staff to assess and 
manage patients in primary care 
and in the community 
Age: 65 + 
CFS: 5-7 
General Practice 
(Public), social service 
agencies 
General 
Practice 
(Public), 
social 
service 
agencies 
Geriatrician 
Singapore 
General 
Hospital 
Community nurses screen, assess 
and manage patients in the 
community supported by family 
physicians 
Age: 65 + 
CFS: 5-7 
Community Nurse Posts, 
social service agencies 
Community 
Nurse Posts 
Family 
Doctor and 
Nurse 
Sengkang 
General 
Hospital 
Geriatrician and multidisciplinary 
team supports primary care 
clinicians to assess and manage 
patients in the primary care 
setting 
Age: 65 + 
CFS: 5-7 
Includes 
patients 
with 
dementia 
General Practice 
(Private, Public), social 
service agencies, 
national senior care 
coordination agency 
 
General 
Practice 
(Private, 
Public), 
community 
geriatrics 
nursing and 
rehabilitation 
facilities 
Geriatrician 
Legend: CFS = Clinical Frailty Score 
170 
Although all five sites are designed and implemented differently, they share the common 
171 
goal of providing comprehensive healthcare services to frail older adults in Singapore.  Therefore, 
172 
this evaluation serves as a common platform to holistically assess the pilot GSH models using a 
173 
standardised process and outcomes framework but yet incorporating contextual information about 
174 
the implementation and care experience across the five sites. For complex interventions, it is 
175 
important to go beyond evaluating outcomes and to create a response loop to feed qualitative 
176 
insights back to the development and improvement of the care models.  
177 
Study Aims 
178 

Our purpose is to describe the evaluation protocol. Through this evaluation, we hope to gain 
179 
a comprehensive understanding of the factors influencing the effectiveness of the implementation 
180 
approaches adopted by the GSH programmes and their impacts. Our specific evaluation objectives 
181 
are:  
182 
1) To assess the development and implementation process of the new programme; 
183 
2) To assess the health status, quality of life, and user experience effects on care recipients 
184 
and the burden of care on primary caregivers; and 
185 
3) To determine the impact on healthcare utilisation and cost impact 
186 
Methods 
187 
Evaluation Design  
188 
The GSH is a complex intervention with multiple interacting components involving different 
189 
organisational partners providing various types of care to frail older persons living in the 
190 
community. We chose to adopt a mixed methods approach, relying on the principle of 
191 
complementarity, to achieve the specific study objectives (Table 2). The qualitative approach has 
192 
the advantage of supporting our understanding of circumstantial and programmatic factors that 
193 
influence the implementation outcomes (34). This allows the evaluation team to examine the 
194 
processes adopted to deliver frailty care in the community, the different configurations of the 
195 
intervention aimed at facilitating this, and perceived conditions that facilitated or impeded its 
196 
implementation. This will be complemented by quantitative approaches to measure the processes 
197 
and impacts stemming from the GSH.  
198 
The quantitative and qualitative components are equally weighted, designed, and analysed 
199 
(35). The Framework on Implementation Research developed for Client-Centred Medical Homes 
200 
(36) is used as a sensitising framework to structure the work phases of this evaluation, and for the 
201 

reporting of the results. In adopting this pragmatic approach, we recognise that there are singular, 
202 
as well as multiple realities, that can be empirically observed (37). Quantitative research 
203 
approaches are complemented with an in-depth exploration of contextual factors using a 
204 
qualitative approach (35, 38). Quantitative data allows for the assessment of processes and 
205 
multidimensional impact outcomes.  
206 
Table 2. Objectives and methods  
207 
Evaluation objectives 
Focus of inquiry 
Methods 
To assess the process of 
development and implementation 
of the new programme 
• 
To examine the perceptions, roles, 
responsibilities, and experience in 
developing and implementing each 
programme 
• 
To understand the workflow from 
the perspective of each profession 
for each of the new programmes 
• 
To improve competency of primary 
care providers in identifying, caring 
and managing geriatric patients in 
the community 
• 
To explore differences in 
contextual factors across the five 
sites that have influenced 
implementation experiences and 
outcomes 
• 
To understand care recipients’ 
experiences in receiving frailty care 
in the community 
Qualitative - Semi-structured 
in-depth interview with key 
policy and programme 
decision-makers 
 
Qualitative - Semi-structured 
focus group discussions with 
health and social care 
professionals and care 
recipients 
 
Qualitative – Participant 
observations 
 
Quantitative – Longitudinal 
monitoring of process 
indicators 
To assess the health, quality of life, 
and user experience effects on care 
recipients and the burden of care on 
primary caregivers 
• 
To compare outcomes between 
baseline, 3- and 6-months for each 
programme 
Quantitative - Pre-test post-test 
design using survey-based data 
collection 
To determine the impact on 
healthcare utilisation and cost 
impact of the new programme 
• 
To assess the use of healthcare 
services and costs between 
participants and non-participants 
for each programme 
Quantitative - Retrospective 
cohort design with propensity 
score matched comparators 
 
208 
Based on the components the sites are funded for (Table 1), we will evaluate the GSH 
209 
program conducted by Ng Teng Fong General Hospital (NTFGH) based on evaluation objective 1 
210 
– to assess the process of development and implementation of the new programme. The focus is 
211 
on competency building, with the funding being utilised to cover the time spent by trainers and 
212 
trainees. In addition, the funding will not be used to cover healthcare costs incurred by patients.  
213 

Based on these considerations, evaluation objectives 2 and 3 would not be representative of the 
214 
results of the program.  The remaining four sites will be evaluated on all three evaluation objectives 
215 
as their funding structures includes subsidising the healthcare costs of patients.  
216 
Qualitative Evaluation 
217 
Given the multi-agency and multi-professional setup of the new programme, it is important 
218 
to consider the competency building process and implementation experiences across organisations 
219 
and professional groups (39). We aim to elicit the perspectives of professional stakeholders playing 
220 
critical roles in the development and implementation of the new model of community-based frailty 
221 
care (Table 2). The conceptual framework for integrated care by Kodner and Kyriacou (40) guided 
222 
the qualitative inquiry. We aim to conduct interviews first to gain an understanding of the 
223 
programme details before moving on to the focus group discussion to understand the 
224 
implementation experiences, and finally carrying out participant observations to clarify and 
225 
substantiate our understanding of the processes of care. 
226 
First, we will conduct one-on-one semi-structured interviews with key decision-makers to 
227 
explore their perspectives and experiences (41). This will allow insights into the complex decisions 
228 
involved in the development of the programme and its implementation. An interview format will 
229 
allow the collection of information across a diverse range of opinions and experiences, and easier 
230 
expression of non-conformity.  
231 
Second, we will conduct focus group discussions with the core implementation team and 
232 
staff members from partner provider organisations. A focus group format was selected to bring 
233 
participants together to discuss and comment upon their personal experience in implementing the 
234 
new programme, and observe the process of collective sense-making (42).  
235 

Third, we will carry out participant observation to collect more detailed information about 
236 
each site’s culture and workflow to supplement the in-depth interviews and focus group 
237 
discussions. The observation will help us generate a more holistic understanding of the context 
238 
and operations (43) involved in the new programme and treatment processes including but not 
239 
limited to the conduct of the CGAs, development of the individualised care plan and execution, 
240 
observing team dynamics and multidisciplinary team meetings, and interaction with patient and 
241 
caregivers. 
242 
Quantitative Evaluation 
243 
The quantitative evaluation comprises three different components. First, process indicators 
244 
describe the key processes, which can be observed and documented to support the achievement of 
245 
desired outcomes. We will monitor quantitative process outcomes longitudinally on a quarterly 
246 
basis over the three-year pilot timeframe to assess the level of receptivity towards the new 
247 
programme, partner organisations’ ability to accurately screen for frailty, provision of personalised 
248 
and goal-oriented care, and care-continuity.   
249 
Second, a single cohort, pre-test post-test design (pre-experimental design) will be used to 
250 
quantify the impact of participation in the programme on care recipient-reported health outcomes, 
251 
experience of care, and caregiver’s burden. There will be no randomised parallel control group 
252 
due to programmes’ reluctance, as they expect high refusal rates from patients. In addition, the 
253 
new model encompasses different care components tailored to each patient’s needs, which renders 
254 
it impractical to conduct a ‘true’ experimental design with randomisation and a separate control 
255 
group (44). 
256 
Third, a retrospective pre-post, matched-groups design will be implemented to compare 
257 
healthcare utilisation and cost. Due to the absence of a parallel control group, we have opted for a 
258 

quasi-experimental design to determine the cost impact of introducing the new programme within 
259 
the current healthcare landscape.  
260 
Study Sample 
261 
Qualitative Evaluation  
262 
For semi-structured in-depth interviews, at least two participants representing each of the 
263 
five implementation sites and the Ministry of Health will be purposefully sampled. They will 
264 
include decision-makers who are higher-level administrators or clinicians who have the authority 
265 
to make policy and implementation decisions. It will also include individuals who are familiar with 
266 
the hospital’s overall frailty strategy, and those who led the development and implementation of 
267 
the programme.  
268 
We will conduct a series of focus group discussions where multiple stakeholders are 
269 
identified through purposive sampling. First, we will conduct focus group discussions with 
270 
members of the core implementation team at each of the five sites– defined as individuals with 
271 
time funded through the programme who have been providing services in the GSH for at least six 
272 
months. They may include doctors, nurses, allied health professionals, and administrative or 
273 
operations personnel. Second, for each site, at least five members of the staff from partner 
274 
organisations including primary care clinics, and aged care facilities will be identified for another 
275 
round of focus group discussions. They may include the heads of doctors, nurses and 
276 
administrative staff who provided services within the new programme for at least six months. 
277 
Third, we aim to identify at least ten participants who receive care under the new model for at least 
278 
three months for the last round of focus group discussions. They must be able to take part in 
279 
conversation for at least 60 minutes. For individuals who do not have the capacity to participate, 
280 
his/her primary caregiver is eligible for participation.  
281 

We will conduct participant observations to better understand the processes involved by 
282 
shadowing 3 to 5 members of the staff over the course of one week per site to account for day-to-
283 
day variations and job role differences. 
284 
Quantitative Evaluation 
285 
First, quality indicators will be developed to measure the processes of care for all individuals 
286 
enrolled into each programme. The relevant sub-populations accessing each care component will 
287 
form the denominator for computation of the indicators.  
288 
Second, we aim to recruit patients enrolled in the new programme. A target sample of 300 
289 
participants per site will be recruited. The sample was calculated based on the Barthel Index (100 
290 
points) – a scale to assess physical functioning, the key health outcome expected to improve as a 
291 
result of participation in the GSH. We computed the sample size for each participating site using 
292 
the dependent t-test to detect a small effect size of 0.2 (based on ß = .80, a = .05) (45). The result 
293 
was a minimum sample size of 156 per participating site. Allowing 20% rejection rate at the first 
294 
instance and a subsequent attrition rate of 30%, we would need to approach 300 individuals at the 
295 
first instance. 
296 
(Insert) Figure 1 Study sample for quantitative evaluation 
297 
Third, propensity score matching will be used to match programme enrollees and a 
298 
constructed comparator group. Both groups will be matched based on their probability of being 
299 
enrolled into the programme conditional on baseline covariates (46). The propensity score can be 
300 
used to mitigate selection bias in a non-randomised study. All programme enrollees will be 
301 
included as cases. Data on comparators will be derived from a Ministry of Health dataset 
302 
containing information on the health status of community-dwelling older adults that were collected 
303 

as part of the government’s efforts to communicate government policies to older adults and 
304 
connect them to relevant healthy ageing programmes.  
305 
Study Procedure 
306 
Qualitative Evaluation 
307 
Written informed consent will be obtained from all participants. They will first be invited to 
308 
complete a simple survey packet that consists of a basic demographic questionnaire and a scale to 
309 
assess their experiences in the programme. To obtain a thick description of the programme at each 
310 
site, possible explanations for why things are such, and to identify key strengths and challenges of 
311 
implementation at each site, interview guides were developed based on 12 out of 15 factors 
312 
identified by Kodner and Kyriacou (2000) to be integral to the development and implementation 
313 
of integrated care (40). We chose this framework as it provided sufficient components to evaluate 
314 
the different GSH site holistically. The framework also provides structure to evaluate similarities 
315 
and differences in the operations of the different GSH sites.  
316 
The 15 factors include patient screening, multidisciplinary assessment, primary care, 
317 
comprehensive service package, network relationships, care management, continuity and coverage 
318 
of care, seamlessness/ease of transitions, teamwork, information sharing, focus on continuum of 
319 
care, strategic planning, governance and management, funding mechanism and system outcomes. 
320 
Of the 15 factors, the research team deliberated and identified 12 that were key to understanding 
321 
the GSH implementation across the four sites. We renamed one of the factors “focus on continuum 
322 
of care” as “patient-centred care” to reflect work done by providers to align with patients’ needs 
323 
and preferences but at the same time to better differentiate this factor from “continuity of coverage 
324 
and care” or having control over transitions between services and providers. A brief description of 
325 
their relevance to the GSH is outlined in Table 3 below.  
326 

Table 3. Factors integral to understanding GSH implementation 
327 
Factors 
Brief description 
Patient screening 
Identification of frail older persons in the community.  
Multidisciplinary 
assessment  
Performing a comprehensive, multidimensional patient evaluation and developing a care 
plan to meet identified needs. 
Comprehensive 
service package 
Putting together a broad range of health and social care services to meet identified needs. 
Care management  
Planning of care, coordination of care and follow-up across time, place and discipline.  
Continuity of 
coverage and care 
Provider’s ability to help patients access the broad range of health and social care services 
across different settings and providers.  
Ease of transition  
Patient’s ability to access the broad range of health and social care services and to navigate 
between different settings and providers.  
Patient-centred care 
The extent to which clinicians and patients work together to make decisions and select 
tests, treatments and care plans based on evidence that balances risks and intended 
outcomes with patient preferences and values. 
Teamwork  
Roles and responsibilities of GSH core team members; ongoing communication and 
collaboration among the multidisciplinary group of providers. 
Network relationships 
Nature of working arrangements among and between institutions and providers, including 
information sharing. 
Strategic planning 
Stakeholder involvement in joint planning and community needs assessment 
System outcomes 
Overall responsibility for the intended outcomes. 
Funding mechanism 
Structure of funding for health and social care. 
 
328 
All interviews and focus group discussions will be audio-recorded and transcribed verbatim. 
329 
Each interview is expected to last 90 to 120 minutes, whereas each focus group discussions are 
330 
expected to last 120 minutes. For the focus group discussions, we aim to balance homogeneity 
331 
against the need for constructive tension by having separate sessions for the core implementation 
332 
team and community-based providers. Focus group discussions will not be conducted according 
333 
to professional groupings but each group will have similar frames of reference based on their job 
334 
roles. This will allow for group interaction but prevent the situation where participants may have 
335 
to defend their viewpoints (47).  
336 
For the participant observation component of this study, the research team will embed 
337 
themselves in the day-to-day operations of the programme environment and take extensive field 
338 
notes. Informal interviews will also be conducted to support our observations of the activities. The 
339 

observational components serve to explore professional practices in service implementation, 
340 
coordination, and collaborative interactions.  
341 
Quantitative Evaluation 
342 
Quantitative process indicators will be developed based on the logic model within inputs 
343 
from each of the five implementing sites to ensure agreement that they accurately describe the key 
344 
processes, which can be observed and described to support the achievement of desired outcomes. 
345 
Indicators are computed for each month and at quarterly intervals for each GSH site. Table 4 
346 
outlines the full list of requested process indicators.  
347 
Table 4. List and description of process indicators 
348 
Indicators 
Measure 
Definition 
Data 
collection 
time-points 
Patient recruitment 
 
 
Number of patients recruited by 
GSH sites 
Receptivity 
towards GSH 
No. of enrollees recruited into each 
GSH site after being referred  
Monthly 
Number, proportion of referred 
patients who fall within CFS 5-7 
Accurate 
identification of 
frailty 
CFS profiles of patients referred as 
scored by referral sources  
Monthly 
Patient-focused care 
management 
 
 
 
Number, proportion of CGA 
completed 
Personalised care 
No. of CGA completed vis-à-vis no. 
of assessments initiated  
Monthly 
Number, proportion of ICP 
developed 
Personalised, 
goal-oriented 
care 
Total no. of ICP developed vis-à-vis 
the no. of CGA completed  
Monthly 
Coordination of care 
 
 
Number of multidisciplinary 
rounds/discussions 
Team-based care 
No. of multidisciplinary team 
discussions conducted  
Monthly 
Number, proportion of referrals to 
services 
Efficiency in care 
continuity 
No. of referrals made to different 
services and the share of each service 
to the total no. of referrals 
Monthly 
Number, proportion of actualised 
referrals 
Care continuity 
No. of actualised first referrals at 
respective services vis-à-vis the no. 
of referrals made to each service  
Monthly 
Appointment waiting time to first 
appointment  
Efficiency in care 
continuity 
Waiting time for a first appointment 
to a referred service  
Monthly 
Competency building 
 
 
Number of community-based staff 
trained to conduct specific 
activities (CGA, exercise) 
Competency 
building 
No. of community-based healthcare 
participants in training sessions 
organised by the GSH1 
Quarterly 

Legend: CFS = Clinical Frailty Score; CGA = Comprehensive Geriatric Assessments; ICP = Individualised Care 
349 
Plans; GSH = Geriatric Service Hub 
350 
1Training sessions include preceptorship-based training and case discussions  
351 
 
352 
The evaluation aims to assess the impact of GSH. We hypothesised that  
353 
1. GSH sites offer a range of medical, social and other services through either direct provision 
354 
or referrals. Given that the model is intended to bridge service gaps (conduct of CGA in 
355 
the community), in the short-term, we hypothesis an increase in the utilisation of 
356 
appropriate services (rehabilitation, ambulatory services) in this time-limited programme.  
357 
2. GSH participants are expected to benefit from the comprehensive package of health and 
358 
social services and multidisciplinary team approach. With better care coordination and 
359 
improved access, it is likely to elicit a higher level of satisfaction relative to comparator 
360 
groups.  
361 
3. Education of the client about self-care and making decisions about potential care options 
362 
with inputs from a multidisciplinary team is expected to result in higher level of shared 
363 
decision making and engagement relative to comparator groups. 
364 
4. It is hypothesised to result in better functional status and health outcomes, might reduce 
365 
the healthcare utilisation (emergency hospitalisation, nursing home admission), 
366 
caregiver burden and the associated indirect cost. In turn, we might expect overall costs 
367 
to be lower compared to the comparison group. 
368 
An interviewer-administered survey will be conducted to collect data on demographic and 
369 
baseline health status, and data measuring the impact of the programme on patient activation, user 
370 
experience and satisfaction with care, health status, quality of life, the burden of care on primary 
371 
caregivers, and health utilisation and cost. The data are collected at baseline, 3-months and 6-
372 
months (Table 5).  After obtaining informed consent, participants will be asked to complete a 
373 
survey estimated to take about 45 to 90 minutes. Participants will complete the survey up to three 
374 

time-points – baseline (within 1 month of programme enrollment) and at 3-months and 6-months. 
375 
The primary caregiver will also be asked to complete a 5 to 10-minute survey at the same three 
376 
time points on caregiver burden. In the event a participant is clinically certified to have dementia, 
377 
we will allow a proxy respondent to answer on behalf of the main participant. Participants will 
378 
receive a token of appreciation for their involvement in the study.  
379 
Table 5. List of indicators for measuring programme impacts 
380 
Impact outcomes 
Assessment 
Measure 
Patient engagement 
Shared decision making 
collaboRATE For Patient – 5-
point anchor scale 
Patient’s experience of shared decision 
making  
Patient activation 
13-item Patient Activation 
Measure (PAM-13) 
Level of patient activation, including ability 
to self-manage, maintain functioning, 
collaborate with healthcare providers, and 
access healthcare services  
Healthcare experiences 
Experience of care delivered 
Consumer Assessment of 
Healthcare Providers and 
System Clinician & Group 
Survey Version 3.0 (CG-
CAHPS)  
Patients’ experience with healthcare 
providers and staff in doctors’ offices 
Health status, adverse outcomes and quality of life 
Functional status 
Barthel Index of Activities of 
Daily Living (ADL) 
Functional independence in ADL such as 
feeding, bathing, and continence 
Frequency of falls 
Count of falls 
Marker of poor health and declining 
function 
Health-related quality of life 
EuroOol-5D-5L  
Health-related quality of life in domains 
including mobility, self-care, usual 
activities, pain, anxiety and depression 
19-item Quality of Life Scale 
(CASP-19) 
Quality of life in later life in domains 
including control, autonomy, self-
realisation, and pleasure 
Impact on caregivers 
Level of caregiver burden 
Zarit Burden Interview (ZBI) 
Level of burden experienced by primary 
caregivers of older adults with dementia 
Direct and indirect cost 
Societal cost  
Client Service Receipt 
Inventory (CSRI) 
Health, social and informal care use and 
cost 
 
381 
To compare the healthcare utilisation and cost between programme enrollees and a 
382 
comparator group, an anonymised analytical dataset comprising existing information drawn from 
383 
datasets maintained by each of the five GSH sites, the evaluation team and the Ministry of Health 
384 

will be established. Given that all persons residing lawfully in Singapore are issued a unique 
385 
National Registration Identity Card (NRIC) number by the government, the NRIC for each 
386 
enrollee will be assigned a unique identifier by a third party. This unique identifier will 
387 
subsequently be used to merge data across the datasets. The anonymised data set will comprise 
388 
sociodemographic variables, all information collected from the above survey, and information on 
389 
the use and system cost of healthcare services (primary care, specialist care, emergency services, 
390 
inpatient care) across all public healthcare institutions in Singapore as well as mortality data.  
391 
Data Analysis 
392 
Qualitative evaluation 
393 
The Framework Analysis approach (48) will be used to analyse the data to generate 
394 
important categories and themes that encapsulate the elements that have influenced the 
395 
development and implementation of the new programme. Key steps outlined by Gale et al. (2013) 
396 
will be adopted (49). To begin, we will familiarise ourselves with the data by thoroughly reading 
397 
the transcript and listening back to the recorded interviews, if necessary. Field notes made during 
398 
and after the interviews will be read alongside the transcripts to ensure that the context was taken 
399 
into consideration. Based on the 12 factors outlined used to describe the model and process of care 
400 
at each of the five sites (Table 3), we will identify overarching categories where codes that are 
401 
conceptually related will be grouped.   
402 
A preliminary list of codes was derived from the literature (28, 50-54) as well as from our 
403 
initial first impressions. Initially, three members of the research team will independently code the 
404 
same two transcripts, allowing the codes to emerge inductively from the data in this open coding 
405 
process. Subsequently, the team will compare and discuss the codes, agree on a set of codes, assign 
406 
code labels, and provide each with a brief definition. This will form the working analytical 
407 

framework which will be applied to subsequent transcripts using NVivo (Release 1.0) (released in 
408 
March 2020). Once the data has been coded using the analytical framework, we will group codes 
409 
that are conceptually similar under the identified categories in a matrix. We will refine and create 
410 
new categories, if necessary. Key themes will be generated from the codes by reviewing the matrix 
411 
and making connections within and between categories. This process will be guided by the 
412 
research objectives and analytical framework as well as new concepts generated inductively from 
413 
the data. During the interpretation stage, we plan to take the analysis beyond describing the 
414 
implementation at each site towards developing themes to offer possible explanations for what was 
415 
happening across sites. The interpretation of findings will be discussed within the team and 
416 
presented to selected respondents from the implementation sites for member checking.  
417 
Quantitative evaluation  
418 
The process indicators will be computed for each month and at quarterly intervals monitor 
419 
the progress of the programme. Baseline characteristics will be described with mean and SD for 
420 
continuous variables, and number and percentage for categorical variables.  
421 
For survey-collected outcomes, Generalised Linear Models (GLM) will be used to allow us 
422 
to make inferences about the population when accounting for the within-subject correlation.  
423 
A propensity score conditional upon observed covariates will be computed for all cases 
424 
(Figure 1). Variables will include sociodemographic information, disease burden measured by the 
425 
Charlson Comorbidity Index (55), and physical functional status. We will match 2 comparators to 
426 
1 case using nearest-neighbour matching. In a Monte Carlo simulation, the mean squared error for 
427 
a 2:1 match was minimised in 84 percent of the simulations compared with 68 percent for a 1:1 
428 
match (56). Nearest neighbour matching based on a calliper of 0.01 of the standard deviation of 
429 
the propensity score will be used (46, 57). Only cases and comparators with propensity scores 
430 

falling within a common support region range will be included in the analysis to ensure 
431 
comparability of the two groups. 
432 
In the multivariable regression analysis of count data, Poisson distribution or Negative 
433 
Binomial distribution (variance greater than the mean) will be used. Negative Binomial regression 
434 
can be used for over-dispersed count data when the conditional variance exceeds the conditional 
435 
mean. GLM will be used for modelling non-normally distributed continuous data such as length 
436 
of stay and healthcare cost. The results will be presented as incidence-rate ratios (IRRs). All 
437 
analyses will be performed using Stata/SE 16.1(58), with the level of signiﬁcance set at 5%. 
438 
Integration of qualitative and quantitative data 
439 
Qualitative and quantitative data are collected concurrently and analysed separately. In the 
440 
final interpretation of the data, we will bring together and triangulate the results from the various 
441 
coordinated parts (59). Findings from each component of a study will be listed to allow the 
442 
evaluation team to look for convergence in findings, offer complementary information on the same 
443 
issues and to highlight discrepancies (60). Findings from interviews, focus group discussions and 
444 
participant observations will be used to contextualise quantitative process indicators and for 
445 
corroboration. Qualitative findings could offer explanations for the effectiveness of the GSH in 
446 
terms of the impacts on health status, quality of life, user experience, and healthcare resource use 
447 
and costs.  
448 
Discussion 
449 
The purpose of the GSH programme is to provide comprehensive and coordinated care for 
450 
frail older adults in the community, and the programme aims to achieve this through collaborative 
451 
working arrangements with various health and social service providers in the community. The 
452 
suite of services provided includes conducting CGA to identify the needs of older adults, and to 
453 

increase accessibility of frail older adults to essential services. Five GSH pilot sites share these 
454 
common goals but due to differences in their innate organisational operations, the design of the 
455 
GSH model was different to harness their existing relationships and organisational strengths.  
456 
The motivation to conduct a multiple methods evaluation is driven by the need to provide 
457 
good quality evidence on the organisation of community-based frailty care. Integration between 
458 
acute care and other forms of care in the community is becoming increasingly advocated (61). As 
459 
GSH is a model of care that aims to provide comprehensive care to frail older adults, it is essential 
460 
to gain rich insights into the development and implementation of the programme. The 
461 
understanding of underlying mechanisms will help us determine each programme’s effects on 
462 
patients’ health outcomes, programme cost-effectiveness, and the programme replicability. 
463 
Variations in the operations at each site demands that our evaluation methods be rigorous and 
464 
consistent across all sites. This will enable us to identify essential links in each model and 
465 
contribute more effectively to the planning of future models that provide comprehensive care.  
466 
Strengths and limitations 
467 
The study protocol has several strengths. Firstly, the multiple method approach allows in-
468 
depth understanding of the programme from the point of view of different stakeholders, as well as 
469 
the programme’s attributed health outcomes and costs impact. Secondly, this approach also 
470 
enables us to triangulate our findings, and cross compare the validity and reliability of our findings 
471 
through the different mediums of data collection. Finally, the choice of propensity score to match 
472 
patients will allow a close match in baseline characteristics between treatment and comparator 
473 
groups. This method attempts to replicate the balance in characteristics achieved by randomisation, 
474 
and thereby minimise selection bias. As a result, it contributes to the validity of conclusions to be 
475 
made on healthcare utilisation and cost outcomes.   
476 

The research protocol also has several limitations. Firstly, the data collection process will 
477 
only account for enrollees who agree to receive services delivered by the GSH programme. This 
478 
inadvertently excludes patients who seek alternative care in other care services in Singapore. As a 
479 
result, this exclusion might lead to an underestimation of the true treatment effect. Secondly, 
480 
although we can reduce selection bias through propensity score matching, there might be 
481 
unmeasured confounders that introduce hidden biases in our analyses.  
482 
In summary, the GSH aims to provide comprehensive health care services to frail older 
483 
adults in Singapore. They achieve this by partnering with health and social service providers in the 
484 
community and aim to increase accessibility and ease transition across different health services. 
485 
We believe that this evaluation study can provide valuable evidence of the impact and effectiveness 
486 
of the GSH. Lessons learnt from this study will be disseminated to programme implementers and 
487 
inform the design of similar programmes nationally.  
488 
 
489 
Declarations 
490 
Ethics approval and consent to participate 
491 
This protocol describes the proposed evaluation of the Geriatrics Services Hub programme. The 
492 
described study has received ethical approval from the NHG Domain Specific Review Board 
493 
(DSRB Ref: 2019/00925).  
494 
Consent for publication 
495 
Not applicable 
496 
Availability of data and materials 
497 
Not applicable 
498 
Competing interest 
499 

The authors declare no conflict of interest with respect to the research, authorship and/or 
500 
publication of this article.  
501 
Funding 
502 
This study was funded by the Ministry of Health, Singapore (HSDP Project Number 19X01). The 
503 
funder has played no role in the study design or preparation of the manuscript.  
504 
Authors’ Contribution 
505 
TWS conceived and designed the study, drafted and revised the article. NZL, THTR, SN, RC, 
506 
MLG, ET and PHJT designed the study, drafted and revised the article. DYY, WCH and LWS 
507 
conceived the study, obtained funding and revised the article. All authors gave final approval of 
508 
the manuscript to be published.  
509 
 
510 
Acknowledgements 
511 
We thank colleagues from participating programmes for supporting the discussions leading to the 
512 
design of this research protocol and in the implementation of the evaluation plan for the Geriatric 
513 
Services Hub.  
514 
 
515 
Members of the Geriatric Services Hub Programme Group 
516 
Santhosh Kumar Seetharaman, Healthy Ageing Programme, Alexandra Hospital, Singapore, 
517 
Singapore; Christopher Tsung Chien Lien, Department of Geriatric Medicine, Changi General 
518 
Hospital, Singapore, Singapore; Barbara Helen Rosario, Department of Geriatric Medicine, 
519 
Changi General Hospital, Singapore, Singapore; Shou Lin Low, Department of Geriatric 
520 
Medicine, Changi General Hospital, Singapore, Singapore; Arron Seng Hock Ang, Accident and 
521 
Emergency Department, Changi General Hospital, Singapore, Singapore; Mei Foon Yap, 
522 

Integrated Care, Changi General Hospital, Singapore, Singapore; Milawaty Nurjono, Health 
523 
Services Research, Changi General Hospital, Singapore, Singapore; Lydia Au, Geriatric Medicine, 
524 
Ng Teng Fong General Hospital, Singapore, Singapore; Lian Leng Low, Population Health and 
525 
Integrated Care Office, Singapore General Hospital, Singapore, Singapore; Su Fee Lim, RHS-
526 
Community Nursing (Population Health and Integrated Care Office) , Singapore General Hospital, 
527 
Singapore, Singapore; Esther Li Ping Lim, Population Health and Integrated Care Office, 
528 
Singapore General Hospital, Singapore, Singapore; Laura Bee Geok Tay, Department of General 
529 
Medicine, Sengkang General Hospital, Singapore, Singapore; Germaine Hwui San Chng, Strategic 
530 
Planning and Performance, Sengkang General Hospital, Singapore, Singapore; Melvin Peng Wei 
531 
Chua, Department of General Medicine, Sengkang General Hospital, Singapore, Singapore; Yee 
532 
Sien Ng, Rehabilitation Medicine, Sengkang General Hospital, Singapore, Singapore.  
533 
 
534 
 
535 

References  
536 
 
537 
1. Xue Q-L. The frailty syndrome: Definition and natural history. Clinics in Geriatric Medicine. 
538 
2011;27(1):1-15. 
539 
2. Vermeiren S, Vella-Azzopardi R, Beckwée D, Habbig AK, Scafoglieri A, Jansen B, et al. 
540 
Frailty and the Prediction of Negative Health Outcomes: A Meta-Analysis. . Journal of the 
541 
American Medical Directors Association. 2016;17(12):1163.e1–.e17. 
542 
3. Bandeen-Roche K, Seplaki CL, Huang J, Buta B, Kalyani RR, Varadhan R, et al. Frailty in 
543 
Older Adults: A Nationally Representative Profile in the United States. . The journals of 
544 
gerontology Series A, Biological sciences and medical sciences. 2015;70(11):1427–34. 
545 
4. Department of Statistics Singapore. Population trends 2019. In: Industry MoT, editor. 
546 
Republic of Singapore2019. 
547 
5. Malhotra R, Bautista MAC, Müller AM, Aw S, Koh GCH, Theng Y-L, et al. The Aging of a 
548 
Young Nation: Population Aging in Singapore. The Gerontologist. 2018;59(3):401-10. 
549 
6. Merchant RA, Chen MZ, Tan LWL, Lim MY, Ho HK, van Dam RM. Singapore Healthy 
550 
Older People Everyday (HOPE) Study: Prevalence of Frailty and Associated Factors in Older 
551 
Adults. J Am Med Dir Assoc. 2017;18(8):734.e9-.e14. 
552 
7. Wei K, Nyunt MSZ, Gao Q, Wee SL, Ng TP. Frailty and Malnutrition: Related and Distinct 
553 
Syndrome Prevalence and Association among Community-Dwelling Older Adults: Singapore 
554 
Longitudinal Ageing Studies. J Am Med Dir Assoc. 2017;18(12):1019-28. 
555 
8. Vaingankar JA, Chong SA, Abdin E, Picco L, Chua BY, Shafie S, et al. Prevalence of frailty 
556 
and its association with sociodemographic and clinical characteristics, and resource utilization 
557 
in a population of Singaporean older adults. Geriatr Gerontol Int. 2017;17(10):1444-54. 
558 
9. Lim WS, Wong CH, Ding YY, Rockwood K, Lien C. Translating the Science of Frailty in 
559 
Singapore: Results from the National Frailty Consensus Discussion. Annals of the Academy 
560 
of Medicine, Singapore. 2019;48(1):25-31. 
561 
10. Travers J, Romero-Ortuno R, Bailey J, Cooney MT. Delaying and reversing frailty: a 
562 
systematic review of primary care interventions. . The British Journal of General Practice: The 
563 
Journal of the Royal College of General Practitioners. 2019;69(678):e61-e9. 
564 
11. Lim WS, Wong SF, Leong I, Choo P, Pang WS. Forging a Frailty-Ready Healthcare System 
565 
to Meet Population Ageing. Int J Environ Res Public Health. 2017;14(12):1448. 
566 
12. Hopman P, de Bruin SR, Forjaz MJ, Rodriguez-Blazquez C, Tonnara G, Lemmens LC, et al. 
567 
Effectiveness of comprehensive care programs for patients with multiple chronic conditions 
568 
or frailty: A systematic literature review. . Health policy (Amsterdam, Netherlands). 
569 
2016;120(7):818–32. 
570 

13. Briggs AM, Valentijn PP, Thiyagarajan JA, Araujo de Carvalho I. Elements of integrated care 
571 
approaches for older people: a review of reviews. . BMJ Open. 2018;8(4):e021194. 
572 
14. Turner G, Clegg A. Best practice guidelines for the management of frailty: a British Geriatrics 
573 
Society, Age UK and Royal College of General Practitioners report. Age and ageing. 
574 
2014;43(6):744-7. 
575 
15. Clegg A, Bates C, Young J, Ryan R, Nichols L, Ann Teale E, et al. Development and 
576 
validation of an electronic frailty index using routine primary care electronic health record 
577 
data. Age and ageing. 2016;45(3):353-60. 
578 
16. Sadler E, Potterton V, Anderson R, Khadjesari Z, Sheehan K, Butt F, et al. Service user, carer 
579 
and provider perspectives on integrated care for older people with frailty, and factors 
580 
perceived to facilitate and hinder implementation: A systematic review and narrative 
581 
synthesis. PloS one. 2019;14(5):e0216488. 
582 
17. Puts M, Toubasi S, Andrew MK, Ashe MC, Ploeg J, Atkinson E, et al. Interventions to prevent 
583 
or reduce the level of frailty in community-dwelling older adults: a scoping review of the 
584 
literature and international policies. . Age and Ageing. 2017;46(3):383–92. 
585 
18. Di Pollina L, Guessous I, Petoud V, Combescure C, Buchs B, Schaller P, et al. Integrated care 
586 
at home reduces unnecessary hospitalizations of community-dwelling frail older adults: a 
587 
prospective controlled trial. BMC Geriatrics. 2017;17(1):53. 
588 
19. Cameron ID, Fairhall N, Langron C, Lockwood K, Monaghan N, Aggar C, et al. A 
589 
multifactorial interdisciplinary intervention reduces frailty in older people: randomized trial. 
590 
. BMC Medicine. 2013;11(65). 
591 
20. Baxter S, Johnson M, Chambers D, Sutton A, Goyder E, Booth A. The effects of integrated 
592 
care: a systematic review of UK and international evidence. . BMC Health Services Research. 
593 
2018;18(1):350-62. 
594 
21. Apóstolo J, Cooke R, Bobrowicz-Campos E, Santana S, Marcucci M, Cano A, et al. 
595 
Effectiveness of interventions to prevent pre-frailty and frailty progression in older adults: a 
596 
systematic review. . JBI Database of Systematic Reviews and Implementation Reports. 
597 
2018;16(1):140–232. 
598 
22. Frost R, Belk C, Jovicic A, Ricciardi F, Kharicha K, Gardner B, et al. Health promotion 
599 
interventions for community-dwelling older people with mild or pre-frailty: a systematic 
600 
review and meta-analysis. BMC geriatrics. 2017;17(1):157. 
601 
23. Berntsen G, Dalbakk M, Hurley JS, Bergmo T, Solbakken B, Spansvoll L, et al. Person-
602 
centred, integrated and pro-active care for multi-morbid elderly with advanced care needs: a 
603 
propensity score-matched controlled trial. BMC Health Services Research. 2019;19(682). 
604 
24. Hoogendijk EO, van der Horst HE, van de Ven PM, Twisk JW, Deeg DJ, Frijters DH, et al. 
605 
Effectiveness of a Geriatric Care Model for frail older adults in primary care: Results from a 
606 

stepped wedge cluster randomized trial. European Journal of Internal Medicine. 2016;28:43-
607 
51. 
608 
25. Ruikes FG, Zuidema SU, Akkermans RP, Assendelft WJ, Schers HJ, Koopmans RT. 
609 
Multicomponent program to reduce functional decline in frail elderly people: a cluster 
610 
controlled trial. . The Journal of the American Board of Family Medicine. 2016;29(2):209-17. 
611 
26. Uittenbroek RJ, Kremer H, Spoorenberg S, Reijneveld SA, Wynia K. Integrated Care for 
612 
Older Adults Improves Perceived Quality of Care: Results of a Randomized Controlled Trial 
613 
of Embrace. . Journal of General Internal Medicine. 2017;32(5):516-23. 
614 
27. Béland F, Hollander MJ. Integrated models of care delivery for the frail elderly: international 
615 
perspectives. . Gaceta Sanitaria. 2011;25(2):138-46. 
616 
28. Kirst M, Im J, Burns T, Baker GR, Goldhar J, O’Campo P, et al. What works in 
617 
implementation of integrated care programs for older adults with complex needs? A realist 
618 
review. International journal for quality in health care : journal of the International Society for 
619 
Quality in Health Care. 2017;29(5):612-24. 
620 
29. Walston J, Buta B, Xue Q-L. Frailty Screening and Interventions: Considerations for Clinical 
621 
Practice. Clin Geriatr Med. 2018;34(1):25-38. 
622 
30. Ministry of Health S. Healthcare 2020: Improving accessibility, quality and affordability 2012 
623 
[Available 
from: 
https://docplayer.net/12829195-Health-healthcare-2020-improving-
624 
accessibility-quality-affordability-healthcare-2020.html. 
625 
31. Chen CY, Gan P, How CH. Approach to frailty in the elderly in primary care and the 
626 
community. Singapore Med J. 2018;59(5):240-5. 
627 
32. Rockwood K, Song X, MacKnight C, Bergman H, Hogan DB, McDowell I, et al. A global 
628 
clinical measure of fitness and frailty in elderly people. CMAJ : Canadian Medical 
629 
Association journal = journal de l'Association medicale canadienne. 2005;173(5):489-95. 
630 
33. Kenis P, Provan KG. Towards an exogenous theory of public network performance. Public 
631 
Administration. 2009;87(3):440-56. 
632 
34. Astbury B, Leeuw FL. Unpacking black boxes: mechanisms and theory building in evaluation. 
633 
American journal of evaluation. 2010;31(3):363-81. 
634 
35. Creswell JW, Clark VLP. Designing and conducting mixed methods research: Sage; 2011. 
635 
36. Smith LR, Ashok M, Dy SM, Wines RC, Teixeira-Poit S. Contextual frameworks for research 
636 
on the implementation of complex system interventions. 2014. 
637 
37. Feilzer YM. Doing Mixed Methods Research Pragmatically: Implications for the Rediscovery 
638 
of Pragmatism as a Research Paradigm. Journal of Mixed Methods Research. 2009;4(1):6-16. 
639 

38. Johnson RB, Onwuegbuzie AJ. Mixed methods research: A research paradigm whose time 
640 
has come. Educational researcher. 2004;33(7):14-26. 
641 
39. Ochoa-Arias AE. An Interpretive-Systemic Framework for the Study of Community 
642 
Organizations. Systemic Practice and Action Research. 1998;11(5):543-62. 
643 
40. Kodner DL, Kyriacou CK. Fully integrated care for frail elderly: two American models. Int J 
644 
Integr Care. 2000;1:e08-e. 
645 
41. Johnson JM. In-depth interviewing. Handbook of interview research: Context and method. 
646 
2002;1. 
647 
42. Wilkinson S. Focus group methodology: a review. International Journal of Social Research 
648 
Methodology. 1998;1(3):181-203. 
649 
43. Kurz DE. The use of participant observation in evaluation research. Evaluation and Program 
650 
Planning. 1983;6(2):93-102. 
651 
44. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 
652 
1996;312(7040):1215-8. 
653 
45. Mayr S, Erdfelder E, Buchner A, Faul F. A short tutorial of GPower. Tutorials in quantitative 
654 
methods for psychology. 2007;3(2):51-9. 
655 
46. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies 
656 
for causal effects. Biometrika. 1983;70(1):41-55. 
657 
47. Grønkjær M, Curtis T, De Crespigny C, Delmar C. Analysing group interaction in focus group 
658 
research: Impact on content and the role of the moderator. Qualitative Studies. 2011;2. 
659 
48. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. The qualitative 
660 
researcher’s companion. 2002;573(2002):305-29. 
661 
49. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the 
662 
analysis of qualitative data in multi-disciplinary health research. BMC medical research 
663 
methodology. 2013;13(1):117. 
664 
50. Threapleton DE, Chung RY, Wong SYS, Wong E, Chau P, Woo J, et al. Integrated care for 
665 
older populations and its implementation facilitators and barriers: A rapid scoping review. 
666 
International journal for quality in health care : journal of the International Society for Quality 
667 
in Health Care. 2017;29(3):327-34. 
668 
51. Sadler E, Potterton V, Anderson R, Khadjesari Z, Sheehan K, Butt F, et al. Service user, carer 
669 
and provider perspectives on integrated care for older people with frailty, and factors 
670 
perceived to facilitate and hinder implementation: A systematic review and narrative 
671 
synthesis. PLoS One. 2019;14(5):e0216488. 
672 

52. Gwyther H, Shaw R, Jaime Dauden EA, D’Avanzo B, Kurpas D, Bujnowska-Fedak M, et al. 
673 
Understanding frailty: a qualitative study of European healthcare policy-makers’ approaches 
674 
to frailty screening and management. BMJ open. 2018;8(1):e018653. 
675 
53. Buist Y, Rijken M, Lemmens L, Baan C, de Bruin S. Collaborating on Early Detection of 
676 
Frailty; a Multifaceted Challenge. Int J Integr Care. 2019;19(2):6. 
677 
54. Shaw S, Rosen R, Rumbold B. What is integrated care. London: Nuffield Trust. 2011;7. 
678 
55. Charlson ME, Sax FL, MacKenzie CR, Fields SD, Braham RL, Douglas RG. Assessing illness 
679 
severity: does clinical judgment work? Journal of chronic diseases. 1986;39(6):439-52. 
680 
56. Austin PC. Statistical criteria for selecting the optimal number of untreated subjects matched 
681 
to each treated subject when using many-to-one matching on the propensity score. American 
682 
journal of epidemiology. 2010;172(9):1092-7. 
683 
57. Connolly JG, Gagne JJ. Comparison of Calipers for Matching on the Disease Risk Score. 
684 
American journal of epidemiology. 2016;183(10):937-48. 
685 
58. StataCorp. Stata Statistical Software: Release 16. College Station: TX: StataCorp LLC; 2019. 
686 
59. Zhang W, Creswell J. The use of “mixing” procedure of mixed methods in health services 
687 
research. Medical care. 2013;51(8):e51-7. 
688 
60. O’Cathain A, Murphy E, Nicholl J. Three techniques for integrating data in mixed methods 
689 
studies. Bmj. 2010;341:c4587. 
690 
61. WHO. WHO global strategy on people-centred and integrated health services. Interim report. 
691 
. Geneva: World Health Organization; 2015. 
692 
 
693