Development and Validation of an Experience Sampling Method Questionnaire (ESM) for Digital Monitoring of Mental State in Psychiatric Hospitalization: Convergence of ESM Reports, Standard Clinical Assessments, and Smartwatch Data
Conditions: Severe Mental Disorder, Schizophrenia, Personality Disorders, Mood Disorders, Bipolar Disorder, Depressive Disorder, Anxiety Disorders, Psychotic Episode, Schizoaffective Disorder
Sponsor: Sheba Medical Center

1
 
 
 
 
 
 
 
 
 
Development and Validation of an Experience Sampling Method 
Questionnaire (ESM) for Digital Monitoring of Mental State in 
Psychiatric Hospitalization: Convergence of ESM reports, 
Standard Clinical Assessments, and Smartwatch Data 
 
 
 
01.10.2023 

2
 
 
Abstract 
This study presents the development and validation of a unique Digital Experience Sampling 
Method (ESM) questionnaire specifically adapted for monitoring changes in the mental state 
of patients during psychiatric hospitalization. The questionnaire was carefully crafted through 
focus groups involving patients and clinical staff, ensuring its relevance and applicability to 
the unique characteristics of mental state changes in a hospitalization setting. 
To evaluate the validity of the ESM questionnaire, symptom severity trends obtained from 
the questionnaire will be compared with estimates derived from the Positive and Negative 
Syndrome Scale (PANSS) assessment. Data will be collected from 100 subjects over a 14-
day psychiatric hospitalization period. In addition to the ESM questionnaire, smartwatch 
sensors will monitor physiological indicators. 
Feasibility and patient compliance will be assessed by examining patients' willingness to use 
the digital ESM questionnaires and the smartwatch sensors. The study will also cross-
reference self-reported sleep quality and activity levels captured in the ESM questionnaires 
with objective physiological indicators and nursing staff reports, providing insights into the 
reliability of the patient-reported data. 
Furthermore, the study will evaluate the impact of the ESM data on clinical decision-making 
by physicians throughout the 14-day psychiatric hospitalization period. Patient satisfaction 
and satisfaction among the multidisciplinary team with the monitoring model will also be 
assessed. 
This research underscores the potential of digital technologies to enhance patient-centered 
care and facilitate informed treatment decisions in psychiatric hospitalization settings. 

3
 
 
Introduction 
Assessing the mental state of the patient hospitalized in a psychiatric ward is essential for 
providing optimal treatment during the hospitalization and determining the recommendations 
for continued treatment after discharge. This evaluation is mainly based on the assessment of 
the patient's subjective experience in a variety of aspects (such as mood, thoughts, feelings, 
and emotional pain) and the evaluation of his/her functioning (such as behavior, social 
functioning, mental functioning, level of daily activity, sleeping mechanisms, eating). 
Frequent documentation throughout the day of the patient's self-reports on his condition and 
function monitoring is rare in routine care in psychiatric inpatient wards, even though the 
patient's subjective perspective on his mental condition is an integral part of the psychiatric 
evaluation. 
The positive and negative syndrome scale (PANSS) for schizophrenia (Kay et al., 1987) is 
the "gold standard" medical scale for measuring the severity of symptoms in patients with 
schizophrenia spectrum disorders. It is conducted through a brief interview rather than a self-
report. Several valid assessment tools rely on the patient's self-report of his condition and 
focus on psychiatric symptoms (PROM - Patient-reported outcome measure), such as (SCL-
90) The Symptom Checklist 90 (Derogatis et al., 1973), and The Brief Symptom Inventory 
(BSI) (Derogatis & Melisaratos, 1983). Many tools focus on specific disorders, such as The 
Beck Depression Inventory (Beck et al., 1996), to assess depression. Other self-report 
questionnaires focus on the patient's quality of life, which is also essential for evaluating his 
psychiatric condition, such as MANSA - the Manchester Short Assessment of Quality of Life 
(Priebe et al., 1999). Despite the extensive use of self-report questionnaires in research and 
with outpatients, few valid tools focus on self-report of the mental state of the patient during 
his hospitalization in the psychiatric ward, such as the BASIS: The Behavior and Symptom 
Identification Scale (Eisen et al., 1986), focuses on both the presence of symptoms and the 
monitoring of function.  
The patient's report from memory about various symptoms and functional characteristics 
experienced in a period ranging from a week to a month is typical in these standard tools. 
These measurement tools are widely used for research purposes. Still, there is difficulty in 
using them in clinical work within the inpatient ward due to their length, difficulty in 
allocating personnel to deliver the questionnaire, note and analyze the results, and difficulty 
in using the collected data as a tool available for the work of the staff in the ward. Due to 

4
 
 
these difficulties, few studies use PROMS questionnaires among hospitalized psychiatric 
patients. With the advancement of technology and the prevalence of smartphones, it became 
possible to use digital questionnaires to monitor mental states. Wong et al. (2022) described 
the use of digital PROMS questionnaires to characterize the symptoms and the improvement 
in the mental state throughout psychiatric hospitalization; this follow-up was carried out at 
the time of admission to the hospital and at the time of discharge when the standard 
questionnaires demonstrated an improvement in the mental state of the hospitalized patients. 
These findings indicated the importance of using self-report questionnaires in the treatment 
routine throughout psychiatric hospitalization to provide more accurate treatment to 
hospitalized patients. 
There are inherent disadvantages to performing a clinical assessment using standardized 
questionnaires. Using the patient's memory to record symptoms is prone to biases due to a 
tendency to remember complex events more strongly and difficulty recalling moments when 
powerful symptoms were not experienced, or there were no symptoms. In addition, there is 
difficulty in examining slight changes in the patient's condition, which are often below the 
sensitivity threshold of the standard tools (Shiffman et al., 2008; Csikszentmihalyi & Larson, 
2014). 
In the field of psychopathology, there is a growing awareness that the models that describe 
the psychological experience are dynamic, change over time, and their understanding requires 
multiple samples and sometimes continuous monitoring while paying attention to the 
microenvironmental factors that surround the person and influence his condition (Myin-
Germeys et al., 2009; Myin-Germeys et al., 2018). 
Momentary Assessment Strategies  
Momentary assessment strategies use repeated real-time sampling of subjects' behaviors and 
experiences. The sample can be discrete (e.g., by self-reporting about subjective aspects of 
the experience at a given moment), continuous (e.g., by continuous monitoring of 
physiological indicators using sensors), or combined (Varese et al., 2019). The strategies 
often used are the Experience Sampling Method-ESM (Delespaul, 1995; Hektner et al., 2007) 
and EMA - Ecological Momentary Assessment (Stone & Shiffman, 1994; Shiffman et al., 
2008); both methods will be referred to below as ESM. The ESM is a method Structured 
based on a diary, in which the subjects are asked to report their thoughts, feelings, and 
symptoms during their daily life, as well as to describe the context in which the report takes 

5
 
 
place, for example, location, social situation, and activity. They can also assess their 
satisfaction with the current situation that was sampled. ESM is a strategy with significant 
advantages in research in psychiatry and somatic problems and is the basis for studies in 
optimizing clinical decisions in these fields (Verhagen et al., 2016). With the development of 
technology and the frequent use of smartphones, the collection of samples, the analysis of 
their characteristics, and the use of sensors on mobile phones (Ben-Zeev et al., 2016) became 
simple and more available for conducting ESM studies. It was also found that the patient's 
response to electronic diary management is significantly better than using paper and pencil 
for running a diary (Stone et al., 2003). To assess the validity of the ESM questionnaire for 
psychiatric symptoms, it is customary to compare the results obtained from standard 
measures such as the PANSS assessment, widely recognized as the gold standard for 
evaluating psychiatric symptoms, with the observed changes in symptom evaluation using the 
ESM questionnaires (Harvey et al., 2021). 
Use of ESM methodology during psychiatric inpatient stays 
Studies using ESM methodology for ambulatory patients used diverse ESM questionnaires 
(Csikszentmihalyi & Larson, 2014; Hare et al., 2015; Chen et al., 2016a, 2016b; Wilhelm & 
Schoebi, 2007). However, those questionnaires are not useful in examining patients' 
experience in the unique situation of psychiatric hospitalization. For example, questions 
inquire regarding the description of the current situation, such as, "Where are you?" "Who are 
you with?" (Csikszentmihalyi & Larson, 2014), which is often unsuitable to ask in a hospital. 
In addition, it is necessary to refer to the fact that hospitalized patients are in an acute phase 
of their illness, and their condition should be assessed accordingly. The use of ESM models 
during psychiatric hospitalization has been tested and found to be feasible and valid (Kimhy 
et al., 2006; So et al., 2013). Studies have examined momentary changes in the mental state 
of hospitalized psychiatric patients (Houben et al., 2017; Hallensleben et al., 2029;; Lucht et 
al., 2022; Brüdern et al., 2022). However, we did not find a study examining the use of the 
ESM model to support the clinical decisions of the treating staff in the psychiatric ward. 
Use of wearable sensors and self-report of symptoms among psychiatric 
patients 
In a systematic review of 35 studies that use data collected from the mobile phone and 
information collected through wearable sensors, Seppälä et al. (2019) found an association 
between data collected by sensors and symptoms of schizophrenia, bipolar disorders, and 

6
 
 
depression. However, the use of such data in clinical settings to support therapeutic 
interventions is not yet been thoroughly evaluated. 
Lahti and his associates (2021) found that there is a possibility to identify an association 
between data collected by sensors, to the worsening of the mental state of patients diagnosed 
with schizophrenia. These illustrate the need for further research in the unique population of 
psychiatric patients and the lack of research information concerning hospitalized psychiatric 
patients during the acute phase of their illness. 
Zarbo and colleagues (2022) assessed adherence to and usability of the Experience Sampling 
Method (ESM) and actigraph in patients with a schizophrenia spectrum disorder. They found 
differences in compliance between the patients residing in psychiatric rehabilitation housing 
in the community and healthy controls. They reported that the presence of a multi-
disciplinary team is critical to ensure patients' responsiveness to using wearable sensors and 
answering ESM questionnaires. A complete psychiatric hospitalization setting constitutes an 
environment adapted to patients' compliance for monitoring. Zarbo et al. (2022) also pointed 
out that subjects monitored through ESM questionnaires and actigraphs reported that their 
involvement in the monitoring process improved their mental well-being through 
understanding their feelings, attention to daily routines, and rewarding occupation in utilizing 
their time. Participation in the monitoring procedure affects the patient's experience; this 
function deserves further research. 
Research Objectives 
1. General purpose 
The present study aims to develop an Experience Sampling Method (ESM) 
questionnaire tailored specifically for the distinctive context of psychiatric 
hospitalization; an instrument not currently employed in the clinical field. The study 
will outline the various stages involved in questionnaire development. The primary 
objective of this study is to validate the ESM questionnaire by comparing the trends in 
symptom severity as reported through the ESM questionnaire with the PANSS 
assessment of symptom severity during a 14-day psychiatric hospitalization period. 
2. Specific Aims: 
1. Development of an ESM questionnaire specifically tailored to the unique 
circumstances psychiatric patients face during their hospitalization. 

7
 
 
2. Investigation of the feasibility and compliance of patients in participating in the study, 
including their willingness to complete digital ESM questionnaires three times a day 
and wear the smartwatch sensor throughout their hospitalization. Additionally, we will 
explore potential differences in adherence and compliance among subjects with 
different psychiatric diagnoses regarding their willingness to participate in the study, 
complete self-report ESM questionnaires, and wear the smartwatch. 
3. Examination of the validity of the ESM questionnaire in predicting the trends of 
change in symptom severity among psychiatric patients during a 14-day psychiatric 
hospitalization period. This validity will be evaluated by comparing the trends in 
symptom severity as measured by the ESM questionnaire with the trends observed in 
four PANSS assessments conducted throughout the hospitalization. 
4. The validity of subjects' self-reports on sleep quality and activity level will be assessed 
by comparing the self-report ESM data with the physiological indicators collected by 
the smartwatch. Additionally, the ESM indices will be compared to reports from the 
nursing staff on the ward. Furthermore, we will explore whether these comparisons 
yield different outcomes among subjects with different psychiatric diagnoses. 
5. The effectiveness of the monitoring model will be assessed by examining the impact of 
exposing the psychiatrists to ESM data, as well as the physiological indicators 
collected by the smartwatch, on the decision-making process regarding changes in the 
patient's medication plan during psychiatric hospitalization. 
6. Patient satisfaction: 
Assessing patient satisfaction with the monitoring model used throughout their 14-day 
hospitalization. Following the monitoring period, subjects will be asked to provide 
feedback on their satisfaction levels through questionnaires. 
7. The satisfaction of the multi-disciplinary teams (physicians, nursing staff, clinical 
psychologists, social workers, occupational therapists):  
At the end of the study, the teams will report through questionnaires on their 
satisfaction with using this monitoring model and its influence on the team's clinical 
decision-making. 
Description of the research process 
The research will be carried out in two phases: 
1. Development of the monitoring questionnaire - Content validity phase - This 
phase has been fully completed, and its results will be presented in the preliminary 

8
 
 
results chapter. At this research stage, the core content domains of the monitoring 
questionnaire will be developed. The research process in this phase includes the 
following: 
1. Examining the content of scales from the Positive and Negative Syndrome Scale 
(PANSS) for schizophrenia and The Symptom Checklist-90-R (SCL-90-R) 
questionnaire: At this stage, the psychiatric symptoms, and the main criteria for 
assessing a mental state as emerged from those questionnaires were examined for 
discussion in the focus groups. 
2. Focus group No. 1: The first focus group included patients hospitalized in the 
psychiatric ward. The purpose of the focus group is to formulate the ESM core 
and additional content domains and the questionnaire items and to discuss the 
optimal measurement frequency according to the hospitalized patients' opinions. 
The focus group will also examine the wording that best suits how the patients 
describe their mental state. 
Participants 
The focus group recruited nine subjects over age 18 (6 women and three men) 
hospitalized by consent in the psychiatric wards at the Sheba Medical Center, who 
signed informed consent to participate in the study. The criteria for exclusion from 
the study were: patients with a low level of compliance to treatment, patients 
suffering from a significant active physical illness, intellectual disability, or 
complex organic conditions including dementia conditions, patients who cannot 
read and write in the Hebrew language, patients who are not qualified to give 
informed consent. The participants were given a refund of expenses for canceling 
the time involved in participating in the focus group using shopping vouchers of 
100 NIS. 
Indices 
The subjects' point of view regarding the psychiatric symptoms they suffered from 
during their hospitalization was examined during a free discussion. Emphasis was 
placed on the symptoms that change daily throughout the hospitalization and how 
much they indicate a change in their mental state. We also investigated what 
information they would like to report to the treatment team about their mental 
state throughout the hospitalization. After the free discussion, the position of the 

9
 
 
subjects was examined regarding the criteria for assessing mental state as they 
emerged from the monitoring scales of the PROM questionnaires. 
After a qualitative analysis of the information collected at this stage, the main 
themes of the ESM questionnaire were formulated. According to the themes 
found, suggestions were developed for the questionnaire items while noting how 
the patient's symptoms were worded. A professional transcriber transcribed the 
discussion. 
3. Focus group No. 2: The second focus group includes a multi-disciplinary team 
working in the psychiatric inpatient wards. In a free discussion, the multi- 
disciplinary team members examined the core issues that should appear in the 
questionnaire for monitoring the mental state of hospitalized patients, then 
explored and expressed their opinion regarding the issues raised by the patients in 
the first focus group. The result of this step will be an ESM questionnaire adapted 
to the needs of the patients and the clinical staff in the inpatient wards. 
Participants 
At this stage, seven staff members working in the psychiatric hospitalization 
wards in Sheba Medical Center (3 women and four men), who signed informed 
consent to participate in the study, participated in the focus group. Inclusion 
criteria: Staff members work in the inpatient wards and have a valid Israeli 
license. The group was attended by two physicians in the psychiatry internship 
stage, two nursing staff members, an occupational therapist, a social worker, and a 
clinical psychologist. 
Indices 
In the discussion, the participants floated important themes for monitoring 
according to their points of view. After a free discussion, the participants were 
presented with the themes raised by the patients in the previous focus group. They 
were also presented with the possible questionnaire items formulated after the last 
stage, and their position was examined regarding the appropriateness of fitting 
these items to the questionnaire. In addition, the discussion dealt with the optimal 
sampling frequency, in their opinion. 
The initial formulation of the ESM questionnaire items. 

10
 
 
The questionnaire items were formulated considering the information collected in 
the focus groups. An effort was made to use the patients' words to ask about their 
mental state. In addition, existing and valid self-report items marked as necessary 
by the patients and staff in the focus groups discussion were selected. Items from 
these questionnaires were reformulated according to the ESM approach that 
examines the current state. At the end of this stage, an ESM questionnaire was 
developed, which consists of 15 items on a five levels Likert scale. In the 
questionnaire, there was another item of a Visual Analogue Scale type where the 
subject had to mark his mood on a scale ranging from 1-100. 
4. Focus group No.3: The third focus group included hospitalized inpatients for 
cognitive debriefing - Cognitive debriefing is the process by which a patient 
questionnaire is actively tested among representatives of the target population and 
target language group to determine whether respondents understand the 
questionnaire as the authors intended it to be understood.  
Participants 
The focus group recruited six subjects over 18 (two women and four men) 
hospitalized with consent in the psychiatric unit at the Sheba Medical Center, who 
signed an informed consent to participate in the study. The criteria for exclusion 
from the study were: patients with a low level of compliance to treatment, patients 
suffering from a significant active physical illness, developmental, intellectual 
disability, or complex organic conditions including dementia, patients who cannot 
read and write in the Hebrew language, patients who are not qualified to give 
informed consent. The participants were given a refund of expenses for canceling 
the time involved in participating in the focus group using shopping vouchers of 
100 NIS. 
Indices 
For each item in the developed ESM questionnaire, subjects were asked to 
describe freely what they thought the item was testing. In addition, to each item, 
we discussed how appropriate the question was in their opinion to describe their 
mental state throughout the hospitalization and whether they would change 
anything in the wording. At the end of this part of the discussion, the subjects 
were asked how many times a day they would be willing to report their mental 

11
 
 
state, how many items they would be ready to answer, and what additional 
questions they would like to be asked to learn about their condition. In addition, 
we examined the patients' attitudes regarding their agreement to answer questions 
via mobile phone and their agreement to continue answering a similar 
questionnaire after their discharge from the hospital. 
Research Process 
The focus groups were held at Sheba Medical Center on 5/18/22, 08/9/22, and 
11/09/22. The groups were guided by the researcher and another facilitator, who 
work as senior clinical psychologists in the psychiatric inpatient wards at the 
Sheba Medical Center. A professional transcriber attended each group and typed 
the conversation throughout the focus groups. The transcripts of the discussions in 
the focus groups were qualitatively analyzed. After collecting and processing the 
data from the focus group, the ESM questionnaire was improved and refined, 
considering the findings. 
2. Running the monitoring model - Feasibility and Validity phase - this phase will 
begin in the middle of 2023 and is expected to continue for 18 months. 
Participants 
To conduct the monitoring model, we will recruit 100 subjects hospitalized in the 
inpatient wards. To receive feedback on implementing the monitoring model from 
the multi-disciplinary care team working in the wards, we will recruit 30 multi-
disciplinary team members. The running will be carried out in two stages: 
•  Pre-Test - 20 patients and feedback from 5 multi-disciplinary – estimated time – 3 
months. 
•  Field Test - 80 additional patients and feedback from 30 multi-disciplinary staff 
members – estimated 12 months. 
The study inclusion criteria for the patients: adults are consenting hospitalized in a 
psychiatric hospital at the Sheba Medical Center in the open part of the ward who 
owns a mobile smartphone. Exclusion criteria: patients with a low level of 
treatment compliance, patients with a developmental or intellectual disability, 
complex organic conditions including dementia, patients who cannot read and 
write in the Hebrew language, and patients who are incapable of informed 
consent. 

12
 
 
The study inclusion criteria for the multi-disciplinary team: multi-disciplinary 
team members who work in the psychiatric inpatient wards at the Sheba Medical 
Center. The staff members will have a valid Israeli license (from the professions 
of medicine, nursing, psychology, social work, and occupational therapy). 
Indices 
Pre-test indices –  
1. The process involves selecting ESM questionnaire items based on their 
distribution indices to ensure optimal item distribution. It also includes refining 
the wording of items with low variability scores within and between subjects, 
aiming to enhance their variability.  
2. We will evaluate the satisfaction indicators of the clinical staff working in the 
ward based on the monitoring indicators. If needed, we will enhance the 
implementation of the monitoring model within the clinical team in the ward. 
Field test indices – 
3. The validation of the ESM questionnaire will be conducted by comparing the 
change trends in the questionnaire indices during a 14-day psychiatric 
hospitalization period to the change trends in four different measurements of 
psychiatric symptoms assessed by the PANSS diagnostic tool, which is widely 
recognized as the gold standard in psychiatric symptom assessment. The 
subject's assessment using PANSS will be conducted at four different time 
points: 
a. T1 - at study entry (day 1) 
b. T2 - during the first week of the study (between days 4-6) 
c. T3 - during the second week of the study (between days 10-12) 
d. T4 - at the end of the study (day 14) 
4. Feasibility and compliance: 
a. The responsiveness to participation in the monitoring model - The 
number of subjects who agreed to participate in the study out of the 
subjects we offered to participate. 
b. The extent of responding to the self-report questionnaires over 14 days 
multiplied by three times a day. 
c. The degree of responsiveness to wearing and charging the smartwatch 
for 14 days. 

13
 
 
d. The degree of difference between subjects with different psychiatric 
diagnoses (psychotic spectrum, depression, mania, and personality 
disorders) in their adherence and responsiveness to agree to participate 
in the study, answer the self-report ESM questionnaires and wear and 
charge the smartwatch. 
5. The validity of the subject's ESM self-reports on sleep quality and activity 
level:  
a. The validity of the subject's self-reports on sleep quality -The subject's 
self-reported data about his sleep quality and number of awakenings 
will be compared to physiological indicators that the smartwatch will 
collect (number of hours of sleep, number of awakenings) and will be 
equalized in addition to the medical reports of the nursing staff in the 
ward. 
b. The validity of the subject's self-reports on his activity level:  
The subject's self-reported data about his energy and activity level will 
be compared to physiological indicators that the smartwatch will collect 
(number of steps, accelerations) and will be equalized in addition to the 
reports of the nursing staff in the ward. 
c. The degree of difference between subjects with different psychiatric 
diagnoses (psychotic spectrum, depression, mania, and personality 
disorders) in correspondence between the self-report and the 
physiological indices and the reports of the nursing staff. 
6. The effect of exposing the psychiatrists to the patient's monitoring indicators on 
their clinical decisions regarding him will be examined in a short questionnaire 
that the physician will fill out after each medication change and mark on a 
Likert scale of 1 5 - how much the exposure to the monitoring indicators 
influenced the clinical decision. 
7. Patient satisfaction: After 14 days of monitoring, the subjects will report their 
satisfaction with using the monitoring model throughout the hospitalization 
through questionnaires. 
8. The satisfaction of the multi-disciplinary teams: At the end of the study, the 
teams will report through questionnaires on their satisfaction with using this 
monitoring model and its influence on the team's clinical decision-making. 

14
 
 
Research Process 
After recruiting the patient for the study and signing informed consent, the 
monitoring application (Datos Health) and the wearable smartwatch application 
(Garmin) will be installed on the patient's smartphone. The patient will receive an 
extensive explanation from the research team's representative on the self-reporting 
method and will experience using the application. 
From this stage, the hospitalized subject will receive ongoing clinical care, 
alongside participation in monitoring indicators by self-reporting in the ESM 
questionnaires using the Datos Health application and physiological indices 
monitoring by using the Garmin smartwatch. The trial will end after 14 days or 
until the patient's discharge (whichever comes first). The ESM self-report 
sampling will be conducted a frequency of 3 times a day in a semi-random 
manner: 
• The first sample will be randomly sampled between 8-10 am. 
• A second sample will be randomly sampled between 1-3 pm. 
• A third sample will be randomly sampled between 7-10 pm. 
In each clinical discussion of the subject's mental state, the ward staff will watch the 
dashboard of the Datos Health application, which will display the subject's self-report 
measures and physiological indicators. For each change in the patient's clinical status, 
determined by the physician (change of medication, vacation, preparation for 
discharge, discharge), the physician will answer a questionnaire in which he will 
document how much (between 1-5) the clinical decision was influenced by the 
monitoring indicators he observed. 
The feasibility and compliance, Validity, and changes in the patient's condition over 
timeline indicators will be collected for 14 days and analyzed at the study's end. The 
patient satisfaction indicators will be tested after 14 days of monitoring through a 
questionnaire sent to each subject. The satisfaction indicators of the clinical staff in 
the ward will be tested through questionnaires sent to the clinical staff at the end of 
the study. 
Place of conducting the research 
The research will be carried out at the psychiatric inpatient wards A, B, and C in the 
psychiatric division at the Sheba Medical Center. 

15
 
 
References 
1. Attride-Stirling, 
J. 
(2001). 
Thematic 
networks: 
an 
analytic 
tool 
for 
qualitative 
research. Qualitative research, 1(3), 385-405. 
2. Beck, A. T., Steer, R. A., & Brown, G. (1996). Beck depression inventory–II. Psychological 
Assessment. 
3. Brüdern, J., Hallensleben, N., Höller, I., Spangenberg, L., Forkmann, T., Rath, D., ... & 
Glaesmer, H. (2022). Sleep disturbances predict active suicidal ideation the next day: an 
ecological momentary assessment study. BMC psychiatry, 22(1), 1-7. 
4. Carter, L. A., & Emsley, R. (2019). The analysis of experience sampling data. Experience 
Sampling in Mental Health Research. Oxon, UK: Routledge. 
5. Csikszentmihalyi, M., & Larson, R. (2014). Validity and reliability of the experience-sampling 
method. Flow and the foundations of positive psychology (pp. 35-54). Springer, Dordrecht. 
6. Delespaul, P. A. E. G. (1995). We are assessing schizophrenia in daily life. Universitaire Pers 
Maastricht, Maastricht. 
7. Delespaul, P., devries, M., & van Os, J. (2002). Determinants of occurrence and recovery 
from hallucinations in daily life. Social psychiatry and psychiatric epidemiology, 37(3), 97-104. 
8. Derogatis, L. R., Lipman, R. S., & Covi, L. (1973). SCL-90: an outpatient psychiatric rating 
scale–preliminary report. Psychopharmacol Bull, 9(1), 13-28. 
9. Derogatis, L. R., & Melisaratos, N. (1983). The brief symptom inventory: an introductory 
report. Psychological medicine, 13(3), 595-605. 
10. Eisen, S. V., Grob, M. C., & Klein, A. A. (1986). BASIS: the development of a self-report 
measure for psychiatric inpatient evaluation. Psychiatric hospital. 
11. Guest, G., Namey, E., & McKenna, K. (2017). How many focus groups are enough? Building 
an evidence base for nonprobability sample sizes. Field methods, 29(1), 3-22. 
12. Hallensleben, N., Glaesmer, H., Forkmann, T., Rath, D., Strauss, M., Kersting, A., & 
Spangenberg, L. (2019). Predicting suicidal ideation by interpersonal variables, hopelessness 
and depression in real-time. An ecological momentary assessment study in psychiatric 
inpatients with depression. European Psychiatry, 56(1), 43-50. 
13. Harvey, P. D., Miller, M. L., Moore, R. C., Depp, C. A., Parrish, E. M., & Pinkham, A. E. 
(2021). Capturing clinical symptoms with ecological momentary assessment: convergence of 
momentary reports of psychotic and mood symptoms with diagnoses and standard clinical 
assessments. Innovations in clinical neuroscience, 18(1-3), 24. 
14. Hektner, J. M., Schmidt, J. A., & Csikszentmihalyi, M. (2007). Experience sampling method: 
Measuring the quality of everyday life. Sage. 
15. Houben, M., Claes, L., Vansteelandt, K., Berens, A., Sleuwaegen, E., & Kuppens, P. (2017). 
The emotion regulation function of nonsuicidal self-injury: A momentary assessment study in 
inpatients 
with 
borderline 
personality 
disorder 
features. Journal 
of 
Abnormal 
Psychology, 126(1), 89. 
16. Lahti, A. C., Wang, D., Pei, H., Baker, S., & Narayan, V. A. (2021). Clinical utility of wearable 
sensors and patient-reported surveys in patients with schizophrenia: Noninterventional, 
observational study. JMIR mental health, 8(8), e26234. 
17. Lucht, L., Spangenberg, L., Forkmann, T., Hallensleben, N., Rath, D., Strauss, M., & 
Glaesmer, H. (2022). Association of real-time assessed mood, affect and suicidal ideation in 
psychiatric inpatients with unipolar depression. Clinical psychology & psychotherapy, 29(5), 
1580-1586. 
18. Myin-Germeys, I., Oorschot, M., Collip, D., Lataster, J., Delespaul, P., & Van Os, J. (2009). 
Experience sampling research in psychopathology: opening the black box of daily 
life. Psychological medicine, 39(9), 1533-1547. 
19. Kay, S. R., Fiszbein, A., & Opler, L. A. (1987). The positive and negative syndrome scale 
(PANSS) for schizophrenia. Schizophrenia bulletin, 13(2), 261-276. 
20. Kimhy, D., Delespaul, P., Corcoran, C., Ahn, H., Yale, S., & Malaspina, D. (2006). 
Computerized experience sampling method (ESMc): assessing feasibility and validity among 
individuals with schizophrenia. Journal of psychiatric research, 40(3), 221-230. 
21. Seppälä, J., De Vita, I., Jämsä, T., Miettunen, J., Isohanni, M., Rubinstein, K., ... & 
Bulgheroni, M. (2019). Mobile phone and wearable sensor-based mHealth approaches for 
psychiatric disorders and symptoms: systematic review. JMIR mental health, 6(2), e9819. 
22. Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annu. 
Rev. Clin. Psychol., 4, 1-32. 

16
 
 
23. So, S. H. W., Peters, E. R., Swendsen, J., Garety, P. A., & Kapur, S. (2013). Detecting 
improvements 
in 
acute 
psychotic 
symptoms 
using 
experience 
sampling 
methodology. Psychiatry research, 210(1), 82-88. 
24. Stone, A. A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavioral 
medicine. Annals of Behavioral Medicine. 
25. Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2003). Patient 
compliance with paper and electronic diaries. Controlled clinical trials, 24(2), 182-199. 
26. Stone, A., Shiffman, S., Atienza, A., & Nebeling, L. (2007). The science of real-time data 
capture: Self-reports in health research. Oxford University Press. 
27. Varese, F., Haddock, G., & Palmier-Claus, J. (2019). We are designing and conducting an 
experience sampling study. Experience sampling in mental health research, 8. 
28. Wong, S. A., Duffy, L. A., Layfield, S. D., Long, M., Rodriguez-Villa, F., Gelda, S. E., ... & Yip, 
A. G. (2022). Using electronic patient-reported measures to characterize symptoms and 
improvement in inpatient psychiatric units. Psychiatry Research, 317, 114839. 
29. Zarbo, C., Agosta, S., Casiraghi, L., De Novellis, A., Leuci, E., Paulillo, G., ... & de Girolamo, 
G. (2022). Assessing adherence to and usability of Experience Sampling Method (ESM) and 
actigraph 
in 
patients 
with 
Schizophrenia 
Spectrum 
Disorder: 
A 
mixed-method 
study. Psychiatry Research, 314, 114675. 

17
 
 
Ethics 
The study was approved by the Helsinki Committee of the Sheba Medical Center.