Research on Delirium Recognition in Neurocritical Patients Based on Facial Expression Behavior Patterns
Conditions: Delirium, Artificial Intelligence (AI)
Sponsor: Beijing Tiantan Hospital

Project Number(PX2023021) Version Number: V1.0, Date: 2024/9/26
1
Identification of delirium in neurocritically ill patients based
on behavioral patterns of facial expressions Clinical study
protocol of the study
Project Name: Identification of delirium in neurocritically ill patients based on
facial expression behavior patterns Project Leader/Responsible Department:
Huang Huawei/Department of Critical Care Medicine
Project entrusted unit: Beijing Tiantan Hospital Affiliated to Capital Medical University
Project undertaking unit: Beijing Tiantan Hospital Affiliated to Capital Medical
University
Project Sponsor: Beijing Tiantan Hospital Affiliated to Capital Medical University
Study Duration: 2023/01 - 2025/12
Version number: V1.0
Version date: September 26, 2024

2
Summaryoftheplan
Projectname
Identification Study of Delirium in Neurocritically Ill Patients Based on Facial
Expression Behavior Patterns
Research
objectives
Through a three-year project, a large sample of neurocritically ill patients was
included in the Department of Critical Care Medicine of Beijing Tiantan
Hospital Affiliated to Capital Medical University, and the facial expression
behavior of patients with severe neurodelirium was digitally analyzed by
machine learning algorithms, and a delirium classification model based on
facial expression behavior was constructed, so as to realize the accurate and
efficient diagnosis of delirium in neurocritically ill patients through facial
expression behavior recognition.
Study
design
Prospective diagnostic study
Total
numberof
cases
3180 people
Inclusion criteria: Admitted to the comprehensive ICU of Beijing Tiantan
Hospital from January 2023 to January 2025
Patients with severe neurology who are in intensive care in
the
ward,
including
patients
after
neurosurgery,
stroke
patients, and patients admitted to the ICU for monitoring and
treatment due to other neurological diseases, and sign an informed
consent form.

2
Case
selection
Exclusion Criteria: 1) Age less than 18 years; 2) Persistent coma (GCS) before
surgery and within 7 days after surgery（≤8 points) unable to perform
delirium evaluation; 3) Severe dementia or severe cognitive impairment or
mental illness that cannot be evaluated for delirium; 4) Not alive in the ICU
for more than 24 hours; 5) Patients with facial paralysis, post-traumatic facial
deterioration, etc., which may have a serious impact on facial recognition; 6)
Exclude patients with dementia, Parkinson's, depression, etc., which may affect
facial emotional expressions.
Treatment
options
Critically ill patients in the ICU who met the inclusion criteria were evaluated
for delirium and their facial features were recorded using software. Algorithm
technology is used to learn to simulate facial features of patients with
delirium.
Efficacy
evaluation
Effectivenessevaluationindicators(primaryandsecondaryefficacyindicators):
Constructionandevaluationofdeliriummodelinpatientswithsevereneuropathybased
onexpressionbehaviorpatterns

3
Safetyevaluationindex:diagnosticresearch,notinvolvingsafetyevaluation
Statistical
methods
The time frequency of expression behavior and the dynamic change rate of
expression behavior between the delirium group and the non-delirium
group, as well as the differences in different high-level features, were
expressed as mean and standard deviation. The Shapiro-Wilk test is used to
confirm that the data conforms to a normal distribution. Therefore, the
independent sample t-test was used for comparison between the two
groups with normally distributed data, and the one-way analysis of variance
was used for comparison between multiple groups. The Mann-White U test
was used for comparison between the two groups of non-normally
distributed data. P<0.05 is the difference and is statistically significant.
Through machine learning, the advantages and disadvantages of different
models are examined, and three metrics are used: correctness, sensitivity,
and intentionality. Sensitivity represents the number of subjects who are
predicted to be normal and the real situation is normal/the total number of
normal groups, that is, the strength of the model's ability to identify normal
groups. Specificity refers to the number of subjects predicted to have
delirium and the real situation is delirium/review of the delirium group,
that is, the strength of the model to identify the delirium group.
Study
duration
2023/01 - 2025/12

8
Ⅰ.Researchbackground
Through a three-year project, a large sample of neurocritically ill patients was
included in the Department of Critical Care Medicine of Beijing Tiantan Hospital
Affiliated to Capital Medical University, and the facial expression behavior of patients
with severe neurodelirium was digitally analyzed by machine learning algorithms, and a
delirium classification model based on facial expression behavior was constructed, so as
to realize the accurate and efficient diagnosis of delirium in neurocritically ill patients
through facial expression behavior recognition. By summarizing the research results,
writing and publishing academic papers, and participating in domestic academic
exchanges, the delirium face recognition software is promoted, which provides new
ideas for the clinical prevention and treatment of POD.
Ⅱ.Researchobjectives
1. Main objective: To digitally analyze the facial expression behavior of patients with
severe delirium by machine learning algorithm, and to construct a delirium classification
model based on expression behavior.
Ⅲ. Research design types, principles and test steps
1. Study design

8
This study is a prospective diagnostic study
2. Sample size and research plan
According to previous literature studies, based on the advanced characteristics
related to facial expressions and behaviors in patients with severe delirium. The
sensitivity of predicting delirium is between 75% and 85%, and the specificity is between
80% and 90%, respectively, according to our previous studies and previous studies, the
incidence of delirium in patients with severe neuropathy is about 20%, the test level is
a=0.05, the certainty is 80%, and the sample size of patients with severe neuropathy is
calculated according to the PASS software to be 2891 cases. The number is 3180 cases.
At present, the department has 75 beds and treats more than 4,000 patients annually
Basic satisfaction.
3. Research duration: 3 years
4. Subject selection
Inclusion criteria: Critically ill patients admitted to the comprehensive ICU ward of
Beijing Tiantan Hospital for intensive care from January 2023 to January 2025, including
patients after neurosurgery, stroke patients, and patients admitted to the ICU for monitoring
and treatment due to other neurological diseases, and signed the informed consent form.

8
Exclusion criteria include: 1) age less than 18 years; 2) Persistent coma (GCS ≤8 points)
before surgery and within 7 days after surgery, unable to evaluate delirium; 3) Severe dementia or
severe cognitive impairment or mental illness that cannotbe evaluated for delirium; 4) Notalive in
the ICU for more than 24 hours; 5) Patients with facial paralysis, post-traumatic facial damage that
may have a serious impact on facial recognition; 6) Exclude patients with dementia, Parkinson's,
depression, and other conditionsthatmay affect facial emotional expressions.
ⅣResearchmethods
(1) Delirium assessment process: After the patient is admitted to the ICU, delirium
assessment and facial behavior data are collected every day at 8:00 – 10:00 and 20:00 - 22:00.
Assessment Diagnosis by an experienced specialist using the gold standard DSM-V for delirium
measurement. Assessment completed Video data collection of facial expression behavior was
carried out within 5 minutes.
(2) The gold standard DSM-V for delirium measurement specifically includes: 1) attention
and consciousness disorders, respectively, which represent decreased attentional directing,
concentration, maintenance, and switching abilities, and decreased orientation to the
environment; 2) Impaired attention and consciousness progressed within hours or days, and
the condition fluctuated within 1 day; 3) Companion Cognitive dysfunction such as memory,
orientation, language, visuospatial or perceptual impairment; 4) When not in a coma, it
cannot be explained by other pre-existing neurocognitive disorders; 5) There is evidence
related to delirium, such as drug poisoning or withdrawal. Other symptoms that may occur

8
may include altered arousal, disorientation, disturbed thinking, inappropriate speech or mood,
disturbed sleep-wake cycles, hallucinations, and psychomotor alterations.
(3) Identification of facial expression behavior patterns in delirium people
1 Design an experimental paradigm for facial expressions and behaviors of delirium
people for data collection According to the diagnostic criteria of delirium, changes in
attention and consciousness and disturbed thinking are important conditions, so the
experimental paradigm we designed should reflect the feedback of different facial
expressions and behaviors when patients have consciousness disorders, attention
deficits and thought disorders. We used the questions on consciousness, attention,
and thinking assessment in the classic delirium assessment tool CAM-ICU as stimulus
material. These stimulus materials are integrated into a mobile application software to
take pictures of the patient's facial expressions answering a question as subsequent
analysis data. Among them, the questions for examining attention are "I will read you
10 numbers, when you hear the number 8, please pinch my hand, 6-8-5-9-8-3-8-8-4-
7" and "Please stretch out these fingers (the examiner stretches out a few fingers in
front of the patient), and now extend the same finger with the other hand (the
examiner does not demonstrate)"; The question of examining thinking is: "Do stones
float on water?" Are there fish in the sea? Is a pound heavier than two pounds? Can a
hammer be used to drive nails? "or" Can leaves float on water? Are there elephants in
the sea? Can two pounds be used to saw? "While recording the video recording

8
during the question, the examiner also records the correctness of the answer on the
software so that it can be associated with the recorded expression. The consciousness
situation can be dynamically presented throughout the video recording process.
Through the mobile application software, the collected data can also be standardized
and constrained, and preprocessing operations such as face cropping, alignment, and
data enhancement can be completed.
②Digital analysis of expression behavior patterns in people with severe delirium
We will use the collected data to quantitatively analyze the facial expression behavior patterns
of delirium people. Since AU is produced by the movement of facial muscle masses, it may
be more behaviorally interpretable than other traditional image features. Therefore, in this
study, we propose a quantification method for typical expression behavior based on AU
features. The details are as follows: 1) First, through the description of delirium symptoms
through previous literature, compared with normal people, delirium patients have special
behavior patterns in emotional cognition and emotional feedback. Therefore, according to
the classification of delirium and the specific manifestations in each classification, we divided
the facial expressions of the subjects into the following three categories: agitation (including
fright, excitement, euphoria), calmness, and apathy (including drowsiness and lethargy).
According to the AU involved in looking for the above emote in the FACS documentation.
Among them, the calm expression is used as the initial expression and used as a reference for
comparison with the rest of the expressions. The so-called calm expression is defined as when
the facial muscle mass is not very active, or the exercise intensity is low, the face does not

8
produce expression behavior, which is the so-called calm expression. The remaining AUs
involved in the emoji are as follows:
2) The 2400 frames of data of each video are precisely divided into the typical facial
expressions described above, i.e. by realization. Each expression on each frame of the video
data calculates a value, and in each frame. The values of all expressions represent the main
expressions and subordinate expressions produced by the above faces cause. Therefore, we
need to distinguish the main expression at each frame moment, ignore the subordinate
expressions, and label each frame of expression behavior with a classification label. 3) The
number of occurrences of typical facial expressions under stimulation and their proportion in
each video were calculated separately to measure the time-frequency ratio of facial expressions,
so as to explore the emotional feedback of delirium patients. The amplitude characteristics of
AU are used to quantify the dynamic change rate of expression behavior, so as to reflect the
sensitivity of delirium patients' emotional feedback. The purpose of this paper is to verify the
expression behavior patterns of delirium patients from the digital level.
(3) To construct high-level features related to facial expressions and behaviors in patients with
severe delirium. Taking the time-frequency ratio of expression behavior and the dynamic
change rate of expression behavior as objects, we construct the advanced characteristics of
delirium in the time domain and frequency domain of AU signal, which are mainly divided
into three aspects: statistical features, rate characteristics and dynamic characteristics. Among
them, the statistical characteristics include: the time frequency of expression behavior, the

8
number of occurrences of maximal points, and the number of occurrences of singular points;
The rate characteristics include the rate feature of expression behavior change, the average
gradient feature of the maximum point, the average gradient feature of the singular point, and
the HHT frequency feature. Dynamic features are maximal point gradient FDHH features and
singular point gradient FDHH features. A total of 9 types of features are involved. These
features are described in detail below:
1) Characteristics of the time frequency of expression behaviors: We hypothesize that
the number of excited, fearful or euphoric expressions in agitated delirium is higher than
that of normal people, while the number of unconcerned and emotionless expressions in
apathetic delirium is higher than that of normal people. Therefore, we take the expression
ratio number as a high-level feature, that is, the number of times the subject's various
expressions appear under the stimulus question are entered into the classifier to calculate.

10
2) Characteristics of the rate of change in facial expression behavior: The physical
meaning of AU is the movement of muscle masses, which in turn are related to facial
behavior. Since the rate of change of expression behavior is difficult to measure by feature
indexes, in this experiment, we measure the dynamic change rate of expression behavior by
the average gradient of its amplitude and AU, and classify it as a constructed high-level
feature.
3) Characteristics of the maximum value point: In the original video data, there may
be no significant difference in expression between the delirium group and the normal
population in many time periods, and the difference in the corresponding AU signal value
may not be large. Therefore, we should consider capturing the "key points" in the
expression behavior, that is, extracting the maximum value of the AU feature as a
secondary feature. The maximum value point in this study represents the maximum
degree of deformation of the action unit under the stimulus problem. We extract the
relevant AUs involved in different facial expressions, count the number of their maximum
value points, and then classify them as high-level characteristics for delirium recognition.
4) Singularity characteristics: A certain derivative of the original signal suddenly
changes at a certain moment, which is called singular signal, which can represent many
characteristics of the signal and is an indispensable and important part of the original signal.
Detect the singularity of the signal, find and determine the location of the singularity of the

10
signal value, and judge the singularity of the point. However, due to the influence of external
environmental noise, the signal of singular point presents non-stationary characteristics, so it
is very difficult to process such signals. In order to study the changes in the signal curve
carrying important information, we will combine the Lipshitz exponent and wavelet
transform theory to construct singular point features.
5) Hilbert-Huang Transform (HHT) frequency characteristics: In order to study the
relevant information of AU amplitude time signals on the frequency threshold, we perform the
HHT transform on the signal. Firstly, the Empirical Mode Decomposition (EMD) method of
the signal is used to decompose the original AU signal into several IMF functions. Then apply
the Hilbert transform to each IMF component, The corresponding Hilbert spectrum is
obtained, and then the subband energy and center frequency of each IMF component are
calculated. The energy ratio of each IMF component is multiplied by the average frequency
to obtain the final HHT frequency characteristic index.
6) Feature dynamic history histogram features based on gradients (Feature dynamic history
histogram, (FDHH): Firstly, the maximum point gradient sequence and the singular point
gradient sequence of each AU are extracted, and then the absolute value difference is calculated
for the adjacent feature point gradient series, and then the threshold is set, and the absolute value
difference is set as "0" or "1", and the binary value sequence is defined as different PM modes
according to the difference in the number of consecutive "1" in the above figure and the final
histogram features are obtained.

10
④Construction and evaluation of delirium model in patients with severe neuropathy based
on expression behavior patterns. We send the advanced features of each of the above six
different feature sets, including the time frequency feature of expression behavior, the rate
of change of expression behavior, the statistical number of maximum point statistics, the
average gradient feature of the maximum value point, the statistical number of singularity
points, the average gradient feature of singularity, the HHT frequency feature, the FDHH
dynamic feature based on the maximum point gradient, and the FDHH dynamic feature
based on the singularity point gradient into the classifier to establish a model for
classification. In order to reduce randomness and chance bias, the classification process
will strictly abide by the "tenfold cross-validation" criterion, and divide the samples into
training samples and test samples, which are not related to each other and intersect as null.
The training sample is only used to train the classifier and does not participate in any
testing experiments. The test sample is only used to detect the classification effect of
features, and does not participate in any training tasks. Finally, the performance metrics of
the models, namely correctness, sensitivity, and specificity, will be evaluated, and the
advantages and disadvantages of the models will be examined.
Ⅴ. Concomitantmedication
Not.

14
Ⅵ. Observationindicatorsand inspection time
Delirium assessment process: After the patient was admitted to the ICU, delirium
assessment and facial behavior data were collected every day at 8:00 – 10:00 and 20:00 - 22:00
respectively. The assessment is made by an experienced specialist using the gold standard DSM-
V for delirium measurement. Video data collection of facial expression behavior will be carried
out within 5 minutes after the assessment is completed.
ⅦFollow-uptime
not
Ⅷ. Clinical evaluation
(1) Objective efficacy evaluation: construct a delirium classification model based on
facial expression behavior
(2)Safety indicators:None
ⅨPre-assessmentofprojectrisks andrisk disposal plans
Adverse events refer to adverse medical events that occur after the subject receives
the test drug, but it does not necessarily have a causal relationship with the treatment. The
investigator should record all adverse events that occur during drug treatment in the case
report form, including type, diagnosis time, duration, and consequences. Any adverse
events should be followed up until remission or treatment termination.

14
Predictable adverse events in this study included: None
Adverse events were handled according to clinical routine. When the treating physician
deems it necessary, the dose of the trial drug may be temporarily or permanently reduced, or
the trial drug may be temporarily or permanently discontinued. and record these situations in
the case report form.
For predictable adverse events in this study, management measures include: None
Serious adverse events
When serious adverse events occur, theinvestigational drug mustbe stopped immediately
and treatment mustbe given. In this study, serious adverse events are those that result in:
1. Causes death
2. Life-threatening (risk of immediate death)
3. Prolong the subject's hospital stay or readmission to the hospital
4. Persistent or significant disability/incapacity
5. Other important medical events that the investigator deems to be reportable
Any serious adverse events occurring in the trial, whether related to the trial drug or
not, should be reported to the Ethics Committee within 24 hours of being informed. If
the test drug causes death, the test should be stopped immediately, reported to the ethics

14
committee as soon as possible, and recorded in detail, and the information should be
properly preserved. Any adverse events are followed up until remission or treatment
termination.
ⅩQuality control and quality assurance of research
(1)Researchersandmedicalstaff
1. Conduct training on the research plan before the start of the experiment, and strictly
follow the plan during the trial.
2. All drugs and materials used in clinical research must be subject to quality control,
regular inspection,
ensure that it is in good condition;
3. The Ethics Committee has the right to audit clinical research for the purpose of
ensuring the authenticity of clinical research records and complying with the
provisions of the clinical research protocol.
(2)Subjects
1. Explain the benefits and risks of the investigational drug in detail to each potential subject;
2. Informed consent must be signed by the patient or an authorized representative;
3. Subjects can withdraw from the trial at any time after inclusion.

14
(3)Terminationofthetrial
1. In the event of death due to the investigational drug, the trial shall be terminated and
reported to the Ethics Committee. heavy
The New Start Trial requires ethics committee approval;
2. When all patients have completed inclusion and follow-up, and the data collection is
complete, the principal investigator will decide to terminate the trial
Examine.
XI Datapreservation
The cognitive assessment data, imaging data, and postoperative cognitive assessment
results of the patients were stored by the main research.
XII.Datasecuritysupervision
Clinical research will formulate corresponding data security monitoring plans
according to the size of the risk. All adverse events should be recorded in detail, handled
appropriately and tracked until they are properly resolved or the condition is stable, and
serious adverse events and unexpected events should be reported to the Ethics Committee,
competent departments, sponsors and drug administration departments in a timely manner
according to regulations. The principal investigator conducts a regular cumulative review of
all adverse events and holds investigator meetings if necessary to assess the risks and

14
benefits of the study. Studies greater than minimal risk will have an independent data
monitor review the study Data monitoring, high-risk research will establish an
independent data security monitoring committee to monitor the accumulated security
data and efficacy data to make recommendations on whether to continue the study.
XIII.Statisticalprocessing
The time frequency of expression behavior and the dynamic change rate of
expression behavior between the delirium group and the non-delirium group, as well as
the differences between different high-level features, were expressed as mean and
standard deviation. The Shapiro-Wilk test is used to confirm that the data conforms to a
normal distribution. Therefore, the independent sample t-test was used for the
comparison of the normally distributed data between the two groups, and the single-
factor variance analysis was used for the comparison between multiple groups. The
Mann-White U test was used for the comparison of the data between the two groups.
P<0.05 is the difference and is statistically significant. Through machine learning, the
advantages and disadvantages of different models are examined, and three metrics are
used: correctness, sensitivity and intentionality. Sensitivity represents the number of
subjects predicted to be normal and the real situation is normal/the total number of
normal groups, that is, the strength of the model's ability to identify normal groups.
Specificity represents the number of subjects predicted to have delirium and the real

14
situation is delirium/a review of delirium groups, that is, the strength of the model to
identify delirium groups.
XIIIIEthicalconsiderations
Before the study begins, the clinical study is carried out after the ethics committee
approves the trial protocol. Before each subject is selected for this study, the investigator is
responsible for comprehensively introducing the purpose, procedures and possible risks of
this study to the subject or his legal representative, as well as the corresponding information
of alternative treatment, and signing a written informed consent form. The personal privacy
and data confidentiality of the subjects will be protected during the research process.
Describe the study's consideration of the informed process and how to ensure that
the subject is fully informed and given

15
Subjects have sufficient time to consider and voluntarily choose to agree to participate in the
study to avoid inducement and inappropriateness of the subject
Effect.
XV:Dataconfidentiality
Confidentiality of subject information, data protection, intellectual property rights,
research summaries, publication communications, etc.
XVI:Participants
name
Title/Major
task
GCPTraining
(time
Period)
Huang
Huawei
Deputy Chief
Physician/ICU
Project design, delirium
assessment
September
2018
Zhang
Guobin
Associate Chief
Physician/Neurosurgery
Project design, sample
collection, data collation
and analysis, article
writing
August 2019
An
bamboo
forest
Seniorengineer/
imagepreprocessing,
algorithmdesignand
implementation

15
Zhang
Xiaokang
PhD
Candidate/Neurosurgery
Cognitiveassessment,
datacollationand
analysis,andclinical
follow-up
November 2021
LiHao
arrived
PhD
Candidate/Neurosurgery
Cognitiveassessment,
datacollationand
analysis,andclinical
follow-up
Ying
Yuzhe
Master's
Degree/Neurosurgery
Cognitiveassessment,
datacollationand
analysis,andclinical
follow-up

16