# Developing Machine Learning and Statistical Tools to Evaluate the Accessibility of Public Health Advice on Infectious Diseases among Vulnerable People

Authors: Wenxiu Xie; Meng Ji; Mengdan Zhao; Kam-Yiu Lam; Chi-Yin Chow; Tianyong Hao
Journal: Computational Intelligence and Neuroscience
 (2021)

---

## Abstract

Background. From Ebola, Zika, to the latest COVID-19 pandemic, outbreaks of highly infectious diseases continue to reveal severe consequences of social and health inequalities. People from low socioeconomic and educational backgrounds as well as low health literacy tend to be affected by the uncertainty, complexity, volatility, and progressiveness of public health crises and emergencies. A key lesson that governments have taken from the ongoing coronavirus pandemic is the importance of developing and disseminating highly accessible, actionable, inclusive, coherent public health advice, which represent a critical tool to help people with diverse cultural, educational backgrounds and varying abilities to effectively implement health policies at the grassroots level. Objective. We aimed to translate the best practices of accessible, inclusive public health advice (purposefully designed for people with low socioeconomic and educational background, health literacy levels, limited English proficiency, and cognitive/functional impairments) on COVID-19 from health authorities in English-speaking multicultural countries (USA, Australia, and UK) to adaptive tools for the evaluation of the accessibility of public health advice in other languages. Methods. We developed an optimised Bayesian classifier to produce probabilistic prediction of the accessibility of official health advice among vulnerable people including migrants and foreigners living in China. We developed an adaptive statistical formula for the rapid evaluation of the accessibility of health advice among vulnerable people in China. Results. Our study provides needed research tools to fill in a persistent gap in Chinese public health research on accessible, inclusive communication of infectious diseases’ prevention and management. For the probabilistic prediction, using the optimised Bayesian machine learning classifier (GNB), the largest positive likelihood ratio (LR+) 16.685 (95% confidence interval: 4.35, 64.04) was identified when the probability threshold was set at 0.2 (sensitivity: 0.98; specificity: 0.94). Conclusion. Effective communication of health risks through accessible, inclusive, actionable public advice represents a powerful tool to reduce health inequalities amidst health crises and emergencies. Our study translated the best-practice public health advice developed during the pandemic into intuitive machine learning classifiers for health authorities to develop evidence-based guidelines of accessible health advice. In addition, we developed adaptive statistical tools for frontline health professionals to assess accessibility of public health advice for people from non-English speaking backgrounds.


---

## Body

## 1. Introduction

From Ebola, Zika, to the novel coronavirus pandemic, outbreaks of highly infectious diseases continue to reveal severe consequences of social and health inequalities in both developing and developed countries [1, 2]. Vulnerable people are affected more by the uncertainty, complexity, volatility, and progressiveness of public health crises and emergencies [3, 4]. A key lesson that governments have taken from the ongoing coronavirus pandemic is the importance of developing disseminating highly accessible, inclusive, actionable, coherent public health advice [5–8], which represents a critical tool to help people with diverse cultural, educational backgrounds and varying abilities to effectively implement health policies at the grassroots level. High variability in people’s socioeconomic background, education, and health literacy levels, varying intellectual, cognitive abilities, English proficiency, and religious beliefs can cause barriers to access and implement public health advice under health emergencies and crises [9–11]. Increasing inclusiveness and accessibility of health advice recommendations among diverse vulnerable populations has emerged as an important topic in both public health education and domestic and international policy making as a highly cost-effective measure to reduce health inequalities [12–15].Currently, there is a lack of national or international guidelines around the development of accessible and inclusive public health advice, especially for health emergencies and crises. However, the recent outbreak of coronavirus has prompted national health authorities to develop highly accessible health resources on COVID-19. Most current public-oriented health resources on infectious diseases belong to regular health resources (RHR) requiring higher levels of education, English proficiency, health knowledge, and literacy [16–19]. Typical RHR are resources published by the World Health Organisation which are intended for both general and professional readerships [20–22]. However, the practical accessibility of WHO health resources among the public remains unknown, as the international health organisation recently embarked on surveys to establish evidence of the accessibility of its public health resources [20].Vulnerable people-oriented (VPO) health resources are known for their significantly improved language understandability, information relevance (minimal distracting details irrelevant to the readers), social inclusiveness (applicability among diverse people), and information actionability [23–25]. They are developed by medical/health professionals with extensive experiences of working firstly with diverse vulnerable populations to ensure the practical usability of VPO. In English-speaking multicultural societies, intralingual health translations and simplified or accessibility-enhanced English resources provide main sources of public health advice and information for diverse vulnerable populations [8, 26–28], although evidence-based guidelines to inform the development of these materials and associated quality control measures are yet to be established and validated.Increasing amounts of VPO health resources in English, known as easy-to-read or easy read health materials, provide valuable firsthand materials for the development of assessment instruments and techniques to support best practices in the clinic, as well as global health policy making around accessible health recommendation design and social dissemination. The development of language-adaptive evaluation instruments can facilitate evidence-based health policy making by international health authorities. For health policy makers, we developed supervised Bayesian machine learning classifiers, and for frontline health professionals we developed convenient, language-adaptable statistical analysis tools to evaluate the accessibility of public health advice on top infectious diseases, including COVID-19.It should be noted that the machine learning classifiers and statistical tools that we developed using Chinese health resources can be conveniently and reliably adapted to other languages using low-cost, relatively easy-to-obtain natural language annotation tools. In doing so, our study provides practical and useful tools for clinical and health professionals working directly with vulnerable people. The intuitive Bayesian probabilistic evaluation tool that we developed will help advance research-based global accessible health advice design and international benchmarking for the pandemic which continues to spread in many developing countries and any future public health crises that require rapid, effective, and efficient response from governments and health authorities to address the practical needs from diverse vulnerable populations and help minimise the environmental and health impacts on them.


## 2. Methods

### 2.1. Collection and Translation of Regular and Vulnerable People-Oriented Health Resources

We collected two sets of stylistically distinct public health advice on the prevention and self-management of infectious diseases. Regular health resources (RHR) (202) were selected from the website of the World Health Organisation (WHO).Vulnerable People-Oriented (VPO) materials (91), especially public health advice on COVID-19, were collected from websites of health authorities, including Centers for Disease Control and Prevention (CDC), Australian Ministry of Health and Public Health England. These resources were easily recognisable due to being labelled clearly as easy-to-read public health advice and instructions on COVID-19. As an international health authority, WHO provides verified professional translations of original English resources. Since our study was to develop assessment tools to evaluate accessibility of public health advice in Chinese, Chinese translations of regular WHO health advice on infectious diseases were collected. VPO resources collected from sources of CDC, the Australian Ministry of Health and Public Health, England, were translated into Chinese using forward and backward translation recommended by the WHO [29–31].


### 2.2. Morphological-Lexical-Structural (MLS) Features

The Chinese translations of RHR and VPO were annotated using Chinese Readability Index Explorer (CRIE) [32, 33]. CRIE annotation provided 26 morphological-lexical-structural (MLS) features and 46 part-of-speech (POS) features (72 in total) of the annotated Chinese texts for our machine learning classifier development. MLS included average sentence number per paragraph, type token ratio (TTR), low-stroke characters (1–10 strokes), middle-stroke characters (11–20 strokes), high-stroke characters (21 or above), average strokes per character, 2-character words, 3-character words, average words per sentences, ratio of noun phrases, normalised frequency of noun phrases, average number of idioms per sentence, content words (verbs, nouns, adverbs, and adjectives), adverbs of negation, sentences with complex semantic (polysemous) categories, density of content words, average logarithmic frequency of content words, pronouns, personal pronouns, conjunctions, positive conjunctions, negative conjunctions, and difficult words ratio. These MLS features were studied extensively in Chinese readability research.


### 2.3. Parts of Speech (POS) Features

A POS tagging system developed by Academia Sinica as one of the most comprehensive automatic analysers of Chinese was applied. The POS features collected included nonpredicate adjective (A), coordinate conjunction (Caa), conjunctions (Cab), conjunctions (Cba), correlative conjunctions (Cbb), adverbs (D), nominal/adverbial/complement markers (DE), adverbial noun-modifiers (Da), adverbs of degree before verbs (Dfa), adverbs of degree after verbs (Dfb), tense markers (Di), sentential adverbs (Dk), interjections (I), common nouns (Na), proper nouns (Nb), geographical names (Nc), location names (Ncd), time adverbs (Nd), attributive adjuncts (Nep), modifiers of quantitative measures (Nes), quantities (Neu), measure words (Nf), post-positions (Ng), pronouns (Nh), prepositions (P), Be verbs (SHI), auxiliary words (T), intransitive predicates (VA), causative verbs (VAC), transitive verbs placed after objects (VB), transitive verbs (VC), verbs placed before locations (VCL), predicates used with both direct and indirect objects (VD), action verbs as sentence objects (VE), action verbs as predicate objects (VF), classification verbs (VG), modifiers of verbs and nouns (VH), causative modifiers (VHC), adjectives or past particles placed after objects (VI), transitive verbs to describe states (VJ), mental states and processes (VK), causative predicates (VL), and have verbs (V_2).


### 2.4. Statistical Analysis

Table1 shows that, among the 72 natural language features (26 MLS, 46 POS), statistically significant differences (p<0.05 of nonparametric Mann-Whitney U tests) between regular health resources (RHR) and vulnerable people-oriented (VPO) health resources were present in 88.9% of the entire feature set: 96.15% (25/26) of morphological-lexical-structural (MLS) features and 84.78% (39/46) of part-of-speech (POS) features. In addition to 2-sided p values, we computed Hedges’ g [34] as corrected Cohen’s d (1, 36, and 37) and 95% confidence interval of the effect size estimates. We also provided common language effect sizes (CLES), also known as probability of superiority [35–37]. In our study, CLES allowed an intuitive interpretation of the likelihood of the mean of a certain natural language feature randomly selected from VPO as higher than the mean of that feature in RHR on infectious diseases. We calculated Hedges’ g and CLES alongside commonly reported p values for two purposes: first, this was to help us interpret the result of automatic feature selection using Bayesian machine learning classifiers (Gaussian Naïve Bayes selected due to the presence of normally distributed continuous variables as features); second, this facilitated the determination of sample sizes for follow-up studies or the comparison of effects across studies.Table 1
Mann-Whitney U test.
MLS features (1–26) POS (27–72)EHR meanEHR std.RHR meanRHR std.Asymp. sig. (2-tailed)Effect sizedCohen (corrected effect size or Hedges’ g)Common language effect size CLES95% CIMann-Whitney UWilcoxon WZAverage sentences per paragraph0.6010.3453.3161.1630.0002.5880.9662.301 to 2.87452.004238.00−14.669TTR0.4610.1190.6230.0790.0001.8280.9021.568 to 2.0884389.008575.00−10.680Difficult words141.81396.38170.87235.9860.000−1.3110.823−1.558 to −1.0658769.5070897.50−6.657Low-stroke characters641.593444.264284.707140.9300.000−1.5070.857-1.758 to−1.2568108.0070236.00−7.264Middle-stroke characters120.39680.34851.81027.6950.000−1.5620.865−1.814 to−1.317357.0069485.00−7.955High-stroke characters0.0550.2290.4521.6040.0050.2770.5780.045 to 0.50814144.0018330.00−2.815Average strokes per character7.6770.2747.7110.3420.1950.1030.529−0.127 to 0.33414604.0018790.00−1.2972-character words256.637176.783114.34156.2620.000−1.5090.857−1.76 to −1.2588100.5070228.50−7.2713-character words13.00011.8248.2986.3720.008−0.6030.665−0.837 to −0.36913139.5075267.50−2.647Average words per sentences11.5924.96311.8931.9620.0180.1060.530-0.125 to 0.33613438.0017624.00−2.368Single sentences0.8830.1140.4600.1910.000−2.3760.954−2.655 to −2.098949.0063077.00−13.851Ratio of noun phrases0.2670.1040.4150.1980.0000.8100.7170.573 to 1.0468396.5012582.50−6.999Frequency of noun phrases per 10K322.36937.240314.66437.0020.028−0.2080.558−0.439 to 0.02313620.5075748.50−2.200Average idioms per sentences0.0010.0040.0120.0290.0040.4240.6180.192 to 0.65613969.0018155.00−2.894Content words392.582265.421163.07480.5380.000−1.6420.877−1.896 to −1.3876988.5069116.50−8.293Adverbs of negation2.7583.0670.9351.2280.000−1.0320.767−1.272 to −0.7929522.5071650.50−6.283Sentences with complex semantic categories25.39622.0397.6144.7580.000−1.6430.877−1.898 to −1.3886696.5068824.50−8.578Density of content words0.8280.0260.8150.0310.000−0.4330.620−0.665 to −0.211857.5073985.50−3.820Average logarithmic frequency of content words1.7380.1691.3370.1830.000−2.2250.942−2.498 to −1.9521854.0063982.00−13.009Idioms0.0770.2680.2240.5100.0090.3120.5870.081 to 0.54414168.0018354.00−2.623Pronouns41.69235.0431.4691.6520.000−2.5300.963−2.814 to −2.245620.5062748.50−14.398Personal pronouns37.86831.8380.7051.1390.000−2.5770.966−2.864 to −2.291326.0062454.00−15.334Conjunctions18.96716.44011.5206.4070.001−0.7950.713−1.031 to −0.55812507.5074635.50−3.228Positive conjunctions16.82414.1599.0005.1930.000−0.9910.758−1.23 to -0.75110808.0072936.00−4.795Negative conjunctions0.8461.5841.4401.3930.0000.4140.6150.182 to 0.64610710.5014896.50−5.070Difficult words ratio30.3845.72435.8478.1750.0000.7060.6910.471 to 0.9419400.5013586.50−6.077A3.0993.7743.0262.9690.236−0.0230.507−0.254 to 0.20714740.5018926.50−1.184VI0.1430.4850.1590.4240.3480.0370.510−0.194 to 0.26715418.5019604.50−0.938Dk0.0110.1050.0170.1300.6800.0480.514−0.183 to 0.27815919.0020105.00−0.413VG2.1542.4542.0541.9190.505−0.0490.514−0.28 to 0.18115305.0019491.00-0.666Nv9.96710.3909.2906.7110.083−0.0890.525−0.32 to 0.14214132.0018318.00−1.734Neqb0.0330.1800.0570.2770.5900.0920.526−0.138 to 0.32315810.0019996.00−0.539Cab0.1210.3900.3980.7990.0000.3770.6050.145 to 0.60913151.0017337.00−3.534I0.0330.1800.0000.0000.001−0.4060.613−0.638 to −0.17415488.0077616.00−3.414VAC0.1540.3920.0480.2150.001−0.4060.613−0.638 to −0.17414493.0076621.00−3.214Nd4.2537.8722.2983.4080.002−0.4180.616-0.65 to −0.18612764.5074892.50−3.050Nb1.4512.4230.7391.4200.000−0.4250.618−0.657 to −0.19312369.5074497.50−3.789Dfb0.0660.2910.0030.0530.000−0.4510.625−0.683 to −0.21915181.0077309.00−3.831Neu4.5056.3392.3953.2090.001−0.5210.644−0.754 to −0.28812438.0074566.00−3.347VJ10.0118.8366.4555.1250.000−0.5860.661−0.82 to −0.35211896.0074024.00−3.797VL2.9122.9461.7101.7410.001−0.5880.661−0.821 to −0.35412457.0074585.00−3.343Cba0.1100.3790.0030.0530.000−0.6020.665−0.836 to −0.36914652.5076780.50−5.125Caa12.84611.1178.7055.1500.103−0.6080.666−0.842 to −0.37414244.5076372.50−1.631VB0.9121.4880.3210.7220.000−0.6350.673−0.869 to −0.40112839.5074967.50−3.789VHC0.5271.0891.7051.9580.0000.6490.6770.415 to 0.8849100.0013286.00−6.641Da0.5161.0580.1250.3480.000−0.6860.686−0.921 to −0.45112813.0074941.00-4.647Nes2.0332.1000.9601.2780.000−0.7230.696−0.959 to -0.48810783.5072911.50−5.088Dfa3.4733.8541.6081.7520.000−0.7970.713−1.033 to −0.56110952.0073080.00−4.769VH22.15415.91313.7678.2220.000−0.8170.718−1.053 to −0.5811401.0073529.00−4.243Nf7.4188.0713.0943.9560.000−0.8520.727−1.089 to −0.6159110.5071238.50−6.401Nc7.7807.7812.6885.3030.000−0.8640.729−1.101 to −0.6278121.0070249.00−7.430Nep3.5274.3881.3271.6530.000−0.8900.736−1.128 to −0.65312279.5074407.50−3.559T1.1651.6620.2610.6490.000−0.9530.750−1.192 to −0.71510192.0072320.00−6.926Di2.9013.1690.9121.5380.000−1.0030.761−1.243 to −0.7639200.5071328.50−6.755SHI4.3964.9421.6761.7090.000−1.0060.762−1.246 to −0.76610776.0072904.00−4.917VD1.7032.3550.3750.8610.000−1.0120.763−1.252 to −0.7729587.5071715.50−7.241Cbb5.8906.4022.4152.0010.000−1.0220.765−1.263 to −0.78210325.5072453.50−5.297Neqa6.0115.9972.5372.2270.000−1.0340.768−1.274 to −0.79410036.5072164.50−5.553Na113.20977.72164.63932.6060.000−1.0650.774−1.306 to −0.82411326.5073454.50−4.308Ng6.4738.0482.0681.9800.000−1.0900.780−1.331 to −0.8488874.5071002.50−6.668V_24.5605.8791.1281.2140.000−1.1970.801−1.44 to −0.9539003.5071131.50−6.676VE7.8247.2072.6702.9130.000−1.2370.809−1.482 to −0.9938111.5070239.50−7.335DE27.39621.30011.6537.6180.000−1.3360.828−1.583 to −1.098606.5070734.50−6.816P22.57116.5709.3306.1880.000−1.4240.843−1.672 to −1.1758358.0070486.00−7.045Ncd6.7475.9601.8691.9330.000−1.5260.86−1.777 to −1.2747167.5069295.50−8.263VCL5.2425.7300.6931.0330.000−1.6560.879−1.911 to −1.4015248.5067376.50−10.615VC48.79135.98514.75910.5510.000−1.8120.900−2.071 to −1.5525672.5067800.50−9.506D44.11031.49114.4898.2470.000−1.8490.905−2.11 to −1.5895603.0067731.00−9.573VK9.7367.6882.2932.0860.000−1.8890.909−2.151 to −1.6275158.5067286.50−10.096VA11.65910.1402.0882.2140.000−1.9190.913−2.181 to −1.6564616.5066744.50−10.605VF4.1983.9640.2220.6290.000−2.1190.933−2.388 to −1.8494026.5066154.50−13.780Nh41.69235.0431.4691.6520.000−2.5300.963−2.814 to −2.245620.5062748.50−14.398The following formula shows the corrected effect size or Hedges’g:(1)Hedges′gs=Cohen′sds×1−34n1+n2−9.The following formula shows common language effect size (CLES):(2)Z=x1−x2SD12+SD22/2.When computing effect sizes (Hedges’gs) and associated probabilities of superiority (CLES), we used RER as reference class. As a result, positive values indicated that features were statistically higher in more difficult regular health resources; and negative values suggested that features were prevalent in highly accessible, vulnerable people-oriented public health advice. MLS features which had large (absolute value larger than 1), corrected effect sizes, Hedges’ gs, and CLES included the following: average sentences per paragraph (VPO: M = 0.601, SD = 0.345; RHR: M = 3.316, SD = 1.163, p<0.001, Hedges’ gs = 2.588, 95% CI [2.301, 2.874], and CLES = 0.966); type token ratio (TTR) (VPO: M = 0.461, SD = 0.119; RHR: M = 0.623, SD = 0.079, p<0.000, Hedges’ gs = 1.828, 95% CI [1.568, 2.088], and CLES = 0.902); frequency of difficult words (VPO: M = 141.813, SD = 96.381; RHR: M = 70.872, SD = 35.986, p<0.001, Hedges’ gs = −1.311, 95% CI [−1.558, −1.065], and CLES = 0.823); single sentences (VPO: M = 0.883, SD = 0.114; RHR: M = 0.460, SD = 0.191, p<0.001, Hedges’ gs = −2.376, 95% CI [−2.655, −2.098], and CLES = 0.954); pronouns (VPO: M = 41.692, SD = 35.043; RHR: M = 1.469, SD = 1.652, p<0.001, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963); personal pronouns (VPO: M = 37.868, SD = 31.838; RHR: M = 0.705, SD = 1.139, p<0.001, Hedges’ gs = −2.577, 95% CI [−2.864, −2.291], and CLES = 0.966); low-stroke characters (VPO: M = 641.593, SD = 444.264; RHR: M = 284.707, SD = 140.930, p<0.001, Hedges’ gs = −1.507, 95% CI [-1.758, −1.256], and CLES = 0.857); 2-character words (VPO: M = 256.637, SD = 176.783; RHR: M = 114.341, SD = 56.262, p<0.001, Hedges’ gs = −1.509, 95% CI [−1.76, −1.258], and CLES = 0.857); and average logarithmic frequency of content words (VPO: M = 1.738, SD = 0.169; RHR: M = 1.337, SD = 0.183, p<0.001, Hedges’ gs = −2.225, 95% CI [−2.498, −1.952], and CLES = 0.942).Part of speech (POS) features which had large (absolute value larger than 1), corrected effect sizes, Hedges’gs, and CLES included the following: Nh (pronouns) (VPO: M = 41.692, SD = 35.043; RHR: M = 1.469, SD = 1.652, p<0.000, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963); VF (action verbs as predicate objects) (VPO: M = 4.198, SD = 3.964; RHR: M = 0.222, SD = 0.629, p<0.001, Hedges’ gs = −2.119, 95% CI [−2.388, −1.849], and CLES = 0.933); VA (intransitive predicates) (VPO: M = 11.659, SD = 10.140; RHR: M = 2.088, SD = 2.214, p<0.000, Hedges’ gs = −1.919, 95% CI [−2.181, −1.656], and CLES = 0.913); VK (mental states and processes) (VPO: M = 9.736, SD = 7.688; RHR: M = 2.293, SD = 2.086, p<0.000, Hedges’ gs = −1.889, 95% CI [−2.151, −1.627], and CLES = 0.909); D (adverbs) (VPO: M = 44.110, SD = 31.491; RHR: M = 14.489, SD = 8.247, p<0.001, Hedges’ gs = −1.849, 95% CI [−2.11, −1.589], and CLES = 0.905); VC (transitive verbs) (VPO: M = 48.791, SD = 35.985; RHR: M = 14.759, SD = 10.551, p<0.001, Hedges’ gs = −1.812, 95% CI [−2.071, −1.552], and CLES = 0.900); Ncd (location names) (VPO: M = 6.747, SD = 5.960; RHR: M = 1.869, SD = 1.933, p<0.001, Hedges’ gs = −1.526, 95% CI [−1.777, −1.274], and CLES = 0.860); P (prepositions) (VPO: M = 22.571, SD = 16.570; RHR: M = 9.330, SD = 6.188, p<0.001, Hedges’ gs = −1.424, 95% CI [−1.672, −1.175], and CLES = 0.843); and V_2 (have) (VPO: M = 4.560, SD = 5.879; RHR: M = 1.128, SD = 1.214, p<0.001, Hedges’ gs = −1.197, 95% CI [−1.44, −0.953], and CLES = 0.801).


### 2.5. Gaussian Naïve Bayes

Gaussian Naïve Bayes (GNB) is a variant of Naïve Bayes which is supervised machine learning classification algorithm based on the Bayes theorem [38–42]. Various strengths of GNB are its convenience, computation speed (suitability for making real time prediction), scalability, generalisability with small data like most Bayesian machine learning classifiers, and flexibility with continuous and discrete features. In our study, the size of the training (205) and testing data (88) was relatively small. Bayesian machine learning classifiers like GNB, relevance vector machine (RVM), and multinominal Naïve Bayes (MNB) are more suitable, as they are unlikely to overfit small datasets. Furthermore, the two sets of public health resources that we collected, regular and vulnerable people-oriented sets, contained continuous features and their distributions in the Chinese translations of regular health resources followed Gaussian normal distribution (Table 2, Figure 1). As a result, GNB was selected as the most suitable machine learning classifier in our study to ensure the generalisability and reliability of the classifiers.Table 2
Continuous feature distribution in regular health resources (RHR) translated to Chinese.
Continuous MLS featuresMinMaxMeanSE.95% confidence interval for meanSD.SkewnessSE.KurtosisSE.ASPP0.1228.8572.7590.0722.6172.901.5180.1450.116−0.1940.231DCW0.6910.8960.8170.0010.8140.8200.030−0.2100.1160.3820.231RNP0.0001.2710.3850.0090.3670.4030.1920.8260.1161.2070.231ALFCW0.8082.2281.4200.0121.3971.4420.2420.3650.116-0.180.231NFNP200.00458.33316.251.765312.78319.7137.140.2680.1160.6160.231ASPC6.5058.7057.7040.0167.6737.7340.329-0.2640.1161.0590.231ASPP: average sentences per paragraph; DCW density of content words; RNP: ratio of noun phrases; ALFCW: average logarithmic frequency of content words; NFNP: normalised frequency of noun phrases; ASPC average strokes per character.Figure 1
Continuous feature distribution in regular health resources (RHR) translated into Chinese.


### 2.6. Training and Testing Machine Learning Classifiers

To train machine learning classifiers for automatic information accessibility evaluation, the total number of RHR used was 202 and the total number of VPO used was 91. Next, 73.3% (148) of RHR and 62.6% (57) of VPO were used as the training data and the remaining texts (54 RHR and 34 VPO) were used as testing data. We applied 5-fold cross-validation with the training data to produce the mean and standard deviation of area of curve (AUC) of the GNB classifier. Review of the model performance was on the remaining 30% test data in terms of AUC, accuracy, sensitivity, specificity, and macro F1 (Table3).Table 3
Performance of Gaussian Naïve Bayes (GNB) classifiers with different feature sets.
ModelTechniquesTraining (5-fold CV)TestingAUC mean (SD)AUCAccuracyMacro F1SensitivitySpecificity1MLS + POS full (69)0.971 (0.0212)0.9400.9210.9920.9440.88242MLS + POS jointly optimised (6)0.998 (0.0026)0.9930.9400.9430.9630.91183MLS full (26 features)0.997 (0.003)1.00.9660.9631.00.91184MLS optimised (2 features)0.998 (0.004)1.00.9430.9381.00.85295POS full (46 features)0.959 (0.0238)0.9070.8520.8430.8890.79416POS optimised (8 features)0.982 (0.0166)0.9680.9550.9511.00.88247MLS + POS separately optimised (10)1.0 (0)1.00.9550.9511.00.88248Refined MLS + POS separately optimised (2)0.995 (0.0079)0.9990.9890.9880.9821.0


### 2.7. Classifier Optimisation

High dimensional features can reduce the performance of machine learning classifiers due to the forced inclusion of irrelevant parameters in the model. To counter the issue of classifier underperformance caused by the presence of redundant features, we applied different classifier optimisation techniques to reduce the original features collected. First, we applied integral optimisation by selecting the optimised feature set from the combined MLS and POS features (72 in total). This led to a combinedly optimised feature set of 6 features (around 8% of the original total features): average sentences per paragraph (ASPP) (p<0.000, Hedges’ gs = 2.588, 95% CI [2.301, 2.874], and CLES = 0.966), personal pronouns (p<0.001, Hedges’ gs = −2.577, 95% CI [−2.864, −2.291], and CLES = 0.966), Di (tense markers) (p<0.001, Hedges’ gs = −1.003, 95% CI [−1.243, −0.763], and CLES = 0.761), Nd (time adverbs) (p=0.002, Hedges’ gs = −0.418, 95% CI [−0.65, −0.186], and CLES = 0.616), VF (action verbs as predicate objects) (p<0.001, Hedges’ gs = −2.119, 95% CI [−2.388, −1.849], and CLES = 0.933), and V_2 (have verbs) (p<0.001, Hedges’ gs = −1.197, 95% CI [−1.44, −0.953], and CLES = 0.801).These optimised features were also those with the most significant statistical differences (indicated byp values, corrected effect size Hedges’ g, common language effect sizes CLES) between regular and vulnerable people-oriented health resources (Table 1). Next, we applied feature optimisation in the two sets of MLS (26) and POS (46) features separately. This led to an optimised MLS feature set of 2 features only (7.7% of the total MLS features): ASPP and ALFCW (p<0.001, Hedges’ gs = −2.225, 95% CI [−2.498, −1.952], and CLES = 0.942). The optimised POS feature set of 8 features (17.4% of the total POS features): A (nonpredicate adjectives) (p<0.000, Hedges’ gs = −0.023, 95% CI [−0.254, 0.207], and CLES = 0.507), Cbb (correlative conjunctions) (p<0.001, Hedges’ gs = −1.022, 95% CI [−1.263, −0.782], and CLES = 0.765), Dfa (adverbs of degree before verbs) (p<0.001, Hedges’ go = −0.797, 95% CI [−1.033, −0.561], and CLES = 0.713), Nd (time adverbs) (see above), Ng (postpositions) (p<0.001, Hedges’ gs = −1.090, 95% CI [−1.331, −0.848], and CLES = 0.780), Nh (pronouns) (p<0.001, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963), VCL (verbs placed before locations) (p<0.001, Hedges’ gs = −1.656, 95% CI [−1.911, −1.401], and CLES = 0.879), and VHC (causative modifiers) (p<0.001, Hedges’ gs = 0.649, 95% CI [0.415, 0.884], and CLES = 0.677). For both integral and parallel classifier optimisations, we used backforward feature elimination known as recursive feature elimination (RFE) with support vector machine as base estimator. Maximal validation accuracy/minimal classification errors were used as the feature optimisation criteria (Table 4 and Figure 2).Table 4
Criteria of classifier optimisation.
TechniquesCross-validation accuracy (CVA)Minimal classification error (MCE) = 1-CVAJoint optimisation (6)0.98530.0147Optimised MLS features (2)0.99020.0098Optimised POS features (8)0.98540.0146Figure 2
Recursive feature elimination using SVM as base estimator.


## 2.1. Collection and Translation of Regular and Vulnerable People-Oriented Health Resources

We collected two sets of stylistically distinct public health advice on the prevention and self-management of infectious diseases. Regular health resources (RHR) (202) were selected from the website of the World Health Organisation (WHO).Vulnerable People-Oriented (VPO) materials (91), especially public health advice on COVID-19, were collected from websites of health authorities, including Centers for Disease Control and Prevention (CDC), Australian Ministry of Health and Public Health England. These resources were easily recognisable due to being labelled clearly as easy-to-read public health advice and instructions on COVID-19. As an international health authority, WHO provides verified professional translations of original English resources. Since our study was to develop assessment tools to evaluate accessibility of public health advice in Chinese, Chinese translations of regular WHO health advice on infectious diseases were collected. VPO resources collected from sources of CDC, the Australian Ministry of Health and Public Health, England, were translated into Chinese using forward and backward translation recommended by the WHO [29–31].


## 2.2. Morphological-Lexical-Structural (MLS) Features

The Chinese translations of RHR and VPO were annotated using Chinese Readability Index Explorer (CRIE) [32, 33]. CRIE annotation provided 26 morphological-lexical-structural (MLS) features and 46 part-of-speech (POS) features (72 in total) of the annotated Chinese texts for our machine learning classifier development. MLS included average sentence number per paragraph, type token ratio (TTR), low-stroke characters (1–10 strokes), middle-stroke characters (11–20 strokes), high-stroke characters (21 or above), average strokes per character, 2-character words, 3-character words, average words per sentences, ratio of noun phrases, normalised frequency of noun phrases, average number of idioms per sentence, content words (verbs, nouns, adverbs, and adjectives), adverbs of negation, sentences with complex semantic (polysemous) categories, density of content words, average logarithmic frequency of content words, pronouns, personal pronouns, conjunctions, positive conjunctions, negative conjunctions, and difficult words ratio. These MLS features were studied extensively in Chinese readability research.


## 2.3. Parts of Speech (POS) Features

A POS tagging system developed by Academia Sinica as one of the most comprehensive automatic analysers of Chinese was applied. The POS features collected included nonpredicate adjective (A), coordinate conjunction (Caa), conjunctions (Cab), conjunctions (Cba), correlative conjunctions (Cbb), adverbs (D), nominal/adverbial/complement markers (DE), adverbial noun-modifiers (Da), adverbs of degree before verbs (Dfa), adverbs of degree after verbs (Dfb), tense markers (Di), sentential adverbs (Dk), interjections (I), common nouns (Na), proper nouns (Nb), geographical names (Nc), location names (Ncd), time adverbs (Nd), attributive adjuncts (Nep), modifiers of quantitative measures (Nes), quantities (Neu), measure words (Nf), post-positions (Ng), pronouns (Nh), prepositions (P), Be verbs (SHI), auxiliary words (T), intransitive predicates (VA), causative verbs (VAC), transitive verbs placed after objects (VB), transitive verbs (VC), verbs placed before locations (VCL), predicates used with both direct and indirect objects (VD), action verbs as sentence objects (VE), action verbs as predicate objects (VF), classification verbs (VG), modifiers of verbs and nouns (VH), causative modifiers (VHC), adjectives or past particles placed after objects (VI), transitive verbs to describe states (VJ), mental states and processes (VK), causative predicates (VL), and have verbs (V_2).


## 2.4. Statistical Analysis

Table1 shows that, among the 72 natural language features (26 MLS, 46 POS), statistically significant differences (p<0.05 of nonparametric Mann-Whitney U tests) between regular health resources (RHR) and vulnerable people-oriented (VPO) health resources were present in 88.9% of the entire feature set: 96.15% (25/26) of morphological-lexical-structural (MLS) features and 84.78% (39/46) of part-of-speech (POS) features. In addition to 2-sided p values, we computed Hedges’ g [34] as corrected Cohen’s d (1, 36, and 37) and 95% confidence interval of the effect size estimates. We also provided common language effect sizes (CLES), also known as probability of superiority [35–37]. In our study, CLES allowed an intuitive interpretation of the likelihood of the mean of a certain natural language feature randomly selected from VPO as higher than the mean of that feature in RHR on infectious diseases. We calculated Hedges’ g and CLES alongside commonly reported p values for two purposes: first, this was to help us interpret the result of automatic feature selection using Bayesian machine learning classifiers (Gaussian Naïve Bayes selected due to the presence of normally distributed continuous variables as features); second, this facilitated the determination of sample sizes for follow-up studies or the comparison of effects across studies.Table 1
Mann-Whitney U test.
MLS features (1–26) POS (27–72)EHR meanEHR std.RHR meanRHR std.Asymp. sig. (2-tailed)Effect sizedCohen (corrected effect size or Hedges’ g)Common language effect size CLES95% CIMann-Whitney UWilcoxon WZAverage sentences per paragraph0.6010.3453.3161.1630.0002.5880.9662.301 to 2.87452.004238.00−14.669TTR0.4610.1190.6230.0790.0001.8280.9021.568 to 2.0884389.008575.00−10.680Difficult words141.81396.38170.87235.9860.000−1.3110.823−1.558 to −1.0658769.5070897.50−6.657Low-stroke characters641.593444.264284.707140.9300.000−1.5070.857-1.758 to−1.2568108.0070236.00−7.264Middle-stroke characters120.39680.34851.81027.6950.000−1.5620.865−1.814 to−1.317357.0069485.00−7.955High-stroke characters0.0550.2290.4521.6040.0050.2770.5780.045 to 0.50814144.0018330.00−2.815Average strokes per character7.6770.2747.7110.3420.1950.1030.529−0.127 to 0.33414604.0018790.00−1.2972-character words256.637176.783114.34156.2620.000−1.5090.857−1.76 to −1.2588100.5070228.50−7.2713-character words13.00011.8248.2986.3720.008−0.6030.665−0.837 to −0.36913139.5075267.50−2.647Average words per sentences11.5924.96311.8931.9620.0180.1060.530-0.125 to 0.33613438.0017624.00−2.368Single sentences0.8830.1140.4600.1910.000−2.3760.954−2.655 to −2.098949.0063077.00−13.851Ratio of noun phrases0.2670.1040.4150.1980.0000.8100.7170.573 to 1.0468396.5012582.50−6.999Frequency of noun phrases per 10K322.36937.240314.66437.0020.028−0.2080.558−0.439 to 0.02313620.5075748.50−2.200Average idioms per sentences0.0010.0040.0120.0290.0040.4240.6180.192 to 0.65613969.0018155.00−2.894Content words392.582265.421163.07480.5380.000−1.6420.877−1.896 to −1.3876988.5069116.50−8.293Adverbs of negation2.7583.0670.9351.2280.000−1.0320.767−1.272 to −0.7929522.5071650.50−6.283Sentences with complex semantic categories25.39622.0397.6144.7580.000−1.6430.877−1.898 to −1.3886696.5068824.50−8.578Density of content words0.8280.0260.8150.0310.000−0.4330.620−0.665 to −0.211857.5073985.50−3.820Average logarithmic frequency of content words1.7380.1691.3370.1830.000−2.2250.942−2.498 to −1.9521854.0063982.00−13.009Idioms0.0770.2680.2240.5100.0090.3120.5870.081 to 0.54414168.0018354.00−2.623Pronouns41.69235.0431.4691.6520.000−2.5300.963−2.814 to −2.245620.5062748.50−14.398Personal pronouns37.86831.8380.7051.1390.000−2.5770.966−2.864 to −2.291326.0062454.00−15.334Conjunctions18.96716.44011.5206.4070.001−0.7950.713−1.031 to −0.55812507.5074635.50−3.228Positive conjunctions16.82414.1599.0005.1930.000−0.9910.758−1.23 to -0.75110808.0072936.00−4.795Negative conjunctions0.8461.5841.4401.3930.0000.4140.6150.182 to 0.64610710.5014896.50−5.070Difficult words ratio30.3845.72435.8478.1750.0000.7060.6910.471 to 0.9419400.5013586.50−6.077A3.0993.7743.0262.9690.236−0.0230.507−0.254 to 0.20714740.5018926.50−1.184VI0.1430.4850.1590.4240.3480.0370.510−0.194 to 0.26715418.5019604.50−0.938Dk0.0110.1050.0170.1300.6800.0480.514−0.183 to 0.27815919.0020105.00−0.413VG2.1542.4542.0541.9190.505−0.0490.514−0.28 to 0.18115305.0019491.00-0.666Nv9.96710.3909.2906.7110.083−0.0890.525−0.32 to 0.14214132.0018318.00−1.734Neqb0.0330.1800.0570.2770.5900.0920.526−0.138 to 0.32315810.0019996.00−0.539Cab0.1210.3900.3980.7990.0000.3770.6050.145 to 0.60913151.0017337.00−3.534I0.0330.1800.0000.0000.001−0.4060.613−0.638 to −0.17415488.0077616.00−3.414VAC0.1540.3920.0480.2150.001−0.4060.613−0.638 to −0.17414493.0076621.00−3.214Nd4.2537.8722.2983.4080.002−0.4180.616-0.65 to −0.18612764.5074892.50−3.050Nb1.4512.4230.7391.4200.000−0.4250.618−0.657 to −0.19312369.5074497.50−3.789Dfb0.0660.2910.0030.0530.000−0.4510.625−0.683 to −0.21915181.0077309.00−3.831Neu4.5056.3392.3953.2090.001−0.5210.644−0.754 to −0.28812438.0074566.00−3.347VJ10.0118.8366.4555.1250.000−0.5860.661−0.82 to −0.35211896.0074024.00−3.797VL2.9122.9461.7101.7410.001−0.5880.661−0.821 to −0.35412457.0074585.00−3.343Cba0.1100.3790.0030.0530.000−0.6020.665−0.836 to −0.36914652.5076780.50−5.125Caa12.84611.1178.7055.1500.103−0.6080.666−0.842 to −0.37414244.5076372.50−1.631VB0.9121.4880.3210.7220.000−0.6350.673−0.869 to −0.40112839.5074967.50−3.789VHC0.5271.0891.7051.9580.0000.6490.6770.415 to 0.8849100.0013286.00−6.641Da0.5161.0580.1250.3480.000−0.6860.686−0.921 to −0.45112813.0074941.00-4.647Nes2.0332.1000.9601.2780.000−0.7230.696−0.959 to -0.48810783.5072911.50−5.088Dfa3.4733.8541.6081.7520.000−0.7970.713−1.033 to −0.56110952.0073080.00−4.769VH22.15415.91313.7678.2220.000−0.8170.718−1.053 to −0.5811401.0073529.00−4.243Nf7.4188.0713.0943.9560.000−0.8520.727−1.089 to −0.6159110.5071238.50−6.401Nc7.7807.7812.6885.3030.000−0.8640.729−1.101 to −0.6278121.0070249.00−7.430Nep3.5274.3881.3271.6530.000−0.8900.736−1.128 to −0.65312279.5074407.50−3.559T1.1651.6620.2610.6490.000−0.9530.750−1.192 to −0.71510192.0072320.00−6.926Di2.9013.1690.9121.5380.000−1.0030.761−1.243 to −0.7639200.5071328.50−6.755SHI4.3964.9421.6761.7090.000−1.0060.762−1.246 to −0.76610776.0072904.00−4.917VD1.7032.3550.3750.8610.000−1.0120.763−1.252 to −0.7729587.5071715.50−7.241Cbb5.8906.4022.4152.0010.000−1.0220.765−1.263 to −0.78210325.5072453.50−5.297Neqa6.0115.9972.5372.2270.000−1.0340.768−1.274 to −0.79410036.5072164.50−5.553Na113.20977.72164.63932.6060.000−1.0650.774−1.306 to −0.82411326.5073454.50−4.308Ng6.4738.0482.0681.9800.000−1.0900.780−1.331 to −0.8488874.5071002.50−6.668V_24.5605.8791.1281.2140.000−1.1970.801−1.44 to −0.9539003.5071131.50−6.676VE7.8247.2072.6702.9130.000−1.2370.809−1.482 to −0.9938111.5070239.50−7.335DE27.39621.30011.6537.6180.000−1.3360.828−1.583 to −1.098606.5070734.50−6.816P22.57116.5709.3306.1880.000−1.4240.843−1.672 to −1.1758358.0070486.00−7.045Ncd6.7475.9601.8691.9330.000−1.5260.86−1.777 to −1.2747167.5069295.50−8.263VCL5.2425.7300.6931.0330.000−1.6560.879−1.911 to −1.4015248.5067376.50−10.615VC48.79135.98514.75910.5510.000−1.8120.900−2.071 to −1.5525672.5067800.50−9.506D44.11031.49114.4898.2470.000−1.8490.905−2.11 to −1.5895603.0067731.00−9.573VK9.7367.6882.2932.0860.000−1.8890.909−2.151 to −1.6275158.5067286.50−10.096VA11.65910.1402.0882.2140.000−1.9190.913−2.181 to −1.6564616.5066744.50−10.605VF4.1983.9640.2220.6290.000−2.1190.933−2.388 to −1.8494026.5066154.50−13.780Nh41.69235.0431.4691.6520.000−2.5300.963−2.814 to −2.245620.5062748.50−14.398The following formula shows the corrected effect size or Hedges’g:(1)Hedges′gs=Cohen′sds×1−34n1+n2−9.The following formula shows common language effect size (CLES):(2)Z=x1−x2SD12+SD22/2.When computing effect sizes (Hedges’gs) and associated probabilities of superiority (CLES), we used RER as reference class. As a result, positive values indicated that features were statistically higher in more difficult regular health resources; and negative values suggested that features were prevalent in highly accessible, vulnerable people-oriented public health advice. MLS features which had large (absolute value larger than 1), corrected effect sizes, Hedges’ gs, and CLES included the following: average sentences per paragraph (VPO: M = 0.601, SD = 0.345; RHR: M = 3.316, SD = 1.163, p<0.001, Hedges’ gs = 2.588, 95% CI [2.301, 2.874], and CLES = 0.966); type token ratio (TTR) (VPO: M = 0.461, SD = 0.119; RHR: M = 0.623, SD = 0.079, p<0.000, Hedges’ gs = 1.828, 95% CI [1.568, 2.088], and CLES = 0.902); frequency of difficult words (VPO: M = 141.813, SD = 96.381; RHR: M = 70.872, SD = 35.986, p<0.001, Hedges’ gs = −1.311, 95% CI [−1.558, −1.065], and CLES = 0.823); single sentences (VPO: M = 0.883, SD = 0.114; RHR: M = 0.460, SD = 0.191, p<0.001, Hedges’ gs = −2.376, 95% CI [−2.655, −2.098], and CLES = 0.954); pronouns (VPO: M = 41.692, SD = 35.043; RHR: M = 1.469, SD = 1.652, p<0.001, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963); personal pronouns (VPO: M = 37.868, SD = 31.838; RHR: M = 0.705, SD = 1.139, p<0.001, Hedges’ gs = −2.577, 95% CI [−2.864, −2.291], and CLES = 0.966); low-stroke characters (VPO: M = 641.593, SD = 444.264; RHR: M = 284.707, SD = 140.930, p<0.001, Hedges’ gs = −1.507, 95% CI [-1.758, −1.256], and CLES = 0.857); 2-character words (VPO: M = 256.637, SD = 176.783; RHR: M = 114.341, SD = 56.262, p<0.001, Hedges’ gs = −1.509, 95% CI [−1.76, −1.258], and CLES = 0.857); and average logarithmic frequency of content words (VPO: M = 1.738, SD = 0.169; RHR: M = 1.337, SD = 0.183, p<0.001, Hedges’ gs = −2.225, 95% CI [−2.498, −1.952], and CLES = 0.942).Part of speech (POS) features which had large (absolute value larger than 1), corrected effect sizes, Hedges’gs, and CLES included the following: Nh (pronouns) (VPO: M = 41.692, SD = 35.043; RHR: M = 1.469, SD = 1.652, p<0.000, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963); VF (action verbs as predicate objects) (VPO: M = 4.198, SD = 3.964; RHR: M = 0.222, SD = 0.629, p<0.001, Hedges’ gs = −2.119, 95% CI [−2.388, −1.849], and CLES = 0.933); VA (intransitive predicates) (VPO: M = 11.659, SD = 10.140; RHR: M = 2.088, SD = 2.214, p<0.000, Hedges’ gs = −1.919, 95% CI [−2.181, −1.656], and CLES = 0.913); VK (mental states and processes) (VPO: M = 9.736, SD = 7.688; RHR: M = 2.293, SD = 2.086, p<0.000, Hedges’ gs = −1.889, 95% CI [−2.151, −1.627], and CLES = 0.909); D (adverbs) (VPO: M = 44.110, SD = 31.491; RHR: M = 14.489, SD = 8.247, p<0.001, Hedges’ gs = −1.849, 95% CI [−2.11, −1.589], and CLES = 0.905); VC (transitive verbs) (VPO: M = 48.791, SD = 35.985; RHR: M = 14.759, SD = 10.551, p<0.001, Hedges’ gs = −1.812, 95% CI [−2.071, −1.552], and CLES = 0.900); Ncd (location names) (VPO: M = 6.747, SD = 5.960; RHR: M = 1.869, SD = 1.933, p<0.001, Hedges’ gs = −1.526, 95% CI [−1.777, −1.274], and CLES = 0.860); P (prepositions) (VPO: M = 22.571, SD = 16.570; RHR: M = 9.330, SD = 6.188, p<0.001, Hedges’ gs = −1.424, 95% CI [−1.672, −1.175], and CLES = 0.843); and V_2 (have) (VPO: M = 4.560, SD = 5.879; RHR: M = 1.128, SD = 1.214, p<0.001, Hedges’ gs = −1.197, 95% CI [−1.44, −0.953], and CLES = 0.801).


## 2.5. Gaussian Naïve Bayes

Gaussian Naïve Bayes (GNB) is a variant of Naïve Bayes which is supervised machine learning classification algorithm based on the Bayes theorem [38–42]. Various strengths of GNB are its convenience, computation speed (suitability for making real time prediction), scalability, generalisability with small data like most Bayesian machine learning classifiers, and flexibility with continuous and discrete features. In our study, the size of the training (205) and testing data (88) was relatively small. Bayesian machine learning classifiers like GNB, relevance vector machine (RVM), and multinominal Naïve Bayes (MNB) are more suitable, as they are unlikely to overfit small datasets. Furthermore, the two sets of public health resources that we collected, regular and vulnerable people-oriented sets, contained continuous features and their distributions in the Chinese translations of regular health resources followed Gaussian normal distribution (Table 2, Figure 1). As a result, GNB was selected as the most suitable machine learning classifier in our study to ensure the generalisability and reliability of the classifiers.Table 2
Continuous feature distribution in regular health resources (RHR) translated to Chinese.
Continuous MLS featuresMinMaxMeanSE.95% confidence interval for meanSD.SkewnessSE.KurtosisSE.ASPP0.1228.8572.7590.0722.6172.901.5180.1450.116−0.1940.231DCW0.6910.8960.8170.0010.8140.8200.030−0.2100.1160.3820.231RNP0.0001.2710.3850.0090.3670.4030.1920.8260.1161.2070.231ALFCW0.8082.2281.4200.0121.3971.4420.2420.3650.116-0.180.231NFNP200.00458.33316.251.765312.78319.7137.140.2680.1160.6160.231ASPC6.5058.7057.7040.0167.6737.7340.329-0.2640.1161.0590.231ASPP: average sentences per paragraph; DCW density of content words; RNP: ratio of noun phrases; ALFCW: average logarithmic frequency of content words; NFNP: normalised frequency of noun phrases; ASPC average strokes per character.Figure 1
Continuous feature distribution in regular health resources (RHR) translated into Chinese.


## 2.6. Training and Testing Machine Learning Classifiers

To train machine learning classifiers for automatic information accessibility evaluation, the total number of RHR used was 202 and the total number of VPO used was 91. Next, 73.3% (148) of RHR and 62.6% (57) of VPO were used as the training data and the remaining texts (54 RHR and 34 VPO) were used as testing data. We applied 5-fold cross-validation with the training data to produce the mean and standard deviation of area of curve (AUC) of the GNB classifier. Review of the model performance was on the remaining 30% test data in terms of AUC, accuracy, sensitivity, specificity, and macro F1 (Table3).Table 3
Performance of Gaussian Naïve Bayes (GNB) classifiers with different feature sets.
ModelTechniquesTraining (5-fold CV)TestingAUC mean (SD)AUCAccuracyMacro F1SensitivitySpecificity1MLS + POS full (69)0.971 (0.0212)0.9400.9210.9920.9440.88242MLS + POS jointly optimised (6)0.998 (0.0026)0.9930.9400.9430.9630.91183MLS full (26 features)0.997 (0.003)1.00.9660.9631.00.91184MLS optimised (2 features)0.998 (0.004)1.00.9430.9381.00.85295POS full (46 features)0.959 (0.0238)0.9070.8520.8430.8890.79416POS optimised (8 features)0.982 (0.0166)0.9680.9550.9511.00.88247MLS + POS separately optimised (10)1.0 (0)1.00.9550.9511.00.88248Refined MLS + POS separately optimised (2)0.995 (0.0079)0.9990.9890.9880.9821.0


## 2.7. Classifier Optimisation

High dimensional features can reduce the performance of machine learning classifiers due to the forced inclusion of irrelevant parameters in the model. To counter the issue of classifier underperformance caused by the presence of redundant features, we applied different classifier optimisation techniques to reduce the original features collected. First, we applied integral optimisation by selecting the optimised feature set from the combined MLS and POS features (72 in total). This led to a combinedly optimised feature set of 6 features (around 8% of the original total features): average sentences per paragraph (ASPP) (p<0.000, Hedges’ gs = 2.588, 95% CI [2.301, 2.874], and CLES = 0.966), personal pronouns (p<0.001, Hedges’ gs = −2.577, 95% CI [−2.864, −2.291], and CLES = 0.966), Di (tense markers) (p<0.001, Hedges’ gs = −1.003, 95% CI [−1.243, −0.763], and CLES = 0.761), Nd (time adverbs) (p=0.002, Hedges’ gs = −0.418, 95% CI [−0.65, −0.186], and CLES = 0.616), VF (action verbs as predicate objects) (p<0.001, Hedges’ gs = −2.119, 95% CI [−2.388, −1.849], and CLES = 0.933), and V_2 (have verbs) (p<0.001, Hedges’ gs = −1.197, 95% CI [−1.44, −0.953], and CLES = 0.801).These optimised features were also those with the most significant statistical differences (indicated byp values, corrected effect size Hedges’ g, common language effect sizes CLES) between regular and vulnerable people-oriented health resources (Table 1). Next, we applied feature optimisation in the two sets of MLS (26) and POS (46) features separately. This led to an optimised MLS feature set of 2 features only (7.7% of the total MLS features): ASPP and ALFCW (p<0.001, Hedges’ gs = −2.225, 95% CI [−2.498, −1.952], and CLES = 0.942). The optimised POS feature set of 8 features (17.4% of the total POS features): A (nonpredicate adjectives) (p<0.000, Hedges’ gs = −0.023, 95% CI [−0.254, 0.207], and CLES = 0.507), Cbb (correlative conjunctions) (p<0.001, Hedges’ gs = −1.022, 95% CI [−1.263, −0.782], and CLES = 0.765), Dfa (adverbs of degree before verbs) (p<0.001, Hedges’ go = −0.797, 95% CI [−1.033, −0.561], and CLES = 0.713), Nd (time adverbs) (see above), Ng (postpositions) (p<0.001, Hedges’ gs = −1.090, 95% CI [−1.331, −0.848], and CLES = 0.780), Nh (pronouns) (p<0.001, Hedges’ gs = −2.530, 95% CI [−2.814, −2.245], and CLES = 0.963), VCL (verbs placed before locations) (p<0.001, Hedges’ gs = −1.656, 95% CI [−1.911, −1.401], and CLES = 0.879), and VHC (causative modifiers) (p<0.001, Hedges’ gs = 0.649, 95% CI [0.415, 0.884], and CLES = 0.677). For both integral and parallel classifier optimisations, we used backforward feature elimination known as recursive feature elimination (RFE) with support vector machine as base estimator. Maximal validation accuracy/minimal classification errors were used as the feature optimisation criteria (Table 4 and Figure 2).Table 4
Criteria of classifier optimisation.
TechniquesCross-validation accuracy (CVA)Minimal classification error (MCE) = 1-CVAJoint optimisation (6)0.98530.0147Optimised MLS features (2)0.99020.0098Optimised POS features (8)0.98540.0146Figure 2
Recursive feature elimination using SVM as base estimator.


## 3. Results

Table3 shows the performance of GNB classifiers using different feature sets. Overall, Bayesian classifiers using optimised features outperformed those using original, larger feature sets. For example, on the testing data, the integrally optimised GNB with 2 MLS and 6 POS features achieved a higher AUC (0.993), sensitivity (0.963), specificity (0.9118), and accuracy (0.940) than the classifier using the full feature set which had AUC (0.940), sensitivity (0.944), specificity (0.8824), and accuracy (0.921). The separately optimised POS feature set (8 POS features) achieved a higher AUC (0.968), sensitivity (1.0), specificity (0.8824), and accuracy (0.955) than the original POS feature set (46 POS features) (AUC = 0.852, sensitivity = 0.889, specificity = 0.7941, and accuracy = 0.852). After comparing the 3 optimised feature sets: MLS/POS jointly optimised (6), MLS optimised (2), and POS optimised (8), we added the 2 sets of separately optimised features of 2 MLS features (ASPP, ALFCW) and 8 POS features (A, Cbb, Dfa, Nd, Ng, Nh, VCL, and VHC) together and further refined the feature set to 2 features only: ALFCW and Nh (pronouns) using the same backward elimination RFE_SVM procedure. This resulted in a highly simplified model (model 8) which achieved largely comparable performance to the best-performing model, the one which integrated the 2 separately optimised features (model 7). Optimised model 8 achieved higher specificity (1.0) than the model 7 (0.8824), which indicated better detection of public health resources and advice suitable for use under health emergencies and crises for maximal social accessibility.


### 3.1. Bayesian Probabilistic Outputs

The outputs of Bayesian machine learning classifiers are in the form of a probability of belonging to the regular public health advice training data. In our study, for the best-performing classifier (model 8: refined MLS and POS separately optimised: 2 features: ALFCW and Nh), average mean output (probability) was 0.9557 (SD = 0.152; range: 0.02, 1; 95% CI: 0.915, 0.996) for regular public health resources and 0.022 (SD = 0.069; range: 0, 0.28; 95% CI: −0.00118, 0.04518) for vulnerable people-oriented public health advice. The differences among public health advice on infectious diseases (translated to Chinese) in terms of public accessibility (high: vulnerable people-oriented resources; low: regular resources) were statistically significant (p<0.001, Hedges’ gs = 7.367, 95% CI: 6.197, 8.536, and CLES = 1). Figure 3 is a histogram which shows the number of regular (restricted accessibility) and vulnerable people-oriented (high accessibility) health pieces of advice that fell into each 10% probability bin based on the GNB outputs. One hundred percent of vulnerable people-oriented health advice was assigned a probability of highly accessible health resource equal to or smaller than 50% (specificity = 1); and 98.15% of regular public resources were assigned a probability of public advice of restricted/low accessibility larger than 50% (sensitivity = 98.15%). Around 2% of regular public health resources were misclassified as highly accessible information and advice for the public.Figure 3
Percentage of vulnerable people-oriented (VPO) materials and regular health resources (RHR) assigned by GNB to 10 probability bins.


### 3.2. Thresholds and Positive/Negative Likelihood Ratios (LR+)

Although it is intuitive to use 0.5 as the probability threshold (Figure3), this is not the case in real life scenarios, because the criterion for a meaningful cut-off depends on the desired pair of sensitivity and specificity or the diagnostic utility of the research instrument. In our study, higher classifier sensitivity is indicative of higher precision with the prediction of regular public health advice (from WHO resources) which have restricted public accessibility; and higher classifier specificity implies increased accuracy with the detection vulnerable people-oriented health advice (from 3 national health authorities) which were designed by experienced health professionals to maximally enhance the language, cognitive accessibility, informational actionability, and communicative effectiveness of these emergency pieces of advice among diverse vulnerable populations with limited education, health literary, socioeconomic abilities, and varying intellectual/cognitive capabilities.Table5 shows various probability cut-offs and their associated sensitivity and specificity pairs using the best-performing GNB classifier that we developed using 2 features only (ALFCW: average logarithmic frequency of content words; Nh: pronouns). It shows that when setting the cut-offs lower than 0.1, sensitivity was the highest (0.9815) and specificity was 0.9118. This means that if this machine learning system were used to assist with public health advice design and dissemination, less than 2% of public health advice with restricted accessibility would be misclassified as accessible information and less than 10% of highly accessible materials would be misclassified as ineffective public health advice, potentially increasing the budgetary burden to hire experts to review health resources or extend the timeframe of releasing information to the public. When increasing the threshold from 0.1 to 0.2, sensitivity remained unchanged, and specificity increased. From 0.2 to 0.5, specificity (cost related) remained the same, but sensitivity deceased from 0.98 to 0.96, suggesting that 2% of health public with limited accessibility to vulnerable populations would be misclassified as suitable health advice and information. In high-risk scenarios such as the outbreaks of highly infectious diseases among some of the most deprived communities, this misjudgement in health advice planning can be very costly, as vulnerable people would be given less effective, accessible, and protective health advice and information.Table 5
Thresholds, positive/negative likelihood ratio, and 95% CI of the best-performing GNB classifier on the test data.
Probability thresholdsSensitivity (95% CI)Specificity (95% CI)Positive likelihood ratio (LR+) (95% CI)Negative likelihood ratio (LR-) (95% CI)0.10.982 (0.95, 1.0)0.9118 (0.82, 1.0)11.12 (3.77, 32.79)0.02031 (0.003, 0.142)0.20.982 (0.95, 1.0)0.941 (0.86, 1.0)16.69 (4.35, 64.04)0.01968 (0.003, 0.137)0.50.963 (0.91, 1.0)0.941 (0.86, 1.0)16.37 (4.26, 62.87)0.039 (0.010, 0.154)0.60.963 (0.91, 1.0)1.0 (1.0, 1.0)Infinity0.037 (0.010, 0.144)0.70.9444 (0.883, 1.0)1.0 (1.0, 1.0)Infinity0.0556 (0.019, 0.167)0.90.9074 (0.830, 0.985)1.0 (1.0, 1.0)Infinity0.09259 (0.0402, 0.213)Machine learning classifier: refined GNB classifier (with ALFCW and Nh features)Sensitivity continued to decrease when thresholds raised higher than 0.5, despite the fact that specificity reached 1. Positive likelihood ratio (LR+) (the ratio between sensitivity and false positivity) is another measurement of diagnostic utility. A LR + larger than 10 indicates a very large effect on posttest probability of disease or in the case of our study lack of accessibility of public health advice among some of our most vulnerable communities and people, who need accessible, actionable public health advice most. In our study, using the best-performing Bayesian classifier, the highest LR+ 16.685 (95% CI: 4.35, 64.04) was reached when setting the threshold of probability at 0.2 (sensitivity = 0.982, 95% CI: 0.95, 1; specificity = 0.941, 95% CI: 0.86, 1). This represents the safest (highest sensitivity) and the most cost-effective (lowest budget investment in expert hire) model of machine learning-assisted predictive design of accessible public health advice and information for vulnerable people and populations. We hope that this promising result based on machine learning development using low-cost, relatively easy-to-obtain natural language tools would provide more support for evidence-based, user-oriented policy making around inclusive, accessible public health advice design and communication under both regular and emergency circumstances.


## 3.1. Bayesian Probabilistic Outputs

The outputs of Bayesian machine learning classifiers are in the form of a probability of belonging to the regular public health advice training data. In our study, for the best-performing classifier (model 8: refined MLS and POS separately optimised: 2 features: ALFCW and Nh), average mean output (probability) was 0.9557 (SD = 0.152; range: 0.02, 1; 95% CI: 0.915, 0.996) for regular public health resources and 0.022 (SD = 0.069; range: 0, 0.28; 95% CI: −0.00118, 0.04518) for vulnerable people-oriented public health advice. The differences among public health advice on infectious diseases (translated to Chinese) in terms of public accessibility (high: vulnerable people-oriented resources; low: regular resources) were statistically significant (p<0.001, Hedges’ gs = 7.367, 95% CI: 6.197, 8.536, and CLES = 1). Figure 3 is a histogram which shows the number of regular (restricted accessibility) and vulnerable people-oriented (high accessibility) health pieces of advice that fell into each 10% probability bin based on the GNB outputs. One hundred percent of vulnerable people-oriented health advice was assigned a probability of highly accessible health resource equal to or smaller than 50% (specificity = 1); and 98.15% of regular public resources were assigned a probability of public advice of restricted/low accessibility larger than 50% (sensitivity = 98.15%). Around 2% of regular public health resources were misclassified as highly accessible information and advice for the public.Figure 3
Percentage of vulnerable people-oriented (VPO) materials and regular health resources (RHR) assigned by GNB to 10 probability bins.


## 3.2. Thresholds and Positive/Negative Likelihood Ratios (LR+)

Although it is intuitive to use 0.5 as the probability threshold (Figure3), this is not the case in real life scenarios, because the criterion for a meaningful cut-off depends on the desired pair of sensitivity and specificity or the diagnostic utility of the research instrument. In our study, higher classifier sensitivity is indicative of higher precision with the prediction of regular public health advice (from WHO resources) which have restricted public accessibility; and higher classifier specificity implies increased accuracy with the detection vulnerable people-oriented health advice (from 3 national health authorities) which were designed by experienced health professionals to maximally enhance the language, cognitive accessibility, informational actionability, and communicative effectiveness of these emergency pieces of advice among diverse vulnerable populations with limited education, health literary, socioeconomic abilities, and varying intellectual/cognitive capabilities.Table5 shows various probability cut-offs and their associated sensitivity and specificity pairs using the best-performing GNB classifier that we developed using 2 features only (ALFCW: average logarithmic frequency of content words; Nh: pronouns). It shows that when setting the cut-offs lower than 0.1, sensitivity was the highest (0.9815) and specificity was 0.9118. This means that if this machine learning system were used to assist with public health advice design and dissemination, less than 2% of public health advice with restricted accessibility would be misclassified as accessible information and less than 10% of highly accessible materials would be misclassified as ineffective public health advice, potentially increasing the budgetary burden to hire experts to review health resources or extend the timeframe of releasing information to the public. When increasing the threshold from 0.1 to 0.2, sensitivity remained unchanged, and specificity increased. From 0.2 to 0.5, specificity (cost related) remained the same, but sensitivity deceased from 0.98 to 0.96, suggesting that 2% of health public with limited accessibility to vulnerable populations would be misclassified as suitable health advice and information. In high-risk scenarios such as the outbreaks of highly infectious diseases among some of the most deprived communities, this misjudgement in health advice planning can be very costly, as vulnerable people would be given less effective, accessible, and protective health advice and information.Table 5
Thresholds, positive/negative likelihood ratio, and 95% CI of the best-performing GNB classifier on the test data.
Probability thresholdsSensitivity (95% CI)Specificity (95% CI)Positive likelihood ratio (LR+) (95% CI)Negative likelihood ratio (LR-) (95% CI)0.10.982 (0.95, 1.0)0.9118 (0.82, 1.0)11.12 (3.77, 32.79)0.02031 (0.003, 0.142)0.20.982 (0.95, 1.0)0.941 (0.86, 1.0)16.69 (4.35, 64.04)0.01968 (0.003, 0.137)0.50.963 (0.91, 1.0)0.941 (0.86, 1.0)16.37 (4.26, 62.87)0.039 (0.010, 0.154)0.60.963 (0.91, 1.0)1.0 (1.0, 1.0)Infinity0.037 (0.010, 0.144)0.70.9444 (0.883, 1.0)1.0 (1.0, 1.0)Infinity0.0556 (0.019, 0.167)0.90.9074 (0.830, 0.985)1.0 (1.0, 1.0)Infinity0.09259 (0.0402, 0.213)Machine learning classifier: refined GNB classifier (with ALFCW and Nh features)Sensitivity continued to decrease when thresholds raised higher than 0.5, despite the fact that specificity reached 1. Positive likelihood ratio (LR+) (the ratio between sensitivity and false positivity) is another measurement of diagnostic utility. A LR + larger than 10 indicates a very large effect on posttest probability of disease or in the case of our study lack of accessibility of public health advice among some of our most vulnerable communities and people, who need accessible, actionable public health advice most. In our study, using the best-performing Bayesian classifier, the highest LR+ 16.685 (95% CI: 4.35, 64.04) was reached when setting the threshold of probability at 0.2 (sensitivity = 0.982, 95% CI: 0.95, 1; specificity = 0.941, 95% CI: 0.86, 1). This represents the safest (highest sensitivity) and the most cost-effective (lowest budget investment in expert hire) model of machine learning-assisted predictive design of accessible public health advice and information for vulnerable people and populations. We hope that this promising result based on machine learning development using low-cost, relatively easy-to-obtain natural language tools would provide more support for evidence-based, user-oriented policy making around inclusive, accessible public health advice design and communication under both regular and emergency circumstances.


## 4. Discussion

### 4.1. Retrospective Assessment of Accessibility of Public Health Advice in Other Languages

Another major strength of our study was that we translated the best practices of accessible health advice on infectious diseases developed by English health authorities during the pandemic into adaptive analytical instruments (low-cost, fast-to-build machine learning algorithms) that can be used for the retrospective assessment of existing health advice in other languages. This was achieved by translating the English materials that we collected to the Chinese language using the forward and backward translation method recommended by WHO and comprehensively annotated the translations using Chinese natural language processing tools to allow automatic feature engineering and machine learning classifier development. Table5 and Figure 4 illustrate how probabilistic outputs might assist with decision making among policy makers and health/medical professionals when designing public health advice for vulnerable people and communities speaking Chinese or migrants, foreigners living in China or greater China regions. Next, we adapted the machine learning classifier to a convenient statistical tool which can be effectively used in the clinic by health/medical professionals and educators to assess whether a certain piece of health advice on infectious diseases is accessible to vulnerable people (both Chinese and migrants) with limited education and health literacy or Chinese proficiency while living in Chinese speaking regions. We fitted a binary Logistic Regression (LR) model with the 70% training data collected. The model contained two independent variables which were borrowed from the best-performing classifier: ALFCW (average logarithmic frequency of content words) and Nh (number of pronouns). The fitted model with the regression coefficients was shown as follows.Figure 4
Receiver operating characteristic (ROC) of Gaussian Naïve Bayes classifiers.The following is the binary regression formula:(3)scoreLR=−2.67×ALFCW−0.663×Νh+7.536.We then used the Sigmoid Function to scale the scoreLR to the region of [0, 1]:(4)Sigmoidx=11+e−x.The following formula shows Chinese health advice accessibility assessment tool (CHAAAT):(5)score=100×Sigmoid−2.67×ALFCW−0.663×Nh+7.536.We examined the performance of the CHAAAT formula on the remaining 30% test data which contained 34 highly accessible COVID-19 prevention resources translated from 3 national health authorities in the USA, Australia, and UK and 54 regular health resources on infectious diseases developed by the WHO for the public. Average mean output (transformed score using the Sigmoid Function) was 5.8958 (SD = 18.128, range: 1.51908E-32, 73.5, and 95% CI: −0.194, 12) for highly accessible, vulnerable people-oriented public health advice and resources and 88.363 (SD = 2.34, range: 12.119, 99.466, and 95% CI: 83.665, 93.062) for regular WHO public health resources with limited accessibility among vulnerable people with low education, limited health literacy, and limited Chinese proficiency such as foreigner migrants or people with cognitive impairments.Differences between the two sets of test data measured by CHAAAT formula were statistically very significant:p<0.001, Hedges’ gs = 7.248, 95% CI: 6.094, 8.401, and CLES = 1. These effect sizes were comparable to those (p<0.000, Hedges’ gs = 7.367, 95% CI: 6.197, 8.536, and CLES = 1) of probabilistic outputs of the best-performing GNB classifier using the same feature pair: AFLCW and Nh. Figure 5 shows the number of regular (restricted accessibility) and vulnerable people-oriented (high accessibility) health pieces of advice that fell into each 10-score bin based on the transformed scores computed using the CHAAAT regression formula. Ninety-four percent of vulnerable people-oriented health advice were assigned a transformed score of highly accessible health resource smaller than 50 (specificity = 94.12%) and 96.3% of regular public resources were assigned an indicative score of public advice of restricted/low accessibility equal to or larger than 50 (sensitivity = 96.3%).Figure 5
Percentage of vulnerable people-oriented (VPO) or regular health resources (RHR) assigned by CHAAAT to each 10-score bin.Around 4% regular (requiring higher literacy and Chinese proficiency) resources were misclassified as highly accessible information and advice for vulnerable communities and people living in Chinese speaking regions. Since the 4% errors can still result in a costly overestimate of the accessibility or usability of public health advice among vulnerable populations who require highly accessible (simple, actionable, and implementable) public health information and advice, like with machine learning classifiers, we could adjust the thresholds of the CHAAAT formula to obtain the desired sensitivity and specificity pair according to the practical needs in the clinic. Table6 shows that, like the threshold of the best-performing GNB classifier, setting the threshold of the transformed score (Formula (5)) to 29.906 will allow the regression calculator to achieve the same sensitivity and specificity as the GNB classifier. According to practical needs for accessible public health advice under different circumstances, further decreasing the threshold will lead to increased sensitivity of the assessment tool, which might be needed when the health risks being communicated are complex, and the vulnerability of the target populations is high.Table 6
Thresholds of transformed scores of CHAAAT regression formula.
Score thresholdsSensitivitySpecificityScore thresholdsSensitivitySpecificity1.8281.0000.85364.1180.9260.9417.6851.0000.88266.2220.9070.94116.4620.9810.88268.5420.8890.94125.1440.9810.91270.9640.8700.94129.9060.9810.94172.9940.8700.97143.1600.9630.94173.3270.8520.97159.8990.9440.94176.0010.8521.000The design and assessment of population-oriented public health advice are high-stakes activities which require highly precise, reliable research tools to support informed, evidence-based public health policy planning and delivery. Our study showed that valuable experiences gained in the design of accessible public advice as triggered by the coronavirus pandemic can be translated to useful, much-needed research instruments and tools to support the design of public health advice for any future health events or crises. Furthermore, we developed assessment tools adaptable to other languages to facilitate international benchmarking and support global public health policy planning around accessible health risk communication and public engagement. The limitation of our study is that we used Chinese as an illustrating example which is a distinct language from English. However, the underlying techniques and methods that we discussed and demonstrated can be conveniently modified for other languages or writing systems, especially underresourced languages such as African languages and minority languages, as we developed high performing Bayesian machine learning classifiers based on small datasets. For illiterate populations, our methods need further adaptation for the assessment of oral public advice and information.


## 4.1. Retrospective Assessment of Accessibility of Public Health Advice in Other Languages

Another major strength of our study was that we translated the best practices of accessible health advice on infectious diseases developed by English health authorities during the pandemic into adaptive analytical instruments (low-cost, fast-to-build machine learning algorithms) that can be used for the retrospective assessment of existing health advice in other languages. This was achieved by translating the English materials that we collected to the Chinese language using the forward and backward translation method recommended by WHO and comprehensively annotated the translations using Chinese natural language processing tools to allow automatic feature engineering and machine learning classifier development. Table5 and Figure 4 illustrate how probabilistic outputs might assist with decision making among policy makers and health/medical professionals when designing public health advice for vulnerable people and communities speaking Chinese or migrants, foreigners living in China or greater China regions. Next, we adapted the machine learning classifier to a convenient statistical tool which can be effectively used in the clinic by health/medical professionals and educators to assess whether a certain piece of health advice on infectious diseases is accessible to vulnerable people (both Chinese and migrants) with limited education and health literacy or Chinese proficiency while living in Chinese speaking regions. We fitted a binary Logistic Regression (LR) model with the 70% training data collected. The model contained two independent variables which were borrowed from the best-performing classifier: ALFCW (average logarithmic frequency of content words) and Nh (number of pronouns). The fitted model with the regression coefficients was shown as follows.Figure 4
Receiver operating characteristic (ROC) of Gaussian Naïve Bayes classifiers.The following is the binary regression formula:(3)scoreLR=−2.67×ALFCW−0.663×Νh+7.536.We then used the Sigmoid Function to scale the scoreLR to the region of [0, 1]:(4)Sigmoidx=11+e−x.The following formula shows Chinese health advice accessibility assessment tool (CHAAAT):(5)score=100×Sigmoid−2.67×ALFCW−0.663×Nh+7.536.We examined the performance of the CHAAAT formula on the remaining 30% test data which contained 34 highly accessible COVID-19 prevention resources translated from 3 national health authorities in the USA, Australia, and UK and 54 regular health resources on infectious diseases developed by the WHO for the public. Average mean output (transformed score using the Sigmoid Function) was 5.8958 (SD = 18.128, range: 1.51908E-32, 73.5, and 95% CI: −0.194, 12) for highly accessible, vulnerable people-oriented public health advice and resources and 88.363 (SD = 2.34, range: 12.119, 99.466, and 95% CI: 83.665, 93.062) for regular WHO public health resources with limited accessibility among vulnerable people with low education, limited health literacy, and limited Chinese proficiency such as foreigner migrants or people with cognitive impairments.Differences between the two sets of test data measured by CHAAAT formula were statistically very significant:p<0.001, Hedges’ gs = 7.248, 95% CI: 6.094, 8.401, and CLES = 1. These effect sizes were comparable to those (p<0.000, Hedges’ gs = 7.367, 95% CI: 6.197, 8.536, and CLES = 1) of probabilistic outputs of the best-performing GNB classifier using the same feature pair: AFLCW and Nh. Figure 5 shows the number of regular (restricted accessibility) and vulnerable people-oriented (high accessibility) health pieces of advice that fell into each 10-score bin based on the transformed scores computed using the CHAAAT regression formula. Ninety-four percent of vulnerable people-oriented health advice were assigned a transformed score of highly accessible health resource smaller than 50 (specificity = 94.12%) and 96.3% of regular public resources were assigned an indicative score of public advice of restricted/low accessibility equal to or larger than 50 (sensitivity = 96.3%).Figure 5
Percentage of vulnerable people-oriented (VPO) or regular health resources (RHR) assigned by CHAAAT to each 10-score bin.Around 4% regular (requiring higher literacy and Chinese proficiency) resources were misclassified as highly accessible information and advice for vulnerable communities and people living in Chinese speaking regions. Since the 4% errors can still result in a costly overestimate of the accessibility or usability of public health advice among vulnerable populations who require highly accessible (simple, actionable, and implementable) public health information and advice, like with machine learning classifiers, we could adjust the thresholds of the CHAAAT formula to obtain the desired sensitivity and specificity pair according to the practical needs in the clinic. Table6 shows that, like the threshold of the best-performing GNB classifier, setting the threshold of the transformed score (Formula (5)) to 29.906 will allow the regression calculator to achieve the same sensitivity and specificity as the GNB classifier. According to practical needs for accessible public health advice under different circumstances, further decreasing the threshold will lead to increased sensitivity of the assessment tool, which might be needed when the health risks being communicated are complex, and the vulnerability of the target populations is high.Table 6
Thresholds of transformed scores of CHAAAT regression formula.
Score thresholdsSensitivitySpecificityScore thresholdsSensitivitySpecificity1.8281.0000.85364.1180.9260.9417.6851.0000.88266.2220.9070.94116.4620.9810.88268.5420.8890.94125.1440.9810.91270.9640.8700.94129.9060.9810.94172.9940.8700.97143.1600.9630.94173.3270.8520.97159.8990.9440.94176.0010.8521.000The design and assessment of population-oriented public health advice are high-stakes activities which require highly precise, reliable research tools to support informed, evidence-based public health policy planning and delivery. Our study showed that valuable experiences gained in the design of accessible public advice as triggered by the coronavirus pandemic can be translated to useful, much-needed research instruments and tools to support the design of public health advice for any future health events or crises. Furthermore, we developed assessment tools adaptable to other languages to facilitate international benchmarking and support global public health policy planning around accessible health risk communication and public engagement. The limitation of our study is that we used Chinese as an illustrating example which is a distinct language from English. However, the underlying techniques and methods that we discussed and demonstrated can be conveniently modified for other languages or writing systems, especially underresourced languages such as African languages and minority languages, as we developed high performing Bayesian machine learning classifiers based on small datasets. For illiterate populations, our methods need further adaptation for the assessment of oral public advice and information.



---

*Source: 1916690-2021-12-17.xml*