GENERATING AND TESTING HYPOTHESES AND UPDATING A PREDICTIVE MODEL OF PANDEMIC INFECTIONS

A system that generates and testing hypotheses about the spread of pandemic infections and updates a predictive model of the disease to reflect newly identified hypotheses and/or determinations that previously identified hypotheses are no longer suggested by the latest data. By coding, organizing, and sorting newly received data in a non-biased way, the disclosed system rapidly identifies new insights about the disease (and evidence challenging previously held assumptions about that disease) that can be communicated to public health officials, policymakers, and clinicians to better understand the nature of the disease and the effectiveness of clinical and public health interventions that are being used—or may be used—to control and treat the disease. The disclosed system also uses those new hypotheses (and evidence that previous hypotheses can be discounted) to adjust the predictive model to more accurately reflect the latest understanding of the disease and the effectiveness of potential interventions.

FEDERAL FUNDING

BACKGROUND

Early in the spread of newly recognized or emergent pathogens, disease characteristics are often unknown and poorly understood in terms of transmission, agent durability in the environment, inoculation/infectious dose, host susceptibility, and—importantly— effective medical and public health control and intervention measures. Much in the same way as the recognized phases of other natural disasters, a hallmark of newly emergent diseases is that early information is often confused, limited, incorrect, and skewed. In the case of SARS-CoV-2, for example, early information on COVID-19 implicated highest risk for older adults with comorbid conditions, producing the assumption/implication that younger persons were not at risk for severe disease and death. That assumption has proven tragically untrue.

Other assumptions that have proven untrue over time in the COVID-19 experience include that the disease is mild in adults under the age of 65, that COVID-19 is transmitted by droplets and not aerosols, the efficacy of masks in preventing transmission, and that vaccinated persons do not shed meaningful concentrations of virus and, therefore, do not participate in the disease transmission cycle.1Reliance on those incorrect assumptions has proven to be a significant impediment to effective control and management of the COVID-19 pandemic in the United States and elsewhere.1Barker, Hartley, Beck et al, Rethinking Herd Immunity Managing the Covid-19 Pandemic in a Dynamic Biological and Behavioral Environment, NEJM Catalyst, 10 Sep. 2021, https://catalyst.nejm.org/doi/full/10.1056/CAT.21.0288

For newly emerged infections, what is learned early from a small number of observations often influences decisions in other circumstances incorrectly. In the case of COVID-19, for example, the Wuhan experience suggested controls that apparently worked in China but were later shown to be inaccurate.2Nevertheless, the mistaken belief that those controls were effective influenced U.S. policies and thinking regarding interventions.32See, e.g., Pan et al, Association of Public Health Interventions with the Epidemiology of the COVID-19 Outbreak in Wuhan, China, JAMA, 19 May 2020, https://pubmed.ncbi.nlm.nih.gov/32275295/; Hartley and Perencevich, Public Health Interventions for COVID-19: Emerging Evidence and Implications for an Evolving Public Health Crisis, JAMA, 19 May 2020, https://pubmed.ncbi.nlm nih gov/32275299/3Auger, Shah, Richardson, Hartley et al, Association Between Statewide School Closure and COVID-19 Incidence and Mortality in the US, JAMA, 1 Sep. 2020, https://pubmed.ncbi.nlm.nih.gov/32745200/

Accordingly, to identify effective medical and public health control and intervention measures in the emergent stages of each pandemic, it is vitally important to correctly ascertain the characteristics of a novel disease and the effectiveness of each measure.

Additionally, the coronavirus pandemic revealed the extent to which policymakers rely on predictive models, which attempt to predict the future of virus spread, to decide what actions are best to take.4Although better than relying on intuition or flying completely blind into a crisis, predictive models rely on assumptions about disease characteristics and the effectiveness of public health and medical interventions. For instance, of the 28 probabilistic forecasts evaluated in a recent paper, seven made explicit assumptions that social distancing and other behavioral patterns would change over the prediction period.5As additional information is collected over time, the data may challenge or contradict some of the assumptions used by those predictive models to predict future virus spread. Additionally, emerging data may suggest additional elements that, if incorporated into the predictive model, would improve the accuracy of the predictive model. Reliance on COVID-19 models that failed to adjust in view of new4Sample, I., Coronavirus exposes the problems and pitfalls of modelling, The Guardian 2020 Mar. 25, https://www.theguardian.com/science/2020/mar/25/coronavirus-exposes-the-problems-and-pitfalls-of-modelling5Cramer et al, Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States, PNAS, 8 Apr. 2022, https://doi.org/10.1073/pnas.2113561119 evidence may have led to several missteps.6For example, some early COVID-19 models did not consider the possible effects of mass “test, trace, and isolate” strategies or potential staff shortages on transmission dynamics.7Including those factors in predictive models may have led to an earlier focus on testing capacity and providing appropriate protective equipment for frontline workers. Accordingly, correctly ascertaining the characteristics of a novel disease and the effectiveness of interventions is also vitally important when modeling the future spread of the novel disease.6Ahmed, N., Covid-19 special investigation, part 1, The politicized science that nudged the Johnson government to safeguard the economy over British lives, Byline Times 2020 Mar. 23, https://bylinetimes.com/2020/03/23/covid-19-special-investigation-part-one-the-politicised-science-that-nudged-the-johnson-government-to-safeguard-the-economy-over-british-lives/7Sridhar et al., Modelling the pandemic, BMJ 21 Apr. 2020, https://doi.org/10.1136/bmj.m1567

Newly emergent “learning health systems” and “learning networks,” which rapidly learn from data and disseminate learning to system stakeholders, are regarded as an important advance in US healthcare.8That advance has the potential to rapidly disseminate critical information in emergent pandemic situations, but also runs the risk of promulgating incorrect conclusions and information. Currently lacking in the art but critically needed—especially in the case of pandemics9—is the ability to learn rapidly in a non-biased way, revise that learning as new data are observed, and rapidly communicate insights to stakeholders such as hospitals and public health departments throughout medicine. More so, that ability must support the identification of new insights that challenge or contradict previous conclusions and assumptions as additional information is obtained.8Ardura, Hartley, Dandoy et al, Addressing the Impact of the Coronavirus Disease 2019 (COVID-19) Pandemic on Hematopoietic Cell Transplantation: Learning Networks as a Means for Sharing Best Practices, Biol Blood Marrow Transplant, July 2020, https://pubmed.ncbi.nlm.nih.gov/32339662/9Beck, Hartley, Kahn et al, Rapid, Bottom-Up Design of a Regional Learning Health System in Response to COVID-19, Mayo Clin Proc, 16 Feb. 2021, https://pubmed.ncbi.nlm nih gov/33714596/; Hartley, Beck, Seid et al, 16. Multi-sector Situational Awareness in the COVID-19 Pandemic: The Southwest Ohio Experience, 2021, https://www.springerprofessional.de/en/multi-sector-situational-awareness-in-the-covid-19-pandemic-the-/19551082

An especially important need exists in the area of pandemic detection and early warning,10currently a major focus of interest.11That can be seen in the case of Project Argus,12which examined massive amounts of unstructured, multilingual textual data to detect leading indicators of infectious disease outbreaks globally. ProjectArgusmade observations of disease or potential disease incidents, enabling human analysts to form hypotheses regarding the correct interpretation of such events on an ad hoc basis. Importantly, those hypotheses often changed over the course of days to weeks to months as events evolved and spread, and as additional data became available. More recently, systematic machine methods were developed that generate and rank a universe of relevant hypotheses.13However, there was no systematic way to test the assumptions made earlier in the assessment of a novel disease and determine whether, as data emerge over time, the newly received data challenge or contradict those assumptions.10Nelson, Brownstein, and Hartley, Event-based biosurveillance of respiratory disease in Mexico, 2007-2009: connection to the 2009 influenza A(H1N1) pandemic?, Euro Surveill, 29 Jul. 2010, https://pubmed.ncbi.nlm nih gov/20684815/; Hartley, Nelson, Arthur et al, An overview of internet biosurveillance, Clin Microbiol Infect, 19 Nov. 2012, https://pubmed.ncbi.nlm.nih.gov/23789639/11CDC Stands Up New Disease Forecasting Center, https://www.cdc.gov/media/releases/2021/p0818-disease-forecasting-center.html12Hartley et al, Landscape of international event-based biosurveillance, Emerg Health Threats, 19 Feb. 2010, https://pubmed.ncbi.nlm nih gov/22460393/; U.S. Pat. No. 10,002,034 to Li, Torii, Hartley and Nelson13e.g., U.S. Pat. Nos. 10,521,727 and 11,106,878 to Frieder and Hartley; Parker, Wei, Yates, Frieder and Goharian, A framework for detecting public health trends with Twitter, Proceedings of the 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 25 Aug. 2013, https://dtacm.org/doi/10.1145/2492517.2492544

The uncertainty associated with early assumptions regarding pandemics is often not recognized. That uncertainty may stem from imprecise reporting, unintentional (and, at times, intentional) misleading information, political agendas, general lack of understanding, incomplete information due to the novelty of the pathogen, etc. Thus, a need exists to combine uncertain information based on early observations with new observations as disease spreads to new areas to avoid being misled and surprised.

SUMMARY

The disclosed system addresses the problem of how to transform what is learned from early observations into new information based on data in new areas to which emerging infections spread, resulting in forecasts and interventions tailored to local areas (e.g., villages, towns, cities, counties, zip codes, states, countries, etc.) as well as updated guidelines. The disclosed system learns more rapidly than is possible at present by iteratively combining data from multiple sources (using machine learning based on massive longitudinal and geographic data collection and data fusion). The disclosed system securely manages and resolves data conflict (e.g., noise effect reduction) to support an epidemic response tailored to local circumstances and populations (i.e., precision public health). The disclosed system supports the extraction, integration, and reconciliation of multiple local population segments to yield, analyze, propagate, and disseminate global guidelines.

DETAILED DESCRIPTION

Reference to the drawings illustrating various views of exemplary embodiments is now made. In the drawings and the description of the drawings herein, certain terminology is used for convenience only and is not to be taken as limiting the embodiments of the present invention. Furthermore, in the drawings and the description below, like numerals indicate like elements throughout.

FIG.1is a block diagram of an architecture100of a system200for generating and testing hypotheses and updating a predictive model of pandemic infections according to an exemplary embodiment.

As shown inFIG.1, the architecture100may include a server120that communicates with client devices180, for example via one or more networks130such as the Internet. The server120includes one or more hardware computer processors160and non-transitory computer readable storage media140. The server120receives data212from data sources110. The server120may be any suitable computing device including, for example, an application server or a web server. As described below, in some embodiments the data212may be publicly available and the data sources110may be accessible via the Internet. In other embodiments, some of the data212may be proprietary and/or sensitive. In those embodiments, the server120may be the secure computing environment for co-analyzing proprietary data, for example described in U.S. patent application Ser. No. 16/663,547, which is hereby incorporated by reference.

FIG.2is a block diagram illustrating the system200, which is realized by software modules executed by the hardware computer processor(s)160, generating and distributing initial hypotheses268and an initial prediction246according to an exemplary embodiment.

As shown inFIG.2, the system200may include a data collection module210, a validation/weighting module220, a machine learning module240, a hypothesis generation module260, and a dissemination module280.

The data212may include information in a variety of formats from a variety of data sources110. For instance, the data212may include spatio-temporal data regarding a disease, symptoms of that disease, human behavior contemporaneous with and/or in response to the disease, environmental conditions, meteorologic conditions, demographic and/or cultural data regarding geographical areas, and other data types in textual, numeric, image, and other formats. The data212may include structured or unstructured alphanumeric and non-alphanumeric elements, grammatically or ungrammatically structured text, non-text components (e.g., tables, figures, annotations, logos, images, or other element conveying information). The data212may include composite data (e.g., graphs, charts, spreadsheets, etc.). The data212may include medical and scientific literature (e.g., published peer reviewed studies including rapid review), open-source information (e.g., social media posts, public health reports, etc.), etc. The data212may include electronic health records from health information exchanges, regional health information organizations (e.g., The Health Collaborative,14Chesapeake Regional Information System for our Patients,15etc.), etc. The data212may include an aggregation of data collected from wearable devices (activity or fitness trackers, smart watches, etc.), personal communication devices (e.g., smartphones), Internet of Medical Things (IoMT) devices (e.g., remote patient monitoring devices, medication trackers, etc.), etc.14https://healthcollab.org/15https://crisphealth.org/data

The validation/weighting module220validates the data212and assigns a weight to each document in the data212to form and output validated and weighted data214. While all of the data214may be of interest, some of the data214may have different associated weights depending on characteristics of the data214such as the nature, source of capture, volume, uniqueness, and variance of the data214. Additionally, documents in the data214may be weighted based on the quality of the source of that data214(e.g., trustworthiness, authority, target audience, writing/reading level, number of references cited, domain of interest, etc.). As such, some documents in the data214may be treated as being more valuable than others. For instance, the validation/weighting module220may assign a higher weight to a study published in the Lancet or a Weekly Epidemiological Report from the World Health Organization than a blog post.

In some embodiments, the validation/weighting module220may weight data214from data sources using existing (qualitative or quantitative) measures of the reliability of those sources, such as the impact factor of a journal (a measure of the frequency with which the average article in the journal has been cited in a particular year), a reputation score of a website as determined by a web reputation service,16etc.16e.g., https://www.brightcloud.com/tools/url-ip-lookup.php

In emergent situation like a pandemic, however, data sources may emerge that are not rated by existing measures of reliability but nevertheless provide valid, reliable data214.17Accordingly, in some embodiments, the validation/weighting module220may weight data214using heuristics and/or subjective determinations of the reliability of specific data sources. For example, the validation/weighting module220may store a table of trusted data sources and weight the data214from those trusted data sources higher than data214received from other data sources. For instance, the validation/weighting module220may weight the data214from each of those trusted data sources equally or may store individual weights for each trusted data source (that are all higher than weights applied to data214received from other data sources). Similarly, the validation/weighting module220may store a table of data sources considered untrustworthy and either de-weight the data214from those untrustworthy data sources or may invalidate and ignore the data214from those untrustworthy data sources. In those embodiments, the validation/weighting module220may provide functionality for authenticated users (i.e., subject matter experts) to specify trustworthy and untrustworthy data sources (e.g., journals, epidemiological data sources, etc.) identified using their preferred criteria (e.g., transparency, reliability of past data, or other criteria). By providing functionality to identify trustworthy data sources (and, in some instances, weights to apply to data214received from those trustworthy data sources), the validation/weighting module220enables subject matter experts to weight data214using new criteria that may suggest itself in the moment. For instance, data214from health exchange organizations may be rated very highly because those health exchange organizations have access to electronic health records.17e.g., the Covid Tracking Project (https://covidtracking.com/about), the COVID-19 School Data Hub (https://www.covidschooldatahub.com), etc.

Modeling Pandemic Infections

Using the initial data214provided by the validation/weighting module220, the machine learning module240develops a predictive model242that predicts the future spread of the disease based on the data214output by the validation/weighting module220and associations, identified by the machine learning module240, between predictor variables that are identifiable in the data214and a dependent variable. For example, the predictive model242may predict the magnitude of one or more disease-related metrics (e.g., infections, hospitalizations, deaths, etc.) by applying weights and biases to predictor variables included in the data214. In another example, the predictive model242may be a probabilistic model (e.g., a Bayesian belief network) that calculates the probability of one or more disease events based on associations between predictor variables included in the data214and the probability of those future disease events.

The predictive model242generated by the machine learning module240may be a machine learning model, a mathematical model (e.g., an epidemic model, a contagion model, a hospital needs model, etc.), etc. As shown inFIGS.2and4, the predictive model242uses the data214output by the validation/weighting module220to generate predictions246regarding the spread of the disease. The predictions246may be probabilistic or deterministic forecasts of the magnitude of a dependent variable (e.g., an infection rate, hospital capacity, availability of personal protective equipment, etc.), the probability of a dependent variable (e.g., a disease-related event), etc. The predictor variables identified by the machine learning module240as associated with the dependent variable may include numerical values (having magnitudes, rates over time, rates of change, rates of acceleration, ratios relative to other numeric variables, etc.), whether certain conditions are true or false (e.g., whether certain public health interventions have been implemented, etc.), etc. The associations, identified by the machine learning module240, between the predictor variables and the dependent variable may include any predictive associational relationships and/or causal relationships between the predictor variables and the dependent variable, including correlations between the predictor variables and the dependent variable, non-linear mappings of the predictor variables onto the dependent variable, and/or any other relationships between the predictor variables onto the dependent variable.

To generate the predictive model242, the machine learning module240is trained using the data214to learn both the predictor variables in the data214that may be associated with the future spread of the disease and the associations (e.g., weights, Bayesian probabilities, etc.) between those predictor variables and the predicted disease-related metric or event. The machine learning module240may utilize any or all supervised, unsupervised, or semi-supervised learning approaches. The machine learning module240may utilize approaches that include classification, regression, regularization, decision-tree, Bayesian, clustering, association, neural networks, deep learning algorithms, etc. Deep learning algorithms may include recurrent models, convolutional models, transformer models with or without attention, etc. The machine learning module240may employ various machine learning algorithms known in the art, for instance pre-train transformers (used as global data), one or more final layers (trained while maintaining previous layers for localization),18etc.18MacAvaney, Nardini, Perego, Tonellotto, Goharian, and Frieder, Efficient Document Re-Ranking for Transformers by Precomputing Term Representations, ACM Forty-Third Conference on Research and Development in Information Retrieval (SIGIR), July 2020, https://dl.acm.org/doi/abs/10.1145/3397271.3401093

As shown inFIG.2, the predictive model242uses the initial data214to generate an initial prediction246of how the disease will spread in one or more locations.

Generating and Testing Hypotheses

Using the initial data214provided by the validation/weighting module220, the hypothesis generation module260generates a ranked list of initial hypotheses268.

FIGS.3A and3Bare flowcharts of a hypothesis generation process300according to an exemplary embodiment.

As shown inFIG.3A, the initial data212are collected from the data sources110in step310as described above. In some embodiments, each document in the data212is validated and weighted to form validated and weighted data214as shown inFIG.3A.

An ontology324is identified in step320. An ontology324is a set of possible event descriptions. That ontology can be understood to represent a formal conceptualization of a particular domain of interests or a definition of an abstract view of a world a user desires to present. Such conceptualization or abstraction is used to provide a complete or comprehensive description of events, interests, or preferences from the perspective of a user who tries to understand and analyze a body of information.

Each ontology324includes a number of elements. An ontology324with three elements, such as {subject, verb, object} for example, is used to detect all data corresponding to the notion “who did what to whom.” A 6-element ontology324may include {what, who, where, indicators, actions, consequences}. Each element includes choices of terms for that element of the ontology324, known as a “vocabulary.” If each element in a 6-element ontology324has a 100-term vocabulary, for example, then the ontology324defines 1006descriptions of distinct, mutually exclusive (although possibly related) events. Accordingly, the ontology324constitutes the set of all distinct combinations of hypotheses considered during the hypothesis generation process300. Each combination of elements in an ontology324is referred to as a “ontological vector.”

For many vocabulary terms, synonyms exist that refer to the same real-world concept. Accordingly, the ontology324may include synonym collections that each correspond to one of the vocabulary terms.

The ontology324may be supplied by a user or may be constructed by the system200using datasets being analyzed using machine methods. The ontology324identified in step320is preferably specific to an infectious disease. Accordingly, a subject matter expert (SME) preferably vets the ontology324to ensure that it accurately represents the domain knowledge of the data214under consideration.

The data214are coded using the ontology324to form coded data335at step330. Specifically, the computer processor(s)160executing the hypothesis generation module260search the data214using one or more entity extraction schemes that are known in the art to determine which ontological vectors in the ontology324appear in the data214. Each ontological vector identified in the data214represents a hypothesis268. For example, an analysis of reports on public health using a 3-element {subject, verb, object} ontology324may identify the following ontological vectors representing the following hypotheses:

In some embodiments, the hypothesis generation module260also assigns each ontological vector identified in the data214to the corresponding elements of text in the data214that include the ontological vector.

The ontology324can be graphically represented as an ontology space346, for example with as many dimensions as there are elements in the ontology324. The ontological vectors identified in the data214form an ontology space346at step340. A one-element ontology324, for example, forms an ontology space346with only one dimension (i.e., a line), which is readily understandable by a human analyst. Each point along the line represents a vocabulary term in the ontology324. It can be imagined that each time a vocabulary term is identified in the data214, a bar graph at that point along the line gets higher (or lower). The vocabulary terms found most often in the data214are represented by the highest peaks (or lowest troughs) along the one-dimensional ontology space346. Two-element and three-element ontologies324may form two-dimensional and three-dimensional ontology spaces346, which are more complicated but may still be visualized and comprehended by an analyst. However, when the ontology324has more than three elements and forms a 4-dimensional, 5-dimensional, or even 100-dimensional ontology space346, the ontology space346becomes so complex that no human analyst could ever intuitively understand it.

Regions of the initial ontology space346are populated as the documents in the data214are coded. The populated ontology space346is a geometric representation of possible events that are encoded by that particular corpus of data214according to that particular ontology324. The ontological vectors identified in the data214, which are assigned to the corresponding coordinates in the ontology space346, form structures in the ontology space346. In particular, points in the ontology space346that are populated by successive occurrences in the data214are assigned a value corresponding to a larger weight (described above as a higher peak or lower trough) than points in the ontology space346that are found less often in the data214. When all documents are coded, the ontology space346is populated by clusters (i.e., neighborhoods of points) of differing weights. The clusters of points of highest weight in the ontology space346correspond to the most likely hypotheses of what the data214are describing.

As described above, an ontology324with N elements may be depicted graphically in an N-dimensional ontology space346, where each dimension of the N-dimensional ontology space346represents one of the N elements of the ontology324. In other embodiments, however, the hypothesis generation module260may perform dimension reduction such that the ontology space346has fewer dimensions than the number of elements in the ontology. For example, the hypothesis generation module260can separate the N elements of the ontology324into R groups and then depict them graphically in the coded data335in an R-dimensional ontology space346. Depending on the nature of the ontology324, the hypothesis generation module260may perform lossless dimension reduction to preserve semantic content or perform dimension reduction with an acceptable loss across dimensions.

As described above, the data214may be weighted by the validation/weighting module220based on characteristics of the data214(e.g., the source, nature, volume, uniqueness, and variance of the data214). Accordingly, the hypothesis generation module260may weight each of the ontological vectors identified in the data214based on the weight of the data214from which each ontological vector was identified. Additionally, each attribute of the ontology324may be weighted based on the significance of that attribute. For example, attention may be placed on one or more dimensions of the ontology space346to place additional weight on the magnitude of each ontological vector along those one or more dimensions. Additionally, within each attribute of the ontology324, some of all of the vocabulary terms may be weighted based on the significance of those vocabulary terms. For example, the hypothesis generation module260may assign higher weights to ontological vectors that include more specific vocabulary terms than ontological vectors that include more generic vocabulary terms. Additionally, as described in U.S. Pat. No. 11,106,878, ontological vectors may be weighted based on the profile of a particular user. For example, if a user is interested in Asia and not Africa, ontological vectors with Africa as a component may be de-valued or excluded. Alternately, ontological vectors with Africa as a component may could be weighted more heavily as they may suggest connections to foreign nations that are of interest.

The hypothesis generation module260may also group or merge ontological vectors describing similar or related concepts into neighborhoods in the ontology space346. For example, the hypothesis generation module260may identify ontological vectors that describe similar or related concepts—for example, {masks, prevent, new infections} and {masks, stop, viral spread}— that are not distinct events. If the ontology324is ordered, meaning similar or related choices for each ontology element appear in order, the similar or related ontological vectors in the coded data335will appear close together in the ontology space346. That is, the embeddings or representations of the coded data335will map to a near vicinity, i.e., neighborhood, within the ontology space346. In one embodiment, the embeddings or representations of the coded data335will map to a near vicinity, i.e., neighborhood, within the ontology space346. Accordingly, the hypothesis generation module260may merge similar and/or related ontological vectors (e.g., via clustering hierarchies, filters/thresholds, topic models, conditional random fields, deep learners, etc.).

An optimization algorithm identifies hypotheses268in the ontology space346populated by the ontological vectors found in the data214(and ranks those identified hypotheses268) at step350. The computer processor(s)160executing the hypothesis generation module260identify and rank the hypotheses268by identifying the clusters of highest weights in the ontology space346. Identifying that set of clusters in the ontology space346is not a trivial problem for ontologies324of significant size and structure. However, it is a moderately well-defined optimization problem that can be solved using an iterative optimization algorithm (such as coordinate or gradient descent) or a heuristic optimization algorithm (such as simulated annealing, a Monte Carlo-based algorithm, a genetic algorithm, etc.).

Simulated annealing, for example, identifies the highest weighted clusters in an efficient and robust manner by selecting a random point in the ontology space346and letting simulated annealing govern a random “walk” through the weighted ontology space346via a large number of heat-cooling cycles. The computer processor(s)160executing the hypothesis generation module260build up an ensemble of such cycles for a large number of randomly chosen initial points. An accounting of the most highly weighted regions in the weighted ontology space346then corresponds to a ranked list of the hypotheses268that potentially explain the material in the data214, which may be presented to an analyst to test. In another example, the ontology space346can graphically depict populations and a genetic algorithm can be used to identify and rank the highest weighted ontological vectors or neighborhoods in terms of fitness of population.

In some instances, the dataset of ontological vectors identified in the data214may be so numerous that it is impractical or even infeasible for the server120to rank each ontological vector (or group of similar or related ontological vectors) using a computationally intensive optimization routine. Accordingly, in some embodiments the hypothesis generation module260may use a first optimization function to perform a coarse ranking of the ontological vectors or groups and a second optimization function to perform a more precise ranking of the ontological vectors or groups ranked highest by the first optimization function. In some of those embodiments, the hypothesis generation module260may use a first optimization function (e.g., a heuristic optimization function) that is less computationally intensive than the second optimization function to process the entire dataset of ontological vectors or groups and a second optimization function (e.g., an iterative optimization function) that is more computationally intensive than the first optimization algorithm to process the smaller subset of ontological vectors or groups ranked highest by the first optimization function. In those instances, using a first optimization function that is less computationally intensive routine to perform the coarse ranking may make the process of ranking the entire dataset of hypotheses268tractable for the server120. Meanwhile, reducing the amount of data needed to be examined in detail may make it tractable for the server120to use a second, more computationally intensive optimization routine to refine and improve the accuracy of the coarse ranking. In other embodiments, both optimization functions may be of similar complexity but functionally differ. Therefore, using an optimization algorithm that includes two separate optimizations functions may enable the hypothesis generation module260to both process the entire dataset of ontological vectors identified in the data214while also accurately and precisely ranking the hypotheses268in accordance with the weight of their associated ontological vectors (or groups of similar or related ontological vectors).

In some embodiments, the hypothesis generation module260may rank the hypotheses268based on the weight of each ontological vector or group of similar or related ontological vectors (e.g., using the first optimization function as described above), adjust the weight the ontological vectors or groups (e.g., by placing attention on one or more dimensions of the ontology space346to place additional weight on the magnitude of each ontological vector along those one or more dimensions as described above), and re-rank the hypotheses268according to the adjusted weights of the ontological vectors or groups corresponding to those embodiments (e.g., using the second optimization function as described above).

The hypotheses268may be filtered at step360to generate a filtered set of ranked relevant hypotheses268. Trivial hypotheses (such as tautologies) and/or nonsensical hypotheses may be discarded. Techniques from information retrieval and natural language procession (e.g., term frequency, scope and synonym analysis, etc.) may be used to identify and discard trial and/or nonsensical hypotheses. A hypothesis268that only contains frequent words, for example, is most likely too general to be of interest. In some embodiments, additional weighting can be placed on particular dimensions to rescore and possibly reorder the hypotheses268.

Local minima effects can sometimes provide a solution even when a better solution exists in another neighborhood. Random variations or mutations in the optimization algorithm (e.g., simulated annealing or genetic process) can be used to prevent the incorrect determination of a desired solution (e.g., a hypothesis of limited value) due to local minima effects. Those variations or mutations may be guided. At each proposed mutation, the neighborhood can be assessed for fitness. In an annealing process, for example, fitness can be assessed by the rate of change (e.g., the slope of descent or accent). In a genetic process, the fitness of a population member can be computed. In either process, a mutation can be rejected if the mutation results in an ontology space346that is deemed highly anticipated. Additionally, the rate of mutation can be modified to be a function of the anticipation level of the neighborhood initially in (e.g., a nonlinear mapping, a simple proportional dependence, etc.). Still further, the level of anticipation can be based on the profile of the analyst receiving the hypotheses.

The hypothesis generation module260may determine and output a degree of certainty as to the likelihood of each generated hypothesis268. The degree of certainty as to the likelihood of each generated hypothesis268is related to the confidence in—and support for—each generated hypothesis268. The hypothesis generation module260may determine a degree of certainty for each hypothesis268based on (e.g., proportional to) the weight ontological vector or neighborhood associated with that hypothesis268, which is based on (e.g., proportional to) the number of documents within the data214(and the weight of those documents) that, when coded, are found to contain the ontological vector or an ontological vector within that neighborhood.

As alluded to above, the system200repeatedly performs the hypothesis generation process300to generate updated hypotheses268′ based on updated data214′ (and discard initial hypotheses268that are no longer supported by the updated data214′). As shown inFIG.3B, updated data212′ are collected (and, in some embodiments, validated and weighted to form updated data214′) at step310and coded according to the selected ontology324to form updated coded data335′ at step330. The initial ontology space346is populated with ontological vectors in the updated coded data335′ at step340to augment the initial ontology space346and form an updated ontology space346′. The optimization algorithm identifies updated hypotheses268′ in the updated ontology space346′ (and ranks those updated hypotheses268′) at step350as described above. Those updated hypotheses268′ may be filtered at step360as described above.

Referring back toFIG.2, the initial hypotheses268are provided to the dissemination module280. The hypotheses268may include, for example, locally vulnerable and seemingly resistant population segments, local population factors potentially representing hitherto unobserved risk and resilience factors, speed of spread in unknown populations, etc. The hypotheses268may identify likely (pharmaceutical and/or nonpharmaceutical) public health interventions relevant for local populations. The hypotheses268may identify the likely impacts on local healthcare organizations, such as the need for field hospitals/care centers, the requirement of medical supplies (such as personal protective equipment), supply chain dynamics, etc. Via the dissemination module280, these respective healthcare organizations may be forewarned of potential impending crisis, and they, in turn, can commence precautionary measures.

To use a specific example, the initial hypotheses268identified in the ontology space346populated by the ontological vectors identified in the initial data214may include:Viral disease causes pneumonia in persons >70Bacterial illness causes death in persons >60Influenza-like illness causes unknown in persons <50

The dissemination module280distributes the prediction246generated by the predictive model242and the hypotheses268generated by the hypothesis generation module260to the relevant stakeholders and policy makers in the field of infectious disease. The dissemination module280may be any software program suitably configured to distribute information (using text, charts, graphics, etc.). The dissemination module280may include one or more specialized dashboards (for example, dashboards similar to those described in U.S. patent application Ser. No. 17/059,985, which is incorporated by reference). The distribution module280may be, for example, a web server that publishes one or more websites viewable via the client devices180over the one or more networks130using a web browser. Additionally, or alternatively, the distribution module280includes an email server configured to output email messages. The dissemination module280may include security features to securely disseminate information (e.g., the hypotheses268and the prediction246) only to authorized users. Additionally, or alternatively, the distribution module280may publish information and make that information viewable to the public via the Internet.

Evaluating the Initial Hypotheses268and Identifying Newly Emerging Hypotheses268

FIG.4is a diagram of the system200ofFIG.2, at a later point in time, generating and distributing the updated hypotheses268′ and an updated prediction246′ according to an exemplary embodiment.

As shown inFIG.4and described above with reference toFIG.3B, the data collection module210receives updated data214′ and the validation/weighting module220validates and assigns a weight to each document in the updated data214′. The predictive model242generates an updated prediction246′ based on the updated data214′. The updated prediction246′ is provided to the dissemination module280for distribution. The updated data214′ and updated prediction246′ are provided to the hypothesis generation module260. Using the updated data214′ and updated prediction246′, the hypothesis generation module260populates an updated ontology space346′ and generates updated hypotheses268′.

A hypothesis space difference evaluation module490compares the updated hypotheses268′ to the initial hypotheses268. For example, the hypothesis space difference evaluation module490determines whether updated hypotheses268′ identified in the updated data214′ were previously identified in the initial data214and, if so, may compare the rankings assigned to corresponding initial and updated hypotheses268and268′ by the optimization algorithm and/or the weights of the ontology vectors in the initial and updated ontology spaces346and346′ corresponding to those initial and updated hypotheses268and268′. If an updated hypothesis268′ was not previously identified in the initial data214—or if the corresponding initial hypothesis268was ranked lower than the updated hypothesis268′ because the ontology vector in the initial ontology space346corresponding to the initial hypothesis268was lower weighted than the ontology vector in the updated ontology space346′ corresponding to the updated hypothesis268′— then the updated hypothesis268′ represents a new insight that may help understand, control, and treat the disease.

Similarly, the hypothesis space difference evaluation module490determines whether initial hypotheses268identified in the initial data214are also identified in the updated data214′ and, if so, may compare the rankings assigned to corresponding initial and updated hypotheses268and268′ and/or the weights of the corresponding ontology vectors. A determination that an initial hypothesis268is not identified in the updated data214′— or corresponds to an updated hypothesis268′ that is lower ranked and lower weighted than the initial hypothesis268—is evidence that the initial hypotheses268represents an assumption that may no longer be supported by the latest data214′.

The updated hypotheses268′, relative to the initial hypotheses268, can then be delivered to users for consideration and further investigation. Accordingly, the system200can be used to inform public health officials and medical practitioners if the newly received data214′ suggests new inferences about the characteristics of the disease, the effectiveness of medical and/or public health interventions, the impacts on healthcare organizations in geographic areas, etc. Perhaps even more critically, the system200can also be used to inform those officials and practitioners if the newly received data214′ challenges or contradicts previous inferences drawn from earlier data214. Since the hypotheses268and268′ are combinations of English words, the new hypotheses268′ identified by the system200(and the previous hypotheses268challenged or contradicted by the system200) are immediately understandable to human users. Meanwhile, the hypotheses268and268′ identified by the system200can be traced back to the data214or214′ from which those hypotheses268and268′ were identified, enabling public health and medical researchers to evaluate those data sources.

Using the Difference Between Newly Identified and Previous Hypotheses for Optimization

As described above, if additional ontological vectors (that were not detected in the initial data214) are identified in the updated data214′ (e.g., previously unexhibited symptoms, unaffected geographical regions, etc.), the system200augments the initial ontology space346(that generated the initial hypotheses268) to form the updated ontology space346′, which generates the updated hypotheses268′. In addition to better informing officials, practitioners, and policymakers, the difference between the updated hypotheses268′ and the initial hypotheses268(as determined by the hypothesis space difference evaluation module490) can also be used by the optimization algorithm described above to more efficiently and effectively identify and rank hypotheses using future data.

For instance, if the set of updated hypotheses268′ subsumes the set of initial hypotheses268, then the updated ontology space346′ subsumes the initial ontology space346and only the updated ontology space346′ needs to be maintained. Accordingly, instances where the set of updated hypotheses268′ subsume the set of initial hypotheses268, the system200may discard the initial ontology space346and augment only the updated ontology space346′ using future data.

Alternatively, if the set of initial hypotheses268subsumes the set of updated hypotheses268′, then the additional ontological vectors in the updated ontology space346′ (that were not present in the initial ontology space346) lead to contradictory or inconsistent hypotheses268′. In those instances, the additional ontological vectors in the updated ontology space346′ limit the possibility of identifying valid hypotheses268′. Also, if those additional ontological vectors are used as the basis for public health regulations, those regulations will be overly restrictive and unsupported by the data214and214′. Accordingly, in instances where the set of initial hypotheses268subsumes the set of updated hypotheses268′, the system200may discard additional ontological vectors in the updated ontology space346′ that were not present in the initial ontology space346(or assign those additional ontological vectors lower weights than to the ontological vectors in both the initial ontology space346and the updated ontology space346′).

Finally, if the set of initial hypotheses268is equal to the set of updated hypotheses268′, then the additional ontological vectors in the updated ontology space346′ (that were not present in the initial ontology space346) are redundant and may be removed by the system200.

Returning back to the specifical example above, the difference between the updated hypotheses268′ and the initial hypotheses268may reveal:A new hypothesis268′ identified in the updated ontology space346′ that was not present in the initial ontology space346, such as:Viral disease causes rash in persons >15A persistent hypothesis268identified in the initial ontology space346that remains in the updated ontology space346′, such as:Bacterial illness causes death in persons >60 Anomalous hypotheses268′, such as:Viral disease causes pneumonia in persons >70Viral disease causes pneumonia in persons >20Influenza like illness causes unknown in persons <50Influenza like illness causes unknown in persons <10

Using Newly Identified and Recently Evaluated Hypotheses to Update the Predictive Model

As described above, identifying new hypotheses268′ in newly received data214′ can help medical practitioners and public health officials identify additional medical and public health interventions that may treat and control the spread of a disease. Also, determining whether initial hypotheses268continue to be suggested by the latest data214′ helps those practitioners and officials evaluate whether the interventions that are currently being implemented are as effective as originally assumed.

Additionally, identifying new hypotheses268′ (and discarding previous hypotheses268that are no longer suggested by the latest data214′) can help predictive models more accurately predict the future spread of a disease by providing those predictive models with the latest understanding of characteristics of the disease and the effectiveness of various interventions. Accordingly, if the updated hypotheses268′ significantly differ from the initial hypotheses268, the system200uses those updated hypotheses268′ to inform the predictive model242generated by the machine learning module240.

As described above, the predictive model242predicts the future spread of the disease based on predictor variables identified in the data214(e.g., numerical metrics, Boolean conditions, etc.) and associations (e.g., weights, Bayesian probabilities, etc.) between those predictor variables and the future spread of the disease. The machine learning module240is trained using the initial data214to learn the predictor variables that are associated with the future spread of the disease and the extent of those associations. As updated data214′ are received, however, new hypotheses268′ in the updated data214′ that were not detected in the initial data214(e.g., previously unexhibited symptoms, unaffected geographical regions, etc.) may identify additional predictor variables in the updated data214′ that, if incorporated in the predictive model242, would improve the accuracy of the predictive model242. Similarly, new hypotheses268′ in the updated data214′ may suggest adjustments to the associations (e.g., weights, Bayesian probabilities, etc.) used by the predictive model242, which were initially learned by the machine learning module240while being trained using the initial data214, to better reflect the updated hypotheses268′ in the updated data214′. By contrast, an initial hypothesis268(identified in the initial data214) failing to appear in the updated data214′ (or having significantly less weight in the updated ontology space346′ relative to the initial ontology space346) is an indication that the initial hypothesis268is less relevant than the initial data214suggested. Accordingly, the predictive model242may be updated to discount that initial hypothesis268, for example by reducing the weight (or adjusting the probability) previously applied to a variable that the initial hypothesis268suggested was predictive of the future spread of the disease (or no longer using that variable at all when generating predictions246).

In some embodiments, the machine learning module240is trained on the newly identified hypotheses268′ (and/or indications that previously identified hypotheses268should be discounted) to learn adjusted associations and/or additional predictor variables indicative of those newly identified hypotheses268′, as well as variables (previously viewed as predictive) that can be de-weighted or no longer considered. Alternatively, the machine learning module240may be trained on the updated hypotheses268′ to generate a new predictive model242to replace the predictive model242generated using the initial data214. In either embodiment, providing the machine learning module240with the difference between the updated hypotheses268′ and the initial hypotheses268enables the machine learning module240to perform back propagation and readjust the predictor variables and associations (and/or the model structure, initial conditions, boundary conditions, etc.) to make the predictive model242represent and classify the current state of knowledge.

To adjust the predictive model242, for example, the machine learning module240may utilize deep learning, for instance with attention on more recent data214′ and/or on data214that are higher weighted by the validation/weighting module220to support greater intuition regarding classification results derived by deep learners and/or graph-oriented models to provide interpretability via derivation graphs.1919See, e.g., U.S. Pat. No. 11,238,966 to Frieder et al.

In addition to improving the accuracy of the predictive model242, providing the machine learning module240with updated hypotheses282′ that better reflect the latest understanding of the disease enables the predictive model242to generate predictions246′ that are tailored to local geographic areas based on predictor variables that are specific to those geographic areas (e.g., the current disease metrics in those areas, the demographic composition of those areas, whether public health interventions are required and the level of compliance in those areas, etc.) and the associations between those predictor variables and the spread of the disease suggested by the updated hypotheses282′.

As the disease continues to spread, the system200repeatedly captures updated data214′ to generate updated hypotheses268′ and uses those updated hypotheses268′ to update the predictive model262. While the process performed by the system is logically viewed as sequential, the data collection, analytics, and dissemination can overlap, either pairwise or in totality. That is, partial analysis and partial dissemination may occur while additional data collection and analysis proceeds.

By combining hypothesis generation/testing and predictive modeling, the disclosed system200provides important technical benefits that cannot be realized using separate hypothesis generation systems and predictive models. As described above, prior art predictive models often rely on assumptions to model pandemic infections,20such as assumptions about the characteristics of a disease, the effectiveness of medical and/or public health interventions, potential changes in human behavior over the prediction period, etc. If the assumptions embedded in those prior art predictive models are inaccurate, those inaccurate assumptions will negatively impact the accuracy of every subsequent prediction generated by those prior art predictive models, even as those prior art predictive models incorporate new data, until those prior art predictive models are updated to no longer rely on those assumptions. Critical in early-stage diagnostic predictions is the ability to forget or “be forgotten.”. Prior art predictive models are either insufficiently powerful to learn the associations needed to derive the ranked hypotheses268described above or do not provide sufficient intuition to enable change, including replacing variables and associations previously considered predictive or simply forgetting those previously considered variables.20See Cramer et al., supra, wherein seven probabilistic COVID-19 forecasts made explicit assumptions that social distancing and other behavioral patterns would change over the prediction period.

By contrast, the disclosed system200uses initial data214to generate a predictive model242that outputs an initial prediction246and then evaluates the assumptions embedded in that predictive model242by repeatedly collecting updated data214′, identifying the hypotheses268′ in the new data214′, and comparing those updated hypotheses268′ to the initial hypotheses268identified in the initial data214used to generate the predictive model242. Accordingly, as new data214′ emerge that challenge or contradict the assumptions embedded in the predictive model242, the disclosed system200is configured to adjust the predictive model242to more accurately reflect the most recent understanding of the disease and the public health and medical interventions to control and treat the disease.2121See Sridhar et al., supra, wherein some early COVID-19 models did not consider the possible effects of mass “test, trace, and isolate” strategies or potential staff shortages on transmission dynamics.

Additionally, while prior art machine learning algorithms can use newly received data to identify unexpected predictor variables and the associations between those predictor variables and potential outcomes, those prior art deep learning algorithms fail to provide any insight as to why predictions change over time. Accordingly, rather than merely identifying numerical metrics based on their fit to past data, the disclosed system200goes a step further by coding the new data214′ (including textual information, etc.) according to an ontology324, organizing the coded data335in an ontology space346, and using an optimization algorithm to identify and rank hypotheses268′ found in the new data214′. In doing so, the system200provides human-comprehensible reason(s) for each suggested update to the predictive model242and human-comprehensible actions (e.g., public health or clinical interventions) that can be implemented to better control and/or treat the disease (and, therefore, generate predictions246that are reflective of more desirable health outcomes). Accordingly, the disclosed system200enables researchers to identify the change to our understanding of the disease that triggers each change to the predictive model242and, for instance, the probability that each change is permanent, the predicted duration of any change believed to be transient, the likelihood that any change will be repeated, whether any change can be mitigated via a public health or clinical intervention, the probability that a suggested intervention will mitigate the identified issue, other issues that may be caused by the suggested intervention, etc. Those new insights, in addition to their value for keeping public officials and clinicians better informed, also enable the machine learning module240to more accurately predict the current trajectory of a disease and the effectiveness of current and potential interventions.

Example

An illustrative and instructive case in early days of the COVID-19 pandemic concerns the use of masks as a public health intervention to slow/mitigate spread.

Early in the pandemic, masks were not thought to be an effective public health or personal protection measure. On Jan. 29, 2020, the World Health Organization (WHO) noted that “a medical mask is not required, as no evidence is available on its usefulness to protect non-sick persons.”22That continued to be the definitive guidance on mask wearing. A month later, on Feb. 26, 2020, the U.S. Centers for Disease Control and Prevention (CDC) confirmed the first likely instance of community spread of COVID-19 in the U.S.23On Feb. 27, 2020, in a Congressional hearing, CDC Director Robert Redfield was asked whether healthy people should wear a face covering and responded “No.”24Additional official guidance was disseminated by U.S. Surgeon General Jerome Adams on Feb. 29, 2020. On Twitter, Adams urged Americans to “STOP BUYING MASKS!”, asserting that masks are “NOT effective in preventing general public from catching coronavirus” and that rushing to buy masks would deplete mask supplies for healthcare providers. Indeed, the former assertion had some evidence in the research literature, which presented mixed results in evaluations of the effectiveness of masks in preventing community respiratory illness.22https://apps.who.int/iris/handle/10665/33098723https://www.cdc.gov/media/releases/2020/s0226-Covid-19-spread html24https://www.c-span.org/video/?469566-1/house-hearing-coronavirus-response

On Feb. 29, 2020, then-Vice President Pence, speaking as head of the coronavirus task force at a White House press conference, noted that the “average American does not need to go out and buy a mask.” The message was so consistent and ubiquitous that, a week later, on Mar. 8, 2020, Anthony Fauci said in a 60 Minutes interview that “there's no reason to be walking around with a mask,” adding that he was not “against masks” but rather was worried about health care providers and sick people “needing them.” He also mentioned possible “unintended consequences” of mask wearing, including people touching their face frequently when adjusting their masks, posing contamination hazards to themselves.

Given an ontology324of respiratory infection and public health interventions, an ontology space346populated using data214that includes documents corresponding to COVID-19 (such as those alluded to above and others) would have resulted in clusters reflecting that guidance (that masks were not an effective public health or personal protection measure), which was offered universally (outside of China) at this stage of the pandemic. However, applying the hypothesis generation process to the issue of appropriate interventions based on what was known about respiratory infections may have revealed and challenged the (obvious) conflict between the assertion that masks were unlikely to be effective at preventing or slowing community disease with the assertion that they needed to be conserved for healthcare workers, who would be protected by wearing them.

As more was learned, guidance changed. Importantly, in March and April 2020, evidence emerged implicating asymptomatic and presymptomatic transmission of COVID-19 and the implications for mask wearing were recognized. On March 29, former Food and Drug Administration Commissioner Scott Gottlieb published a paper outlining a “roadmap” for emerging from widespread “lockdowns.” Mask use was a prominent recommendation. “Face masks will be most effective at slowing the spread of SARS-CoV-2 if they are widely used, because they may help prevent people who are asymptomatically infected from transmitting the disease unknowingly.”2525https://www.aei.orgkesearch-productskeport/national-coronavirus-response-a-road-map-to-reopening/and

On Mar. 31, 2020, Fauci said he was in “very active discussion” with health officials about reversing guidance on mask use when the U.S. gets in a “situation” where it has a sufficient mask supply, alluding to the emerging evidence that COVID-19 spreads via the air among asymptomatic people who do not cough or sneeze. On Apr. 3, 2020, the CDC updated its guidance on masks and facial coverings, recommending wearing facial coverings “in public settings when around people outside their household, especially when social distancing measures are difficult to maintain.” The WHO followed suit on Apr. 6, 2020, citing presymptomatic transmission and noting that “The use of masks is part of a comprehensive package of prevention control measures that can limit the spread of certain respiratory viral diseases, including COVID-19.”2626World Health Organization, Advice on the use of masks in the context of COVID-19, 1 Dec. 2020, https://www.who.int/publications/i/item/advice-on-the-use-of-masks-in-the-community-during-home-care-and-in-healthcare-settings-in-the-context-of-the-novel-coronavirus-(2019-ncov)-outbreak

Applying hypothesis generation methods to the biomedical and epidemiology literature during this period or before would have revealed the appearance of new clusters in ontology space346corresponding to these new data (regarding transmission mechanisms). Specifically, the emergence of evidence implicating viral shedding in respiratory droplets before COVID-19 symptoms appeared immediately suggests the importance of intervention measures such as mask wearing among others for the general population. The appearance of such hypotheses268in the ontology space346could have cued a search for implications much more quickly than happened, perhaps even instantaneously if the ontology324was sufficiently connected or linked to control measures.

Guidance remained consistent for a time of low transmission but then became more important as a new wave hit. The U.S. saw a dramatic acceleration of COVID-19 transmission in the fall of 2020. In the pre-COVID-vaccine era, nonpharmaceutical interventions (NPIs) continued to be the only means available to prevent increasing morbidity and mortality. Modeling and other epidemiology studies became available implicating the importance of such NPIs for the coming wave. On Oct. 14, 2020, Fauci, discussing the upcoming holidays and associated dangers of the cold weather, said “Don't be afraid to wear a mask in your house if you're not certain that the persons in the house are negative.”27He reiterated that advice more strongly roughly a week later, saying “ . . . if people are not wearing masks, then maybe we should be mandating it”28in a CNN interview. “There's going to be a difficulty enforcing it, but if everyone agrees that this is something that's important and they mandate it, and everybody pulls together and says, you know, ‘we're going to mandate it but let's just do it,’ I think that would be a great idea to have everybody do it uniformly.”27CBS News, Dr. Fauci on COVID surge, Trump's recovery, holiday travel and more—Full interview, 14 Oct. 2020, https://www.cbsnews.com/video/dr-fauci-on-covid-surge-trumps-recovery-holiday-travel-and-more-full-interview/28CNN, Fauci says it might be time to mandate masks as Covid-19 surges across US, 23 Oct. 2020, https://www.cnn.com/2020/10/23/health/fauci-covid-mask-mandate-bn/index.html

Applying the ontology324described above to the continuing epidemiology and biomedical literature among other sources during this period would have detected new clusters in the ontology space324surrounding efficiency and performance of masks and mask type, calling attention to the importance of mask material and mask-use strategies. Meanwhile, a predictive model242adjusted to reflect the newly recognized correlation between mask usage/materials and lower transmission rates would have estimated the public health benefit of those interventions and illustrated their importance.

As the most intense wave of the pandemic in the U.S. receded, due in no small part to the appearance of an effective vaccine, new guidance was published on double masking. On Feb. 10, 2021, the CDC released research finding that wearing a cloth mask over a surgical mask offers more protection against the coronavirus, as does tying knots on the ear loops of surgical masks. The lateness of this and other updated guidelines is tragic and could have been issued earlier if learning had occurred more rapidly, as outlined in the method of this application.

During the summer and early fall of 2021, numerous documents in the literature have examined the importance of not only vaccination, and not only masking in prevention of COVID-19, which began increasing over the summertime, but combining those measures. Using the hypothesis generation module260, that data214would undoubtedly result in additional clusters in ontology space324and should cue policy guidance to the public that the need for wearing a mask has not yet passed.

While preferred embodiments have been set forth above, those skilled in the art who have reviewed the present disclosure will readily appreciate that other embodiments can be realized within the scope of the invention. For example, disclosures of specific numbers of hardware components, software modules and the like are illustrative rather than limiting. Accordingly, the present invention should be construed as limited only by any appended claims.