AUTOMATICALLY DETECTING ERRORS IN COMPUTER-BASED ANALYTIC METHOD ROUTING OBJECTS

A method for diagnosing errors in an analytic method definition is described. The facility accesses the definition, which specifies a set of pages, one or more of which include at least one question. Each question has two or more possible answers. One or more pages each include a swimlane control having a plurality of lanes, each lane identifying a combination of possible answers to two or more of the questions included among the pages and specifying routing in an iteration of the analytic method in which that combination of answers has been given. For each of the swimlane controls, among the included questions any of whose answers have been included in the combination of answers identified by a lane of the swimlane, the facility determines any combinations of possible answers not identified by any lane of the swimlane.

BACKGROUND

Medical professionals who treat patients (“caregivers”) rely on medical knowledge. For many caregivers, this medical knowledge can extend to an extensive variety of different kinds of information: normal and anomalous body function; condition diagnosis, including relevant forms of physical examination and testing; referral to specialists; condition treatment options and attendant benefits and risks, including by prescribing medicines or placebos, performing surgery, undergoing physical therapy; surgical and anesthesiological techniques; epidemiological updates; techniques for interacting with patients and cooperating caregivers; etc.

Medical knowledge is obtained by caregivers from a variety of sources, such as: medical school classes and textbooks; clinical rotations, internships, residencies, and fellowships; treating patients; ongoing medical education seminars and conferences; informal discussions with other caregivers; reading journal articles; reviewing pharmaceutical data sheets; consulting treatment or referral guides and other information published by or within a caregiver's organization; etc.

DETAILED DESCRIPTION

The inventors have recognized significant disadvantages in the conventional ways that medical knowledge is created and disseminated to medical caregivers such as physicians and applied by them.

In particular, a staggering volume of medical knowledge, in widely-varying subject matter areas, can be beneficially applied to the treatment of many caregivers' patients. This beneficial medical knowledge is often both within the caregiver's specialty area, and in adjacent or related specialty areas. Caregivers are unable to reliably retain all of the medical knowledge they have encountered.

Also, whether consumed in advance of needing it or directly in response to needing it, medical knowledge is spread over a wide variety of sources and modalities that makes it difficult to locate pieces of medical knowledge relevant to a patient situation. Also, in many cases it is difficult to assess the accuracy of a particular piece of medical knowledge, or select the most useful of multiple sources for particular medical knowledge. There is often also a trade-off between reliability and latency of sources of medical knowledge, in light of the significant length of the authoring, review, and publication cycle of medical journal articles. Additional factors can also make medical knowledge obsolete as soon as it is made available, or shortly thereafter.

It is common for sources of medical knowledge to be difficult to apply to a caregiver's needs. For example, certain sources may be focused on theory, to the exclusion of practice, or practical aspects may be difficult to locate among content that is less practical. Some sources that seek to provide practical guidance achieve limited success based on using ill-suited language, formatting, or organization, for example.

Because of the high level of dynamism of many medical fields, caregivers often face the challenge of medical knowledge “changing out from under them”: both caregiver memories and static sources of medical knowledge can become outdated as new discoveries are made, new technologies emerge and mature, new techniques and best practices are developed, etc.

In response to recognizing these disadvantages, the inventors have conceived and reduced to practice a software and/or hardware facility for constructing and applying computer-based analytic methods representing medical knowledge (“the facility”). These analytic methods can represent a wide variety of different kinds of medical knowledge, including, for example, disorder diagnoses, referral recommendations, medical procedures or protocols, selection among condition treatment options, etc.

In some embodiments, the facility provides a creator user interface for creating and revising analytic methods, as well as one or more other user interfaces for consuming and/or visualizing analytic methods.

In some embodiments, a “creator” user—such as a domain expert, or another qualified person who is transforming a conventional source of medical knowledge into an analytic method—interacts with controls presented by the facility in order to create an analytic method for use by consumer users, such as caregivers who wish to rely on the medical knowledge embodied in the analytic method. In particular, the creator creates a set of “pages” that make up the analytic method, which can correspond to individual “blocks of knowledge.” In each page, the creator can specify questions, their possible answers, and whether it is valid for more than one of the possible answers to be selected at the same time by a consumer. For a particular question, the creator can specify routing that is to occur in response to each of the question's possible answers being given. This routing can be to destinations of various types, including (1) certain text or other content that the creator specifies within the context of the page; (2) another page generated by the facility, either part of the current analytic method or a different one, either one that already exists or that the creator will create; (3) a document created outside the facility; or (4) an action, such as against an electronic medical record (“EMR”) software system with respect to the patient who is the subject of a consumer's use of the consumer user interface. In some embodiments, the facility permits the creator to add small supplemental information icons to particular portions of a page, which can be expanded to display supplemental informational content about that portion of the page in the consumer user interface, such as by hovering over or touching or clicking on the information icon.

In some embodiments, the facility provides a “swimlane” control in which the creator can use a visual grid scheme to easily specify routing for different combinations of answers to a group of multiple questions. For example, where a page in analytic method relating to joint pain contains a first question asking how many joints are affected—whose answers include one, two, and more than two—and a second question asking whether those joints are swollen—whose answer include yes and no—the creator can use the swimlane to specify one routing target for the combination of the “one” answer to the first question and the “yes” answer to the second question; another routing target for the combination of the “one” answer to the first question and the “no” answer to the second question; etc. In some embodiments, the facility enables the creator to group multiple answer combinations together with the same routing target.

In some embodiments, as the creator uses the creator user interface to construct the analytic method, the facility concurrently displays and updates a graph diagram visualization of the analytic method in which each page is shown as a node, and each routing path to another page is shown as an edge between nodes. In some embodiments, the creator can select one of the nodes of the graph diagram to jump to the corresponding page in the creator user interface. In some embodiments, the creator can interact with one of the nodes of the graph diagram to delete or otherwise alter the corresponding page of the analytic method.

In some embodiments, as part of the creator interface, the creator can switch on an interview mode in order to preview the experience that consumers will have in using the analytic method. If the creator identifies an issue with a page in the interview mode, the creator can switch the interview mode off in order to alter the definition of the page in the creator user interface.

In some embodiments, throughout the creation of the analytic method, the facility continuously updates an internal representation of the analytic method that is stored by the facility to maintain the persistence of the analytic method, and is a basis for rendering all of the ways supported by the facility to visualize and interact with the analytic method, including the creator user interface, the graph diagram visualization, and consumer interview user interfaces. In various embodiments, this internal representation is stored locally in a browser and/or on a server.

By using the facility to represent medical knowledge as an analytic method in the ways described above, the facility renders that medical knowledge “computable”—i.e., usable in the facility's consumer user interface to directly and systematically apply to a patient.

In some embodiments, the facility provides one or more consumer user interfaces for applying analytic methods created using the facility, such as interview user interfaces in which an analytic method's questions are presented to a consumer; answers to the questions are received from the consumer; and content, additional questions, and/or actions routed to according to the received answers are presented to the consumer. In various embodiments, these interview user interfaces conduct their interviews in a variety of modalities. These can include a touch- or click-based visual user interface in which answers to questions are displayed as buttons that the consumer can select by touching them or clicking on them; a text-based visual user interface in which consumers type answers to their questions, or shorter text strings representing those answers; a voice-based audio user interface in which the facility speaks questions and their answers using speech synthesis, and consumers speak their answers; or a combination of these, e.g., a consumer speaks in response to seeing questions and their possible answers displayed in the touch- or click-based visual user interface.

In some embodiments, the consumer user interface enables a consumer to select a particular analytic method from an inventory of analytic methods contained in a repository; enables a consumer to search for or otherwise discover analytic methods of this inventory appropriate to a particular patient scenario; etc.

In some embodiments, the consumer user interface concurrently displays the graph diagram visualization of the selected analytic method, enabling the consumer to follow their progress through the analytic method, skip forward or backward among the pages of the analytic method; switch to a different route among the pages of the analytic method, jump to a different analytic method, etc.

The inventors have further noted certain errors that creators sometimes make in creating an analytic method. These include (1) creating pages in the analytic method that are unreachable from the analytic method's root page; (2) creating loops among the connections between pages in the analytic method; (3) when creating a swimlane control for specifying routing for different combinations of answers to a group of multiple questions, omitting lanes that provide routing for certain possible combinations of answers; (4) creating unreachable swimlanes; (5) linking to external documents that don't exist; (6) linking to a specific page that doesn't exist in an external document; and (7) linking to an external document in a different audience (e.g. an adult guide accidentally linking to a pediatric-only guide, without explicitly marking it as such).

In response to this recognition, in some embodiments, the facility automatically performs diagnostic assessments of created analytic methods, such as continuously during their creation, or at certain junctures, such as when each page is saved, or when the analytic method's creation is completed. In various embodiments, these diagnostic assessments include (1) whether any page of the analytic method is unreachable from the root page; (2) whether the connections between the pages of the analytic method form any loop; and/or (3) in each swimlane control, whether lanes that provide routing for any possible combinations of answers have been omitted.

By performing some or all of the ways described above, the facility makes it straightforward to embody medical knowledge in a computer-based analytic method, either medical knowledge originated by the creator of the analytic method or medical knowledge originated by one or more third-party sources. This analytic method can be quickly and easily employed by vast numbers of caregivers as part of their treatment of patients, or their ongoing medical studies. The analytic method can later be updated or otherwise improved, to the benefit of this large number of caregivers and their patients. In addition, in some embodiments the facility helps creators to identify and resolve errors in the analytic methods they create.

Additionally, the facility improves the functioning of computer or other hardware, such as by reducing the dynamic display area, processing, storage, and/or data transmission resources needed to perform a certain task, thereby enabling the task to be permitted by less capable, capacious, and/or expensive hardware devices, and/or be performed with lesser latency, and/or preserving more of the conserved resources for use in performing other tasks. For example, by concentrating easily-applied medical knowledge in a central repository of analytic methods, the facility reduces the network and processing resources that would otherwise have been used to search for, retrieve, evaluate, and seek to apply conventional sources of medical knowledge. Also, by using a compact internal representation of analytic methods, the facility reduces the amount of storage that would otherwise have been used to store less data-efficient conventional sources of medical knowledge. Further, by automatically assisting creators to identify and correct errors in creating analytic methods, the facility reduces the processing load on computing systems by (1) preventing the unsuccessful use in production of analytic methods that have these errors, which would have to later be repeated once the errors are found and corrected, and (2) obviating the user interface load imposed by creators manually searching for and resolving these errors.

FIG.1is a block diagram showing some of the components typically incorporated in at least some of the computer systems and other devices on which the facility operates. In various embodiments, these computer systems and other devices100can include server computer systems, cloud computing platforms or virtual machines in other configurations, desktop computer systems, laptop computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, etc. In various embodiments, the computer systems and devices include zero or more of each of the following: a processor101for executing computer programs and/or training or applying machine learning models, such as a CPU, GPU, TPU, NNP, FPGA, or ASIC; a computer memory102for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel, and device drivers; a persistent storage device103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer-readable medium; and a network connection105for connecting the computer system to other computer systems to send and/or receive data, such as via the Internet or another network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like. While computer systems configured as described above are typically used to support the operation of the facility, those skilled in the art will appreciate that the facility may be implemented using devices of various types and configurations, and having various components.

FIG.2is a data flow diagram providing a high-level view of the operation of the facility in some embodiments. A creator user210interacts with a creator UI220provided by the facility in order to create or revise an internal representation of an analytic method230. The facility stores this internal representation persistently, such as in a repository of analytic methods. The facility provides a variety of interpreters to transform an analytic method's internal representation into a visualization or user interface for the analytic method. For example, a visual interview interpreter241interprets the internal representation to generate a visual interview user interface for the analytic method that can be used by any number of consumers290, as well as the creator to preview the consumer experience of the analytic method. Similarly, a diagram interpreter251interprets the internal representation to generate a graph diagram user interface252for the analytic method in which the pages of the analytic method are represented as graph nodes; this diagram user interface can similarly be used by consumers or the creator. Further, the facility provides an audio interview interpreter261to transform the internal representation of the analytic method into an audio interview user interface for the analytic method, which can be used both by consumers and the creator. In various embodiments, the facility provides additional interpreters as basis for making additional user interfaces available for the analytic method.

FIG.3is a flow diagram showing a process performed by the facility in some embodiments to build an analytic method. In act301, the facility creates a new analytic method. In some embodiments, act301involves receiving metadata for the analytic method from its creator, such as a name, subject matter area, keywords, creator name, creator organization, creator title, creator qualifications or designations, groups of consumer users that can or should use it, etc. The facility initiates storage for the analytic method, and stores the collected metadata for the analytic method. In act302, the facility receives user input adding a page to the analytic method. This page poses one or more questions, and performs routing based on the answers to those questions. In various embodiments, the routing can be to any of text or other content that is stored as part of the page; a different page-one that already exists or can be subsequently created by the creator-which can contain its own questions and routing; an external document that is not part of an analytic method; or an action performed by an EMR with respect to a patient on whose behalf the analytic method is being applied. In some embodiments, certain pages added to the analytic method by the facility contain no questions; in some such embodiments, routing from these pages is performed unconditionally to a single target, while in others, routing is performed based upon factors other than answers to questions in the page, such as answers to questions in previous pages; in other cases, such a page may be a “leaf” from which no routing to another page occurs.

In act303, the facility updates the internal representation of the analytic method to include the new page added in act302. In act304, the facility applies one or more interpreters to the internal representation updated in act303to reflect the new page in visualizations of the analytic method that are presently being used. After act304, the facility continues in act302to receive user input needed to add the next page to the analytic method. Though not shown, in some embodiments, the facility performs acts302-303to delete an existing page from the analytic method, revise the content of an existing page, alter routing among pages, split a page into multiple pages, etc.

Those skilled in the art will appreciate that the acts shown inFIG.3and in each of the flow diagrams discussed below may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.

FIG.4is a display diagram showing a sample display presented by the facility in some embodiments to begin the creation of a first page of a new analytic method using the creator user interface. The display400includes an indication407of a name specified for the analytic method by the creator, as well as controls408and409for adding questions to the analytic method's first page. For example, the user can click control408in order to add a single choice question to the first page.

FIG.5is a display diagram showing a sample display presented by the facility in some embodiments to add textual content to a page. The display500is divided into a creator UI column501and a graph diagram column502. The creator UI column includes a pane510that in turn includes a name for the page511; a selectable type512of the page; a content entry area514for entering content for the page; and formatting controls513for formatting parts of the content, such as text in the content. The graph diagram column shows a graph diagram visualization of the in-construction analytic method. In some embodiments, the graph diagram shown is a directed acyclic graph. Presently the graph diagram includes only a single starting node580for the starting page of the analytic method, at which the analytic method will begin each time it is applied by a consumer.

WhileFIG.5and each of the display diagrams discussed below show a display whose formatting, organization, informational density, etc., is best suited to certain types of display devices, those skilled in the art will appreciate that actual displays presented by the facility may differ from those shown, in that they may be optimized for particular other display devices or device sizes or other device capabilities, or have shown visual elements omitted, visual elements not shown included, visual elements reorganized, reformatted, revisualized, or shown at different levels of magnification, etc.

FIG.6is a display diagram showing a sample display presented by the facility in some embodiments to enter information about a question to be posed by an analytic method page. The display600that the user has entered content614for the page. Also, the creator has used a control provided by the facility to insert a supplemental information icon611in this content, and enter the following supplemental information text relating to the portion of the content where the creator placed the icon: “including septic, viral (COVID, Parvo), lyme, RMSF, Acute onset, fever, rash, arthritis, must r/o infectious diseases.” The display includes control615for specifying the question, here “How many joints are involved?” The display also includes a field616into which the facility has automatically populated a variable name whose value during application of the analytic method will be the consumer's answer to this question; the user can edit this prepopulated variable name. The display also includes an expandable area621into which the user can enter information about each of the possible answers to this question.

FIG.7is a display diagram showing a sample display presented by the facility in some embodiments to define possible answers to a question. The display700shows that the user has populated three answers to question725, answers721,722, and723. For each of the answers, from left to right, the creator has populated button text—“1” for the first button—summary text—“monoarticular” for the first button—and link. The link specifies a routing target when this answer is selected by the consumer. Here, the creator has not entered routing targets, anticipating the later use of a swimlane to route flow based upon the answer to this question in combination with the answer to a following question that the creator is about to establish, by clicking control708to add another single choice question to the page.

FIG.8is a display diagram showing a sample display presented by the facility in some embodiments to define an additional question for the sample analytic method's starting page. It can be seen that, in response to the creator's selection of the add single choice question control708shown inFIG.7, the facility has displayed fields825and826. The creator has entered text “Is the joint red, hot, swollen?” into the question text field825. The facility has automatically populated variable field826with a corresponding variable name. The facility has further displayed section831for entering the possible values to this second question.

FIG.9is a display diagram showing a sample display presented by the facility in some embodiments to enter possible answers to the second question. The creator has populated a “yes” possible answer931and a “no” possible answer932, including summary text for these possible answers. As the result of this input by the creator, for example, if the consumer presses the yes button, the variable “IS THE JOINT RED, HOT, SWOLLEN?” will be set to the value “YES”. The user has also populated a field941to specify that flow should proceed from the first page to a “Jt pain swimlane” page. In response, the facility has added a node981to the graph diagram corresponding to this new page. It has colored new node981red to indicate that its definition as part of the analytic method is not yet complete.

FIG.10is a flow diagram showing a process performed by the facility in some embodiments to include a swimlane in an analytic method to specify routing based upon the answers to multiple questions. In step1001, the facility receives user input establishing questions in a page, and their possible answers. Act1001is illustrated inFIGS.4-9. In act1002, the facility receives user input creating a swimlane in a page. In the swimlane, each row represents one possible answer for one of the questions previously posted, either in the current page or a page that would be traversed before the current page. In acts1003-1006, the facility loops through each column of the swimlane. In act1004, the facility receives user input selecting a valid combination of answers in the column. In act1005, the facility receives user input specifying an outcome for the column, either content, a target page to route to, a non-page document to display, or an EMR action to take with respect to the patient. In act1006, if additional columns are to be added to the swimlane, the facility continues in act1003to add the next column. After act1006, this process concludes.

FIG.11is a display diagram showing a sample display presented by the facility in some embodiments to begin definition of a swimlane. The display1100shows the name1142of the new page that will contain the swimlane. The highlighting of graph diagram node1181shows that the Jt pain swimlane page is the current page in the creator UI. The display further includes a selection1143of a swimlane control; a name1144of the swimlane, and rows1151-1154of the swimlane each usable to specify a particular answer to one of the questions, i.e., a particular value of the variable defined for one of the questions. The facility is ready for the creator to define a first column1155of the swimlane. In particular, in this column, the creator can: select the first row1151to specify that the combination of answers includes that the answer to the number of joints question is 1; select the second row1152to specify that the answer to the number of joints question is 2-4; select the third row1153to specify that the answer to the number of joints question is more than 4, and select the last row1154once to specify that the answer to the swollen question is yes, and select the last row twice to indicate that the answer to the swollen question is no. In some embodiments, the facility permits a creator to select a point in a row multiple times to cycle through different possible answers for the question to which the row singularly corresponds. For example, by clicking a first time in row1154, the creator specifies the answer yes for the corresponding question, which is indicated by a green checkmark. By selecting that point twice, the creator specifies the value no for this question. By clicking a third time, the creator returns to specifying no value for this question. After this point, the cycle repeats.

FIG.12is a display diagram showing a sample display presented by the facility in some embodiments to define a first column of a swimlane. The display shows that, for first column1255, the creator has selected among rows1251-1253row1253to specify that, in the combination of answers representing column1255, the number of joints question is answered with the more than 4 answer. The creator has also selected row1254once to indicate that the answer to the swollen question is yes, represented by a green checkmark. If the creator wanted to specify that the answer to the swollen question is no, the creator would select row1254again, changing the green checkmark to a red X.

FIG.13is a display diagram showing a sample display presented by the facility in some embodiments to define additional columns in a swimlane. By comparing display1300shown inFIG.13to display1200shown inFIG.12, it can be seen that the creator has added additional swimlane columns1356-1360to represent additional combinations of answers to the number of joints and swollen questions. Additionally, the facility displays at the bottom of these columns controls61-64for specifying routing targets for these columns. The user can select one of these controls in order to specify the routing target for the combination of answers represented by the column that it is in. For example, the user can select control1361to specify routing for the combination of answering the number of joints question with answer more than 4, and answering the swollen question with yes.

FIG.14is a display diagram showing a sample display presented by the facility in some embodiments to specify the routing target for a particular swimlane column. In some embodiments, the facility presents display1400as a pop-up window over the creator user interface. The display1400includes a checkbox1401that the user can select in order to merge this column with the column to the left, if there is one. Where the creator does this, the creator need not enter any routing information, as the routing target specified for or reused by the column to the left will be used. The display also includes a field1403into which the creator can enter content, such as text, to be displayed in response to the consumer submitting the answers to which the swimlane column corresponds. The display also includes a field1404where the creator can specify a name of a different page to which the consumer should be routed when the combination of answers to which the swimlane column corresponds are submitted.

FIG.15is a display diagram showing a sample display presented by the facility in some embodiments to enter content to display if a consumer submits the combination of answers to which the leftmost swimlane column corresponds. It can be seen that the user has entered content into field1503to be displayed to consumers that enter the combination of answers to which the swimlane column corresponds. After entering this content, the creator selects a done control1505to dismiss the pop-up.

FIG.16is a display diagram showing a sample display presented by the facility in some embodiments to display content previously defined for a swimlane column. In the display, can be seen that, in response to the creator hovering over routing target control1663for the third swimlane column1657the facility has displayed a compact pop-up window1699containing the content earlier specified by the creator for this column. At this point, the creator can select routing target control1664to specify a routing target for columns1658-1660, which have previously been grouped by an action of the creator.

FIG.17is a display diagram showing a sample display presented by the facility in some embodiments to specify a routing target for the fourth through sixth columns of the swimlane. It can be seen that the creator has entered the page name “No signs of inflammation” into next page field1704. The creator selects the done control1705to close display1700.

FIG.18is a display diagram showing a sample display presented by the facility in some embodiments to create a new page that was specified as the target of the fourth through sixth columns of the swimlane. It can be seen that the facility has added node1881to the graph diagram representing the completed swimlane. In some embodiments, the creator can hover over the routing target icons at the bottom of the node to see the routing target specified for each of the swimlane columns. Additionally, the facility has added a placeholder node1882for the No Signs of Inflammation page implicated by the routing target specified for the fourth through sixth columns of the swimlane. The user can activate a control1869to create this page.

FIG.19is a display diagram showing a sample display presented by the facility in some embodiments to enter content for the no signs of inflammation page. The creator has specified a questions type1912for this page, and entered content for the page into content field1914. At this point, the facility has replaced the placeholder node in the graph diagram with normal node1982, reflecting that the no signs of inflammation page has been adequately defined. The user can add a single choice question to this page by activating control1908.

FIG.20is a display diagram showing a sample display presented by the facility in some embodiments to add a question to the no signs of inflammation page. It can be seen that the creator has added a “Which joint?” question2015to this page, for which the facility has populated a suggested variable name into variable field2016. The creator has also specified answers2021-2023for this question as part of defining these answers, the creator has specified in the link column the name of a different document external to the analytic methods for each answer. For example, though not fully visible, the creator has added in the link column for answer2021#KneePainReferralGuide. In response to entering this information, the facility has created new nodes2083-2085in the graph diagram, each for corresponding to an external document specified for a different one of the three answers. For example, node2083corresponds to answer2021. The green halo around node2082in the graph diagram reflects that the corresponding page of the analytic method is presently being defined or edited in the creator user interface to the left.

Table 1 below shows an internal representation generated by the facility in JSON for the example analytic method whose creation is shown inFIGS.4-20and discussed above.

Lines 1-90 of the internal representation provide information about the analytic method as a whole. Line 3 contains a unique identifier that can be used to distinguish the analytic method from other analytic methods. Line 4 shows the current stage of the analytic method within an analytic method production cycle. Line 5 shows the name specified for the analytic method. Lines 6-17 show tags that can be used to search for or otherwise locate this analytic method in situations where it may be needed. Lines 19-40 identify subject matter categories to which the analytic method belongs, which can be used to find the analytic method in situations in which it might be useful. Lines 41-62 identify people who have participated in creation, revision, proofreading, maintenance, etc. of the analytic method. Lines 63-66 identify a governance model for the analytic method. Lines 67-90 identify a variety of metadata for the analytic method, including creation and updating timestamps shown in lines 89-90.

Line 93 contains the unique identifier for the Chronic Joint Pain page. Line 94 indicates that this page is an active part of an analytic method, as opposed to a “guide” page constituting a plain document. Line 95 specifies that this page's node is the root of the graph, where the analytic method beings. Lines 96-99 identify the consumer type for which the analytic method is intended; lines 100-103 identify a patient care mode for which the analytic method; and lines 104-107 identify the patient type for which the analytic method is intended. Line 108 specifies the name of the page, which is shown in the display of the page and the page's graph node, and is used by other pages of the analytic method to refer to the page. Lines 109-110 specify the content of this page, shown at reference number614inFIG.6, in an encoded form. Inside, tag delimiter symbols “<” and “>” are encoded using their ASCII values “/u003c” and “/u003e,” respectively. It can be seen that an action tag (“<action>”) specifies the supplemental information icon611, as well as the following text to be displayed when the user hovers over it: “including septic, viral (COVID, Parvo), lyme, RMSF. Acute onset, fever, rash, arthritis, must r/o infectious diseases.” Line 111 specifies the tooltip text that is to be displayed if the user hovers anywhere over the page. Line 112 specifies that multiple question controls occur on the page. Line 113 specifies that this page routes to the Jt pain swimlane page after its questions are answered.

Line 114-156 specify the two questions in the page and their possible answers. For the first question, Line 136 specifies its text as “How many joints are involved?”. Lines 119-123 specify the first possible answer (“option”) for the first question as “1”; lines 124-129 specify as the second possible answer to the first question as “2-4”; and lines 130-134 specify the third possible answer to the first question as “more than 4.” Lines 138-155 similarly define the second question “Is the joint red, hot, swollen?” and its two possible answers, “Yes” and “No.”

For the Jt pain swimlane page defined in lines 161-248, lines 161-177 specify metadata for this page similar to the metadata discussed above in connection with the Chronic Joint Pain page. Also, line 243 identifies the type of the page as “swimlane,” i.e., one that performs differential routing based upon different combinations of answers to questions. Lines 180-189 relate to the first column1355of the swimlane; lines 183-185 indicate that this first column is for the combination of the answer “More than 4” to the question “How many joints are involved?” with the answer “Yes” to the question “Is the joint red, hot, swollen?”. Line 182 indicates that the swimlane does not route to a separate page corresponding to the empty next page field1504left by the creator inFIG.15. Line 187 includes an encoded version of the content1503entered by the creator inFIG.15. Line 188 shows that the first column has not been merged with any column to the left-indeed, there is no column to the left of the first column.

Lines 190-199 relate to the second column1356of the swimlane, which lines 193-195 relate to a selection of the answer “2-4” to the question “How many joints are involved?” with the answer “Yes” to the question “Is the joint red, hot, swollen?” Like the first column, line 192 specifies that the second column does not route to a separate page of the analytic method. Line 197 contains the encoded content to be displayed when the choices for the second column are selected.

Lines 200-209 relate to the third column1357of the swimlane, corresponding to the combination of the answer “1” to the question “How many joints are involved?” and the answer “Yes” to the question “Is the joint red, hot, swollen?.” Line 202 specifies that the third column does not route to a different page, and line 207 contains the content displayed when the combination of answers to which this column corresponds are given.

Lines 210-219 relate to the fourth column1358of the swimlane, corresponding to the combination of the answer “1” to the question “How many joints are involved” and the answer “No” to the question “Is the joint red, hot, swollen?.” Line 217 specifies that no content is displayed when the combination of answers to which this column corresponds is given, but line 212 specifies that, when this combination of answers is given, the facility should route to the “No signs of inflammation” page.

Lines 220-229 relate to the fifth column1359of the swimlane, corresponding to the combination of the answer “2-4” to the question “How many joints are involved?” with the answer “No” to the question “Is the joint red, hot, swollen?” Line 228 specifies that the fifth column has been merged with the column to its left, the fourth column. Therefore, when the fifth column's combination of answers is given, the facility routes to the “No signs of inflammation” page in accordance with line 211 mentioned above.

Lines 230-239 relate to the sixth column of the swimlane1360, which corresponds to a combination of the answer “more than 4” to the question “How many joints are involved?” with the answer “No” to the question “Is the joint red, hot, swollen?”. Like the fifth column, line 238 marks the sixth column as being merged with the column to the left; thus, like the fourth and fifth columns, when the combination of answers to which the sixth column corresponds, the facility routes to the “No signs of inflammation” page.

Lines 249-300 relate to the “No sign of inflammation” page—as indicated in line 265. Lines 250-264 contain metadata for this page. Line 267 contains the content of the page, shown as content2379and2479inFIGS.23and24. Line 269 specifies that this page contains a question, which line 273 specifies has mutually exclusive possible answers—in other words, only one can be selected at a time. Lines 275-280 relate to the first answer to the question, “Knee” as specified in line 278. Line 277 specifies that, if this first answer is selected, the facility routes to the “#Knee Pain Referral Guide” document. Lines 281-285 relate to the second answer, “Hip” as specified in line 284. Line 283 specifies that, when the second answer is selected, the facility routes to the “#Hip Pain Referral Guide” document. Lines 286-291 relate to the third answer, specified as “Lateral hips” in line 288. Line 287 specifies that, when this answer is selected, the facility routes to the “#Trochanteric Bursitis Referral Guide” document. Also, line 290 specifies that this button has a supplemental information icon that, when hovered over, displays the text “If pain is isolated to lateral hip and tender to palpation, consider trochanteric bursitis.” While the question corresponding to these answers could have been specified in line 293, instead in this case it was specified near the end of the content for the page in line 267 as “Would you like to see more about a specific joint?”.

FIG.21is a flow diagram showing a process performed by the facility in some embodiments in order to permit navigation of the analytic method via the graph diagram. In act2101, the facility displays a directed graph diagram visualization of an analytic method, which shows pages of the analytic method as nodes of a graph diagram, and shows routing among the pages as edges between nodes. The graph diagram shown inFIG.20is an example of such a graph diagram visualization. In act2102, the facility receives user input selecting one of the nodes of the graph diagram, such as by touching it or clicking on it. In act2103, the facility changes the state of a user interface other than the graph diagram to make the page corresponding to the node selected in the graph diagram the current page in the other user interface. In various embodiments, this other user interface could be the creator interface or one of the consumer interfaces. After act2103, the facility continues in act2102to receive user input selecting another node of the graph diagram.

FIG.22is display diagram showing a sample display presented by the facility in some embodiments to begin navigating via the graph diagram visualization. Display2200shows the display after the user has clicked on node2080shown in the graph diagram that appears inFIG.20. It can be seen that the corresponding node2280, corresponding to the analytic method's starting page, has a green halo showing it to be the current node and showing the corresponding page of the analytic method to be the current page. Also, the facility has switched the display mode to show the consumer preview mode, which shows how pages of the analytic method will be displayed to the consumer as they are presently defined. It can be seen that the consumer view of the analytic method's first page2270is displayed, including content2271, a first question2272and its possible answers2273-2275, a second question2276and its possible answers2277-2278. In this state, to navigate to the page corresponding to node2281, the creator can either click on this node in the graph diagram, or answer the questions in the consumer mode of the creator user interface, such as by selecting possible answers2273and2278. The effect of doing either is shown inFIG.23discussed below.

FIG.23is a display diagram showing a sample display presented by the facility to reflect navigation to the second page of the analytic method. It can be seen that the graph diagram node2381corresponding to the second page—the Jt pain swimlane page—has a green halo to indicate that this page is the current page. Also, the facility has displayed content2379for this page.

FIG.24is a display diagram showing sample contents of a display presented by the facility in some embodiments to navigate to the third page of the analytic method. It can be seen that the creator has scrolled down the contents of the second page79to see its question94, and that question's possible answers2495-2497. Because the user selected possible answer2495, the facility indicates that the external “KneePainReferralGuide” document is to be opened. In some embodiments (not shown), the facility may add a halo to node2483to reflect this routing.

FIG.25is a flow diagram showing a process performed by the facility in some embodiments to apply an analytic method on behalf of a consumer user, such as one using the facility with respect to a particular patient. In act2501, the facility begins at the starting page for the analytic method. In act2502, the facility presents the content for the present page and any questions it contains. In some embodiments, the facility performs act2502by applying an interpreter to the internal representation of the analytic method. As discussed above, in various embodiments, the facility performs this presentation using a variety of output modalities. In act2503, the facility receives user input specifying one or more answers to any questions presented in act2502. As discussed above, in various embodiments, the facility receives this input using a variety of input modalities. In act2504, on the basis of any answers specified in act2503, the facility routes among multiple targets in accordance with the analytic method, such as presenting content specified by the creator for the specified answers, routing to a different page of the analytic method based on the specified answers, displaying a document external to the facility specified by the creator for the specified answers, and/or performing EMR action with respect to the patient specified by the creator for the specified answers. Such routing may also be unconditional, to a single target. After act2504, the facility continues in act2502to present the next page and any content or questions it contains.

FIG.26is a display diagram showing a display presented by the facility in some embodiments to begin application of an analytic method. The display2600shows the page first of the sample analytic method whose definition is described above in connection withFIGS.3-20, and whose internal representation is shown in Table 1. Its name2601is shown in the display, as is its content2610. It can be seen that the content includes an information icon2611that the consumer can select in order to expand it, such as by clicking on it, touching it, or hovering over it. In some cases, similar information icons may be automatically expanded either every time they are displayed or in response to a certain state such as an answer to an earlier question or a property of the consumer or the patient.

FIG.27is a display diagram showing a sample display presented by the facility in some embodiments to expand an information icon. In display2700, it can be seen that the information icon2611has been expanded to show additional informational content2712relating to the portion of the content identified by the location of the information icon. The display also includes question2720and its possible answers2721-2723, as well as question2730and its possible answers2731and2732. It can be seen that possible answer2733has its own information icon, which can be similarly expanded to provide informational content about this possible answer.

FIG.28is a display showing a sample display presented by the facility in some embodiments to display a second page of the sample analytic method in response to the consumer's answering of the questions of the first page. In particular, it can be seen in display2800that the consumer has selected answer2822to question2820, and answer2831to question2830. In response, the facility has displayed the content2840that the creator specified inFIG.15for the third column of swimlane1350.

FIG.29is a display diagram showing a sample display presented by the facility in some embodiments to make visible the remainder of the second page of the sample analytic method. In particular, after scrolling down, display2900shows all of contents2950specified for the second page, as well as question2960and its possible answers2961-2963, among which the consumer can select.

FIG.30is a display diagram showing a sample display presented by the facility in some embodiments to show the routing flow taken by the facility in the sample analytic method based upon the consumer's interactions in the third page. It can be seen that, in response to the consumer's selection of answer3061to question3060, the facility has, in accordance with the creator's definition of that possible answer2021shown inFIG.20, displayed the name3069of the external document specified by the creator for this possible answer, as well as the content3070of this external document. As shown, the external document is displayed in a tabbed interface, in which tabs3071-3074permit the consumer to select among different sections of the external document. In various embodiments, this display can be used by various other users, such as coauthors or non-author experts seeking to review the method.

In various embodiments, the facility supports various kinds of questions, and various approaches to routing in response to their answers. For example, in some embodiments, the facility prompts the consumer to enter a number, such as patient body mass index, and routes based on which of two or more numerical ranges the number falls within. In some embodiments, the facility prompts the consumer to provide a medical imaging artifact captured from the patient, and routes based on an automatic interpretation of the medical image artifact. In some embodiments, the facility prompts the consumer to provide a photograph or sketch of part of the patient's body, and routes based on an automatic analysis of this image. In some embodiments, the facility displays a diagram of a human body, prompts the consumer to touch or mark an affected area of the body on the diagram, and routes based upon the body area identified by the consumer. In some embodiments, the facility prompts the consumer to provide audio containing speech by the patient, and routes based on an automatic analysis of this audio.

In various embodiments, the facility supports a variety of types of routing targets. For example, in some embodiments, the facility supports routing to programs separate from the facility—such as those for displaying or interpreting medical imaging artifacts, those for making a live connection audio, video, and/or text connection with a person such as a specialist in a certain area of medicine, those for automatically generating text satisfying a text generation specification, etc.

FIG.31is a flow diagram showing a process performed by the facility in some embodiments to perform a diagnostic assessment of an analytic method. In act3101, the facility collects the pages of the analytic method. In act3102, the facility initializes a current page of the analytic method to the analytic method's root page. The facility further initializes to empty both an answer combination and two sets of visited pages. In act3103, the facility calls a visit_page function, passing the current page, the answer combination, and the second set of visited pages—as initialized in act3102—as arguments. In act3104, after the call to the visit_page function returns, the facility displays the results of the diagnostic assessment: any loops among the connections between pages in the analytic method, any pages of the analytic method that are unreachable from the root page, any lanes among the swimlane controls in the analytic method that correspond to impossible combinations of answers, and any combinations of answers that are missing from any of the swimlane controls in the analytic method. The facility does this based on the analytical results accumulated by recursive calls to the visit_page function. After act3104, this process concludes.

FIG.32is a flow diagram showing a process performed by the facility in some embodiments to handle a call to the visit_page function. The visit_page function receives arguments identifying a page that is being visited, an answer combination that is being tested, and a set of visited pages used to check for loops (the “second set of visited pages”). In act3201, the facility marks the page identified by the page argument as visited, to exclude it from any list of unreachable pages in the analytic method. In act3202, the facility detects any loops in the directed acyclic graph of the analytic method's pages, as connected by connections between pages. In act3203, if the page contains any questions, then the facility continues in act3204, else the facility continues in act3208.

In acts3204-3207, the facility loops through each permutation of answers to the questions contained by the page. In act3205, the facility creates a new answer combination reflecting the current permutation in this loop iteration. In act3206, the facility recursively calls the visit_page function, passing it the identity of the current page, as well as the new answer combination created in act3205in this iteration of the loop, and the second set of visited pages. In act3207, if other permutations of answers remain to be processed, then the facility continues in act3204to process the next permutation of answers, else the facility continues in act3208.

In act3208, if the page is or contains a swimlane, then the facility continues in act3209, else the visit_page function returns. In act3209, if any lanes of the swimlane of the page match the answer combination that was the argument value in the call to the visit_page function, then the facility continues in act3211, else the facility continues in act3210. In act3210, the facility logs that no lanes match the answer combination with which the visit_page function was called, identifying the combination as one for which a lane should be added to the swimlane in the analytic method. After act3210, the call to the visit_page function returns.

In acts3211-3214, the facility loops through each lane that matches the answer combination. In act3212, the facility marks the lane as visited. In act3213, the facility recursively calls the visit_page function with the argument values corresponding to the page that is linked-to from the matching swimlane that is the subject of the current iteration of the loop, and the answer combination. In act3214, if additional matching lanes remain to be processed, then the facility continues in act3211to process the next matching swimlane, else the facility returns from this call to the visit_page function.

FIG.33is a display diagram showing a first display presented by the facility in some embodiments to show the results of the facility's diagnostic assessments for a sample analytic method. The display3300includes a graph of the pages making up the diagnostic method: a background questions page3320, a symptoms questions page3330, and a swimlane page3340. The symptoms question page shows two questions3231and3232that it poses, as well as the possible answers to each of these questions. The swimlane page3340shows the two lanes that are presently defined for the swimlane control, lane3341and lane3346. For example, lane3341performs routing within the analytic method for the combination of the answer no fever to the fever question and the answer headache to the headache question.

The display further includes an editing region3310for the swimlane control of the swimlane page. In particular, editing control shows lane3311corresponding to lane3341and lane3316corresponding to lane3346. In lane3311, it can be seen that this lane specifies the answer no fever3312to the fever question, and the answer headache3313to the headache question. This lane further includes a link3314to the page or other routing target that is routed-to by this lane of the swimlane, i.e., in iterations of the analytic method where the combination of answers of lane3311have been given.

The display also includes a compact notification3350that the facility's diagnostic assessment has identified errors with the analytic method in its present state. The user can activate control3351to expand the notification and receive details about the results of the diagnostic assessment.

FIG.34is a display diagram showing a second display presented by the facility in some embodiments to show the results of the facility's diagnostic assessment for a sample analytic method. It can be seen that compact notification3350has been expanded to robust notification3450in response to the creator user activating control3351. The expanded notification includes a control3451that the user can activate in order to collapse the notification. The notification further includes a section3460of errors related to connections between pages: any pages that are unreachable from the root page3330—here, the background questions page3320identified by reference3461—and any cycles or loops among the pages (none shown). In some embodiments, the reference3461to the background questions page is a link that the creator user can follow, such as to display the editing view of the background questions page, or other controls that the creator user can use to make the background questions page reachable either directly or indirectly from a root page of the analytic method.

The expanded notification further contains a section3470showing missing combinations of answers for which the facility can add lanes to the swimlane. In particular, the sections shows missing combinations of answers3471and3476. The creator user can activate control3481to add a lane to the swimlane for missing combination3471; activate control3486to add a lane for missing combination3476; or activate control3489to add lanes to the swimlane for both missing combinations.

In some embodiments, the facility permits creators to specify one or more additional components of a swimlane. A first such additional component-a “supplemental content” additional swimlane component-available to creators in some embodiments is text or other content, or a reference to a page containing such content, that is always displayed to the consumer in addition to content displayed based on the matching lane of the swimlane. The presence in a swimlane of a supplemental content swimlane component does not affect the facility's checking for missing lanes in a swimlane.

A second such additional component—a “default destination” additional swimlane component—available to creators in some embodiments is a destination that the swimlane routes to in the event that no lane of the swimlane matches the combination of answers given by the consumer. The presence in a swimlane of a default destination swimlane component does affect facility's checking for missing lanes in a swimlane: once a default destination swimlane component is added to a swimlane, the swimlane can no longer have missing lanes, because the swimlane will have the effect of routing for any possible combination of answers.

In various embodiments, the facility uses various approaches in the creator user interface to enable creators to specify supplemental content and/or default destination swimlane components for a swimlane, including separate fields for specifying the destinations for these components, or additional columns for these components that specify their destinations. In various embodiments, the facility uses various approaches to identifying a column of the swimlane as an additional column representing a specify supplemental content or default destination components, such as adding additional rows to the swimlane controls for supplemental content and/or default destination, whose checking in a particular column specifies that that column has the corresponding significance.