Patent Description:
An electronic medical record (EMR) or other repository of medical data includes a number of lists compiled according to a controlling lexicon that are routinely viewed by a variety of healthcare practitioners in the delivery of patient care to individual patients. Examples of lists include a patient problem list, a list of patient medications, a surgical history list, a list of lab values, and the like. Each list includes an element with a plurality of occurrences. Each occurrence includes data represented according to a controlling lexicon. Examples of controlling lexicons include International classification of Diseases (ICD)-<NUM>, ICD-<NUM>, Systematized Nomenclature of Medicine (SNOMED), Current Procedural Terminology (CPT), compilations of pharmaceutical names such as a physician's desk reference, RadLex® Playbook, Nursing Outcomes Classification (NOC), NANDA-I, geographical adaptations of controlling lexicons, and the like.

For example, a patient problem list includes an element which is a patient problem, with occurrences of individual problems reported by the patient. Each occurrence of a problem can be listed according to ICD-<NUM> code and/or corresponding description. Each list is presented in its entirety. That is, the list is unfiltered and/or is not a subset. For example, the patient problem list is presented with all the reported problems to a healthcare practitioner. Problems are typically only removed from the list after careful review by a healthcare practitioner, which typically indicates that the listed problem is in error or that the patient is no longer experiencing the problem.

The lists are typically ordered in reverse chronological order. For example, as new problems are reported by a patient, the new problems are added to the top of the patient problem list, e.g. formed with a default chronological order. The lists can be lengthy and are important sources of information to healthcare practitioners, who are generally expected in a standard of patient care to be aware of any relevant occurrence on the list in delivering care. For example, a healthcare practitioner is reasonably expected to be aware of chronic conditions, which may be highly relevant and are naturally aged to the bottom of the list.

Occurrences on each list can have different relevance to different healthcare practitioners, who can be involved in different aspects of patient care. For example, "Falls frequently" is less relevant to a radiologist, than "Diabetes mellitus", while the reverse may hold for an orthopedist. Furthermore, the relevance of any one occurrence relative to another occurrence on the list is difficult to identify with precision by any one practitioner. That is, a healthcare practitioner may have difficulty in precisely reordering each occurrence in the list.

One approach to facilitate understanding in a viewing of relevant occurrences first can be to rank or re-order the list according to importance defined using an algorithm. However, such an approach does not consider individual or institutional preferences.

Another approach to facilitate understanding in a viewing of occurrences can be to use machine learning in how healthcare practitioners would reorder the list. However, such an approach is typically mutually exclusive of the approach of ranking by importance according to an algorithm, and healthcare practitioners are typically severely constrained with time to reorder each list. That is, there is limited time by healthcare practitioners to provide electronic feedback, which includes re-ordered lists of each individual patient. The reordering may involve decisions of determining precisely a relevance of each occurrence to others in the list, which diverts time and attention away from actual delivery of patient care. Moreover, the use of machine learning involves time for training the machine on appropriate learning, such as for example, a learning phase.

<CIT> discloses a method of transacting medical information which includes receiving medical information from medical sources, identifying, mapping, and consolidating the received medical information by a back-end medical processor, providing access to specific relevant data, based on a user's security privileges, within the identified, mapped, and consolidated medical information, based on user-specific functions or roles by a front-end medical processor, and generating user-customized processed medical information to a plurality of users.

<CIT> Al discloses system and method for problem list categorization and management in an electronic health or medical record including matching each entry in the list with an interface terminology concept, grouping related concepts together into one or more categories, and grouping entries into one or more nested sets of problems.

<CIT> Al discloses systems, methods, and apparatus provided to facilitate analysis, presentation, and comparison of clinical information. An example system includes a processor configured to provide a patient library interface. The interface displays a plurality of events along a patient timeline and a list of items for comparison to a clinical scenario.

<CIT> Al discloses a sliding tab tabbed pane user interface component implementable in an electronic device. The user interface component provides a user interface of the electronic device with items belonging to a plurality of different categories, each category being associated with a different sliding tab. The sliding tabs are arranged such that at least a portion of the items of at least two sliding tabs is visible.

<CIT> Bl discloses a system and method for managing search results available from a network resource using a data feed incorporating a processor programmed to retrieve user unsorted search results matching search terms provided by a user, transmit the search results along with a user selectable sorting identifier over the network for display on a graphical user interface and further programmed to sort each search result according to the user selected sorting identifier, update the sorting status of each search result, and transmit for display on the graphical user interface the search results in accordance with the sorting status and the user selected sorting identifier.

<CIT> Bl discloses a system and method for updating a source copy of an ordered list (the source list) comprising a plurality of list items according to modifications made to the order of the list items in a local copy of the ordered list (the local list). Each list item includes an order value.

<NPL> discloses rearranging list columns on all instances of a List.

Aspects described herein address the above-referenced problems and others. The following describes contextual list viewing with sparse feedback.

Dependent claims represent beneficial embodiments.

The following describes the invention according to one aspect of the invention. The following steps relate to a system, a method and non-transitory computer-readable storage medium claims. Exemplary, a system is described, comprising: a ranking engine comprising a first processor configured to receive a list for a patient which includes a plurality of occurrences, compute a relevance score for each occurrence in the list, wherein the computed relevance score is according to a relevance scheme that maps relevance scores from a lexicon controlling the list to each of the plurality of occurrences; and a user interface comprising a second processor. The user interface is configured to display the list on a display device of a local computing device ordered by a presented computed relevance score, wherein each displayed occurrence of the plurality of occurrences includes a feedback indicator; receive feedback comprising an input for one displayed occurrence of the plurality of occurrences according to the feedback indicator which indicates the one displayed occurrence is to be displayed higher or lower in the list than a current position, wherein the input is a binary indicator. The user interface is further configured to compute a feedback relevance score according to the binary indicator and a set of rules, wherein the feedback comprises the feedback relevance score and the system further comprises a feedback database comprising electronic storage and one or more processors configured to receive and store the feedback. The ranking engine is further configured to compute an adjusted relevance score according to the feedback for at least one occurrence of the plurality of occurrences, wherein the ranking engine computes the adjusted relevance score according to a function of all the feedback in the feedback database for the at least one occurrence of the plurality of occurrences in the list, the function including an average or mean of the feedback relevance scores and the function including only the last N feedback relevance scores; and the presented computed relevance score is the adjusted relevance score, where the feedback has been received, or the computed relevance score, where no or insufficient feedback has been received.

In a further aspect of the invention, a computer program product comprising instructions is described, wherein the instructions cause the computer to carry out the steps of the method of the present invention, when the program is executed by a computer.

An embodiment of a system <NUM> configured for contextual list viewing with sparse feedback (e.g., a contextual list viewing with sparse feedback system) is disclosed. A ranking engine <NUM> in response to a request <NUM> from a user receives or retrieves a list <NUM>, such as a patient problem list, a list of patient medications, a surgical history list, a list of lab values, and the like. The request <NUM> includes identification of the list and the user requesting the list <NUM>. The list <NUM> can be received from an EMR or other patient medical repository. The list <NUM>, L, includes a plurality of occurrences, Li. In some optional embodiments, the occurrences in the list <NUM> include a time/date stamp, which indicates when each occurrence was entered into the list <NUM>. In some optional embodiments, the occurrences in the list <NUM> include an identifier of the user that entered the occurrence.

For illustration purposes, the ranking engine <NUM> computes a relevance score for each occurrence, Li, according to a relevance scheme <NUM>. A relevance scheme <NUM> includes a mapping between occurrences in the controlling lexicon and computed relevance scores. For example, ICD-<NUM> codes of a patient problem list can be mapped to relevance scores represented as an interval [<NUM>,<NUM>], where higher values in the interval indicate greater relevance. The mappings of the relevance scheme <NUM> can be constructed from medical expert opinions, literature, data analysis, and the like. In some embodiments, the relevance scheme <NUM> is selected by a context manager <NUM> according to a user context <NUM>.

The ranking engine <NUM> adjusts the computed relevance score according to feedback stored in a feedback database <NUM>. The feedback can be modified or weighted by the context manager <NUM> according to the user context <NUM>. In some embodiments, the user context <NUM> of the user requesting the list <NUM> is used to select and/or weight feedback. In some embodiments, the user context <NUM> of the user requesting the list <NUM> and the user context <NUM> of each user providing corresponding feedback is used to select and/or weight the feedback. In some embodiments, the user context <NUM> includes clinical context of a patient being care for at a time of the request or the feedback.

For illustration purposes, the following embodiment is provided. A user interface <NUM> presents a ranked list <NUM> with occurrences, Li, ordered by or ranked by a presented relevance score. The presented relevance score is the adjusted relevance score, where feedback <NUM> has been received, or the computed relevance score, where no or insufficient feedback <NUM> has been received. The presented ranked list <NUM> includes indicators <NUM> for which the feedback <NUM> can be indicated through an input, such as a single touch of a display <NUM>. In some embodiments, the ranked list <NUM> is displayed again according to current feedback and prior feedback. That is, the ranking engine <NUM> receives the feedback <NUM> and re-computes the adjusted relevance score with the feedback <NUM> and that previously stored in the feedback database <NUM>, and the user interfaces <NUM> presents the re-ranked list <NUM>. In some embodiment, the re-ranking occurs with the next access of the list <NUM>.

The context manager <NUM> can select a mapping for use by the ranking engine <NUM> from among a plurality of mappings of the relevance scheme <NUM> according to the user context <NUM>. In some optional embodiments the relevance scheme can include mappings based on the identifiers of users entering occurrences and the user context <NUM>. The user context <NUM> can include elements, such as healthcare specialty or service domain of the user, a clinical context, a healthcare role of the user, and/or combinations thereof. The healthcare specialty or service domain, such as Radiology, cardiology, ICU, oncology, etc., and healthcare role of the user, such as nurse, technician, resident, attending physician, etc., can be included in the request <NUM> or can be retrieved based on a user identifier from a user profile, security database, and the like. The clinical context, such as imaging study, bedside, discharge, etc., can be inferred by a location, type of examination scheduled, and the like. The clinical context can include an anatomical identification, such as identified from an imaging modality of a scheduled imaging study. The location of the user can be obtained from a user or local computing device <NUM>, such as according to a global positioning system (GPS) component <NUM> or computer network location. The elements can be received from the user or local computing device <NUM> or other data stores and/or systems.

In some optional embodiments, each of the healthcare specialty or service domain of the user, the clinical context, or the healthcare role of the user can include a domain ontology <NUM>, which can provide a hierarchical relationship between individual occurrences in the domain. For example, a hierarchical context of a physician (parent) healthcare role can include a resident physician (child), and an attending physician (child). A relevance scheme <NUM> exists for the physician (parent) and one for the attending physician (child) and not one for the resident (child). Thus, a hierarchical reasoning by the context manager <NUM> for a resident (child) may select a physician (parent) relevance scheme <NUM>, while selecting the more specific relevance scheme <NUM> for an attending physician (child) over the physician (parent) for an attending physician. Another example includes radiology (parent) specialties, which can include subspecialties, such as by anatomy (children) and/or imaging modality (children): computed tomography (CT), magnetic resonance (MR), positron emission tomography (PET), single proton emission computed tomography (SPECT), ultrasound (US) and the like. In some instances, the clinical context according to the healthcare specialty or service domain of the user, the clinical context, or the healthcare role of the user with respect to the domain ontology <NUM> is such that healthcare practitioners more experienced and/or with better training for a given clinical and patient context receives more weight. For example, in caring for a critically ill patient in an ICU, feedback of an attending physician receives more weight than a critical care fellow. The critical care fellow receives more weight than a critical care intern. The critical care intern receives more weight than a medical student.

In addition, the elements can be inter-related, and a relevance scheme <NUM> determined from the combination based on a proximity of elements to an occurrence in the list <NUM>. For example, a proximity can be derived between a clinical context of a brain MR exam and a healthcare role of a neurosurgeon with a selected relevance scheme <NUM>.

The feedback database <NUM> electronically stores the feedback <NUM>, which can be received from one or more local or user devices <NUM> for each list by the ranking engine <NUM>. Examples of the feedback <NUM> as a structured stored feedback n-tuple can include (user identifier, occurrence identifier, presented relevance score, feedback relevance score, binary feedback, age of occurrence, patient clinical context), or (user identifier, occurrence identifier, presented relevance score, feedback relevance score, binary feedback). The user identifier can be related to the user role, user specialty, and/or combinations thereof, such as through a user profile. The occurrence identifier, Li, is an occurrence according to the lexicon, such as an ICD-<NUM> code, a CPT code, and the like. The binary feedback is an indicator or value that indicates whether the feedback indicates that the occurrence identifier is to be listed higher or lower in relevance relative to a current position according to a presented relevance score for occurrences in the ranked list <NUM>. The binary feedback can be indicated with binary values, such as <NUM> and <NUM>, binary labels, such as "Up" and "Down", and the like. The age of occurrence, such as number of days, can be obtained from the date/time stamp according to the entry of the occurrence into the list.

In some optional embodiments the feedback database <NUM> can include a relationally structured format accessed by structured query language (SQL). In some embodiments, the feedback database <NUM> can include unstructured formats, such as storing contextual information of an imaging examination from a digital imaging and communication in medicine (DICOM) header of a study, which can include free text. In some embodiments, a combination of structured and unstructured database formats can be used.

The ranking engine <NUM> computes the adjusted relevance score according to a function of all the feedback in the feedback database <NUM> for the occurrence, Li in the list <NUM>, the function including an average or mean of the feedback relevance scores Fi. The function includes only the last N feedback relevance scores. For example, if feedback for a patient problem list for emphysema (ICD-<NUM> code of <NUM>) includes i=<NUM> occurrences of feedback <NUM>, then the <NUM> feedback relevance scores most recent in time can be used. In some embodiments, the feedback from the feedback database <NUM> includes a minimum number of occurrences of the feedback <NUM>. That is, for Fi, i > X, a predetermined threshold, for the ranking engine <NUM> to compute the adjusted relevancy score. In some embodiments, the function can include only the feedback relevance scores within a fixed time range or from a most recent entry to the list <NUM>. For example, the feedback relevance scores within a most recent <NUM>-daytime period can be used, or the feedback relevance scores received after a last update of the list <NUM>. In some embodiments, the ranking engine <NUM> can use a decaying factor α to avoid abrupt changes in the feedback relevance scores between a presentation of the ranked list <NUM>. For example, with feedback relevance scores temporally ordered by F<NUM>,. , FM, the adjusted relevance score is the sum of Exp(α, i) × Fi for <NUM> ≤ i ≤ M divided by the sum of Exp(α, i) for <NUM> ≤ i ≤ M.

In some otional embodiments, the ranking engine <NUM> uses hierarchical reasoning to select or supplement the feedback from the feedback database <NUM> that is used to compute the adjusted relevance score. The ranking engine <NUM> uses the list <NUM> that includes a domain ontology <NUM> with a supporting hierarchical structure, such as with ICD codes. For example, if an occurrence Li has not received feedback <NUM>, then the feedback <NUM> for the parent of Li according to the domain ontology <NUM>, can be used by the ranking engine <NUM> to adjust the relevancy score. The parent relationship can be used recursively using the domain hierarchy to obtain feedback from the occurrence to a root. In some embodiments, the ranking engine <NUM> can use feedback at different levels in the hierarchy until a sufficient number is received. The feedback according to occurrences at each level in the hierarchy can be weighted according to a distance from the occurrence, Li, to the level of the feedback. In some instances, the ranking engine <NUM> leverages sparse feedback across the domain ontology for a particular occurrence.

For illustration purposes, the context manager can weigh the feedback <NUM> in the feedback database <NUM> according to the user context <NUM>. For a set of feedback relevance scores, F<NUM>,. , FM, for an occurrence, Li, received from corresponding users, U<NUM>,. , UM, weights, w<NUM>,. , wM, can be computed according to a proximity of the user context <NUM> of each user providing feedback to user context <NUM> of the user requesting the list <NUM>. Thus, the ranking engine <NUM> computes the adjusted relevance score using a set of weighted feedback relevance scores, F<NUM> x w<NUM>,. For example, the weight for feedback from a neuroradiologist, wi, may be greater for a neurosurgeon viewing a problem list, than the weight for feedback from an x-ray technician, wj, where wi>wj. The weight can be represented as a continuous number of a distance between the user context <NUM> of the feedback <NUM> and the user context <NUM> of the user requesting the list <NUM>. The weight can be computed as an inverse of the distance. For example, the closer the user context <NUM> of the feedback <NUM> and the user context of the requesting user, the lower the distance, the higher the weight, where the weight is expressed as (<NUM>/D). In some instances, this weighting can be used with hierarchical reasoning of the user context <NUM> according to one or more of domain ontologies <NUM>.

In some optional embodiments the context manager <NUM> can filter the feedback <NUM> according a set of rules of varying contextual granularity of the user context <NUM>. In some embodiments, the feedback <NUM> filtered from the feedback database <NUM> according to a list <NUM>, such as a patient problem list, is for a plurality of patients. In some embodiments, the feedback <NUM> filtered from the feedback database <NUM> according to a list <NUM>, is for a single patient.

For example, filtering can include a fine granularity of filtered feedback <NUM> from the feedback database <NUM> according to the user context <NUM> of the user requesting the list <NUM> that filters for (user and anatomy and imaging modality). The user is filtered for the specific user matching the user requesting the list <NUM> with the feedback <NUM> from the feedback database <NUM>, such as by user identification, and an anatomy and an imaging modality according to the clinical context are also filtered according to the match. The feedback used by the ranking engine <NUM> to compute the adjusted relevance score would then be limited to a set of feedback specific to the user, anatomy and imaging modality. A more coarse filter filters according to (user and anatomy). A further more coarse filter filters according only to (anatomy). A ladder of decreasing contextual granularity from (user, anatomy, modality) to (user, anatomy) to (anatomy) is established. The context manager <NUM> can filter records from the feedback database <NUM> at varying granularities to obtain a sufficient number of feedback for the ranking engine <NUM> to compute the adjusted relevance score. That is, if filtering at one level of granularity does not produce a number of feedback records, F<NUM>,. , Fn, that exceeds a predetermined threshold, the filter can be reapplied with a next decreased level of granularity until sufficient feedback is obtained.

Filtering at different granularities can include the clinical context with data obtained from DICOM headers, such as anatomy, protocol, and/or modality. For example, the granularities can be extended to include (user, anatomy, protocol, modality), (user, anatomy, modality), (user, anatomy) and (anatomy). In some instances, the filtering at different levels of granularity can provide for pooling of feedback between users, and/or pooling of feedback from more generalized clinical contexts and the like.

The ranking engine <NUM>, the context manager <NUM> and the user interface <NUM> are suitably embodied by one or more configured processors, such as one or more processors <NUM> of the user or local computing device <NUM> and one or more processors <NUM> of a computer server <NUM>. The configured processor(s) <NUM>, <NUM> execute at least one computer readable instruction stored in computer readable storage medium, such as the memory <NUM> of the user or local computing device <NUM> or server <NUM>, which excludes transitory medium and includes physical memory and/or other non-transitory medium to perform the disclosed relevance score computing, ranking, contextual determination, hierarchical reasoning, feedback and display techniques. The configured processor may also execute one or more computer readable instructions carried by a carrier wave, a signal or other transitory medium. The user or local computing device <NUM> can comprise a workstation, laptop, tablet, smart phone, body worn computing device, combinations and the like. The server <NUM> can comprise one or more computer servers known in the art. The lines between components represented in the diagram represent communications paths, which can be wired or wireless.

The relevance scheme <NUM>, feedback database <NUM> and the user context <NUM> are suitably embodied by computer storage media, such as local disk, cloud storage, remote storage, and the like, accessed by one or more configured computer processors. The user or local computing device <NUM> includes the display device <NUM>, such as a computer display, projector, body worn display, and the like, and one or more input devices <NUM>, such as a mouse, keyboard, microphone, touch or gesture interface, and the like. The local or user computing device <NUM> includes processors <NUM>, such as a digital processor, a microprocessor, an electronic processor, an optical processor, a multi-processor, a distribution of processors including peer-to-peer or cooperatively operating processors, client-server arrangement of processors, and the like.

Referring to an embodiment, an example of a contextual list view of a patient problem list <NUM> with sparse feedback is disclosed. The patient problem list <NUM> is a presented list <NUM> that includes occurrences <NUM> of problems reported for a patient. For example, occurrences include "Falls frequently," "Aneurysm of anterior cerebral artery," and "Diabetes mellitus.

Each occurrence <NUM> includes the indicators <NUM>, which are illustrated as "[Down]," "[Up]. The indicators <NUM>, such as two touch sensitive areas, two buttons, and the like, are used to indicate binary feedback in an input that the corresponding occurrence is to be re-ranked up, is more relevant, or is to be positioned higher in the list, or is to be re-ranked down, is less relevant, or is to be positioned lower in the presented list <NUM>. The input can be a single input, such as a single touch, single gesture, single mouse click, and the like.

In some optional embodiments, the indicators <NUM> are sticky, so that if the user presses "Up" a second time, the indicator <NUM> is de-selected or non-indicated. In some embodiments, the indicators <NUM> interact like radio buttons such that if one is selected, the other is automatically de-selected. In some embodiments, once the indicator <NUM> is selected, the indicators <NUM> are removed for all occurrences until a next presentation of the ranked list <NUM>. The view displaying the ranked list <NUM>, such as the patient problem list <NUM>, can include other visual aids, such as a scroll bar. That is, the view of the ranked list <NUM> can include a displayed portion of the ranked list <NUM>, while the entire ranked list <NUM> remains accessible through the visual aid. The view displaying the ranked list <NUM> can include a position of an occurrence in the ranked list <NUM>, such as numbered from <NUM> to N of a list of N occurrences, and the ranked list <NUM> is ranked according to the presented relevancy score.

In an example, rules that adjust relevance scores through feedback are disclosed. The example rules are formatted in a table according to the binary feedback or feedback indicators <NUM> and a position of the occurrence in the ranked list <NUM>. The rules provide a mapping from a position of an occurrence in the ranked list to a feedback relevance score as a function of presented relevance scores of one or more occurrences.

In some optional embodiments, the example rules are generalized according to a top X occurrences of the presented relevance score and a bottom Y occurrences of the presented relevance score, where X and Y are integers. For example, X and Y can be <NUM>. X and Y can be the same or different. In some embodiments X and Y can be based on the length of the list <NUM>, that is, the number of occurrences, Z, in the list <NUM>. For example, where Z < <NUM>, the top Z/<NUM> can be used rounded down to a nearest integer.

A first set of example rules address a feedback indicator <NUM> value of "Up" with the feedback for an occurrence in the top X presented occurrences. Two alternative rules are presented. A first rule includes the feedback relevance score computed as the sum of highest presented relevance score plus a constant. For example, with the top <NUM> presented relevance scores of (. <NUM>), and a constant of. <NUM>, the feedback relevance score is computed as. A second rule computes the feedback relevance score as a square root of the highest presented relevance score. Using the above set of top <NUM> scores, the feedback relevance score is computed as the SQRT (.

A second set of example rules address a feedback indicator <NUM> value of "Up" with the feedback for an occurrence not in the top X presented relevance score occurrences. The position of the occurrence with the binary feedback is lower in the list than the top X presented relevance score occurrences. The example rule of a computed feedback relevance score is an average of the presented relevance scores of the top X occurrences. Using the above example set of top <NUM> presented relevance score occurrences, the average of (.

A third set of example rules address a feedback indicator <NUM> value of "Down" with the feedback for an occurrence among in the top X presented relevance occurrences. An example rule computes the feedback relevance score as an average or a median of presented relevance scores for all presented relevance scores but the top X occurrences. That is, a set of presented relevance scores for computing the average includes those presented relevance scores for the ranked list <NUM> excluding the top X occurrences of the presented relevance scores. Another example alternative rule computes the feedback relevance score as an average or a median of presented relevance scores for the bottom Y presented relevance scores.

A fourth set of example rules address a feedback indicator <NUM> value of "Down" with the feedback for an occurrence not among in the top X occurrences. An example rule computes the feedback relevance score as a constant, such as zero.

An illustrative embodiment of a method of viewing a contextual list with sparse feedback is disclosed.

At <NUM>, the list <NUM> is received. The list <NUM> can be received in response to the request <NUM> from the user or local computer device <NUM>. The request can include the user context <NUM> or data, such as user identification, GPS location, and the like, which is used to identify the user context <NUM> according to a profile or other data stores.

At <NUM>, a relevance score is computed for each occurrence of the list <NUM>. The computed relevance score is computed according to a relevance scheme <NUM> that maps relevance scores for a lexicon controlling the list <NUM> to individual occurrences. The relevance scheme <NUM> can be selected according to the user context <NUM> by a context manager <NUM>. The selection can include hierarchical reasoning and/or a proximity measurement between the user context <NUM> and the selected relevance scheme <NUM>.

At <NUM>, an adjusted relevance score is computed according to feedback stored in a feedback database <NUM> for the list <NUM> by the ranking engine <NUM>. The feedback can be filtered, weighted, and/or supplemented by the context manager <NUM> according to the user context <NUM> of the user requesting the list <NUM> and the user context <NUM> of a user that provided the feedback <NUM> stored in the feedback database <NUM>. The filtering, weighting, and/or supplementing can use hierarchical reasoning according to the domain ontology, which establishes relationships between occurrences of the elements of the user context <NUM>. The filtering, weighting, and/or supplementing can use a distance measurement between elements of the user context <NUM> of the user requesting the list <NUM> and the user context <NUM> of a user that provided the feedback <NUM> stored in the feedback database <NUM>.

At <NUM>, the list <NUM> is presented as the ranked list <NUM>, ordered by a presented relevance score. The presented relevance score for each occurrence is the adjusted relevance score if sufficient feedback is present in the feedback database <NUM>. Otherwise, the presented relevance score is the computed relevance score. That is, if the feedback in the feedback database <NUM> is null or insufficient (does not exceed a predetermined threshold), the presented relevance score is the computed relevance score, computed at <NUM>, otherwise is the adjusted relevance score, computed at <NUM>. The presented ranked list <NUM> is displayed on the display device <NUM> of the user or local computing device <NUM> and includes feedback indicators <NUM> for each occurrence of the ranked list <NUM>.

At <NUM>, feedback <NUM> is received for one occurrence of the presented ranked list <NUM>. The feedback <NUM> includes a binary indicator or value, which indicates that the corresponding occurrence is to be ranked higher or lower relative to the entire. A feedback relevance score is computed according to a set of rules, which is included in the feedback and stored in the feedback database <NUM>.

The above may be implemented by way of computer readable instructions, encoded or embedded on computer readable storage medium, which, when executed by a computer processor(s), cause the processor(s) to carry out the described acts. Additionally or alternatively, at least one of the computer readable instructions is carried by a signal, carrier wave or other transitory medium.

Claim 1:
A system, comprising:
a ranking engine comprising a first processor configured to receive a list for a patient which includes a plurality of occurrences, compute a relevance score for each occurrence in the list, wherein the computed relevance score is according to a relevance scheme that maps relevance scores from a lexicon controlling the list to each of the plurality of occurrences; and
a user interface comprising a second processor configured to:
display the list on a display device of a local computing device ordered by a presented computed relevance score, wherein each displayed occurrence of the plurality of occurrences includes a feedback indicator;
receive feedback comprising an input for one displayed occurrence of the plurality of occurrences according to the feedback indicator which indicates the one displayed occurrence is to be displayed higher or lower in the list than a current position, wherein the input is a binary indicator; and
wherein the user interface is further configured to compute a feedback relevance score according to the binary indicator and a set of rules, wherein the feedback comprises the feedback relevance score and
the system further comprises a feedback database comprising electronic storage and one or more processors configured to receive and store the feedback, and
wherein the ranking engine is further configured to compute an adjusted relevance score according to the feedback for at least one occurrence of the plurality of occurrences,
wherein the ranking engine computes the adjusted relevance score according to a function of all the feedback in the feedback database for the at least one occurrence of the plurality of occurrences in the list, the function including an average or mean of the feedback relevance scores and the function including only the last N feedback relevance scores; and
the presented computed relevance score is the adjusted relevance score, where the feedback has been received, or the computed relevance score, where no or insufficient feedback has been received.