CURATION SYSTEMS AND METHODS THAT IMPROVE THE PERFORMANCE AND ACCURACY OF ARTIFICIAL INTELLIGENCE SYSTEMS

Provided herein are systems and methods that improve the performance and accuracy of artificial intelligence (AI) systems and enhance real-world uses thereof. For example, provided herein are expert curation systems and methods that prevent or reduce the frequency of AI hallucinations; allow for rapid identification of errors, misinformation, and out of date information; enable faster and easier corrections; and provide accurate and actionable results.

FIELD

Provided herein are systems and methods that improve the performance and accuracy of artificial intelligence (AI) systems and enhance real-world uses thereof. For example, provided herein are expert curation systems and methods that prevent or reduce the frequency of AI hallucinations; allow for rapid identification of errors, misinformation, and out of date information; enable faster and easier corrections; and provide accurate and actionable results.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Nos. 63/633,507, filed Apr. 12, 2024, and 63/633,517, filed Apr. 12, 2024, the contents of which are herein incorporated by reference in their entirety.

BACKGROUND

Artificial intelligence (AI) and machine learning (ML) offer the promise of faster results, greater precision, and the recognition of previously unappreciated complex correlations between variables with real-world impact. Recent years have seen significant achievements in AI/ML. However, AI/ML models notoriously hallucinate or provide incomplete and inaccurate information. Depending on the application, hallucinations may be harmless and/or manageable. In other applications, hallucinations can have dire consequences. The opacity and uncertainty on the veracity of underlying training data for Large Language Models (LLMs) and Large Multimodal Models (LMMs), and unresolved issues on fair attribution and compensation for intellectual property used for model training, also present long term questions on sustainable solutions to realize the full potential of AI

Improved systems are needed.

SUMMARY

In some embodiments, provided herein are hybrid/human “expert in the loop” AI systems and methods. In some embodiments, the systems and methods include granular attribution and revenue sharing for curation participants. These approaches, alone and/or in combination facilitate the generation of more accurate, trustworthy, actionable, and sustainable decision support guidance.

In some embodiments, provided herein are systems for expert curation of source materials (e.g., training data source materials) for an artificial intelligence (AI) system. In some embodiments, the systems comprise a computer processor that tracks a plurality of individuals wherein any or all of the plurality of individuals validate each source material for a given subject matter to generate a curated library of validated source materials for the given subject matter for use as training data for the AI system. In some embodiments, the validated source materials are relevant and accurate to the given subject matter as reviewed and analyzed by the plurality of individuals.

In some embodiments, source materials are identified by one or more of the plurality of individuals or a user of the AI system.

In some embodiments, the plurality of individuals are defined into two or more tiers based on qualifications in the given subject matter. In some embodiments, any or all of the plurality of individuals name, authenticate and/or remove individuals in lower tiers

In some embodiments, the plurality of individuals is selected by an adjudication board. In some embodiments, the adjudication board comprises two or more top-tier experts in fields comprising the given subject matter. In some embodiments, the adjudication board defines subject matter specific databases or knowledge bases (e.g., databases or knowledge bases accessible by a generative AI inference system, curated databases or knowledge bases, language model training databases, etc.).

In some embodiments, the two or more tiers comprises an advisory board comprising subject matter experts selected by the adjudication board. In some embodiments, the advisory board manages the curation of source material (e.g., training data source material) for the given subject matter.

In some embodiments, the two or more tiers comprises one or more tiers comprising administrators and/or curators named by members of the advisory and/or adjudication boards. In some embodiments, the advisory board creates and assigns responsibilities to the one or more tiers of administrators and/or curators. In some embodiments, the administrators and/or curators comprise leading practicing individuals in the given subject matter. In some embodiments, the administrators and/or curators: contribute to defining and auditing topic specific databases or knowledge bases (e.g., databases or knowledge bases accessible by a generative AI inference system, curated databases or knowledge bases, language model training databases, etc.); audit, select, ingest, update, and/or removes source material; and/or collate and review comments from any or all of the plurality of individuals in the curation system and users of the AI system.

In some embodiments, the two or more tiers comprises one or more commentators to review, rate, recommend source materials (e.g., training data source materials) and responses from any or all of the plurality of individuals in the curation system.

In some embodiments, the plurality of individuals comprises one or more moderators to review, rate and recommend user responses and questions input into the AI system.

In some embodiments, the system further comprises users of the AI system. In some embodiments, the users of the AI system provide feedback on AI system and/or identifies source materials.

In some embodiments, the system comprises a quality control system to organize, standardize, tokenize, and/or render machine readable each validated source material. In some embodiments, the quality control system processes, updates, and/or corrects any or all metadata, citations, attributions, notes, or recommendations for each source material.

In some embodiments, the system further comprises a non-transitory computer-readable medium and/or one or more processors for storing validated source materials and any or all associated metadata, citations, attributions, notes, or recommendations for each source material in the curated library.

In some embodiments, provided herein are methods for generating a curated library of source materials for an artificial intelligence (AI) system comprising providing one or more source materials for a given subject matter to the curation system as described herein for validation.

DEFINITIONS

As used herein, terms and phrases such as “having,” “may have,” “include,” or “may include” a feature (such as a number, function, operation, or component, such as a component) indicate the presence of that feature, and do not preclude the presence of other features. Further, as used herein, the phrase “a or B,” “at least one of a and/or B,” or “one or more of a and/or B” may include all possible combinations of a and B. For example, “a or B,” “at least one of a and B,” and “at least one of a or B” may indicate all of the following: (1) comprises at least one A, (2) comprises at least one B, or (3) comprises at least one A and at least one B. Furthermore, as used herein, the terms “first” and “second” may modify various components without regard to importance, and do not limit the components. These terms are only used to distinguish one component from another. For example, the first user device and the second user device may indicate user devices that are different from each other regardless of the order or importance of the devices. A first component may be termed a second component, and vice-versa, without departing from the scope of the present disclosure.

It will be understood that when an element (such as a first element) is referred to as being (operatively or communicatively) “coupled/coupled” or “connected/connected” to another element (such as a second element), it can be directly coupled or connected/coupled or connected to the other element (such as the second element) or via a third element. Conversely, it will be understood that when an element (such as a first element) is referred to as being “directly coupled”/“directly coupled to” or “directly connected”/“directly connected” to another element (such as a second element), there is no other element (such as a third element) intervening between the element and the other element.

As used herein, the phrase “configured (or set) to” may be used interchangeably with the phrases “adapted to,” “having . . . capability,” “designed to,” “adapted to,” “made to,” or “capable,” as the case may be. The phrase “configured (or set) to” does not substantially mean “specially designed in hardware.” Rather, the phrase “configured to” may indicate that a device is capable of performing an operation with another device or component. For example, the phrase “a processor configured (or arranged) to perform A, B and C” may refer to a general-purpose processor (such as a CPU or an application processor) or a special-purpose processor (such as an embedded processor) that may perform operations by executing one or more software programs stored in a memory device.

The various functions described below may be implemented or supported by one or more computer programs, each formed from computer-readable program code and embodied in a computer-readable medium. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof adapted for implementation in suitable computer readable program code. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as Read Only Memory (ROM), Random Access Memory (RAM), a hard disk drive, a Compact Disc (CD), a Digital Video Disc (DVD), or any other type of memory. A “non-transitory” computer-readable medium does not include a wired, wireless, optical, or other communication link that transmits transitory electrical or other signals. Non-transitory computer readable media include media that can permanently store data as well as media that can store data and later rewrite the data, such as rewritable optical disks or erasable memory devices.

Various functions described below may be implemented or supported by one or more natural language communication systems (“NLCS”), which function as networks of interconnected components designed to accept, process, and generate human language. Such systems may include one or more of the following characteristics or structure: input processing, language understanding, knowledge representation, language generation, output presentation, and feedback loops.

NLCS may receive input in the form of text or speech. Inputs not in the form of text, for example, audio, video, images, databases can be converted into text as appropriate. Text input is typically tokenized, while speech input undergoes transcription into textual form through speech recognition algorithms before being tokenized. “Tokenized” refers to the process of segmenting a sequence of text into smaller units, typically words, subwords, or characters, known as tokens. Tokenization involves identifying word boundaries and separating punctuation marks, whitespace, and other delimiters to create a structured representation of the text that can be processed by the NLCS and serves as the basis for further analysis and processing. NLCS may employ various techniques such as statistical models, deep learning architectures, and semantic analysis to understand the meaning of the input text. This includes tasks like named entity recognition, part-of-speech tagging, syntactic parsing, and semantic role labeling to extract relevant information and comprehend the context of the input. Structured databases, knowledge graphs, or embeddings may be utilized to represent information and knowledge extracted from text data.

Inference mechanisms may be used to derive conclusions, make predictions, or answer questions based on the input and various heuristics. This involves various reasoning techniques such as deductive, inductive, or abductive reasoning, as well as probabilistic reasoning to deal with uncertain information. After processing the input and performing any necessary reasoning, NLCS may generate responses or output in natural language form. Generation techniques may include template-based approaches, rule-based systems, or more advanced methods like sequence-to-sequence models with attention mechanisms. The generated output may be presented to the user in a human-readable format, which may involve text rendering for text-based interactions or speech synthesis for voice-based interactions. The generated output may also be presented in non-text based formats e.g., audio, video, images, and the like. Output presentation may also include formatting, summarization, and other post-processing tasks to enhance readability, usability, and relevance. NLCS may also incorporate feedback mechanisms to improve their performance over time. This feedback may come from user interactions, explicit corrections, or implicit signals such as user satisfaction metrics, which may be used to update and refine the system's models and algorithms.

NLCS may include or be supported by a “neural network,” or a computational model consisting of interconnected nodes, or “neurons,” which receive individual input signals, process them, and produce an output signal. Information may flow through the network from an input layer, through hidden layer(s), and then to the output layer. The input layer is the first neuron layer, where input data is fed into the network. Each neuron in the input layer may represent a feature or attribute of the input data. Hidden layers are intermediate layers between the input and output layers in a neural network, which perform transformations on the input data using weighted connections and activation functions. The output layer of a neural network is the final layer, where the network produces its output predictions or classifications. The number of neurons in the output layer may correspond to the number of output classes or dimensions of the prediction. An activation function is a mathematical function applied to the weighted sum of inputs at each neuron in a neural network. Weights and biases are parameters within a neural network that are learned during the training process. Weights may be understood to represent the strength of connections between neurons, determining the influence of one neuron's output on another. Biases are additional parameters added to each neuron that shift the activation function. Neural networks may use various training techniques such as backpropagation. Backpropagation based training may use an algorithm to update the weights of a neural network based on the error between the predicted output and the true output and may involve calculating the gradient of the error with respect to the network's weights and adjusting the weights to minimize the error.

As used herein, the term “database” refers to an organized collection of structured information, or data, typically stored electronically in a computer system.

As used herein, the term “knowledge base” refers to a store of information that is available to draw on. When used in reference to curated knowledge bases, the knowledge bases can include not only text, other information contained in curated documents (e.g. in for the form of images, charts, graphs, etc.), or other curated media (e.g., audio, video, images, databases), but also curator annotations that guide when (e.g., for what types of questions) each knowledge base is used to generate responses, and how portions of the knowledge base are used.

The terms and phrases used herein are used only to describe some embodiments of the present disclosure and do not limit the scope of other embodiments of the present disclosure. It is to be understood that the singular includes plural referents unless the context clearly dictates otherwise. All terms and phrases used herein (including technical and scientific terms and phrases) have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the present disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. In some instances, the terms and phrases defined herein may be construed to exclude embodiments of the disclosure.

Definitions for other specific words and phrases may be provided throughout this patent document. Those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope. The scope of patented subject matter is defined only by the claims. Any other term used in the claims, including, but not limited to, “mechanism,” “module,” “device,” “unit,” “assembly,” “element,” “member,” “apparatus,” “machine,” “system,” “processor,” or “controller,” is understood by the applicants to refer to structures known to those of ordinary skill in the relevant art.

DETAILED DESCRIPTION

Human-conducted empirical studies have significantly advanced our understanding of the world over the previous centuries, decades, and years. But progress is slow and is often narrowly focused on specific sub-parameters of a specific problem or on simple correlations between a limited number of variables. Artificial intelligence (AI) and machine learning (ML) offer the promise of faster results and the recognition of previously unappreciated complex correlations between variables. Recent years have seen significant achievements in AI/ML, and a huge rise in generative AI and its applications. However, AI/ML systems are limited by the quality of information used to train them. This is particularly evident in applications that generate output containing facts. AI systems with gaps or inaccuracies in their training data may notoriously “hallucinate,” or fail to produce output at all. Depending on the application, mistakes or hallucinations may be harmless and/or manageable. In other applications, hallucinations can have dire consequences. Other issues include problems with attribution, safety, and bias, in addition to the misinformation/hallucinations (see e.g., Menz et al., Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis, BMJ, 2024, 384: e078538; Tyson and Kennedy, Many Americans think generative AI programs should credit the sources they rely on, Pew Research Center, Mar. 26, 2024; and Editorial, How to support the transition to AI-powered healthcare, Nature Medicine, 30:609-610 (2024); each of which is herein incorporated by reference in its entirety).

Several recent stories have highlighted the problem of hallucinations. A Dec. 3, 2022, post on the Hacker News forum highlighted a hallucination by ChatGPT. The user queried ChatGPT to “provide references that deal with the mathematical properties of lists.” ChatGPT returned five citations by title, author, and hyperlink. The user was “pretty surprised and happy” because searches using the GOOGLE search engine had failed to produce any useful results. It turned out that everyone one of ChatGPT's citations was made up. The references did not exist, and the links were not real. The cited authors never published papers with the recited titles. In this instance, time was wasted and perhaps trust was lost. But there were no dire consequences.

U.S. News reported a story on Jun. 22, 2023, explaining that a federal judge imposed a $5000 fine on two lawyers and a law firm based on submission of legal documents containing fictitious legal citations created by ChatGPT.

A study entitled “Medical Hallucination in Foundation Models and Their Impact on Healthcare” from researchers at Massachusetts Institute of Technology, Harvard Medical School, University of Washington, Carnegie Mellon University, Seoul National University Hospital, Google, Columbia University, and Johns Hopkins University published on Mar. 3, 2024, states that “non-trivial levels of hallucination persist. These findings underscore the ethical and practical imperative for robust detection and mitigation strategies, establishing a foundation for regulatory policies that prioritize patient safety and maintain clinical integrity as AI becomes more integrated into healthcare. The feedback from clinicians highlights the urgent need for not only technical advances but also for clearer ethical and regulatory guidelines to ensure patient safety.”

When it comes to health care, the risk of hallucinations, incorrect information or out of date source material can be more consequential. For example, Ross (Why the early tests of ChatGPT in medicine miss the mark, Health Tech, Apr. 3, 2023) notes that:

When it comes to health care, the risk of hallucinations can be more consequential. For example, a large multi-modal model (LMM) trained on all available data sources could be one where a prompt query asking for treatments of syphilis results in recommendations from long-debunked medieval treatments such as leeches or mercury. While outrageous, this is a realistic scenario if an LMM had been trained on academic manuscripts from the Middle Ages.

If an AI is designed or coerced to return a complete response versus an incomplete one or none at all, a profoundly inaccurate hallucination that is presented as factual (often with fabricated references) can occur and have real-world and very dangerous consequences both to individual safety, trust, and liability.

Subtle instances of misinformation can equally be dangerous and consequential, as they may go unnoticed. One example of a dangerous hallucination that could do real harm and breed mistrust would be AI recommendations based on out-of-date or non-peer-reviewed research. For example, OpenAI's ChatGPT model is based on training data only up until April 2023, so any new or updated research and findings in the medical field would not be reflected in its responses, and there is still little transparency on what data its model is based on and how frequently it's updated. For instance, new research published by the WHO in the Lancet (Anderson et al., Health and cancer risks associated with low levels of alcohol consumption. Lancet Public Health. 2023 January; 8 (1): e6-e7) states that there is no safe threshold for alcohol consumption and the risk of cancer and heart disease correlates to alcohol consumption even at low levels. If an end user or health practitioner relied on recommendations from an AI agent based on outdated or training data not certified by medical experts, it could incorrectly suggest that moderate drinking is still safe or ok when the latest scientific evidence recommends otherwise to maximize healthspan and longevity. This could be considered malpractice and present real health risks and expose any companies who misrepresent medical recommendations to liability.

These and other problems are addressed by the technology provided herein. For example, embodiments of the present technology reduce, minimize, and/or eliminate AI/ML shortcomings through use of expert curation systems and methods. The systems and methods employ one or more levels of expert curation to manage information content used in AI system training and, in some embodiments, to audit AI system performance and make changes, as necessary or desired, to maximize performance.

For example, in some embodiments, the systems and methods employ an administrator-controlled, secured, tiered provisioning system to allow only authenticated and verified users (“curators”) to select, upload, ingest, edit, and update content (e.g., websites, papers, articles, tables, charts, audio, media files, transcripts, books, and other digital and analog materials (“sources”)) into a database or knowledge base accessible by a generative AI inference system (e.g., a vector index accessible by a novel RAG (retrieval-augmented-generation)-based AI inference system), an AI language model training database, or the like. In some embodiments, this model is purpose-built only to be based on source data related to predetermined subject matter and/or to only use details from within curated source data and to provide visibility when sources are used outside of the context in which they were curated by “Administrators” to ensure relevance accuracy, and mitigate incorrect responses or hallucinations. “Administrator” and “curators” roles can have a range of customizable permissions, controls, access, and influence on source data and metadata.

The number of tiers and the qualifications of individuals within the tiers will vary depending on the subject matter. In some embodiments, an adjudication board or individual sits at a top level and supervises one or more sub-specialties within the general subject matter area. In some embodiments, the adjudication board or individual nominates, votes for, authenticates, and/or revokes access for tiers that reside below it. In some embodiments, the adjudication board or individual is provided the ability to define, provision, and create discrete databases or knowledge bases (e.g., databases or knowledge bases accessible by a generative AI inference system, curated knowledge bases, language model training databases, etc.) across the sub-specialties that it supervises. In some embodiments, the adjudication board or individual roles are populated by top-tier experts in the field, ideally with organizational management experience.

In some embodiments, residing under the adjudication board is a specialized advisory board or individual that manages a sub-specialty within the general subject matter area managed by the adjudication board. In some embodiments, this tier invites, authenticates, and revokes access for administrators that oversee the recruitment and management of curators for their given field of expertise. In some embodiments, the specialized advisory board or individual is populated by respected and established, and where appropriate, certified, subject matter experts.

In some embodiments, residing under the specialized advisory board is one or more super administrators. In some embodiments, super administrators define, provision, and audit discrete databases or knowledge bases (e.g., databases accessible by a generative AI inference system, curated knowledge bases, language model training databases, etc.) and are authorized to name, invite, edit, and/or revoke administrator roles. In some embodiments, the super administrator has all administrator functionalities. In some embodiments, the super administrator is an experienced subject matter expert with administrative experience, for example, a department dean at a top tier academic institute, or equivalent, in the particular subject matter domain.

In some embodiments, residing under the super administrator is one or more administrators. In some embodiments, administrators audit authorized training databases (e.g., an administrator and associated curators can only access corresponding databases that they have been invited to (e.g., that relate to the subject matter sub-specialty)) and nominate, invite, approve, and revoke super curator roles. In some embodiments, the administrator has all super curator functionalities. In some embodiments, the administrator is an experienced subject matter expert, for example a tenured academic professor, or equivalent, in a subject-specific sub-category.

In some embodiments, residing under the administrator is one or more super curators. In some embodiments, super curators audit, select, ingest, update, and remove training data from authorized training databases. In some embodiments, super curators review all recommendations, ratings, responses, flags, and comments from all user roles. In some embodiments, super curators name, invite, edit, and review curator roles. In some embodiments, super curators have all curator functionalities. In some embodiments, the super curators are subject matter experts, for example, associate or non-tenured professors, or equivalent, in a subject-specific sub-category.

In some embodiments, residing under the super curator is one or more curators. In some embodiments, the curators audit, review, recommend, and rate data sources, prompts, and responses for authorizing training databases. In some embodiments, the curators annotate the source materials. In some embodiments, the curators name, invite, edit, and revoke commenter roles. In some embodiments, the curators have all commentator functionalities. In some embodiments, the curators are subject matter knowledgeable, for example professionals or researchers working in the field of the subject matter sub-category.

In some embodiments, residing under the curator is one or more commentators. In some embodiments, the commentators review, recommend, and rate source data, prompts, and responses for authorized training databases. In some embodiments, the commentators name, invite, edit, and revoke moderator roles. In some embodiments, the commentators have all moderator functionalities. In some embodiments, the commentators are graduate students, or equivalent, in the field of the subject matter sub-category.

In some embodiments, residing under the commentator is one or more moderators. In some embodiments, moderators review end-user prompts and responses, flags, ratings, and recommendations. In some embodiments, moderators name, invite, and revoke end user roles. In some embodiments, moderators are graduate students, or equivalent, in any field related to the subject matter sub-category.

In some embodiments, moderators interact with end users. In some embodiments, end users answer, edit survey questions, and upload personal health data and information. In some embodiments, end users enter text-based prompts and questions and rate and comment on responses and recommendations based on prompts and questions. In some embodiments, end users can enter prompts and questions and rate and comment on responses and recommendations based on prompts and questions using non-text-based means, e.g., audio and video. End users include any user interested in interacting with the system and include professionals, students, researchers, service providers, service users, individuals associated with advocacy groups, government employees, and general individuals.

In some embodiments, failures by the AI system to generate answers, or answers that end-users rate as low-quality, are provided as feedback to the curation system. This feedback refines the model and informs future curation of information to train future iterations of the AI system.

An example of a curation system focused on the field of health care and on the sub-topic of atherosclerosis is provided in Example 1 below to illustrate an implementation of such systems and methods.

The systems and methods may employ any one or more of the curation tiers identified above. In some embodiments, responsibilities for the above tiers are consolidated into a single tier. For example, a single administrator tier may take on the activities and responsibilities of the above listed super administrator and administrator tiers. Likewise, a single curator tier may take on the activities and responsibilities of the super curator and curator tiers.

In some embodiments, the system implements a topic-based architecture for organizing source materials, wherein each topic corresponds to a primary curator's domain of expertise. This architectural approach permits a distributed curation model in which a primary curator can programmatically delegate access privileges to additional curators for collaborative contribution within a specific topic domain. In some embodiments, the query processing subsystem employs multiple algorithmic techniques to match incoming queries to the most relevant topic or topics, including: qualification-based matching that evaluates curator expertise profiles against query content; vector-based semantic similarity calculations between query embeddings and topic source material embeddings; structured tag and keyword association systems; and other computational matching methodologies.

In some embodiments, within each topic, the system maintains a document collection architecture wherein curator-assembled source materials undergo a multi-layer validation process. The validation processing system captures both document-level annotations and granular annotations at the subsection, paragraph, and sentence levels. The annotation system implements a rating schema that programmatically determines content inclusion or exclusion parameters when generating responses related to the topic. In some embodiments, the system also processes natural language annotations, which are computationally incorporated as contextual instructions or supplementary information during response generation. In some embodiments, the annotation subsystem is configured to permit real-time updates, with changes propagating through the system's vector indices and retrieval mechanisms. In some embodiments, the system includes a collaboration module that collates annotation contributions for systematic review by the primary curator.

In some embodiments, the system additionally implements automated annotation generation capabilities. The automated annotation subsystem employs computational logic to identify and exclude document sections with low information value for training or response generation, such as pagination elements, standard disclaimers, or funding acknowledgments that do not contribute substantive content.

In some embodiments, the system incorporates a content coverage analysis module wherein curators define a structured outline of required content components for a given topic. For example, a topic focused on “increasing healthspan via improved musculoskeletal health” might specify a computational outline including parameters for strength training methodologies, cardiovascular exercise protocols, mobility enhancement techniques, and related subcategories. This structured outline permits the system to algorithmically analyze the coverage provided by source documents, accounting for inclusion/exclusion parameters derived from annotations. When document exclusions result in coverage gaps, the system generates automated alerts to curators identifying specific content domains requiring additional source materials.

Information may be processed and managed by the systems and methods in any manner that achieves the desired outcomes. In some embodiments, administrators or curators identify and nominate curated material (e.g., training data material (e.g., a newly published peer research paper in a particular journal on a specific topic)). Sources of discovery can come from individual news intake, professional networks, or “bubbled up” from end user, moderator, or commentor recommendations.

Once a “quorum” of endorsements on nominated source material by a “committee” (e.g., a predefined number or percent) of curators or administrators in a given specialization field is reached (e.g., based on an internal “Peer Reviewed” protocol), source material is sent to a “Data Quality Control Team” (Data QC Team) or an automated QC system to ingest into a training database. In some embodiments, the Data QC Team or automated QC system transforms source material into standardized, tokenized, machine-readable format with all associated metadata (see below), including detailed source material attribution. Once source material is ingested, curators/administrators who nominated and voted in favor are notified to prompt LLM to test and verify that source material is correctly referenced in responses. The Data QC team or automated QC system verifies that all appropriate metadata attributions and citations are correctly tracked and displayed and that the changelog reflecting these updates references all modifications, contributions, and attributions. The curators and the Data QC team or automated QC system provides expert validation, compares views to identify contradictions between sources, tracks for evolving guidelines, and provides a mechanism for gauging and ensuring confidence in the source materials.

Source materials can originate from multiple inputs and vectors. The source materials are not limited by media type or source. Source material may be from websites, papers, media files, transcripts, books, articles, presentations, and other digital and analog materials and may include tables, charts, audio, videos, images, and the like. Exemplary source materials include, but are not limited to, journal publications, surgery or lecture videos, conference presentations including text, audio, video, or images therefrom, podcasts, videos or other multimedia accompanying educational materials or courses, diagnostic images, medical calculators or decision aids, virtual models, exercise videos based on individual health conditions, medical pamphlets, recorded consultations with experts, and the like.

Traditional “top-down” discovery may be via journal publications and professional networks through which curators find and nominate peer-reviewed studies or other verified source materials directly. As the number of end users grows to thousands and millions of users, bottom-up or user-generated discovery and selection of source material occurs through the analysis of user interactions, prompts, responses, ratings, and comments at scale; enabled by AI but with humans in the loop at every phase.

Source materials can be selected using a variety of different overarching approaches. In some embodiments, the source materials are selected based on an evidence-based hierarchical approach. Such an approach, for example, prioritizes peer-reviewed source materials, includes guidelines from major organizations, incorporates educational materials and reference works, and selectively adds high-quality websites in conjunction with systematic reviews and meta-analyses. Additionally or alternatively, diversity considerations may be used when selecting source materials. These considerations ensure representation across different specialties, include information relevant to diverse populations (e.g., age, gender, ethnicity), balance technical content with user-friendly materials, and consider regional variations, when appropriate.

Source materials can be annotated during, after, or concurrently with discovery and selection. For example, the source materials can be annotated to for a particular specialty category (e.g., cardiology, pediatrics, etc.), evidence level classification (e.g., randomized controlled trials, observational study, expert consensus), publication date and currency indicators, target audience (e.g., clinicians vs. patients), geographical relevance, confidence/certainty level of medical claims.

In some embodiments, the quality control system comprises a document processing pipeline optimized for handling documents such as PDFs, common in scientific literature and medical/health research journals. In some embodiments, each page of a PDF is first converted to an image using conversion tools. Concurrently, text and non-text components (e.g., images, charts, graphs) are extracted. In some embodiments, the system analyzes the page layout using tools such as AWS Textract, which identifies how text flows across page layout elements such as columns, with possible components such as tables, charts, and captions interrupting the flow of text, and extracts text and components into blocks. For blocks which are not text components (e.g., charts, images), the system may convert to text by using an LLM to describe the component. Optionally, the text surrounding this component may be provided as context in generating a description of the chart, image, etc. In some embodiments, the system extracts text from ordered blocks and combines into a single, accurately flowing body of text, organized by pages of the original document. The page number of the original document is stored as page-level metadata.

In some embodiments, the quality control system converts page data into document chunks for reference by the AI system. Several chunk strategies may be used in parallel, to optimize access to the document text for a variety of purposes. In one example strategy, the system splits document text into “chunks” based on semantic similarity, using tools such as llama-index. This results in chunks based on a similar/cohesive meaning of text within each chunk. This approach ensures that the document text is split into pieces, each of which contain adequate context around information within that text. In another example strategy, the system splits document text by sentences or paragraphs, resulting in a more consistent sizing of chunks, which is important for balancing tradeoffs during retrieval. In some embodiments, each chunk (including chunks from multiple chunking strategies) is converted into a vector representation and stored in a vector index such as OpenSearch, representing a standard approach for retrieval augmented generation (RAG).

An example of top-down material that would pass peer review: Paradigm shifts are occasionally discovered (e.g., prions are responsible for bovine spongiform encephalopathy (BSE)). Such shifts may be met with initial skepticism but are eventually accepted based on the weight of data. The source material, absent curation, will include historical inaccurate information and skepticism. Expert curation permits selection of training data or editing of training data consistent with current knowledge and understanding of the topic in question.

An example of top-down material that would not pass peer review: an on market, regulatory approved drug is found to be dangerous once used in the general population and is pulled off the market. An AI-system may still recommend the drug based on the historical record, which may include peer reviewed publications suggesting that it is safe. The curation employed in the present technology provides a system that would not make such a recommendation.

An example of bottom-up material that would pass peer review: ayurvedic medicine for centuries recommended an herb with a snake-like root called Rauwolfia serpentina to reduce blood pressure. It was found that an extract from Rauwolfia serpentina is a safe and effective treatment for hypertension. The plant was used by many physicians throughout India in the 1940s and then was used throughout the world in the 1950s, including in the United States and Canada. If a critical mass of end-users recommended Rauwolfia serpentina, and it “bubbled up” to moderators and curators, after verifying its medical efficacy and passing internal “peer review” as described above, it would be selected to be ingested as training data source material.

An example of bottom-up material that would not pass peer review: an end-user recommends using the herb “Mau Huang” for lowering blood pressure (BP) rather than the FDA approved drug Losartan Potassium because the patient is experiencing side effects. If a critical mass of end-users recommended “Mau Huang” and it “bubbled up” to moderators and curators, they would conduct an internal “peer review” and the medical efficacy of “Mau Huang” would not be verified. The curators would find that the herb contains ephedra and raises blood pressure. The combination of quitting the known BP-lowering drug and taking the Chinese herb can cause extremely high BP and strokes. If there was no filtering and moderating of user recommendations, or the training data somehow got “contaminated” or “confused” by a surge of potentially harmful or unproven recommendations and did not take into account other medications or preexisting conditions (or at least raise these factors and risks to the end user) it could pose real harm. The expertise provided by the curation system ensures that the training data would not lead to such a recommendation.

In some embodiments, a participant in the curation system would be responsible for inputting all databases and training the system how to use, analyze, and interpret the data in the database. Examples in health include the Framingham study, HANES, Odyssey, databases, the Harvard nutrition study database, or the Norwegian Mother/Child database. Participants in the curation system would be responsible for choosing which databases would be input and which excluded in addition to which technologies or professional skills would be needed to be input into the system to train it in interpreting the database. Such as, as exemplified in the above case, epidemiology, biostatistics, operational research, genomics, longevity science, actuarial science, etc.

Widely implemented, the technology benefits from numerous participants in the curation system. An effective system does not require that individual participants in the curation system dedicate full-time efforts. Traditional employment models are not optimal for staffing such a curation system. In some embodiments, provided herein are systems and methods for incentivizing individuals and groups to participate in the curation systems and methods and to provide high quality contributions to the curation systems and methods. In some embodiments, the incentivization comprises financial rewards (e.g., via revenue sharing, direct financial benefits, indirect financial benefits) and/or reputational rewards (e.g., via ranking, scoring, or other reputational indices that provide status, prestige, or recognition).

For example, in some embodiments, provided herein are systems and methods for revenue sharing based on an evaluation of participant contributions, work quality, and content quality. The systems and methods may evaluate and provide revenue sharing for any and all participants in the expert curation system, including end-users as well as any other person or entity that interacts with the systems or methods or generates or provide content evaluated by the systems and methods (e.g., in some embodiments, is applied to sources (e.g. publishers, researchers, universities, intellectual property (IP) owners, etc.)

In some embodiments, the revenue sharing systems and methods employ a “share of voice” assessment to weigh contributions and work quality. For example, in some embodiments, various inputs (“sources” and “curators” surrounding metadata) are assessed to derive responses to user prompts and to calculate the values of data input, based on one or more of the proportion of underlying citations and attributions, the aggregate metadata of user interaction with the AI (including proportionally weighted rating of response from “curators” and “consumers”), to determine the distribution of proportional revenues to “curators” and “sources” derived from any “customer” activity under a range of revenue models, including paid subscriptions, software as a service, affiliate sales, and 3rd party software licensing.

In some embodiments, data usage metadata is used to calculate revenue sharing distributions.

In some embodiments, each piece of data is placed in a retrieval augmented generation output module modified to increase a counter every time the retrieval algorithm considers the data. This is an inline tally sheet of use within the model. The counter can be retrieved at any time to compute appropriate payout.

In some embodiments, a token is issued for every data input, and this token stores all interactions with it on a ledger. The size of the ledger in bits is then the value of the token. The aggregate size of all tokens associated with the curator is the curator's value.

In some embodiments, the revenue sharing assessment includes selection of one or more separate denominators, representing, for example, profit, EBITDA, or top-line revenue, associated with a group of curators. In some embodiments, a single denominator, representing total revenue from all sources, is selected for use in assessing revenue sharing for all participants in the curation systems and methods. In some embodiments, multiple different denominators are selected. For example, separate denominators may be selected based on different revenue sources or based on different subject matter, geographic areas, or any other desired demarcations.

In some embodiments, different curation system participant tiers or individuals (e.g., advisory board members, administrators, curators, moderators, etc.) are assigned a royalty rate appropriate for their level of expertise and responsibility. In some embodiments, the royalty rate is independent of the amount of or quality of work. For example, a super administrator and curator that otherwise have identical contribution scores may receive a different percentage of the revenue based on the royalty rate assigned to them or their position.

In some embodiments, a scorecard is generated for each participant that conveys both the quantity and quality of work they provided. A score can be based on a variety of factors, including, but not limited to, a percentage of users that cite to data vetted by the participant, a rating of usefulness (e.g., determined by surveys or other user feedback, measured user outcomes, reliance/stability over time, tested outcomes, audits, etc.), a deduction for errors found, a boost (e.g., 50%, 100%, 200%, etc.) for top performance within a peer group (e.g., top 5% curator, top 5 curator, etc.), a reduction (e.g., 10%, 20%, 50%, etc.) for low performance within a peer group (e.g., bottom 5%, 5 most mistake prone curators, etc.), the number of responses source material was cited, in response to end-user prompts, and the nature of subject matter domain or account in which the work was performed (e.g., paid accounts, free accounts, high value accounts, high liability risk accounts, etc.).

In some embodiments, curation system participants are given feedback (e.g., reports) that provide regular (e.g., monthly, weekly, daily, real-time) feedback on the factors that are evaluated in determining their share of the revenue. In some embodiments, the feedback includes a list of errors or important affirmative actions required (e.g., removing out-of-date information) to improve the system. In some embodiments, distribution of payment is contingent on correction of errors and/or completion of required affirmative actions.

In some embodiments, the systems and methods utilize an audit process to ensure the participants receiving a reward for participation are not gaming the system to maximize rewards without provide a commensurate contribution. In some embodiments, the auditing comprises comparing one or more performance parameters, that are not included in the reward calculation system, to confirm that the participant's contributions merited the associated reward. In some embodiments, the auditing comprises a review of one or more individual factors that are employed in the reward calculation system, to confirm the merit of the reward. The audit can include automated and/or human evaluations. In some embodiments, top performers are audited more frequently than other participants. In some embodiments, participants that have significant increases in earned rewards over two or more time periods, independent of the scale of the reward, are more frequently audited than other participants.

In some embodiments, the systems and methods employ a tracking system that tracks all user (e.g., curator, administrator, end user, etc.) metadata. User metadata includes, but is not limited to, username, time, location, device, documents, files, filenames, URLs, interactions, and all other implicit and explicit data related to curator activity. In some embodiments, the data is stored in one or more databases. In some embodiments, the tracking system generates a comprehensive and auditable log of all curator contributions and interactions to inform the refinement of the AI/ML system and subsequent citations, attributions, and revenue distributions. In some embodiments, the tracking system collects and/or stores information about curation system participants, including, but not limited to, username, role, date/time of login, location of login (e.g., IP address), device of login, source material submitted, reviewed, nominated, rated, and/or rejected, and interactions of users with data, and metadata about the context of the interaction. In some embodiments, some or all of this participant information (e.g., personal information) is not directly used in adjusting relevance scores, weighting relevance calculations, or retraining or refinement of the AI/ML system. In some embodiments, participants can select privacy settings that control the amount and nature of participant information that is stored or utilized.

In some embodiments, the system implements a specialized metadata collection and processing system focused particularly on training data source materials typical of published research and scientific journals, which are often in PDF format. In some embodiments, the metadata collection system extracts and stores multiple categories of metadata that enhance the verification and traceability of information, including: publication name and journal-impact-factor metrics, which provide quantifiable indicators of publication credibility; author information and associated h-index values, which serve as quantitative measures of author scholarly impact; precise publication date information, which permits chronological assessment of information currency; and standardized identifiers such as PubMed ID or DOI, which facilitate programmatic citation generation and source verification. This structured metadata architecture permits the system to generate machine-verifiable citations for each response produced by the AI system, which is particularly valuable for applications in scientific, healthcare, and medical domains where source verification is important.

In some embodiments, the metadata extraction subsystem employs multiple retrieval pathways. When source documents are imported through integration with established document databases such as PubMed or DOI.org, the system automatically queries and retrieves associated metadata through standardized APIs. In cases where complete metadata cannot be programmatically obtained, such as when a curator directly uploads a PDF document, the system employs a dedicated language model specifically configured for metadata extraction. This extraction model processes the textual content, document structure, headers, footers, and reference sections of the uploaded document to derive the required metadata parameters. The extraction process employs specialized prompting techniques optimized for academic and scientific document structures.

The collection and analysis of such data facilitates refinement of AI systems. For example, if the date of the most recent login of an expert on a particular topic (e.g., prostate cancer) was one year old, then a flag is sent to an oversight individual or group to determine whether certain analyses should or should not be performed by the system (e.g., providing information about the difficult choice between getting surgery, medicine or adopting a “wait and see” to slow-growing prostate cancer). In some embodiments, such time gap flags set a default “do not provide decision support” setting that prevents use of the system for particular purposes unless overridden by an appropriately senior supervisor. In some embodiments, for curators whose cumulative metadata indicate a high “rating” of quality of contributions and have demonstrated a regular engagement, their subsequent contributions would have an algorithmically calculated higher weighted impact on adjusting relevance scores, weighting relevance calculations, or otherwise fine tuning of the system for any errors they correct or training data than curators who have a lower “rating.” In some embodiments, for data that often yields high engagement (“clicks,” “views,” “further prompts”), the system initiates an additional chain of expert reinforcement so that these higher impact requests undergo more thorough vetting. In some such embodiments, the system is adjusted to suggest high engagement content more often, which creates a virtuous cycle.

In some embodiments, the systems and methods comprise a blind spot detection component that identifies gaps in the quality and/or quantity of data (e.g., questions from users that are not supported by sufficient curated research) and reports gaps to curators. Utilization of a blind spot detection component, by routing gap information into the expert curation process and incentivizing curators to find information that reduces or eliminates the gap, increases system accuracy and associated effectiveness and trust.

In some embodiments, the systems and methods comprise an interactive artificial intelligence language system interface, where users in a range of designated roles (e.g., “administrators”, “curators” and “customers”) can enter prompts, ask questions, provide individual information (e.g., for health care indications: age, race, gender, location, existing conditions, genomics, medications, biomarkers, etc.) receive answers, and rate responses from the AI language system trained on and generated from the cumulative “source” data entered by “curators”, and personalized according to information provided by users (e.g., customers) and the context of previous prompts, answers provided information. In some embodiments, answers are accompanied by citations and metadata illustrating the “sources” and “curators” whose training data inputs inform the AI language systems' response to a given prompt.

In some embodiments, individual user information, prompts, and responses provided by users while interacting with AI are used to tailor responses unique to their experience, and are not used to train the system writ large. In some embodiments, anonymized metadata surrounding their interactions, such as the content and structure of their prompts, and ratings of responses are incorporated into calculations to inform research, refinement, and optimization of the system, but only in aggregate and under the supervision of appropriate authorized personnel (e.g., administrators) within the curation system.

In some embodiments, the system generates and displays a flowchart or other information summary showing all contemplated information inputs and outputs for each type of participant that interacts with the system. For example, for an administrator, the system identifies the ranges of information inputs they will query the system with and the ranges of information that will be received from the system. The process is repeated for a curator. The process is repeated for an exemplary end user (e.g., customer). This may be the first time that the end user is exerting an influence on the system-making it important to identify and list their inputs and outputs. The process may be repeated for a number of exemplary individual users.

In some embodiments, the systems and methods comprise a two-stage diagnostic reinforcement learning system. In some embodiments, the two-stage diagnostic reinforcement learning system allows and encourages curation system participants to consistently test, rate, correct, and refine responses, and update, remove, or refine underlying source material AI training data they ingest into the system. In some embodiments, this module is designed to make participants responsible for the life cycle of the source material they input into the AI training database by creating a circular, continuous “rinse and repeat” feedback loop that keeps the source material accurate and contemporary.

In some embodiments, in addition to the automated quality control that the system provides, the curation process includes an iterative testing, evaluation, and annotation cycle which results in consistent, high-quality outputs when the given topic is activated during a query. After uploading source materials, providing curator annotations, or editing scoring rubrics, the system re-processes the document/annotation indexes, which allow immediate ad-hoc testing. Curators can submit questions for the system to answer, similar to how it would function in its final production form. An answer is generated using only sources from the topic being curated. In addition to generating an answer and citations, the system also provides the document chunks and annotations used, and the resulting scores from automated quality control steps.

In some embodiments, the system is configured to generate likely reasons for low scores. For example, the system may identify insufficient source material or applicable source material not present or not found. The system may also suggest remedial actions. For example, when applicable source material could not be found, the system may suggest either uploading additional source material, or providing annotations where an answer might have been derived.

In some embodiments, once the results of an ad-hoc test meet predetermined quality thresholds, a curator can create a formal evaluation. Evaluations provide a measure of the quality of answers over a statistically significant quantity of generated answers. The quantitative results of evaluations are also used as a tool to measure improvements to the system relative to previous evaluation scores. In some embodiments, evaluations can be scored automatically, manually, or through a combination of both approaches. Based on scores produced during evaluations, the system can suggest likely reasons for low scores or low consistency, and generate improvement suggestions. Suggestions may include changes to standard LLM tuning parameters, such as temperature, top_p, and other parameters, or proprietary tuning parameters, such as the weighting of general context versus personalized context. In embodiments where evaluations are scored automatically, the system can run multiple self-tuning cycles to arrive at optimal settings.

In some embodiments, when generating output from a topic, the system provides citations of source material. In some embodiments, the mechanism of providing citations leverages features of existing LLMs. An automated quality control, or evaluation system, can score the output in terms of correctness, adherence to instructions and sources, completeness, and consistency. In addition, the system may add a topic-specific, expert-in-the-loop scoring rubric. For example, for the topic of “healthspan via musculoskeletal health,” a curator-defined scoring rubric might include additional scores around the applicability of answers to an individual's specific musculoskeletal conditions. Generated output might be scored against checks such as “Do answers account for previous injuries? Do answers account for bone density?” and similar topic-specific considerations that would not be generally applicable to all topics.

In some embodiments, by incorporating this automated quality control into the curator workflow, the system can further identify potential gaps in the curated source material. Gaps may be addressed through additional annotation or through additional source material. Additionally, the system can perform automated quality control at the time of generation. The quality control results may be used to regenerate output or portions of output until a sufficiently high-quality response is provided. In some embodiments, when quality fails to pass checks, the system might instead explain that no answer can be given rather than providing potentially inaccurate information.

Use of such a system is exemplified by reference to a specific topical example: Lyme disease. Lyme disease is caused by a bacteria, Borrelia burgdorferi which is carried by black-legged ticks that feed on the blood of animals and humans. These ticks may feed on the blood of mice and chipmunks and become infected with the bacteria. The ticks may also feed on animals with larger territories such as deer, birds, dogs, and cats that can move the infected tick across very long distances to reach and infect new people. Participants may interact directly or indirectly with the system in several ways: (1) Top-down from experts curators who are vertical experts in Lyme disease and Rheumatology related to treatment for arthritis and chronic pain; (2) “Sideways” in cross-cutting areas like Entomology, Infectious diseases, and “One Health” epidemiologists that track the migration of ticks and the deer, birds, skunks, foxes, and other animals that carry them; and (3) Bottom-up from thousands to millions of recommendations of family home treatments, salves, poultices, vitamins, out-of-favor preventive medicine, or niche treatments that the AI can sort and present to the board of medical experts to adjudicate.

In some embodiments, where privacy is required or desired, the systems and methods comprise components to de-identified individual user data. In some embodiments, the de-identification process employs 3rd party mechanisms (e.g., via connected health devices, documents, etc.). In some embodiments, these channels can also be included in revenue sharing calculation. In some embodiments, users opt-in to share their anonymized data to evaluate the efficacy of recommendations and contribute results to future research, which can then funnel back into the system as a novel training data source. In some embodiments, users are compensated with revenue, free usage, recognition in the research, or by other mechanisms (e.g., virtual points that are redeemed to unlock more advanced features or for products, goods, or services).

Uses of the systems and methods described herein prevents, corrects, and reduces the impact of common sources of misinformation and hallucinations, including, contamination from LLMs trained on the wider Internet, unverified sources with no audit trail, outdated source material from AIs whose training data is updated at long intervals, “poison” or “trickery” or “jailbreaking” from bad actors, trolls, or hackers attempting to fool or compromise the LLM with malicious intents and/or prompts, and compromised “rinse and repeat’ cycles. The systems and methods also reduce or eliminate the problems of combining information improperly from multiple sources (which individually may be accurate), overfitting (not being able to adapt to new situations, and overly recommending what it has seen before without understanding differences in the situation), and input bias (e.g., source data may be accurate, but leaning towards certain perspectives) and the potentially associated compounding of such biases by training.

The systems and methods find use with a variety of AI/ML systems including, but not limited to, neural networks (e.g., with reinforcement learning through human feedback), large language models (LLMs) or mall language models (SLMs), large sequence models (e.g., where a sequence is defined as “diagnosis” defined as any of a set of determinations about the causality and each diagnosis has an associated regression model stored separately), fine-tuned large language models, an ensemble of machine learning models of any permutation of the above, and causal inference models (e.g., where interventions are modeled as causal and have associated effect coefficients; where interventions are modeled as causal and associated effect coefficients are determined through a series of prompts passed to any of the above models), Retrieval augmented generation (RAG) techniques (e.g., where prompts and information generated by LLMs is combined with data retrieved directly from data bases and other data sources), Model Context Protocol (MCP) systems (e.g., wherein a universal framework connects tools or other AI/ML models and systems and data sources of diverse contexts while emphasizing security), Agentic AI (e.g., where a system comprised of multiple components work together to autonomously set goals, create action plans, and perform workflows towards achieving them), and Mixture of Experts (MOE) techniques (e.g., where multiple smaller models trained to have expertise in specific fields are employed over a single monolithic model, a gating network is used to route inputs to the expert models and weigh their responses, and a collaboration/answer model is used to combine outputs into a single cohesive response). In some embodiments, the systems and methods utilize existing, deployed research and commercial systems including, but not limited to, OpenAI, ChatGPT, Hugging Face, Google Gemini, DeepMind, Med-PaLM, Mistral, Databricks, Anthropic, and/or Adept.

In some embodiments, the system leverages specific commercial AI components, including commercial LLMs such as Anthropic Claude, provided either directly or via cloud providers such as AWS; vector embedding models such as Cohere; vector search systems such as OpenSearch; OCR and data extraction systems such as Textract; and LLM data frameworks providing tools such as Llama-index semantic splitting.

In some embodiments, the system builds upon specific AI techniques, including but not limited to: Retrieval augmented generation (RAG); semantic similarity matching; in-context learning; chain of thought inference pipelines; one-shot and multi-shot prompting; and prompt engineering. These techniques, when combined with the expert curation systems and methods described herein, enhance the accuracy, reliability, and trustworthiness of the generated outputs.

In some embodiments, the systems and methods find use in the enhancement of healthspan. Numerous factors are involved in improving and maximizing healthspan. Experiments conducted during the development of embodiments of the present technologies identified the most relevant factors that most significantly contribute to healthspan. While some individual factors were expected, some were not, leading to an unexpected priority list. The top identified worldwide factors were: atherosclerosis (and related heart attacks and strokes), cancer, neurodegenerative diseases (e.g., Alzheimer's disease, Parkinsons disease, dementia), infectious disease (e.g., tuberculosis, pneumonia, Sars-Cov2, antibiotic resistance, superbugs, etc.), metabolic syndrome (e.g., obesity, diabetes), sarcopenia/orthopedic (e.g., muscle wasting, hip fractures, disabling joint disease and pain), violence (e.g., accidents (e.g., cars, sports), gun violence, etc.), lower respiratory disease (e.g., COPD, bronchiectasis, asthma), despair (e.g., situational depression, hopelessness, self-medication, addiction, suicide, homicide), maternal morbidity and mortality, menopause, testosterone imbalances, kidney disease (e.g., end stage renal disease), liver disease (e.g., cirrhosis), accidents and injuries, and place factors (e.g., climate change (e.g., antecedent causes, subsequent effects), vector-borne, super storms, floods, migration, drought, crop failures, water-borne disease, air pollution/soot, plastics, ocean warming, loss of fish, species extinctions). The results included surprising findings, such as the significantly greater impact on lifespan related to relatively innocuous-sounding conditions (e.g., broken hips, new blindness due to untreated cataracts, postponing menopause in women) compared to treatments for cancer.

Notably, a number of these factors involve “cross-talk” in that there is some known and likely vast amounts of unknown correlation between them in terms of their impact on healthspan. The systems and methods provided herein, when evaluating these factors individually or in various combinations (e.g., evaluation of all factors), provides significant insight into actionable steps to enhance healthspan.

The systems and methods may further evaluate one or more individual risk factors. Some risk factors are relatively easy to obtain from health records (e.g., electronic medical records) or simple surveys. Such factors include, but are not limited to, age, gender at birth, race, health record data (e.g., clinical history and physical examination, prior illness and treatments, blood tests, prescribed medications, exercise supplements, wearables, etc.), social determinants of health (e.g., class, wealth, education, family background, etc.), geographic location by year and duration. Other risk factors may be determined by laboratory tests, connected devices or other sources, including, but not limited to “omics” (e.g., genomics, epigenomics, proteomics, microbiomics (e.g., gut, lung, oral-pharynx, vaginal, etc.)), “digital biomarkers” (e.g., data collected phones, cameras, voice recordings, wearables, sleep data, etc.), and “exposome” (e.g., place history (e.g., zip codes) over time related to air pollution, proximity to toxins, lead level, noise, etc.).

Assessment of these and other data allow the systems and methods of the present invention to identify both population-based, group-based, and individual-based interventions to improve healthspan. For example, use of the systems and methods identify correlations (e.g., use of hearing aids and/or better oral hygiene to reduce the impact or onset of Alzheimer's disease or dementia) that have relatively simple actionable steps to improve health, well-being, and longevity.

In some embodiments, when the system receives a healthspan query, it uses a variety of techniques, such as semantic similarity, to generate a relevance score between the query and each topic within the system. Each topic represents a collection of source documents, curator annotations, guides for which documents should be used and when/how, scoring rubrics, and related information. To produce a score on a single topic, the query is encoded as an embedding vector using an embedding model. The distance between the query vector and each chunk vector in the topic's RAG index is calculated using a vector database. The search returns the top “nearest” chunks along with their distances, and a relevance score is calculated using these distances. In some embodiments, where queries include personal details such as personal health information, geographic place factors, or other individual characteristics, those details are also encoded as vectors. Thus, answers to the healthspan queries may be personalized to an individual based on personalized health information, e.g., electronic health record, digital twins, etc., as well as population, community, and cohort data related to the individual.

In some embodiments, topics with high relevance scores are selected by the system as eligible to answer the query. For each eligible topic, the system retrieves top document chunks from the topic's associated RAG index. Using a similar vector index comparison, the system retrieves additional annotation data associated with the topic. Using similar vector comparisons, the system retrieves context relevant to personal information provided. This is performed as a separate step to ensure that both general information and personal information are incorporated into the retrieval of context. In some embodiments, the system creates a prompt payload using proprietary prompt engineering which incorporates the original query with relevant document chunks and annotation data. The prompt payload is sent to an LLM such as Anthropic Claude, Open AI ChatGPT, Google Gemini, or other suitable models. Citations are retrieved from the respective LLM system.

In some embodiments, each answer is checked against an automated quality control, which includes checks for accuracy against source material, completeness in answering the query, and consistency within the response. The quality control may include topic-specific scoring rubrics developed by the curation experts. Low scores may result in a full or partial regeneration of the answer. Depending on the scores across multiple topics, a single answer may be provided, or multiple perspectives may be presented. Citations are provided alongside the answer, permitting a human to trace through the applicable sources and reach an informed independent conclusion on the quality of the answer.

Understanding the relevant risk factors and contributors to healthspan provides insight into the characteristics of the personnel involved in the curation systems and methods. For example, the personnel, collectively, should have expertise, wisdom, and common sense in the following areas: preventative medicine, actuarial and longevity sciences, biostatistics, epidemiology (e.g., clinical, infectious disease, chronic disease), genomics, pharmacology, medical care, mental health, public health, health economics, health insurance, health planning, health behavior, environmental health, and climate science. Supporting expertise includes, but is not limited to, coaches, trainers, and therapists in the fields of nutrition, physical training, physical therapy, health-seeking behavior, life doulas, massage therapy, herbalism, integrative medicine, home health care, and mobile/traveling health care.

The analysis of multiple topics and sub-topics together prevents the systems for generating bad judgments that might be considered good judgment when viewed from the perspective of a single dimension. For example, based on information generated by a single-dimensional analysis, a functional medicine doctor might suggest estrogen replacement therapy to postpone the onset of menopause in a woman because data shows one year of postponement is associated with three more years of healthy life. However, an OB-GYN doctor might suggest if the woman has a certain genomic picture, that estrogen would increase her risk of breast or ovarian cancer. By evaluating information in multiple dimensions, with the associated expertise, adjudication of this legitimate debate would be elevated to an appropriate supervisory panel. Also, for example, climate expertise and infectious disease expertise, working together, provides a better evaluation, and if needed adjudication, of risks associated with malaria due to climate change.

In this context, the systems and methods provide a hybrid AI/expert decision support system. The people whose decisions are supported include individual health seekers, physicians and clinicians of many types (e.g., not just MD's but many forms of practitioners: pharmacists, nurses, dentists, community health workers, physical therapists, sports medicine specialists, etc.), hospital operators, health system executives, health insurance companies, governments, enterprises, and countries seeking to equitably and efficiently make resource allocation decisions. The expert support comes not only from providing highly valid information to expert users, but also the ability to access the experts within the curation systems and methods in adjudication processes.

Aggregation of data from large numbers of users allows for better decision-making and more efficient delivery of health services throughout the health care, preventive medicine, and public health systems of any enterprise or jurisdiction where it is used. For example, federal insurance programs, such as Medicare and Medicaid as well as private health and life insurers as well as HMOs have a difficult job predicting disease patterns (e.g., how many patients over 65 living in Mississippi or by the bayous of Louisiana are likely to have malaria and need specialized treatment five years from now or how will changes in the onset dates menopause affect the number of broken hips likely to emerge in their patient or covered populations-which also determines how many MRI and CT scanning machines they need to buy and how many radiologists they will have to have on staff, etc.).

At scale, the experience of users using the systems and methods provided herein creates nations of health seekers armed with the best data available and prevention in their hands (e.g., on the mobile devices) and in their communities to create advocates for better health policy and better use in the first instance of the huge amount of money spent by our governments and healthcare systems. Use of the systems and methods shifts the ratio of expenditures in favor of preventive medicine over curative medicine.

Likewise, the systems and methods provide highly individualized, proactive guidance and prevention integrated into the workflow of clinicians and other healthcare workers who can readily leverage this (e.g., via in-person, virtual and digitally enabled hybrid care) to personalize health optimization (i.e., “precision wellness”), to improve current and future function and healthspan, as well as to best screen patients most appropriately based on insights of their specific patient's relative risk (e.g., colonoscopy screening could be indicated before or after the general age 45 guidelines of today). This facilitates highly personalized proactive and preventative care plans (e.g., that adapt over time based on new data/insights). By considering factors such as genetic predispositions, biomarker profiles (e.g., both lab data and data from wearables) and trends, and treatment response data, the system can recommend tailored preventative or therapeutic medication and/or evidence-based supplement regimens, dosage adjustments, and diet and lifestyle interventions. This lengthens healthspan and lowers downstream medical costs for acute and chronic disease. Further, the system can communicate the guidance to the individual users through modalities and platforms highly attuned to the user's age, culture, language, personality, etc, offer individually tailored insights, recommendations, goals, and progress. The system can act as an “Agent” or “Agentic AI” and seamlessly provide relevant information, products, services, professionals, and resources, and perform tasks to support users on their journey to day to day wellness, increased healthspan. Interactions with the system can occur across a range of devices, platforms, settings, and technologies (web portals, mobile phones, smart watches, connected homes, Internet of Things (IoT), and others) ensuring a seamless and continuous experience with users wherever they go and throughout their lives.

EXAMPLES

Atherosclerosis Curation System

A table of an exemplary tiered provisional system for authenticated individuals to select, upload, ingest, edit, and update content for atherosclerosis is provided in Table 1. Atherosclerosis (sometimes referred to as hardening of the arteries) is a precursor to atheromas, which obstruct blood flow distal to the lesion, leading to large numbers of both strokes (if the atheroma is in the blood vessels serving the brain) and heart attacks (if in the coronary arteries). The tiered provisional system includes: a medical adjudication board comprised of medical experts; a specialized medical advisory board including board certified subject matter experts; one or more super administrators; one or more administrators; one or more super curators; one or more curators; one or more commentators; one or more moderators; and a plurality of end users. Each of the individuals in the tiered systems can carry on the roles and responsibilities of the indicated tier or any tier under the indicated tier. For example, an associate professor of neurology may fulfill the roles or contribute to the responsibilities of a curator or super curator. Internal medicine is a superset of both neurology (strokes) and cardiology (heart attacks) and thus would be more appropriate in this instance.

Role
Responsibilities
Credentials

Medical
Acts as controlling body to adjudicate
Vetted medical

Adju-
when one specialty differs in
experts at the

dication
recommendations from another (e.g.,
top of fields

Board
when orthopedic doctors recommend 
including an

rest for back injuries but the 
adjacent to

and revokes Specialized Medical 
government

Advisory Board Members
health agencies

Edits responsibilities of Specialized
(e.g., Centers

Medical Advisory Board Members
for Disease

Defines, provisions, and creates discrete
Control and

atherosclerosis databases or knowledge
Prevention,

bases (e.g., databases or 
National

knowledge bases
Institutes

accessible by a generative AI inference
of Health,

system, curated knowledge bases,
Food and Drug

and non-

government

scientific

organizations

Academy of

Advisory
Edits responsibilities of Super
PhDs, and

Board
Administrators
subject matter

Super
Defines, provisions, and audits discrete
experts (e.g.,

Admin-
atherosclerosis databases or knowledge
chairpersons

istrator
bases (e.g., databases or 
of leading

knowledge bases accessible by a 
research

language model training databases, etc.)
researchers or

and revokes Administrators
Published

Edits responsibilities of Administrators
Author)

All Administrator functionalities
in medical

fields associated

atherosclerosis

of Neurology,

Lipid Research

at Top 5 US

Medical School

Admin-
Ability to audit atherosclerosis databases
Professors

istrator
or knowledge bases (e.g., databases or
(PhD, MD) in

knowledge bases accessible by a
medical

knowledge bases, language model
Neurology,

training databases, etc.) to which they
Cardiology,

have access
Lipid Research)

Names (e.g., nominates, invites, votes
associated with

and revokes Super Curators
at Top 5 US

Edits responsibilities of Super Curators.
Medical School

All Super Curator functionalities

Curator
removes training data from
(PhD, MD) in

atherosclerosis training databases to
medical fields

which they have access
(e.g., Neurology,

responses, flags, and comments from all
Lipid Research)

roles
associated with

and revokes Curators
Medical

Edits responsibilities of Curators
Schools and

All Curator functionalities
Institutions

rates source materials, prompts, and
and professors

responses for authorized training
(PhD, MD) in

databases
medical fields

and revokes Commentators
Lipid Research)

Edits responsibilities of Commentators
associated with

atherosclerosis

Com-
Reviews, recommends, and rates source
Graduate

men-
materials, prompts, and responses for
Students (PhD,

tator
authorized training databases
MD) in medical

and revokes Moderators
Cardiology,

Edits responsibilities of Moderators
Lipid Research)

associated with

atherosclerosis

flags, ratings, and recommendations
students

Names (e.g., nominates, invites, votes
or degree

and revokes End Users
medical field

End User
Answers and provides suggested 
Doctors,

edits or new survey questions
Professors,

Enters prompts and questions and
Graduate

rates/comments on responses and
Students, Nurses,

recommendations to prompts and
Researchers,

questions
Patients, Health

Connects and Integrates 3rd party health
Seekers, Health

data (from connected devices such as
Providers

personal patient records)

Accesses a dashboard of personalized

and the like

Evaluator/
Tests and evaluates exemplary or

Tester
anticipated End User prompts and

responses to ensure quality, accuracy,

and source attribution

Experiments with prompt engineering to

identify quality improvements in

responses and outputs, including those

identified by other user Roles

errors or quality variance to Data Quality

Control Team

U.S. Case Law Curation System

Role
Responsibilities
Credentials

Legal
Acts as controlling body to adjudicate
Vetted and

Adju-
when one specialty or 
distinguished

dication
legal interpretation differs in 
experts at the

Board
recommendations from another
top of US legal

(e.g., when some case law judgments
fields (e.g.

may overlap, conflict or have different
former local,

jurisdictions)
state and federal

Defines, provisions, and creates discrete
judges and non-

US Case Law databases or knowledge
government

bases (e.g., databases or knowledge bases
legal and

accessible by a generative AI inference
academic

language model training databases, etc.)
(e.g., Bar

and ontology
Association,

and revokes Specialized Legal 
Law Schools,

Advisory Board Members
Non Partisan

Edits responsibilities of Specialized
Not-for-Profit

Legal Advisory Board Members
Organizations

such as

Legal
and revokes Super Administrators
authors,

Advisory
Edits responsibilities of Super
legal scholars,

Board
Administrators
and former

practicing

attorneys in US

Case Law across

a section of

legal fields and

jurisdictions

litigation at state

and federal

Super
Defines, provisions, and audits discrete
Distinguished

Admin-
US Case Law databases or knowledge
bar-certified

istrator
bases (e.g., databases or 
JDs and

knowledge bases accessible by 
specialized

a generative AI inference
US Case

system, curated knowledge bases,
Law subject

language model training databases, etc.)
matter expertise

Names (e.g., nominates, invites, votes
across all US

Edits responsibilities of Administrators
US states and

All Administrator functionalities
federal) in a

cross-section of

specialized

US Case Law

as intellectual

property and

with 15 years

of experience

in US Case

of Top 25

Admin-
Ability to audit US Case Law databases
Distinguished

istrator
or knowledge bases (e.g., databases or
Professors in

knowledge bases accessible by a
US Case Law

generative AI inference system, curated
from Top 10

knowledge bases, language model
US Law Schools

training databases, etc.) to which they
in each state

have access
with 15 years

Names (e.g., nominates, invites, votes
of experience

for, approves), authenticates, 
in US Case Law

and revokes Super Curators

Edits responsibilities of Super Curators

All Super Curator functionalities

Curator
removes training data from
Bar-Certified

atherosclerosis training databases to
Graduates (JD)

which they have been given access
from Top 5

Reviews all recommendations, ratings,
law schools in

responses, flags, and comments from all
each US state

roles
with 10 years

Names (e.g., nominates, invites, votes
of experience

for, approves), authenticates, 
in US Case Law

and revokes Curators

Edits responsibilities of Curators

All Curator functionalities

source materials, prompts, and responses
Graduates (JD)

for authorized training databases
from Top 10

Names (e.g., nominates, invites, votes
law schools in

for, approves), authenticates, 
each US state

and revokes Commentators
with 5 years of

Edits responsibilities of Commentators
experience in

US Case Law

Com-
Reviews, recommends, and rates source
Graduate law

mentator
materials, prompts, and responses for
school students,

authorized training databases
nominated,

and revokes Moderators
by their

Edits responsibilities of Moderators
professors or

peers with

Curator

credentials

or above

flags, ratings, and recommendations
legal students

Names (e.g., nominates, invites, votes
or graduate in

and revokes End Users
nominated,

vetted, and

approved by

their professors

or Graduate

Students who

Commentator

credentials

or above

End User
Enters prompts and questions and
Lawyers, Judges,

citations to prompts and questions
Students,

Answers and provides suggested edits or
Journalists,

new survey questions
Citizens, Legal

Accesses a dashboard of historical
Counsel for

relevant citations and source materials
Public

Sector, and

Organizations

Evaluator/
Tests and evaluates exemplary or

Tester
anticipated End User prompts and

responses to ensure quality, accuracy,

and source attribution

Experiments with prompt engineering

to identify quality improvements in

responses and outputs, including those

identified by other user Roles

errors or quality variance to

Data Quality Control Team

Curation System Workflow

Curation systems for any topic or subject area include a group of individuals, as described in Examples 1 and 2, which contribute to the curation system workflow. As shown in FIG. 2, the curation systems provide a means to identify and curate source materials. Within the curation system, one of the members of the curation system would manage the curation of the library of source materials which is used to formulate the training databases and the AI system.

Any single source material, e.g., book, research paper, presentation, is identified, reviewed, and analyzed within the curation system by the administrators and curators. Each potential source material entering the curation system is referenced and once a quorum of endorsements from the administrators and curators is achieved the source material is sent to a Data Quality Control Team comprising data scientist and quality control engineers, optionally in conjunction with a system of language learning models, for organizing, standardizing, tokenizing, and casting the material in a machine-readable format. The metadata associated with the source materials as well as any citations or attributions to the source material is verified and, if needed, updated by a member of the curation team. Over time, the Data Quality Control Team verifies that all appropriate metadata attributions and citations are correctly tracked and displayed and that the changelog reflecting these updates and references all modifications, contributions, and attributions

Once the source material passes through each of the curation steps above, the source material is curated in the library alongside any other additional curated source materials for use in both the training data and the system as a whole. The administrators and curators confirm and verify that the source material is correctly applied to the AI systems.

Source Materials

As described in Example 3, top-down discovery of a source material, e.g., medical journal publication and professional network of administrators and curators, enters the curation system following their identification by the administrators and curators. However, users of the AI system or other members of the curation team may generate a second, bottom-up, type of source material. This bottom-up source material may be generated through user interactions, prompts, responses, ratings, and comments. As shown in FIG. 3, this data can be fed into the curation system for referencing, and review and verification of the source material and metadata by administrators and curators prior to casting by the data scientists and quality control engineers into materials suitable for use in the training databases and the AI system.

Source materials can be annotated during, after, or concurrently with discovery and selection. The source materials can be annotated and tagged with metadata for a particular specialty category (e.g., cardiology, pediatrics, etc.), evidence level classification (e.g., RCT, observational study, expert consensus), publication date and currency indicators, target audience (e.g., clinicians vs. patients), geographical relevance, confidence/certainty level of medical claims. The source materials can be annotated during the curation to extract key technological features, e.g., conditions, treatments, symptoms, identify causal relationships between concepts, mark contraindications and warnings, highlight or tag process and related outcomes, e.g., diagnostic criteria, treatment protocols, and the resulting treatment outcome.

Formatting and Categorizing Source Materials

The data from the source materials is organized into a relational database facilitating extraction of the key/value pairs into a flat file or by a dynamic schema management tool, such as HarperDB, MongoDB, Pyspark, CrateDB, and the like.

For example, for a journal article or manuscript from a peer-reviewed publication, the research is distilled into a summary by a generative AI agent using an LLM that includes general knowledge of English and terms used in health, medicine, biology, etc. The summary is structured in a standard, tokenized form. For example:

management and uncertainties.

Journal: Manual therapy

Abstract: Rotator cuff related shoulder pain (RCRSP) is

an over-arching term that encompasses a spectrum of shoulder

conditions including subacromial pain (impingement)

Additional tagging may be applied to indicate if the source material contains information relevant to determinants, mediators, interventions, or outcomes. Additional tagging may indicate whether the source material might be specific to certain populations included in the study. Additional tagging may include curator notes, comments, observations, rationalizations, and ratings.

Prompts and Questions

End users (e.g., professionals, students, researchers, service providers, service users, individuals associated with advocacy groups, government employees, and general individuals) interact with the system in a variety of ways, as described above. One aspect of an end user's experience is to be presented with survey questions during or after use as well as enter prompts and questions and rate and comment on responses and recommendations based on prompts and questions. The presentation of these questions as well as the user's prompts, questions, responses, and recommendations may use any form of media, including text-, audio-, video-based.

During use of the system, an end user may be prompted with questions to determine if they are having a good or poor experience. If an end user is having a poor experience, they may be connected to a system assistant to help navigate or integrate with the system and gather more information regarding the user's experience to troubleshoot or guide the user to their goal.

When an end user enters a question and receives a response, a survey may be generated by a survey instrument or rating system within the AI program to ask questions such as:

Users who provide beneficial details and answer can be rewarded (e.g., a free subscription or free products, etc.) for participating in the curation system. The survey or prompts may include questions such as:

Members of the curation system (e.g., curators and administrators) are given regular (e.g., monthly) reports of these answers and required to “fix” any errors, and remove out-of-date info.