AI DRIVEN SMART PATIENT LABELING SYSTEM

In an approach for automatically identifying one or more updates in a Scientific Drug Label (SL) relevant to a patient and incorporating the one or more updates into a Patient Drug Label (PL), a processor receives a pair of documents, wherein the pair of documents include the SL and the PL. A processor converts a complex medical language of the SL into a simplified patient friendly language. A processor identifies one or more words, one or more phrases, or one or more sentences that have been modified, inserted, or deleted. A processor searches for a location in the PL that closely maps to the one or more words, the one or more phrases, or the one or more sentences to the SL. A processor incorporates the one or more words, the one or more phrases, or the one or more sentences in a mapped location of the PL.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of data processing, and more particularly to an artificially intelligent driven smart patient labeling system.

The Food and Drug Administration (FDA) is responsible for determining whether a pharmaceutical drug is safe and effective for use under the conditions prescribed, recommended, or suggested in a proposed labeling. Once a pharmaceutical drug is approved, the FDA requires labeling that sets forth the conditions of use under which the pharmaceutical drug has been shown to meet the relevant standard for marketing. The FDA-required labeling also provides directions and information on how to use the pharmaceutical drug safely and effectively.

For every pharmaceutical drug, there are two labels generated: a Scientific Drug Label and a Patient Drug Label. A Scientific Drug Label is meant for the medical practitioners who prescribe the pharmaceutical drug to patients. The Scientific Drug Label is complex as it contains scientific terms and medical terminology. The Scientific Drug Label contains information relevant to the medical practitioner such as a name of the pharmaceutical drug, a composition of the pharmaceutical drug, a shape and/or appearance of the pharmaceutical drug, and a medical condition treated by the pharmaceutical drug. A Patient Drug Label is meant for the patients who are prescribed the pharmaceutical drug. The Patient Drug Label has simpler language and is devoid of complex medical terms. The Patient Drug Label contains information relevant to the patient such as a method to administer the pharmaceutical drug, one or more side effects a patient may experience when taking the pharmaceutical drug, and one or more precautions a patient should take when using the pharmaceutical drug.

SUMMARY

Aspects of an embodiment of the present invention disclose a method, computer program product, and computer system for automatically identifying one or more updates in a Scientific Drug Label relevant to a patient and incorporating the one or more updates into a Patient Drug Label. A processor receives a pair of documents from a user, wherein the pair of documents include a Scientific Drug Label and a Patient Drug Label. A processor converts a complex medical language of the Scientific Drug Label into a simplified patient friendly language. A processor analyzes the simplified patient friendly language to identify one or more words, one or more phrases, or one or more sentences that have been modified, inserted, or deleted. Responsive to determining the one or more words, the one or more phrases, or the one or more sentences are relevant to a patient, a processor classifies the one or more words, the one or more phrases, or the one or more sentences in one or more categories. A processor searches for a location in the Patient Drug Label that closely maps to the one or more words, the one or more phrases, or the one or more sentences to the Scientific Drug Label. A processor incorporates the one or more words, the one or more phrases, or the one or more sentences in a mapped location of the Patient Drug Label. A processor outputs an updated Patient Drug Label to the user.

In some aspects of an embodiment of the present invention, the one or more categories include a complete insertion of a sentence in the Scientific Drug Label, a complete deletion of the sentence in the Scientific Drug Label, and an insertion or a deletion of a word or a phrase in the Scientific Drug Label.

In some aspects of an embodiment of the present invention, subsequent to outputting the updated Patient Drug Label to the user, a processor requests feedback from the user. Responsive to receiving the feedback from the user, a processor validates the feedback received from the user manually using a confidence score of one or more intermediate outputs. A processor annotates the feedback received from the user.

In some aspects of an embodiment of the present invention, subsequent to annotating the feedback received from the user, a processor identifies one or more engines to be retrained. A processor retrains the one or more engines with the annotated feedback.

In some aspects of an embodiment of the present invention, the feedback received from the user includes an acceptance or a rejection of the one or more changes incorporated into the updated Patient Drug Label.

In some aspects of an embodiment of the present invention, a processor converts the pair of documents from a word format to a Portable Document Format (PDF). A processor extracts content from the PDF of the Scientific Drug Label in a structured format. A processor sorts the extracted content into a corresponding section heading or a corresponding section subheading.

In some aspects of an embodiment of the present invention, subsequent to sorting the extracted content into the corresponding section heading or the corresponding section subheading, a processor extracts one or more keywords and one or more key phrases from the extracted content using a custom-trained Spacey model to understand a concept of each sentence of the extracted content. A processor maps one or more relationships between the Scientific Drug Label and the Patient Drug Label using a knowledge graph. A processor extracts a confidence score for the one or more relationships mapped between the Scientific Drug Label and the Patient Drug Label.

In some aspects of an embodiment of the present invention, the one or more keywords and the one or more key phrases extracted from the structured data includes a name of a drug, a composition of the drug, a shape of the drug, an appearance of the drug, a medical condition treated by the drug, a method to administer the drug, one or more side effects the patient may experience when taking the drug, and one or more precautions the patient should take when using the drug.

In some aspects of an embodiment of the present invention, a processor generates a model for the one or more keywords and the one or more key phrases from the extracted content. A processor determines a position of the one or more keywords and the one or more key phrases from the extracted content. A processor annotates the position of the one or more keywords and the one or more key phrases from the extracted content. A processor trains on the annotated position of the one or more keywords and the one or more key phrases.

In some aspects of an embodiment of the present invention, a processor modifies the one or more words, the one or more phrases, or the one or more sentences in the mapped location of the Patient Drug Label.

In some aspects of an embodiment of the present invention, a processor inserts the one or more words, the one or more phrases, or the one or more sentences in the mapped location of the Patient Drug Label.

In some aspects of an embodiment of the present invention, a processor deletes the one or more words, the one or more phrases, or the one or more sentences in the mapped location of the Patient Drug Label.

DETAILED DESCRIPTION

Embodiments of the present invention recognize that the Food and Drug Administration (FDA) determines whether a pharmaceutical drug is safe and effective for use under the conditions prescribed, recommended, or suggested in a proposed labeling submitted with the pharmaceutical drug's marketing application. In making this determination, the FDA evaluates whether the conditions of use in the proposed labeling are supported by the required levels and types of evidence of safety and effectiveness and whether the benefits of using the pharmaceutical drug under those specific conditions of use outweigh the risks of the pharmaceutical drug.

Embodiments of the present invention recognize that, after the FDA approves or clears a pharmaceutical drug, the FDA requires labeling that sets forth the conditions of use under which the pharmaceutical drug has been shown to meet the relevant standard for marketing. The FDA-required labeling also provides directions and information on how to use the pharmaceutical drug safely and effectively. The FDA-required labeling is the primary tool that communicates the essential information needed for the safe and effective use of the pharmaceutical drug by the user (i.e., a medical practitioner and a patient), and companies have an obligation to update their FDA-required labeling as needed to ensure it is not false or misleading. The essential information may include, but is not limited to, a name of the pharmaceutical drug, a composition of the pharmaceutical drug (i.e., the ingredients of the pharmaceutical drug and the concentration of the ingredients), a shape and/or appearance of the pharmaceutical drug, a medical condition treated by the pharmaceutical drug, a method to administer the pharmaceutical drug, one or more side effects a patient may experience when taking the pharmaceutical drug (i.e., one or more unwanted or unexpected symptoms or feelings that may occur when taking the pharmaceutical drug), and one or more precautions a patient should take when using the pharmaceutical drug. However, a label is not intended to exhaustively address all that is known about the pharmaceutical drug for its approved or cleared uses.

Embodiments of the present invention recognize that providing the FDA-required labeling is a critical step in maintaining compliance for a pharmaceutical drug that is released to the market. For every pharmaceutical drug, there are two labels generated: a Scientific Drug Label and a Patient Drug Label. A Scientific Drug Label is meant for the medical practitioners who prescribe the pharmaceutical drug to patients. The Scientific Drug Label is complex as it contains scientific terms and medical terminology. The Scientific Drug Label contains information relevant to the medical practitioner such as a name of the pharmaceutical drug, a composition of the pharmaceutical drug, a shape and/or appearance of the pharmaceutical drug, and a medical condition treated by the pharmaceutical drug. A Patient Drug Label is meant for the patients who are prescribed the pharmaceutical drug. The Patient Drug Label has simpler language and is devoid of complex medical terms. The Patient Drug Label contains information relevant to the patient such as a method to administer the pharmaceutical drug, one or more side effects a patient may experience when taking the pharmaceutical drug, and one or more precautions a patient should take when using the pharmaceutical drug.

Embodiments of the present invention recognize that simplification of the technical information of a pharmaceutical drug into patient friendly language requires a great deal of manual intervention, especially when there are updates in the Scientific Drug Label that need to be incorporated into the Patient Drug Label. Typically, pharmaceutical and biopharmaceutical companies employ a group of Subject Matter Experts (SMEs) who generate the Patient Drug Label when a new pharmaceutical drug is released to the market and who update the Patient Drug Label when a pharmaceutical drug currently on the market is changed in some way. Currently, the SMEs for the pharmaceutical and biopharmaceutical companies must manually check for updates and must determine whether the update is relevant to the patient and, therefore, should be incorporated into the Patient Drug Label. The SMEs determination of relevancy is based on the SME's knowledge. When the update is deemed relevant to the patient, the SMEs must incorporate the update into the Patient Drug Label manually. This process, however, is not a straightforward process and requires multiple rounds of verification and approval before the Patient Drug Label is updated and released to the public. Therefore, this process can be time consuming. Embodiments of the present invention recognize the need for a system and method to mitigate the potential ineffectiveness of manual updates of Patient Drug Labels.

Embodiments of the present invention provide a system and method to automatically identify one or more updates in a Scientific Drug Label that are relevant to a patient; to identify what type of update (i.e., insert, delete, modify) each of the one or more updates in the Scientific Drug Label are and where (i.e., appropriate position) each of the one or more updates needs to be incorporated into a Patient Drug Label; to transform the complex medical language of the update into simple patient learning language without losing contextual information; to incorporate each of the one or more updates into the Patient Drug Label; and to analyze user feedback data and separate the feedback based on engines to be retrained using an intelligent data annotator for the respective engines training.

Implementation of embodiments of the present invention may take a variety of forms, and exemplary implementation details are discussed subsequently with reference to the Figures.

FIG.1is a block diagram illustrating a distributed data processing environment, generally designated100, in accordance with an embodiment of the present invention. In the depicted embodiment, distributed data processing environment100includes server120and user computing device130, interconnected over network110. Distributed data processing environment100may include additional servers, computers, computing devices, and other devices not shown. The term “distributed” as used herein describes a computer system that includes multiple, physically distinct devices that operate together as a single computer system.FIG.1provides only an illustration of one embodiment of the present invention and does not imply any limitations with regards to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by those skilled in the art without departing from the scope of the invention as recited by the claims.

Network110operates as a computing network that can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN), such as the Internet, or a combination of the three, and can include wired, wireless, or fiber optic connections. Network110can include one or more wired and/or wireless networks capable of receiving and transmitting data, voice, and/or video signals, including multimedia signals that include data, voice, and video information. In general, network110can be any combination of connections and protocols that will support communications between server120, user computing device130, and other computing devices (not shown) within distributed data processing environment100.

Server120operates to run intelligent patient labeling program122and to send and/or store data in database124. In an embodiment, server120can send data from database124to user computing device130. In an embodiment, server120can receive data in database124from user computing device130. In one or more embodiments, server120can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data and capable of communicating with user computing device130via network110. In one or more embodiments, server120can be a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within distributed data processing environment100. In one or more embodiments, server120can be a laptop computer, a tablet computer, a netbook computer, a personal computer, a desktop computer, a personal digital assistant, a smart phone, or any programmable electronic device capable of communicating with user computing device130and other computing devices (not shown) within distributed data processing environment100via network110. Server120may include internal and external hardware components, as depicted and described in further detail inFIG.11.

Intelligent patient labeling program122operates to automatically identify one or more updates in a Scientific Drug Label relevant to a patient; to identify the type of update and the appropriate position of the update in a Patient Drug Label; to transform the complex medical language of the update into simple patient learning language without losing contextual information; and to incorporate the update into the Patient Drug Label. In the depicted embodiment, intelligent patient labeling program122is a standalone program. In another embodiment, intelligent patient labeling program122may be integrated into another software product, such as a pharmaceutical labeling software. In the depicted embodiment, intelligent patient labeling program122resides on server120. In another embodiment, intelligent patient labeling program122may reside on user computing device130or on another computing device (not shown), provided that intelligent patient labeling program122has access to network110. In the depicted embodiment, intelligent patient labeling program122includes PDF Parser Engine122-A, Natural Language Processing (NLP) Engine122-B, Mapping Engine122-C, Content Relevancy Engine122-D, Update Engine122-E, and Feedback Analyzer Engine122-F.

The operational steps of intelligent patient labeling program122are depicted and described in further detail with respect toFIG.2. An exemplary diagram illustrating the operational steps of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.3. An exemplary diagram illustrating the operational steps of PDF parser engine122-A of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.4. An exemplary diagram illustrating the operational steps of NLP engine122-B of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.5. An exemplary diagram illustrating the operational steps of mapping engine122-C of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.6. An exemplary diagram illustrating the operational steps of content relevancy engine122-D of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.7. An exemplary diagram illustrating the operational steps of update engine122-E of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.8. An exemplary illustration of an analysis of an inserted key phrase by intelligent patient labeling program122is depicted and described in further detail with respect toFIG.9A. An exemplary illustration of an analysis of an inserted sentence by intelligent patient labeling program122is depicted and described in further detail with respect toFIG.9B. An exemplary diagram illustrating the operational steps of feedback analyzer engine122-F of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.9. An exemplary flowchart illustrating the operational steps of intelligent patient labeling program122is depicted and described in further detail with respect toFIG.10.

PDF Parser Engine122-A operates to extract content from a portable document format (PDF) document of a Scientific Drug Label in a pre-defined structure and sort the structured data into a corresponding section heading and/or a corresponding section subheading.

NLP Engine122-B operates to extract one or more keywords and/or one or more key phrases from the structured data using a custom-trained Spacey model in order to understand the concept of each sentence of the structured data.

Mapping Engine122-C operates to extract one or more mapping relationships between the Scientific Drug Label (SL) document in JavaScript Object Notation (JSON) format and the Patient Drug Label (PL) document in JSON format in terms of their section and/or sub-section relationships or in terms of their interdependent and parent-child section/sub-section relationships using a knowledge graph.

Content Relevancy Engine122-D operates to determine whether the one or more words, phrases, and/or sentences identified in the simplified SL document that have been modified, inserted, and/or deleted are relevant to a patient.

Update Engine122-E operates to classify the one or more words, phrases, and/or sentences that have been modified, inserted, and/or deleted into one or more categories. When a word, phrase, or sentence is completely inserted in the original SL document, update engine122-E searches for the correct location where the word, phrase, or sentence needs to be inserted into the original PL document based on contextual similarities of PL sentences and inserts the word, phrase, or sentence into the chosen location of the original PL document. When a word, phrase, or sentence is completely deleted from the original SL document, update engine122-E searches for a word, phrase, or sentence in the original PL document which closely matches the word, phrase, or sentence deleted from the original SL document contextually and/or semantically and deletes the word, phrase, or sentence in the original PL document which closely matches the word, phrase, or sentence deleted from the original SL document contextually and/or semantically.

Feedback Analyzer Engine122-F operates to receive and validate feedback from the user. Feedback Analyzer Engine122-F also operates to identify the relevant engines to be retrained, separate the feedback, automatically annotate the feedback, and store the feedback in a database to be used later to retrain the relevant engines.

In an embodiment, the user of user computing device130registers with server120. For example, the user completes a registration process (e.g., user validation), provides information to create a user profile, and authorizes the collection, analysis, and distribution (i.e., opts-in) of relevant data on identified computing devices (e.g., on user computing device130) by server120(e.g., via intelligent patient labeling program122). Relevant data includes, but is not limited to, personal information or data provided by the user or inadvertently provided by the user's device without the user's knowledge; tagged and/or recorded location information of the user (e.g., to infer context (i.e., time, place, and usage) of a location or existence); time stamped temporal information (e.g., to infer contextual reference points); and specifications pertaining to the software or hardware of the user's device. In an embodiment, the user opts-in or opts-out of certain categories of data collection. For example, the user can opt-in to provide all requested information, a subset of requested information, or no information. In one example scenario, the user opts-in to provide time-based information, but opts-out of providing location-based information (on all or a subset of computing devices associated with the user). In an embodiment, the user opts-in or opts-out of certain categories of data analysis. In an embodiment, the user opts-in or opts-out of certain categories of data distribution. Such preferences can be stored in database124.

Database124operates as a repository for data received, used, and/or generated by intelligent patient labeling program122. A database is an organized collection of data. Data includes, but is not limited to, information about user preferences (e.g., general user system settings, such as alert notifications for user computing device130); information about alert notification preferences; a SL-PL pair; a set of trained data; feedback received from the user; and any other data received, used, and/or generated by intelligent patient labeling program122.

Database124can be implemented with any type of device capable of storing data and configuration files that can be accessed and utilized by server120, such as a hard disk drive, a database server, or a flash memory. In an embodiment, database124is accessed by intelligent patient labeling program122to store and/or to access the data. In the depicted embodiment, database124resides on server120. In another embodiment, database124may reside on another computing device, server, cloud server, or spread across multiple devices elsewhere (not shown) within distributed data processing environment100, provided that intelligent patient labeling program122has access to database124.

The present invention may contain various accessible data sources, such as database124, that may include personal and/or confidential company data, content, or information the user wishes not to be processed. Processing refers to any operation, automated or unautomated, or set of operations such as collecting, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, disclosing by transmission, dissemination, or otherwise making available, combining, restricting, erasing, or destroying personal and/or confidential company data. Intelligent patient labeling program122enables the authorized and secure processing of personal data.

Intelligent patient labeling program122provides informed consent, with notice of the collection of personal and/or confidential data, allowing the user to opt-in or opt-out of processing personal and/or confidential data. Consent can take several forms. Opt-in consent can impose on the user to take an affirmative action before personal and/or confidential data is processed. Alternatively, opt-out consent can impose on the user to take an affirmative action to prevent the processing of personal and/or confidential data before personal and/or confidential data is processed. Intelligent patient labeling program122provides information regarding personal and/or confidential data and the nature (e.g., type, scope, purpose, duration, etc.) of the processing. Intelligent patient labeling program122provides the user with copies of stored personal and/or confidential company data. Intelligent patient labeling program122allows the correction or completion of incorrect or incomplete personal and/or confidential data. Intelligent patient labeling program122allows for the immediate deletion of personal and/or confidential data.

User computing device130operates to run user interface132through which a user can interact with intelligent patient labeling program122on server120. In an embodiment, user computing device130is a device that performs programmable instructions. For example, user computing device130may be an electronic device, such as a laptop computer, a tablet computer, a netbook computer, a personal computer, a desktop computer, a smart phone, or any programmable electronic device capable of running user interface132and of communicating (i.e., sending and receiving data) with intelligent patient labeling program122via network110. In general, user computing device130represents any programmable electronic device or a combination of programmable electronic devices capable of executing machine readable program instructions and communicating with other computing devices (not shown) within distributed data processing environment100via network110. In the depicted embodiment, user computing device130includes an instance of user interface132.

User interface132operates as a local user interface between intelligent patient labeling program122on server120and a user of user computing device130. In some embodiments, user interface132is a graphical user interface (GUI), a web user interface (WUI), and/or a voice user interface (VUI) that can display (i.e., visually) or present (i.e., audibly) text, documents, web browser windows, user options, application interfaces, and instructions for operations sent from intelligent patient labeling program122to a user via network110. User interface132can also display or present alerts including information (such as graphics, text, and/or sound) sent from intelligent patient labeling program122to a user via network110. In an embodiment, user interface132is capable of sending and receiving data (i.e., to and from intelligent patient labeling program122via network110, respectively). Through user interface132, a user can opt-in to intelligent patient labeling program122; create a user profile; set user preferences and alert notification preferences; receive an alert notification; input a Scientific Drug Label-Patient Drug Label pair; receive the updated Patient Drug Label; receive a request for feedback; and input feedback.

A user preference is a setting that can be customized for a particular user. A set of default user preferences are assigned to each user of intelligent patient labeling program122. A user preference editor can be used to update values to change the default user preferences. User preferences that can be customized include, but are not limited to, general user system settings, specific user profile settings, alert notification settings, and machine-learned data collection/storage settings. Machine-learned data is a user's personalized corpus of data. Machine-learned data includes, but is not limited to, past results of iterations of intelligent patient labeling program122.

FIG.2is a flowchart, generally designated200, illustrating the operational steps of intelligent patient labeling program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. In an embodiment, intelligent patient labeling program122operates to automatically identify one or more updates in a Scientific Drug Label relevant to a patient; to identify the type of update and the appropriate position of the update in a Patient Drug Label; to transform the complex medical language of the update into simple patient learning language without losing contextual information; and to incorporate the update into the Patient Drug Label. It should be appreciated that the process depicted inFIG.2illustrates one possible iteration of the process flow, which may be repeated each time a Scientific Drug Label (SL)-Patient Drug Label (PL) pair is received. In another embodiment, the process depicted inFIG.2may be repeated each time intelligent patient labeling program122detects an update in a Scientific Drug Label-Patient Drug Label pair.

In step210, intelligent patient labeling program122(hereinafter referred to as “program122”) receives a Scientific Drug Label (SL)-Patient Drug Label (PL) pair. In an embodiment, program122receives a SL-PL pair as an input from a user through a user computing device (e.g., user computing device130) via a user interface (e.g., user interface132). The user may be, but is not limited to, a pharmaceutical company representative or a medical device company representative. The SL-PL pair is a set of documents representing a pharmaceutical drug or medical device (hereinafter “pharmaceutical drug” means either a pharmaceutical drug or a medical device). The SL-PL pair is inputted when a pharmaceutical drug is released to the market and/or when a pharmaceutical drug currently on the market is changed in some way (i.e., one or more keywords and/or one or more key phrases in the Scientific Drug Label of the SL-PL pair have been modified, inserted, and/or deleted, but the Patient Drug Label of the SL-PL pair has not been updated). In another embodiment, program122monitors a database (e.g., database124) for one or more recent updates to SL-PL pairs stored in the database (e.g., database124).

In an embodiment, program122converts the Scientific Drug Label and the Patient Drug Label of the SL-PL pair from a word format to a PDF using a digitization module. The digitization module is a word-to-PDF convertor that converts a document in word format to a document in PDF format, tracks the changes made to the document, and accepts and/or rejects the tracked changes). In an embodiment, program122sends the PDF document of the Scientific Drug Label and the Patient Drug Label of the SL-PL pair (i.e., a SL .pdf and a PL .pdf) to PDF parser122-A of program122.

In an embodiment, PDF parser engine122-A of program122extracts content from the Scientific Drug Label in a pre-defined structure (hereinafter referred to as the “structured data”). The pre-defined structure may be, but is not limited to, an Extensible Markup Language (XML) format. In an embodiment, PDF parser engine122-A of program122processes the extracted content from the document using an XML parsing module. The XML parsing module has two inner modules: a first module to extract a section heading and/or a subsection heading and a second module to extract data.

In an embodiment, PDF parser engine122-A of program122sorts the structured data. In an embodiment, PDF parser engine122-A of program122sorts the structured data into a corresponding section heading and/or a corresponding section subheading. In an embodiment, PDF parser engine122-A of program122sorts the structured data in a structured JSON format using a JSON formatter module. In an embodiment, PDF parser engine122-A of program122sends the structured data to NLP engine122-B of program122.

In step220, NLP engine122-B of program122extracts one or more keywords and/or one or more key phrases from the structured data. In an embodiment, NLP engine122-B of program122extracts one or more keywords and/or one or more key phrases from the structured data using a custom-trained Spacey model. In an embodiment, NLP engine122-B of program122extracts one or more keywords and/or one or more key phrases in order to understand the concept of each sentence of the structured data. The one or more keywords and/or the one or more key phrases extracted from the structured data may include, but are not limited to, a name of a drug (i.e., associated with the SL-PL pair), a composition of the drug, a shape and/or appearance of the drug, a medical condition treated by the drug, a method to administer the drug, one or more side effects a patient may experience when taking the drug (i.e., one or more unwanted or unexpected symptoms or feelings that may occur when taking the drug), and one or more precautions a patient should take when using the drug.

In an embodiment, program122trains NLP engine122-B of program122to generate a model for the one or more keywords and/or the one or more key phrases extracted. In an embodiment, program122determines a position of the one or more keywords and/or the one or more key phrases extracted. In an embodiment, program122determines a category of the one or more keywords and/or the one or more key phrases extracted. In an embodiment, program122creates an annotation for the position and/or the category of the one or more keywords and/or the one or more key phrases. In an embodiment, program122trains NLP engine122-B of program122on the annotation created for the position and/or the category of the one or more keywords and/or the one or more key phrases using a deep learning algorithm. In an embodiment, program122tests and validates the trained data. In an embodiment, program122stores the trained data in a dictionary in the database (e.g., database124).

In an embodiment, NLP engine122-B of program122sends the documents in JSON format (i.e., a SL document in JSON format and a PL document in JSON format) to mapping engine122-C of program122. In an embodiment, NLP engine122-B of program122sends mapping configuration files obtained from a Quality Review of Documents (QRD) template to mapping engine122-C of program122.

In step230, mapping engine122-C of program122extracts one or more mapping relationships between the SL document in JSON format and the PL document in JSON format. In an embodiment, mapping engine122-C of program122extracts one or more mapping relationships between the SL document in JSON format and the PL document in JSON format in terms of their section and/or sub-section relationships using a knowledge graph. In an embodiment, mapping engine122-C of program122extracts one or more mapping relationships between the SL document in JSON format and the PL document in JSON format in terms of their interdependent and parent-child section/sub-section relationships using a knowledge graph. The one or more mapping relationships between the SL document in JSON format and the PL document in JSON format may include, but is not limited to, SL-PL direct section/sub-section, SL-SL interdependent section/sub-section, SL-SL parent-child, and PL-PL parent-child. In an embodiment, mapping engine122-C of program122extracts a confidence score for each mapping relationship between the SL document in JSON format and the PL document in JSON format based on contextual information.

In an embodiment, mapping engine122-C of program122sends the extracted mapping relationships between the SL document in JSON format and the PL document in JSON format to update engine122-E of program122(i.e., as a mapping document in JSON format). In an embodiment, mapping engine122-C of program122sends the SL document in JSON format and the PL document in JSON format to content relevancy engine122-D of program122.

In step240, in a training stage, content relevancy engine122-D of program122converts the original SL document into a simplified patient friendly language using a transformation module. In an embodiment, content relevancy engine122-D of program122analyzes the simplified SL document to identify the one or more words, phrases, and/or sentences that have been modified, inserted, and/or deleted. In an embodiment, content relevancy engine122-D of program122determines whether the one or more words, phrases, and/or sentences identified are relevant to a patient. In an embodiment, content relevancy engine122-D of program122labels as “relevant” the one or more words, phrases, and/or sentences determined to be relevant to the patient. In an embodiment, content relevancy engine122-D of program122labels as “irrelevant” the one or more words, phrases, and/or sentences determined to be irrelevant to the patient. In an embodiment, content relevancy engine122-D of program122determines the label given to the one or more words, phrases, and/or sentences using a smart system. In an embodiment, content relevancy engine122-D of program122enables the smart system to perform unsupervised learning. The labels are not specified. The smart system determines the label based on an intelligent comparison model. The intelligent comparison model is a bidirectional transformer. In an embodiment, content relevancy engine122-D of program122trains the smart decision-making context similarity model for future cases using the labeled data.

In another embodiment, in an inference stage, content relevancy engine122-D of program122converts the original SL document into a simplified patient friendly language using a transformation module. In an embodiment, content relevancy engine122-D of program122sends the simplified SL document to a smart decision-making context similarity module to determine whether the one or more words, phrases, and/or sentences identified are relevant to a patient. The decision-making context similarity module has human-like decision making capabilities. For example, certain information such as drug interactions are relevant to a practitioner, but not to a patient. Therefore, content relevancy engine122-D of program122determines the new sentence in the SL document is not relevant to the patient and, therefore, will be omitted from the PL document.

In an embodiment, content relevancy engine122-D of program122sends the original SL document and the simplified SL document to update engine122-E of program122. The extracted mapping relationships between the original SL document and the original PL document (i.e., the mapping documents in JSON format) were previously sent to update engine122-E of program122in step230.

In step250, update engine122-E of program122classifies the one or more words, phrases, and/or sentences that have been modified, inserted, and/or deleted into one or more categories. There are three (3) categories that include: complete insertion of a sentence in the original SL document, complete deletion of a sentence in the original SL document, and insertion and/or deletion of words and/or phrases in a sentence in the original SL document.

In an embodiment, if a sentence is completely inserted in the original SL document, update engine122-E of program122classifies the sentence into one or more relevant categories. In an embodiment, update engine122-E of program122searches for the correct location where the sentence needs to be inserted into the original PL document based on contextual similarities of PL sentences. In an embodiment, update engine122-E of program122inserts the sentence into the chosen location of the original PL document.

In an embodiment, if a sentence is completely deleted from the original SL document, update engine122-E of program122classifies the sentence into one or more relevant categories. In an embodiment, update engine122-E of program122searches for a sentence in the original PL document which closely matches the sentence deleted from the original SL document contextually and/or semantically. In an embodiment, update engine122-E of program122deletes the sentence in the original PL document that closely matches the sentence deleted from the original SL document contextually and/or semantically.

In an embodiment, if a word is inserted in the original SL document, update engine122-E of program122classifies the word into one or more relevant categories. In an embodiment, update engine122-E of program122searches for a sentence that closely maps to the word inserted on a sentence level. In an embodiment, update engine122-E of program122searches for the correct location where the word needs to be inserted into the original PL document based on contextual similarities. In an embodiment, update engine122-E of program122inserts the word into the chosen location of the original PL document.

In an embodiment, if a word is deleted from the original SL document, update engine122-E of program122classifies the word into one or more relevant categories. In an embodiment, update engine122-E of program122searches for a sentence in the original PL document that closely matches the word deleted from the original SL document contextually and/or semantically. In an embodiment, update engine122-E of program122deletes the word in the sentence in the original PL document that closely matches the word deleted from the original SL document contextually and/or semantically.

In step260, feedback analyzer engine122-F of program122requests feedback from the user. In an embodiment, feedback analyzer engine122-F of program122requests feedback from the user through the user computing device (e.g., user computing device130) via the user interface (user interface132). In an embodiment, feedback analyzer engine122-F of program122enables the user to input feedback through the user computing device (e.g., user computing device130) via the user interface (user interface132). The feedback input by the user may include, but is not limited to, a verification or rejection of the PL documents outputted and an acceptance or rejection of the updates, insertions, and/or deletions. In an embodiment, feedback analyzer engine122-F of program122receives feedback from the user. In an embodiment, feedback analyzer engine122-F of program122validates the feedback received from the user manually using the confidence scores of intermediate outputs (i.e., of each individual engine). In an embodiment, feedback analyzer engine122-F of program122identifies the relevant engines to be retrained. In an embodiment, feedback analyzer engine122-F of program122separates the feedback received from the user (e.g., based on the model that will be used to retrain the relevant engines). In an embodiment, feedback analyzer engine122-F of program122automatically annotates the feedback received from the user. In an embodiment, feedback analyzer engine122-F of program122stores the feedback received from the user in a database (e.g., database124) to be used later to retrain the relevant engines.

FIG.3is an exemplary diagram, generally designated300, illustrating the operational steps of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. The operational steps of program122are the same as described inFIG.2.

FIG.4is an exemplary diagram, generally designated400, illustrating the operational steps of PDF parser engine122-A of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. PDF parser122-A (e.g.,410) receives an input. The input is a PDF document (e.g.,420) of the Scientific Drug Label and the Patient Drug Label of the SL-PL pair (i.e., a SL .pdf and a PL .pdf). PDF parser122-A extracts content from the Scientific Drug Label in an XML format. PDF parser engine122-A processes the extracted content using an XML processing module. The XML processing module has two inner modules: a first module (e.g.,430) to extract a section heading and/or a subsection heading and a second module (e.g.,440) to extract data. PDF parser engine122-A sorts the structured data into a corresponding section heading and/or a corresponding section subheading in a structured j son format using a j son formatter module (e.g.,450). PDF parser engine122-A sends the structured data (e.g.,460) to NLP engine122-B.

FIG.5is an exemplary diagram, generally designated500, illustrating the operational steps of NLP engine122-B of program122, on server120within the distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. NLP engine122-B extracts one or more keywords and/or one or more key phrases (e.g.,510) from the structured data using a custom-trained Spacey model in order to understand the concept of each sentence of the structured data. Program122trains NLP engine122-B to generate a model for the one or more keywords and/or the one or more key phrases extracted. Program122determines a position and/or a category of the one or more keywords and/or the one or more key phrases extracted (e.g.,510). Program122creates an annotation for the position and/or the category of the one or more keywords and/or the one or more key phrases extracted (e.g.,520). Program122trains NLP engine122-B on the annotation created for the position and/or the category of the one or more keywords and/or the one or more key phrases using a deep learning algorithm (e.g.,530). Program122tests and validates the trained data (e.g.,540). Program122stores the trained data in a dictionary (e.g.,550) in the database (e.g., database124).

FIG.6is an exemplary diagram, generally designated600, illustrating the operational steps of mapping engine122-C of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. Mapping engine122-C receives a SL document in JSON format (e.g.,610), a PL document in JSON format (e.g.,620), and mapping configuration files (e.g.,630) obtained from a QRD template (e.g.,640). Mapping engine122-C extracts one or more mapping relationships (e.g.,6501-N) between the SL document in JSON format and the PL document in JSON format in terms of their section and/or sub-section relationships using a knowledge graph. Mapping engine122-C extracts one or more mapping relationships between the SL document in JSON format and the PL document in JSON format in terms of their interdependent and parent-child section/sub-section relationships using a knowledge graph. The one or more mapping relationships between the SL document in JSON format and the PL document in JSON format may include, but is not limited to, SL-PL direct section/sub-section (e.g.,650N), SL-SL interdependent section/sub-section (e.g.,6501), SL-SL parent-child (e.g.,6501), and PL-PL parent-child (e.g.,6502). Mapping engine122-C extracts a confidence score for each mapping relationship between the SL document in JSON format and the PL document in JSON format based on contextual information. Mapping engine122-C sends the extracted mapping relationships between the SL document in JSON format and the PL document in JSON format to update engine122-E (i.e., as a mapping document in JSON format, e.g.,660). Mapping engine122-C sends the SL document in JSON format and the PL document in JSON format to content relevancy engine122-D.

FIG.7is an exemplary diagram, generally designated700, illustrating the operational steps of content relevancy engine122-D of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. Content relevancy engine122-D receives the original SL document in JSON format (e.g.,720) and the PL document in JSON format (e.g.,730).

In a training stage (e.g.,710-A), content relevancy engine122-D converts the original SL document (e.g.,720) into a simplified patient friendly language using a transformation module (e.g.,740). Content relevancy engine122-D analyzes the simplified SL document (e.g.,750) to identify the one or more words, phrases, and/or sentences that have been modified, inserted, and/or deleted. Content relevancy engine122-D determines whether the one or more words, phrases, and/or sentences identified are relevant to a patient. Content relevancy engine122-D of program122labels as “relevant” the one or more words, phrases, and/or sentences determined to be relevant to the patient. Content relevancy engine122-D of program122labels as “irrelevant” the one or more words, phrases, and/or sentences determined to be irrelevant to the patient. For example,760-A represents the one or more words, phrases, and/or sentences labeled as relevant and760-B represents the one or more words, phrases, and/or sentences labeled as irrelevant. Content relevancy engine122-D of program122determines the label given to the one or more words, phrases, and/or sentences using a smart system. Content relevancy engine122-D of program122enables the smart system to perform unsupervised learning. The labels are not specified. The smart system determines the label based on an intelligent comparison model. The intelligent comparison model is a bidirectional transformer. Content relevancy engine122-D of program122trains the smart decision-making context similarity model (e.g.,770) for future cases using the labeled data.

In an inference stage (e.g.,710-B), content relevancy engine122-D converts the original SL document (e.g.,720) into a simplified patient friendly language using a transformation module (e.g.,740). Content relevancy engine122-D sends the simplified SL document (e.g.,750) to a smart decision-making context similarity module (e.g.,770) to determine whether the one or more words, phrases, and/or sentences identified are relevant to a patient. For example,760-A represents the one or more words, phrases, and/or sentences labeled as relevant and760-B represents the one or more words, phrases, and/or sentences labeled as irrelevant.

FIG.8is an exemplary diagram, generally designated800, illustrating the operational steps of update engine122-E of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. Update engine122-E receives the original SL document, the simplified SL document, and the mapping documents in json format. Update engine122-E classifies the one or more words, phrases, and/or sentences that have been modified, inserted, and/or deleted into one or more categories—complete insertion of a sentence in the original SL document, complete deletion of a sentence in the original SL document, and insertion and/or deletion of words and/or phrases in a sentence in the original SL document. Based on the classification of the one or more words, phrases, and/or sentences, the original PL document is updated.

If a sentence is completely inserted in the original SL document, update engine122-E classifies the sentence into one or more relevant categories. Update engine122-E searches for the correct location where the sentence needs to be inserted into the original PL document based on contextual similarities of PL sentences. Update engine122-E inserts the sentence into the chosen location of the original PL document.

If a sentence is completely deleted from the original SL document, update engine122-E classifies the sentence into one or more relevant categories. Update engine122-E searches for a sentence in the original PL document that closely matches the sentence deleted from the original SL document contextually and/or semantically. Update engine122-E deletes the sentence in the original PL document which closely matches the sentence deleted from the original SL document contextually and/or semantically.

If a word is inserted in the original SL document, update engine122-E classifies the word into one or more relevant categories. Update engine122-E searches for a sentence that closely maps to the word inserted on a sentence level. Update engine122-E searches for the correct location where the word needs to be inserted into the original PL document based on contextual similarities. Update engine122-E inserts the word into the chosen location of the original PL document.

If a word is deleted from the original SL document, update engine122-E classifies the word into one or more relevant categories. Update engine122-E searches for a sentence in the original PL document that closely matches the word deleted from the original SL document contextually and/or semantically. Update engine122-E deletes the word in the sentence in the original PL document which closely matches the word deleted from the original SL document contextually and/or semantically.

FIG.9Ais an exemplary illustration, generally designated900, of an analysis of an inserted key phrase by program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention.

FIG.9Bis an exemplary illustration, generally designated950, of an analysis of an inserted sentence by program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention.

FIG.10is an exemplary flowchart, generally designated1000, illustrating the operational steps of program122, on server120within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. The operational steps of program122are the same as described inFIG.2.

FIG.11is a block diagram illustrating the components of computing device1100(e.g., server120) within distributed data processing environment100ofFIG.1, in accordance with an embodiment of the present invention. It should be appreciated thatFIG.11provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments can be implemented. Many modifications to the depicted environment can be made. Computing device1100includes processor(s)1104, memory1106, cache1116, communications fabric1102, persistent storage1108, input/output (I/O) interface(s)1112, and communications unit1110. Communications fabric1102provides communications between memory1106, cache1116, persistent storage1108, input/output (I/O) interface(s)1112, and communications unit1110. Communications fabric1102can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric1102can be implemented with one or more buses or a cross switch. Memory1106and persistent storage1108are computer readable storage media. In this embodiment, memory1106includes random access memory (RAM). In general, memory1106can include any suitable volatile or non-volatile computer readable storage media. Cache1116is a fast memory that enhances the performance of computer processor(s)1104by holding recently accessed data, and data near accessed data, from memory1106.

Program instructions and data (e.g., software and data) used to practice embodiments of the present invention may be stored in persistent storage1108and in memory1106for execution by one or more of the respective processor(s)1104via cache1116. In an embodiment, persistent storage1108includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage1108can include a solid-state hard drive, a semiconductor storage device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

The media used by persistent storage1108may also be removable. For example, a removable hard drive may be used for persistent storage1108. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer onto another computer readable storage medium that is also part of persistent storage1108. Software and data can be stored in persistent storage1108for access and/or execution by one or more of the respective processor(s)1104via cache1116. With respect to user computing device130, software and data includes user interface132. With respect to server120, software and data includes program122.

Communications unit1110, in these examples, provides for communications with other data processing systems or devices. In these examples, communications unit1110includes one or more network interface cards. Communications unit1110may provide communications through the use of either or both physical and wireless communications links. Program instructions and data (e.g., software and data) used to practice embodiments of the present invention may be downloaded to persistent storage1108through communications unit1110.

I/O interface(s)1112allows for input and output of data with other devices that may be connected to each computer system. For example, I/O interface(s)1112may provide a connection to external device(s)1118, such as a keyboard, a keypad, a touch screen, and/or some other suitable input device. External device(s)1118can also include portable computer readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards. Program instructions and data (e.g., software and data) used to practice embodiments of the present invention can be stored on such portable computer readable storage media and can be loaded onto persistent storage1108via I/O interface(s)1112. I/O interface(s)1112also connect to display1120.

Display1120provides a mechanism to display data to a user and may be, for example, a computer monitor.