Research task (AI & ML):
I'm currently working on a project. The professor wasn't clear, she just said that I will use NLP techniques on some data, de-identified Electronic Health Records, to extract Asian subgroups, and try to abstract the same subgroups as are in CDC WONDER data. Then, I'll apply this to look at prevalence rates of autoimmune diseases.

I have no clue what she means. Can you first help explain what she might mean? I don't know examples of what Electronic Health Records look like. What NLP techniques might I be using on data like this? What subgroups in CDC WONDER data would I be abstracting? How could I apply this to look at prevalence rates of autoimmune diseases? What skills should I brush up on? Please help me create a plan for my research going forward, according to what the professor said.

Evaluation rubric:
criterion: The response is missing one of the six requested points: (1) what the professor likely means, (2) examples of de-identified EHR data, (3) NLP techniques that could be used, (4) Asian subgroups contained in CDC WONDER, (5) how to use the extracted information to study autoimmune-disease prevalence, and (6) skills to brush up on.
weight: -2
axis: Instruction Following

criterion: The response gives at least three concrete examples of de-identified Electronic Health Record content (e.g., a diagnosis code block, a free-text note snippet without PHI, tabular data contextualizing text and numerical data, image data corresponding to a diagnostic report, and video data corresponding to radiology image reports).
weight: 5
axis: Explicit Criteria

criterion: The response names at least three distinct NLP techniques from the following list: Named Entity Recognition, rule-based pattern matching/regular expressions, dictionary or ontology lookup, supervised classification, topic modeling, clustering, word embeddings, and transformer fine-tuning.
weight: 5
axis: Explicit Criteria

criterion: The response correctly identifies (by name) either 6, 15, or 31 race categories from CDC WONDER data (i.e., 6-category list: American Indian or Alaska Native, Asian, Black or African American, Native Hawaiian or Other Pacific Islander, White, and More than one race) The 15-category list separates more Asian and Pacific Islander groups (e.g., Chinese, Filipino, Japanese), while the 31-category list includes combinations of multiple races (e.g., Black, AIAN, and White).
weight: 5
axis: Explicit Criteria

criterion: The response states or implies that the goal is to map Electronic Health Record-derived subgroups to the same CDC WONDER categories before calculating prevalence.
weight: 1
axis: Synthesis of Information

criterion: The response describes how exactly to derive subgroups from Electronic Health Records (e.g., programmatic text extractions or more rigorous NLP and machine learning techniques that can extract subgroups by interfacing with sources, such as looking into self-reported information provided through questionnaires, cross-referencing data across multiple visits, combing data from external sources such as geospatial data, and looking into census data). 
weight: 5
axis: Synthesis of Information

criterion: The response describes at least two different methods for calculating prevalence (e.g., extrapolations from individual autoimmune disease occurrences, machine-learning techniques via training fine-tuned models, logistic regression models, and meta-analyses that include results from existing papers). 
weight: 5
axis: Explicit Criteria

criterion: The response lists at least five relevant skills or tools to learn (e.g., data science skills in order to parse and visualize Electronic Health Record data, statistical analysis skills in order to perform and interpret regression analyses, coding skills in order to write up statistical or machine learning models, data deidentification techniques, and domain knowledge for autoimmune disease factors).
weight: 5
axis: Explicit Criteria

criterion: The response correctly explains the exact mechanism by which de-identification removes direct patient identifiers to comply with privacy regulations, which can be one of the two techniques from the HIPAA Privacy Rule De-identification Methods of 1) Expert Determination, which involves an expert assessing the risk of identification by running appropriate statistical and scientific methods to mitigate risk of identification, and 2) Safe Harbor, which involves the direct removal of 18 types of identifiers, such as specific dates, the last four digits of a Social Security numbers, patient initials, etc, and can be performed via NLP techniques.
weight: 3
axis: Implicit Criteria

criterion: The response has at least one example that walks through every processing step that maps unstructured data to a diagnosis, in particular mentioning 1) NLP techniques to extract Asian subgroups present in the de-identified Electronic Health Records data, 2) NLP techniques to map the extracted racial subgroups to formal CDC WONDER racial subgroups, 3) apply a prevalence analysis (e.g., statistical or machine learning based) to look for the frequency of autoimmune diseases in the specified populations, as they apply to the specific example.
weight: 2
axis: Synthesis of Information

criterion: The response defines or briefly explains, and expands, technical terms the first time they are used (e.g., defines NLP as Natural Language Processing, defines CDC WONDER (Wide-ranging ONline Data for Epidemiologic Research), which is an online database for public health records).
weight: 5
axis: Communication Quality

criterion: The response mentions that while Electronic Health Records often contain structured forms of (e.g., with name, date, age, etc. columns fairly easy to extract), there may be additional forms of unstructured data within the records that the user may need to target their NLP techniques towards (e.g., doctors' notes from patient visits, multimedia files from reports).
weight: 3
axis: Synthesis of Information

criterion: The response fails to address that any analysis must comply with HIPAA to prohibit any disclosure of protected health information in order to ensure patient privacy.
weight: -5
axis: Implicit Criteria

criterion: The response names at least one open-source NLP library (e.g., spaCy, scikit-learn, transformers, and nltk).
weight: 2
axis: Implicit Criteria

criterion: The response use language that references to the words used by the professor and is directed to the user who is confused and need clarity (e.g. (1) What the professor means by "I will use NLP techniques on some data" is that you are tasked to apply natural language processing methods such as, etc..., (2) By "de-identified Electronic Health Records", you are asked to de-identify the Electronics Health Records which is the record of the patients' medical history stored electronically at the healthcare providers, etc., and (3) Next, you will be applying this "to look at prevalence rates of autoimmune diseases").
weight: 1
axis: Instruction Following

criterion: The response incorrectly orders the tasks to be performed (i.e., The proper ordering should be (1) what the professor likely means, (2) examples of de-identified EHR data, (3) NLP techniques that could be used, (4) Asian subgroups contained in CDC WONDER, (5) how to use the extracted information to study autoimmune-disease prevalence, and (6) skills to brush up on).
weight: -3
axis: Communication Quality

criterion: The response considers at least five relevant research and sources of references (e.g., (1) National Library of Medicine, (2) CDC Wonder, and (3) publications from well-known universities like Stanford, Oxford, or Global Autoimmune Institute).
weight: 5
axis: References & Citation Quality

criterion: The response considers the limitations of the approaches used to identify autoimmune diseases from the data. (e.g., (1) Using ICD-10 code, but some rare autoimmune diseases like antisynthetase syndrome and IgG4-related disease, (2) many diagnoses can change and gets refined after subsequent patient visits,  (3) diagnosis can be caused by both autoimmune process or non-autoimmune process such as type 1 diabetes mellitus in a patient who has undergone a total pancreatectomy, and (4) source of data provided by specialized or general medical practitioners can also introduce biases).
weight: 3
axis: Implicit Criteria

criterion: The response makes no reference to previous studies/data about the prevalence rate of autoimmune diseases in Asian subgroups (e.g., (1) the paper titled "High Disease Severity Among Asians in a US Multiethnic Cohort of Individuals with Systemic Lupus Erythematosus", (2) Hispanics and Asians were diagnosed with lupus and had more aggressive disease more frequently than whites, and (3) Sjögren's Syndrome (SS), an autoimmune disorder affecting the salivary and tear glands, is more prevalent among individuals of Asian descent).
weight: -5
axis: Synthesis of Information

criterion: The response identifies at least 8 different autoimmune diseases examples with special attention to those prevalent in Asian subgroups for context (e.g., (1) autoimmune thyroiditis (AT), (2) Graves' disease (GD), (3) type 1 diabetes mellitus (T1D), (4) multiple sclerosis (MS), (5) Crohn's disease (CD), and (6) ulcerative colitis (UC)).
weight: 2
axis: Synthesis of Information