Research task (AI & ML):
Please help me do a literature review. I'm looking for papers that use NLP techniques to turn unstructured EHR data into structured data, or extract information from the unstructured EHR data. Specifically, I'm curious how rule-based systems and machine learning based NER differ in results.

Also, do LLMs supersede these techniques in information extraction from EHR? What's the cheapest way to do this with LLMs, can I run a local LLM to do this? I have a Mac Apple M2 Pro with 16 GB of memory. What's the most powerful local LLM I can run for this purpose? Or, if the data is de-identified, what's the cheapest LLM API that I can call in order to get this done, and what will be the cost per word? Would this be a HIPAA violation?

What's the cost of calling an LLM API compared to existing systems that can you can pay to get unstructured EHR data turned into structured data, depending on how many words are in the unstructured data?

Evaluation rubric:
criterion: The response includes at least five literature articles on rule-based NLP techniques for structuring EHR data (e.g., dictionary-based, regex, or systems like cTAKES) and provides a brief summary of each that highlights the method used, the type of EHR data handled (e.g., clinical notes, discharge summaries), and the effectiveness or limitations relevant to information extraction.
weight: 5
axis: Explicit Criteria

criterion: The response includes at least five literature articles on machine-learning-based NER (e.g., CRF, BiLSTM-CRF, BERT, ClinicalBERT) for EHR data and provides a brief summary of each that explains the model type, the entities extracted (e.g., medications, diagnoses, procedures), and the reported strengths or limitations.
weight: 5
axis: Explicit Criteria

criterion: The response mentions at least three quantitative artifacts (e.g., precision, recall, F1 score, accuracy) from the cited literature to illustrate how rule-based systems perform on extracting structured information from unstructured EHR data.
weight: 3
axis: References & Citation Quality

criterion: The response mentions at least three quantitative artifacts (e.g., precision, recall, F1 score, accuracy) from the cited literature to illustrate how machine-learning-based NER systems perform on extracting structured information from unstructured EHR data.
weight: 3
axis: References & Citation Quality

criterion: The response explicitly states that rule-based systems generally achieve higher precision but lower recall than ML-based NER systems and substantiates this with at least one comparative metric from the literature (e.g., rule-based vs CRF on i2b2 notes, rule-based vs BiLSTM-CRF on MIMIC-III, cTAKES rules vs BERT-based NER on discharge summaries, custom rules vs ClinicalBERT on medication extraction).
weight: 4
axis: Explicit Criteria

criterion: The response clearly takes a position on whether current LLMs fully supersede, partially complement, or do not yet supersede rule-based/ML NER for EHR IE (Information Extraction) and justifies it with at least one concrete reason (e.g., benchmark F1 gaps on clinical NER, ontology/terminology mapping needs, hallucination/consistency risks, cost/latency or privacy constraints).
weight: 5
axis: Explicit Criteria

criterion: The response recommends at least three specific local LLM models (e.g., Llama-2-7B-Q4, Mistral-7B, Falcon-7B, Vicuna-7B) and justifies their suitability for extracting structured information from EHR data (e.g., performance on medical benchmarks, parameter size compatible with 16GB RAM, quantization availability).
weight: 5
axis: Explicit Criteria

criterion: The response indicates the estimated hardware requirements for each recommended local LLM (e.g., 8 GB VRAM for a Llama-2-7B-Q4, 16 GB RAM for CPU-only inference) and confirms whether they can be run on the user’s MacBook Pro M2 with 16 GB of unified memory.
weight: 5
axis: Explicit Criteria

criterion: The response mentions at least three existing commercial EHR NLP services (e.g., Amazon Comprehend Medical, Google Cloud Healthcare NLP, Microsoft Text Analytics for Health, IBM Watson Health) that can be used to convert unstructured EHR data into structured data.
weight: 4
axis: Explicit Criteria

criterion: The response states an estimated price per word for each commercial EHR NLP service mentioned.
weight: 5
axis: Explicit Criteria

criterion: The response provides a definite answer on whether sending de-identified EHR data to an LLM API is a HIPAA violation, without using vague phrases (e.g., ‘most of the time,’ ‘should be fine,’ or ‘probably’).
weight: 5
axis: Explicit Criteria

criterion: The response includes at least five literature articles on the use of LLMs for extracting or structuring EHR data and provides a brief summary of their findings on LLMs' performance on this task (e.g., zero-shot NER capabilities, comparison to fine-tuned BERT, hallucination rates).
weight: 3
axis: Implicit Criteria

criterion: The response mentions at least three quantitative artifacts (e.g., precision, recall, F1 score, accuracy) from the cited literature to illustrate how LLMs perform on extracting structured information from unstructured EHR data.
weight: 3
axis: References & Citation Quality

criterion: The response includes at least five literature articles on hybrid systems (rule-based + ML/LLM) for extracting or structuring EHR data and provides a brief summary of each that notes the hybrid components and key results (e.g., rules + CRF for medication extraction, cTAKES dictionaries + BiLSTM-CRF for problem lists, ontology constraints + BERT for clinical concept normalization, rule filters + GPT-4 validator for adverse event extraction).
weight: 4
axis: Implicit Criteria

criterion: The response addresses the difficulties of running a local LLM on a laptop (e.g., slow inference speed, memory limits, overheating or throttling, large disk space requirements).
weight: 4
axis: Implicit Criteria

criterion: The response addresses differences in the type and format of data required by different extraction systems (e.g., rule-based systems needing dictionaries/ontologies, ML NER requiring annotated training corpora, LLMs requiring large unstructured text prompts, hybrid systems requiring both rules and annotated samples).
weight: 4
axis: Synthesis of Information

criterion: The response uses at least one table to clearly compare different approaches for extracting/structuring EHR data, with columns that highlight key dimensions (e.g., rule-based vs ML vs LLM vs hybrid, required data format, performance metrics, hardware or API costs).
weight: 3
axis: Communication Quality

criterion: The response addresses potential HIPAA concerns with commercial EHR NLP services and describes situations where they do not violate HIPAA when used appropriately (e.g., vendor HIPAA compliance certifications, Business Associate Agreements (BAAs), data encryption in transit, processing within secure cloud environments).
weight: 4
axis: Implicit Criteria

criterion: The response includes the authors and year of publication for the articles mentioned in the literature review.
weight: 4
axis: References & Citation Quality

criterion: The response identifies the single cheapest LLM API option for processing de-identified EHR data and includes a clear cost-per-word calculation (e.g., OpenAI GPT-3.5, Anthropic Claude, Cohere Command, Mistral API).
weight: 5
axis: Explicit Criteria

criterion: The response compares the cost per word of the identified cheapest LLM API with the cost per word of existing commercial EHR NLP services and illustrates how the total cost scales with increasing word count (e.g., 1K, 10K, 100K, 1M words).
weight: 5
axis: Explicit Criteria

criterion: The response recommends the overall cheapest approach for extracting structured information from EHR data (e.g., local LLM, API, commercial service), based on both the user’s hardware constraints (MacBook Pro M2 with 16 GB memory) and the comparative cost-per-word analysis.
weight: 4
axis: Implicit Criteria

criterion: The response mentions the necessary software required to run a local LLM on the user’s MacBook Pro M2 with 16 GB memory (e.g., llama.cpp, GPT4All, Ollama, Hugging Face Transformers).
weight: 2
axis: Implicit Criteria

criterion: The response suggests local LLMs that cannot realistically run on the user’s MacBook Pro M2 with 16 GB memory (e.g., unquantized Llama-2-13B, Falcon-40B) without mentioning the necessity of remedy techniques (e.g., quantization, CPU-only inference, model distillation, offloading to disk).
weight: -5
axis: Explicit Criteria

criterion: The response uses a unit of cost other than per word (e.g., per token, per character) to estimate the price of an EHR NLP service without providing a conversion to a per-word cost.
weight: -4
axis: Explicit Criteria

criterion: The response identifies the most powerful local LLM that can realistically run on the user’s MacBook Pro M2 with 16 GB memory, with justification (e.g., parameter size, quantization level, memory fit).
weight: 5
axis: Explicit Criteria

criterion: The response discusses the HIPAA/privacy trade-offs between running LLMs locally versus sending de-identified EHR data to an API.
weight: 2
axis: Implicit Criteria