Update README.md

README.md

@@ -5,37 +5,31 @@ tags: []
 
 # Model Card for Model ID
 
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+The OpenDeid AICUP Suite is a collection of models developed to facilitate deidentification and temporal normalization research (see paper). It contains sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B based on the [Pythia Scaling Suite](https://huggingface.co/collections/EleutherAI/pythia-scaling-suite-64fb5dfa8c21ebb3db7ad2e1). The 2.8B model was trained on the original OpenDeid-AICUP corpus. 
 
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+This model is trained on the full OpenDeid-AICUP corpus released in the [ACIUP 2023 competition](https://codalab.lisn.upsaclay.fr/competitions/15425).
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** [ISLab](https://nkustislab.github.io/)
+- **Model type:** Transformer-based Language Model
+- **Language:** English
+- **License:** Apache 2.0
+- **Finetuned from model:** [EleutherAI/pythia-160m](https://huggingface.co/EleutherAI/pythia-160m)
 
-### Model Sources [optional]
+### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+- **Repository:** [ISLab-git](https://islab.ee.nkust.edu.tw:40000/hjdai/opendeid)
+- **Paper:** [More Information Needed]
+- **Demo:** [More Information Needed]
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+The primary intended use of the OpenDeid AICUP Suite is research on the behavior, functionality, and limitations of large language models for the deidentification and normalization tasks proposed in the [ACIUP 2023 competition](https://codalab.lisn.upsaclay.fr/competitions/15425). This suite is intended to provide a controlled setting for performing scientific experiments. 
+
+The models in the suite work with the Hugging Face Transformers Library. You may also further fine-tune and adapt the model for deployment, as long as your use is in accordance with the Apache 2.0 license and conduct your own risk and bias assessment.
 
 ### Direct Use
 
@@ -43,35 +37,35 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 
 [More Information Needed]
 
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
 ### Out-of-Scope Use
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+Similar to the original Pythia Suite, the OpenDeid AICUP Suite is not intended for deployment. It is not a in itself a product and cannot be used for human-facing interactions. For example, the model may generate harmful or offensive text. Please evaluate the risks associated with your particular use case.
 
-[More Information Needed]
-
-## Bias, Risks, and Limitations
+The OpenDeid models are English-language only, and are not suitable for translation or generating text in other languages.
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+OpenDeid-2.8B has been fine-tuned for the sensitive health information recognition and normalization tasks based on a pre-defined format. This means the OpenDeid AICUP Suite will not respond to a given prompt the way a product like ChatGPT does, which was fine-tuned using methods such as Reinforcement Learning from Human Feedback (RLHF) to better “follow” human instructions.
 
-[More Information Needed]
+## Bias, Risks, and Limitations
 
-### Recommendations
+This OpenDeid AICUP models are based on the Pythia models, which were pre-trained on the Pile, and further fine-tuned on the OpenDeid AICUP corpus, a dataset compiled for the sensitive health information and normalization tasks. The fine-tuned models tend to generate outputs in the manner of a pre-defined output layout which may not suiable for downstream tasks like text summarization or translation.
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+## How to Get Started with the Model
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+Use the code (based on [vLLM](https://github.com/vllm-project/vllm)) below or conventional transformer codes to to get started with the model.
 
-## How to Get Started with the Model
+```
+from vllm import LLM, SamplingParams
+from transformers import AutoTokenizer
 
-Use the code below to get started with the model.
+model = LLM('ISLabResearch/opendeid-2.8b-ft-full')
+seed = 309
+tokenizer = AutoTokenizer.from_pretrained(Name)
+eos = tokenizer.eos_token
 
-[More Information Needed]
+params = SamplingParams(max_tokens = 50, include_stop_str_in_output = True, temperature = 0,  
+                                     ignore_eos = False, stop = [eos], seed=seed)
+preds = model.generate("Hello", params, use_tqdm = False)
+```
 
 ## Training Details
 
@@ -126,47 +120,40 @@ Use the code below to get started with the model.
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
+#### Sensitive Health Information Recognition Results
+
+| Coding Type | Precision | Recall | F-measure | Support |
+|-------------|----------:|:------:|----------:|--------:|
+| MEDICALRECORD | 0.7651331 | 0.8460509 | 0.8035601 | 747 |
+| PATIENT | 0.8121547 | 0.6159218 | 0.700556 | 716 |
+| IDNUM | 0.8933273 | 0.9283019 | 0.9104788 | 2120 |
+| DATE | 0.9843959 | 0.897926 | 0.9391748 | 2459 |
+| DOCTOR | 0.8278772 | 0.7373009 | 0.7799682 | 3327 |
+| CITY | 0.9522388 | 0.8552279 | 0.90113 | 373 |
+| STATE | 0.973384 | 0.7710843 | 0.8605042 | 332 |
+| ZIP | 0.9937888 | 0.9065156 | 0.9481481 | 353 |
+| TIME | 0.9425287 | 0.8723404 | 0.9060773 | 470 |
+| DEPARTMENT | 0.7521866 | 0.6157518 | 0.6771653 | 419 |
+| HOSPITAL | 0.7908788 | 0.5934892 | 0.6781116 | 1198 |
+| AGE | 0.9565217 | 0.4313726 | 0.5945946 | 51 |
+| STREET | 0.8181818 | 0.2093023 | 0.3333333 | 344 |
+| ORGANIZATION | 0.75 | 0.04054054 | 0.07692308 | 74 |
+| DURATION | 0 | 0 | 0 | 12 |
+| SET | 0 | 0 | 0 | 5 |
+| LOCATION-OTHER | 0 | 0 | 0 | 6 |
+| PHONE | 0 | 0 | 0 | 1 |
+| Micro-avg. F| 0.8766754 | 0.7744291 | 0.8223864 | 13007 |
+| Macro-avg. F| 0.6784776 | 0.5178403 | 0.587374 | 13007 |
+
+#### Temporal Information Normalization Results
+| Temporal Type | Precision | Recall | F-measure | Support |
+|-------------|----------:|:------:|----------:|----------:|
+| DATE | 0.7993659 | 0.7177715 | 0.7563745 | 2459 |
+| TIME | 0.6951219 | 0.606383 | 0.6477273 | 470 |
+| DURATION | 0 | 0 | 0 | 12 |
+| SET | 0 | 0 | 0 | 5 |
+| Micro-avg.| 0.7830405 | 0.6958588 | 0.7368799 | 2946 |
+| Macro-avg.| 0.373622 | 0.3310386 | 0.3510436 | 2946 |
 
 ## Citation [optional]
 
@@ -180,12 +167,6 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 
 [More Information Needed]
 
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
 ## More Information [optional]
 
 [More Information Needed]