Automated Digitization of Unstructured Medical Prescriptions
Year: 2023

Abstract:
Automated digitization of prescription images is a critical prerequisite to scale digital healthcare services such as online pharmacies. This is challenging in emerging markets since prescriptions are not digitized at source and patients lack the medical expertise to interpret prescriptions to place orders. In this paper, we present prescription digitization system for online medicine ordering built with minimal supervision. Our system uses a modular pipeline comprising a mix of ML and rule-based components for (a) image to text extraction, (b) segmentation into blocks and medication items, (c) medication attribute extraction, (d) matching against medicine catalog, and (e) shopping cart building. Our approach efficiently utilizes multiple signals like layout, medical ontologies, and semantic embeddings via LayoutLMv2 model to yield substantial improvement relative to strong baselines on medication attribute extraction. Our pipeline achieves +5.9% gain in precision@3 and +5.6% in recall@3 over catalog-based fuzzy matching baseline for shopping cart building for printed prescriptions.

heading: 1 Introduction
text: In recent years, prompted by the COVID pandemic, there has been a rise in the adoption of online pharmaceutical services leading to improved access to medications and health outcomes. However, in emerging markets such as India, online pharmacy ordering continues to be challenging since prescriptions tend to be paper-based, unstructured and often, handwritten, which makes digitization a vital prerequisite. For in-store purchases, customers follow a simple process of presenting a prescription to the store pharmacist who interprets it and fulfills the order. Current e-commerce purchase process, however, imposes a significant cognitive load on customers since they have to explicitly
∗Equal contribution †Work done while at Amazon
specify the medicines. This process is onerous for the customers due to their (a) unfamiliarity with the ordering process, (b) difficulty in understanding prescriptions, and (c) lack of expertise to interpret medical acronyms and identify substitute medicines. Further, most online pharmacies have a post-cart creation workflow where customers upload the prescription to be verified by a remote pharmacist. Lack of pharmacist capacity often leads to long wait time making the process unscalable. Therefore, an automated system that converts prescription images to a digitized form to facilitate search-less shopping is essential for the success of online pharmacies. In particular, we need to extract the medical advice section which contains a list of medication items, each of which is a record of multiple fields such as BRAND-NAME. Challenges. Addressing this problem is non-trivial due to multiple reasons shown in Figure 3a: (a) variability in prescription image quality, background, and orientation, (b) diversity of layouts and doctor styles, (c) high prevalence of typos that create confusion between similar items (e.g., Fibrodone and Firodone), (d) specialized vocabulary of regional prescriptions, and (e) the need for converting dosage-specific instructions to a precise product order. Additionally, there are limited labeled prescriptions due to the high manual effort it entails. Related work. While there have been significant advances in document AI [6, 15, 19] and information extraction [29, 21, 13] techniques, most of these methods are effective only on images of wellformatted documents such as invoices. Besides, these generic methods require significant supervision and are not sufficiently modular to support a phased automation of the prescription processing workflow. Recent work on digitizing medical prescriptions [27] is focused on using named entity recognition (NER) methods for medication attribute detection, but these models perform poorly on non-US prescriptions due to vocabulary gaps
794
and do not utilize the layout or catalog information. We present additional related work in Appendix B. Contributions. In this paper, we present a study on automated digitisation of prescription images with printed content which covers design, data, modeling and evaluation aspects. We discuss experimental results for our emerging modular pipeline which comprises a mix of ML and rule-based components for (a) image to text extraction and normalization (b) segmentation into blocks, medication items and extraction of medication attributes, (c) matching against pharmacy catalog, and (d) shopping cart building. We detail how our approach efficiently combines layout signal, medical ontologies, and semantic embeddings via LayoutLMv2 model to yield substantial improvement relative to strong baselines on medication attribute extraction, and results in +5.9% and +5.6% gain in precision@3 and recall@3 over catalog-based fuzzy matching baseline for printed prescriptions. We discuss key learnings relevant for low data regime document AI systems in addition to presenting component-wise efficacy of our pipeline and results from ongoing experiments (Appendix A.4.2) to highlight future directions. We present safety aspects in Section 9.

heading: 2 Prescription Digitization Problem
text: Given a prescription image, a natural choice for digitization is in terms of conversion to a structured prescription object as per a global standard such as the Fast Healthcare Interoperability Resources (FHIR) framework [3]. Since our objective is to
create a shopping cart for automated medicine ordering we focus on populating the relevant fields only in the FHIR prescription schema (Table 5). To accommodate the nuances of regional medications, we define each medicine in the pharmacy catalog as a unique tuple of BRAND-NAME or GENERICNAME1, FORM, INGREDIENT and STRENGTH. A unique pair of a medicine and package details corresponds to a stock keeping unit (SKU).
Figure 1 depicts the various stages of processing a prescription (denoted by hA, hD, hO) that results in the successive creation of following entities: (a) Annotated Prescription is a visually rich document (VRD) comprising labeled rectangular bounding boxes (BBs). Each BB is associated with text and a list of annotations, which include the start-end offsets and labels corresponding to medication attributes, item boundaries, and block type, (b) Digitized Prescription is a structured object with canonical entries following the FHIR-based prescription schema, (c) Pharmacy Order is a list of SKUs from the prescription along with the recommended quantities. The conversion to a pharmacy order (hO) can be enabled via a deterministic lookup using a medicine-SKU map if the medication codes in the digitized prescription are from the catalog. Hence, we focus on the non-trivial transformations hA and hD that entail a data-driven approach. Let P = {pi}Ni=1 denote the set of the available prescription images for training. For the ith prescription pi, let (ai, di, oi) denote the human annotated prescription, digitized prescription, and pharmacy order obtained from expert pharmacists, i.e., ai = hA(pi), di = hD(ai), oi = hO(di). Typically, pharmacists directly create or validate pharmacy orders from a prescription image without any record of the intermediate annotation and digitization. Since these prescription-order pairs are inadequate for an end-to-end neural model, we explicitly gather supervision for the intermediate stages for a subset of the prescriptions to enable a pipelined approach. Let zAi , z D i , z O i denote binary indicators of the availability of ai, di, oi respectively. Further LA(·, ·), LD(·, ·), LO(·, ·) be suitable loss functions for comparison of pairs of candidate annotated versions, digitized versions, and orders corresponding to a prescription such as the accuracy of annotations, matching with canonical entities, and the constructed order respectively as shown in Table 2. Then, the training objective
1Generic names are globally approved, e.g., paracetamol, while brand names are manufacturer given e.g., Calpol.
is to learn mappings ĥA and ĥD that produce high fidelity reconstructions of the processed versions of the prescription and can be viewed as a loss minimization:
minĥA,ĥD [∑ i|zAi =1 LA(ai, ĥA(pi))
+ ∑
i|zDi =1 LD(di, (ĥA ◦ ĥD)(pi))
+ ∑
i|zOi =1 LO(oi, (ĥA ◦ ĥD ◦ hO)(pi))
]
Given a new prescription, the learned mappings (ĥA, ĥD) along with hO yield a pharmacy order.

heading: 3 Solution Design

heading: 3.1 Design Choices
text: We discuss the key tenets and design choices of our prescription digitization approach (Figure 3b). Modularity. Supporting phased automation of user-driven cart building and pharmacist-driven validation workflows entailed a modular pipeline. Solution choice dependent on input signals. Limited labeled prescription data coupled with access to medical ontologies made it prudent to choose a hybrid combination of rule-based and ML modules instead of an end-to-end deep neural model. Interoperability. The need to interface with other healthcare systems led us to choose a data representation based on global FHIR standards. Extension over reinvention. Fast and scalable implementation required use of existing solutions for sub-problems wherever acceptable and focusing on exploration of the harder sub-problems.

heading: 3.2 Components
text: We describe components of Figure 3b below. Text extraction & VRD Normalization. First, we identify OCR bounding boxes (BBs) and extract the text from these BBs. Then we perform rotation and background cropping, using the position coordinates of BBs, to create normalized VRDs with more homogeneous layouts as shown in Figure 2.
Entity Annotation. Annotating BBs comprises three tasks corresponding to stage (b) of Figure 1: (a) detecting block(s) containing medical advice of doctor, (b) chunking of words, within medical advice block, related to medication(s) into
item(s), and (c) extracting medication attributes such as brand name, duration of consumption from an item. Though a joint model that optimizes∑
i|zAi =1 LA(ai, ĥA(pi)) to simultaneously detect blocks, medication items, and attributes seems like a natural choice, it is prohibitive due to the constraints on amount of supervision, computational effort, and limits on context size of NLP models (usually 512 tokens). We simplify this problem by solving the sub-tasks in the order (a → c → b). Advice block detection reduces sequence length (as shown in figure 4) permitting transformer-based encodings and increasing precision for later tasks. In this task, we construct latent representation of the BBs based on position, semantics, and membership in medical ontologies and learn a supervised classification model to predict whether a BB contains medical advice or not. We perform step (c) and (b) only on BBs predicted as advice blocks. For medication attributes, we label each token in advice BB using our NER model into one of 7 classes (DURATION, FORM, FREQUENCY, INGREDIENT, ITEM-MARKER, BRAND-NAME, and STRENGTH). Sequence of NER predictions are fed into our heuristic algorithm for medication item chunking that leverages relative positions of BRANDNAME, STRENGTH and FORM tokens. Matching and Canonicalization. The next step (ĥD) is to map each annotated medication item in the prescription (e.g., T.[FORM] Crocin[BRAND-NAME], 5 ml[STRENGTH]) to a medicine ID in the pharmacy catalog using extracted attributes. For this we use our Pharmacy product catalog as a reference. This catalog contains all medicine products listed on our website and each product is described by a set of attributes such as BRAND-NAME and FORM. We adopt a two-stage approach comprising: (a) identifying candidates by fuzzy matching predicted BRAND-NAME with that in catalog, (b) computing a match score based on FORM and STRENGTH to identify the best matching medicine ID using either a rule-based or an ML classifier.
Cart Building. The final step (hO) is to construct the pharmacy order, i.e., list of required SKUs and their quantities. To enable this, the standard dosage amount of SKU is computed during catalog creation, e.g., 3 packs of 30 ml bottle maps to 90 ml. From the digitized prescription, total recommended dosage amount can be computed from dosage duration, daily dosage pattern and units to be consumed at a time. Appropriate SKU and its quantity can be
derived to minimally exceed this amount.

heading: 4 Data Collection
text: Due to the sensitive nature of prescriptions and recent emergence of our medicine ordering application, a public dataset of unstructured prescription images does not exist to the best of our knowledge. External benchmarks such as [5] only contain clean text without any layout information. Hence, we use a proprietary dataset of 1359 Indian prescriptions paired with (fully or partial) digitized orders, delinked from customer IDs. These are mostly or fully printed prescriptions and have been validated by our in-house experts. The prescription images are modified as follows prior to modeling. AWS Textract, which is security certified for critical data, is used to extract text from images. The obtained text is then run through AWS Comprehend to detect personally identifiable information such as patient/doctor names, phone numbers and then the corresponding OCR bounding boxes are grayed out. For a subset of prescriptions, we procured in-house human annotations for supervised training of all components. Ground-truth text, BBs for medical advice blocks, labeled text spans for medication attributes, as well labels for pairs of candidate and ground truth SKUs for medication matching were annotated in the prescription image by the annotators. More details on the annotation tasks are given in Table 6. Table 1 lists details of the training and evaluating splits for various components. Given the expensive labeling effort, this data size is realistic for early-stage specialized document AI systems.

heading: 5 Experimental Results
text: We present our evaluation method and results on the efficacy of the full system and various components with focus on medication attribute extraction.

heading: 5.1 Evaluation Methodology
text: Practical systems need to be evaluated during development (offline metrics) and post deployment (online metrics). Table 1 lists our offline evaluation metrics. Most of these are self-explanatory except Brand match which is the percentage of medicine brand names ordered by the customers in the extracted text and indicates the medical text extraction efficacy. The online metrics of our system (not reported for proprietary reasons) depend
2https://en.wikipedia.org/wiki/Word_error_rate 3Strict matching metrics as per SemEval-13 [7].
on whether the digitization is integrated into the pharmacist processing flow or the customer-facing UI. These include rate of correction of automated cart suggestions, reduction in cart-building time, reduction in order rejections during verification stage as well as business metrics on the order volume.

heading: 5.2 Component-wise Efficacy
text: Table 2 lists the metrics of various components of our digitization pipeline, which we discuss below. Text Extraction. Due to limited supervision, we use pretrained off-the-shelf solutions. AWS Textract is our preferred choice as it provides a higher brand match (+7%) than AWS Rekognition as the latter has limit on the number of extractable words. Advice Block detection. To reduce complexity for downstream tasks, we first detect medical advice blocks. We employ a two-stage solution (see Figure 4) of (a) clustering BBs using K-means on their positional coordinates, and (b) classifying each cluster as advice block or not using XGBoost [8] classifier trained on cluster position, and fractions of medical and printed words. Lastly, adjacent advice blocks are merged. This method can be extended to other block types (e.g., header, footer) using a multi-class classifier and block-type indicators. Our solution results in an operational point with 94.8% recall, 88.1% precision, reduction in block size (Figure 5) and mostly homogeneous clusters (homogeneity score: 0.857). Common errors occur due to: (a) sparse text that cannot use local semantic context well leading to false positives, and (b) long lines that are ideally a single cluster but split because of the high divergence in the horizontal dimension. Medication Item Chunking. We exploit the observation that attributes of a medication are contiguous with BRAND NAME preceding STRENGTH and the ordering relative to FORM being flexible. Let tk be the kth detected BRAND NAME token. For each tk, we construct up to two candidate medications with brand based on tk, STRENGTH based on the closest STRENGTH token to tk in the span (tk, tk+1), and FORM derived from the FORM tokens closest to tk on either side in the spans (tk−1, tk) and (tk−1, tk). Our approach yields high accuracy (97.2%) obviating the need for an ML system. Matching and Canonicalization. As discussed in Section 3.2, we employ a two-stage approach of filtering and ranking using match score. For match score computation, we consider two methods using the same attribute fuzzy scores as inputs: 1) rule-
based one with heuristics for handling missing values and 2) XGBoost classifier that predicts whether a chunked item matches a candidate SKU using the same features as rule-based one except heuristics. Empirically, we see that the former approach yields +18% gain over the latter in precision@1 of the top matching item. Note that precision@1 is the same as recall@1 due to stand-alone evaluation.

heading: 5.3 Medication Attribute Extraction
text: Table 3 shows performance of various approaches based on multiple input signals: (a) Catalog Features: These include membership scores of tokens with respect to dictionaries of BRAND-NAME, INGREDIENT, STRENGTH (e.g., mg) and FORM (e.g., tablet) created from the catalog, (b) Semantic features: These include contextual text embeddings derived from transformer models such as BERT [10] and MedBERT [25] pretrained on Wikipedia and PubMed respectively, (c) Layout features: Since the layout provides extra information, e.g., text in the middle is usually medical
advice, we use LayoutLM [32] and LayoutLMv2 [31] models, which have multi-modal Transformer architecture as backbone and utilize layout, visual, and textual features to learn cross-modal interactions, and (d) Collective labeling: We use the linear conditional random field (CRF) loss to exploit relationships amongst labels, e.g. BRAND-NAME often lies between FORM and STRENGTH.
Note than in Table 3, the token level metrics are weighted with token length so that errors on small tokens are less penalized and OTHER tokens are excluded as these are not critical for the application. From the results, we observe that XGBoost trained with catalog features performs comparable to custom Comprehend fine-tuned on our data illustrating the importance of catalog signal. While BERT-based models using semantic features further improve the performance, the best accuracy is seen when we incorporate layout features (LayoutLMV2 variants) as well. Note that models such as Comprehend Medical and MedBERT are not suitable for our problem as these are not trained on the Indian medicine vocabulary.
Ablation Studies. To evaluate the efficacy of the various signals as well as modeling sequential dependencies via CRF, we conducted ablation studies. Since LayoutLMv2 already uses semantic and layout features, we added collective labeling (LayoutLMv2 + CRF) and catalog features (LayoutLMv2 + CF) separately and in a combined setting (LayoutLMv2 + CF + CRF). We note from Table 3 that performance only changes marginally. Similar behavior is observed when using BERTvariants indicating that catalog and collective labeling are subsumed by the semantic and layout encoding. Good performance of XGBoost + catalog features variant points to presence of non-linear interactions and value of catalog signal. Further studies (Appendix A.4.2) indicate that the performance depends on the quality and diversity of supervision more than than the quantity pointing to
the benefits of using active learning approaches. Error Diagnosis. Table 4 presents an error diagnosis of our best model (LayoutLMv2) and areas of improvement such as deducing labels from context (e.g. "tablet once a day"→ Frequency). Figure 9 presents the confusion matrix of different medication attribute classes.

heading: 5.4 Overall Cart Building Efficacy
text: We evaluate the overall pipeline on a test set of 179 orders (71% are partially digitized) consisting of 200 digitized medication items. We predict top K (K=3) SKUs for each medicine identified in the prescription image for customer safety and evaluate our approaches on precision@3 (i.e., fraction of predicted being in the ground truth orders) and recall@3 (i.e., fraction of actual ordered medications being detected). Since precision estimate is based on partial orders, it is pessimistic. The baseline method performs fuzzy matching of attributes (e.g., BRAND-NAME, FORM) of catalog items with ngrams from complete prescription text and selects
top K SKUs for each prescribed medicine. Our proposed approach combines the best version of each component from Section 3.2 and gets +5.9% in precision@3 and +5.6% in recall@3 over the baseline. Error diagnosis. Primary gaps in our approach include: (a) Text extraction errors, e.g., capsule extracted as "apsule" resulting in misclassification as form; (b) Limited semantic understanding of the model, e.g., "once a month" denotes Frequency but was predicted as Duration; (c) Token not exclusively associated with a label, e.g., “syrup” is usually Form, but "corn syrup” is an Ingredient, and (d) Minor variations in medication attributes (e.g., "LosarH" vs "LosarCH") which can be handled by including INGREDIENT during matching.

heading: 6 Learnings
text: Below are our key learnings on building document AI systems for low data regime: Annotation design is critical. Annotation tasks (drawing BBs, text chunking) should be wellspecified with low cognitive load and include all the
relevant input (e.g., raw images) to avoid cascading errors. This is especially true for annotations on VRD output from OCR which could itself be erroneous. Building an annotation UI that leverages existing models but allows for manual corrections as part of a semi-automated workflow is an ideal strategy for progressive improvement. Divide and conquer. Despite the ubiquity of endend neural models, it is vital to choose a solution approach based on application constraints, e.g., data limitations, the need for modularity to support phased development and audibility. We adopted a divide-and-conquer approach by partitioning our problem into sub-tasks which could be solved separately using domain knowledge where possible. Our multi-stage solution is extensible and reusable across different workflows and data segments. Model and problem complexity should match. Ideal performance is obtained when complexity of approach matches that of the problem conditioned on available data and domain knowledge. We noticed in our case that richer ML models were comparable or under-performed simpler ML models and domain heuristic-based approaches in medicine chunking and matching tasks due to less data.

heading: 7 Concluding Remarks
text: Prescription digitization is a critical enabler of online pharmacy services. We present a holistic, modular approach to address this problem in a low data regime using hybrid ML and rule-based components. Our approach uses layout signals, medical ontologies, sequential dependencies, and semantic embeddings to yield significant improvement over
baselines and good performance on unstructured printed prescriptions. Ongoing directions include using active learning to judiciously label data (section A.4.2), pseudo labeling of partially digitized orders and digitizing handwritten prescriptions.

heading: 8 Limitations
text: Our prescription digitization approach has a few limitations but is still effective for a broad enough application domain and permits future enhancements that address these limitations. First, our system uses an off-the-shelf text extraction tool (AWS Textract) that provides accurate extractions on printed prescriptions but has variable performance on hand written data depending on the legibility of the handwriting. In future, we plan to build a specialized extraction model trained to recognise medical practitioner’s handwriting to replace AWS TextExtract. Further, multiple components in our approach (e.g., attribute extraction) have been trained on primarily English transcriptions. Extension to other language prescriptions requires access to medical vocabulary and training data in those languages. Note that AWS Textract supports multiple languages and can be readily paired with an automated translator to convert the content to English. We did not consider this option since multilingual prescriptions in India tend to have mixed content with medications written in English itself. Lastly, the performance of multiple tasks such as advice block detection, medication attribute extraction and matching-canonicalization depends on the coverage of the available medical catalog.

heading: 9 Safety and Ethics Statement
text: Our motivation is to improve access and affordability to online pharmaceutical services in emerging markets such as India through accurate and easy digitization of medical prescriptions. Given the sensitive nature of medical prescriptions and the associated health impact, it was critical to pay attention to multiple aspects that we discuss below: Secure and Privacy-safe Data Collection: Privacy of customer data is paramount to us. Hence, prior to modeling, we remove customer, facility and practitioner information by obscuring the regions containing personally identifiable fields such as names, phone numbers, and addresses, which are identified using security-certified AWS services (AWS Comprehend, AWS Textract). Model Bias: A key limitation of the existing medical NER models is their poor performance on nonUS and EU prescriptions due to bias in the training data, which is almost exclusively based on USEU centric medical content and vocabulary. In our approach, we have deliberately chosen to have explicit dependence on aspects that vary across geographical regions (e.g., medical catalog), which enhances the applicability of our approach. To further limit the model bias and minimize distributional differences between training and production settings, we have trained our models on prescription images that are randomly sampled from customer uploads. These often include low resolution and improperly positioned images. In future, as the scope of deployment changes, we plan to periodically retrain the model with training images by sampling from the production data. Health Safety: One of the primary concerns in prescription digitisation is the impact of errors on patient health and adherence to health regulations. To alleviate adverse outcomes, we have multiple guardrails. First, we present the top three suggestions along with scores for each medication for two-fold review by customer and pharmacist. Second, to avoid prescription abuse (e.g., manipulation of quantities, prescription reuse) and comply with regulations, there are additional checks based on the prescription date, patient purchase history, and recommended limits on medication quantities. Usage for a Limited Scope: Our proprietary system has been trained for a specific-use case, i.e., prescription digitization with acceptable performance on primarily English printed prescriptions for India region. We plan to use the model within
this limited scope and expand usage only after adequate benchmarking. To limit the risks of misuse, we do not plan to release this system externally.

heading: 10 Acknowledgements
text: We would like to thank our product team (Sourabh Jeswani, Jayaramakrishnan Balasubramanian) and engineering team (Varnit Agnihotri, Ranjith Sompalli, Saurabh Gupta, Amir Kamal, Vinay Vaidya) for supporting us during problem formulation, data gathering and providing continuous feedback.

heading: A Additional Solution Details
text: A.1 Prescription Schema Refer to Table 5 for prescription schema.
A.2 Annotation Tasks Refer to Table 6 for details on annotation tasks.
A.3 Advice Block Detection Figure 5 shows the reduction in length of prescriptions with advice block detection.
A.4 Medication attribute extraction A.4.1 Training setup and details For training LayoutLMV2 model (our best performing model), PyTorch [23] is used and the pretrained model is taken from open-source Huggingface library. Batch size of 2 and dropout of 0.1 is used
for model training. Learning rate schedule and loss convergence curves are shown in Figure 6. Model architecture for LayoutLMv2 is shown in Figure 7. Adam optimizer is used with exponential decay rates for first moment and second moment estimated as 0.9 and 0.99 respectively.
A.4.2 Efficient Use of Unlabeled Data. Training the model with an increasing number of randomly chosen prescriptions indicated that there is improvement in performance, but at a relatively slow rate. Since labeling effort is much more expensive than acquisition of unlabeled prescriptions, we explored using common active learning methods [26] to prioritize the selection of prescriptions for labeling. Figure 8 shows the learning curves using increasing training data size with selection based on random sampling, entropy of class posteriors, and product of entropy as well as normalized occurrence frequency in the unlabeled data. The results point to potential benefits of judicious prioritization but more exploration is required to optimally combine the entropy and frequency signals.

heading: B Related Work
text: Our work is primarily related to four areas of research that we briefly review below. Document AI is a multi-disciplinary area centered
on understanding visually rich documents (VRDs) using techniques [22, 20] spanning computer vision, layout understanding, natural language understanding, and information retrieval. Document AI techniques that combine Optical Character Recognition (OCR) [4, 2, 1] with graph neural networks [33, 34, 18, 29, 19] have proven to be effective at extracting structured information from documents images, especially for well-formatted printed documents with tables and headers such as invoices. However, these methods perform poorly on documents with uneven layout and handwritten content, such as medical prescriptions. Recent models such as LayoutLM [32, 31] that jointly
learn the layout, visual, and text signal from a large corpus of document images improve performance with uneven layout. Handwritten text recognition (HTR) remains an open challenge despite advances in multi-dimensional RNNs and transformer models [9, 12, 17] due to the variability in author style and limited supervision. Incorporating domain-specific language models is, thus, critical for domain-specific HTR. We combine ideas from OCR, LayoutLM, and domain catalog-based matching to create a tailored solution for our application. Information extraction techniques [29, 21, 13] that deal with conversion of unstructured text to structured form, especially forming blocks of interest comprising lists of multi-attribute records are
directly relevant to our application. These methods primarily use coupled models for segmentation and attribute detection (i.e. entity recognition (ER)), based on conditional random fields in combination with semantic embeddings derived from seq2seq models such as BERT [10], Bi-LSTMs and require extensive labeled data. Since such supervision is limited in our scenario, we decouple segmentation and attribute extraction tasks, using simpler approaches for the former and exploring the SOTA ER techniques while incorporating ideas on exploiting ontologies [30]. Prescription Digitization has seen rising interest in recent years with standardization of health data resources [3, 14]. Most techniques [28, 24, 11, 16], however, fixate on the ER aspects assuming the input is an unstructured text sequence and present results on benchmark datasets [27, 13, 5] of printed clinical documents from Western marketplaces. These models are inadequate for unstructured prescriptions since these do not account for the extraction errors, layout signals, and the gaps in the vocabulary. Therefore, we focus on developing a holistic approach with raw noisy prescriptions as input.

heading: Future Work
text: In future, as the scope of deployment changes, we plan to periodically retrain the model with training images by sampling from the production data. Health Safety: One of the primary concerns in prescription digitisation is the impact of errors on patient health and adherence to health regulations. To alleviate adverse outcomes, we have multiple guardrails. First, we present the top three suggestions along with scores for each medication for two-fold review by customer and pharmacist. Second, to avoid prescription abuse (e. g. , manipulation of quantities, prescription reuse) and comply with regulations, there are additional checks based on the prescription date, patient purchase history, and recommended limits on medication quantities. Usage for a Limited Scope: Our proprietary system has been trained for a specific-use case, i. e. , prescription digitization with acceptable performance on primarily English printed prescriptions for India region. We plan to use the model within
this limited scope and expand usage only after adequate benchmarking. To limit the risks of misuse, we do not plan to release this system externally