Venue: NAACL 2021

# Translational NLP: A New Paradigm and General Principles

for Natural Language Processing Research

Denis Newman-Griffis\({}^{1}\)  Jill Fain Lehman\({}^{2}\)  Carolyn Rose\({}^{3}\)  Harry Hochheiser\({}^{1}\)

\({}^{1}\)Department of Biomedical Informatics, University of Pittsburgh, USA

\({}^{2}\)Human-Computer Interaction Institute, Carnegie Mellon University, USA

\({}^{3}\)Language Technologies Institute, Carnegie Mellon University, USA

{dnewmangriffs, harryn}@pitt.edu, {jfl, cprose}@cs.cmu.edu

###### Abstract

Natural language processing (NLP) research combines the study of universal principles, through basic science, with applied science targeting specific use cases and settings. However, the process of exchange between basic NLP and applications is often assumed to emerge naturally, resulting in many innovations going unapplied and many important questions left unstudied. We describe a new paradigm of _Translational NLP_, which aims to structure and facilitate the processes by which basic and applied NLP research inform one another. Translational NLP thus presents a third research paradigm, focused on understanding the challenges posed by application needs and how these challenges can drive innovation in basic science and technology design. We show that many significant advances in NLP research have emerged from the intersection of basic principles with application needs, and present a conceptual framework outlining the stakeholders and key questions in translational research. Our framework provides a roadmap for developing Translational NLP as a dedicated research area, and identifies general translational principles to facilitate exchange between basic and applied research.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4125–4138 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Natural language processing (NLP) lies at the intersection of basic science and applied technologies. However, translating innovations in basic NLP methods to successful applications remains a difficult task in which failure points often appear late in the development process, delaying or preventing potential impact in research and industry. Application challenges range widely, from changes in data distributions Elsahar and Galle (2019) to computational bottlenecks Desai et al. (2020) and integration with domain expertise Rahman et al. (2020). When unanticipated, such challenges can be fatal to applications of new NLP methodologies, leaving exciting innovations with minimal practical impact. Meanwhile, real-world applications may rely on regular expressions Anzaldi et al. (2017) or unigram frequencies Slater et al. (2017) when more sophisticated methods would yield deeper insight. When successful translations of basic NLP insights into practical applied technologies do occur, the factors contributing to this success are rarely analyzed, limiting our ability to learn how to enable the next project and the next technology.

We argue for a third kind of NLP research, which we call _Translational NLP_. Translational NLP research aims to understand why one translation succeeds while another fails, and to develop general, reusable processes to facilitate more (and easier) translation between basic NLP advances and real-world application settings. Much NLP research already includes translational insights, but often considers them properties of a specific application rather than generalizable findings that can advance the field. This paper illustrates why _general principles_ of the translational process enhance mutual exchange between linguistic inquiry, model development, and application research (illustrated in Figure 1), and are key drivers of NLP advances.

Figure 1: Interactions between linguistic theory, model development, and applications in NLP research. Solid lines indicate moving from basic research to applications, and dashed lines indicate how applied research feeds back into basic study. Translational NLP develops processes to realize this exchange.



We present a conceptual framework for Translational NLP, with specific elements of the translational process that are key to successful applications, each of which presents distinct areas for research. Our framework provides a concrete path for designing use-inspired basic research so that research products can effectively be turned into practical technologies, and provides the tools to understand why a technology translation succeeds or fails. A translational perspective further enables factorizing "grand challenge" research questions into clearly-defined pieces, producing intermediate results and driving new basic research questions. Our paper makes the following contributions:

* We characterize the stakeholders involved in the process of translating basic NLP advances to applications, and identify the roles they play in identifying new research problems (SS3.1).
* We present a general-purpose checklist to use as a starting point for the translational process, to help integrate basic NLP innovations into applications and to identify basic research opportunities arising from application needs (SS3.2).
* We present a case study in the medical domain illustrating how the elements of our Translational NLP framework can lead to new challenges for basic, applied, and translational NLP research (SS4).

## 2 Defining Translational NLP

### A third type of research

A long history of distinguishing between basic and applied research Bush (1945); Shneiderman (2016) has noted that these terms are often relative; one researcher's basic study is the application of another's theory. In practice, basic and applied research in NLP are endpoints of a spectrum, rather than discrete categories. As _use-inspired_ research, most NLP studies incorporate elements of both basic and applied research. We therefore define our key terms for this paper as follows:

**Basic research** Basic NLP research is focused on universal principles: linguistically-motivated study that guides model design (e.g., Recasens and Hovy (2009) for coreference, Kouloumpis et al. (2011) for sentiment analysis), or modeling techniques designed for general use across different settings and genres. Basic research tends to focus on one problem at a time, and frequently leverages established datasets to provide a well-controlled environment for varying model design. Basic NLP research is intended to take the long view: it takes the time to investigate fundamental questions that may yield rewards for years to come.

**Applied research** Applied NLP research studies the intersection of universal principles with specific settings; it is responsive to the needs of commercial applications or researchers in other domains. Applied research utilizes real-world datasets, often specialized, and involves sources of noise and unreliability that complicate capturing linguistic regularities of interest. Applications often involve tackling multiple interrelated problems, and demand complex combinations of tools (e.g. using OCR followed by NLP to analyze scanned documents). Applied research is concrete and immediate, but may also be reactive and have a limited scope.

**Translational research** The term _translational_ is used in medicine to describe research that aims to transform advances in basic knowledge (biological or clinical) to applications to human health Butte (2008); Rubio et al. (2010). Translational research is a distinct discipline bridging basic science and applications Pober et al. (2001); Reis et al. (2010). We adopt the term _Translational NLP_ to describe research bridging the gap between basic and applied NLP research, and aiming to understand the processes by which each informs the other. Section 4 presents one in-depth example; other salient examples include comparing the efficacy of domain adaptation methods for different application domains Naik et al. (2019) and developing reusable software for processing specific text genres Neumann et al. (2019). Translational research occupies a middle ground in the timeframe and complexity of solutions: it develops processes to rapidly and effectively integrate new innovations into applications to address emerging needs, and facilitates integration between pipelines of NLP tools.

### Translation is bidirectional

In addition to "forward" motion of basic innovations into practical applications, the needs of real-world applications also provide significant opportunities for new fundamental research. Shneiderman's model of "two parents, three children Shneiderman (2016) provides an informative picture: combining a practical problem and a theoretical model yields (1) a solution to the problem, 

(2) a refinement of the theory, and (3) guidance for future research. Tight links between basic research and applications have driven many major advances in NLP, from machine translation and dialog systems to search engines and question answering. Designing research with application needs in mind is a key impact criterion for both funding agencies Christianson et al. (2018) and industry Spector et al. (2012), and helps to identify new, high-impact research problems Shneiderman (2018).

### NLP as a translational field: a historical perspective

The NLP field has always lain at the nexus of basic and applied research. Application needs have driven some of the most fundamental developments in the field, leading to explosions in basic research in new topics and on long-standing challenges.

The need to automatically translate Russian scientific papers in the early years of the Cold War led to some of the earliest NLP research, creating the still-thriving field of machine translation Slocum (1985). Machine translation has since helped drive many significant advances in basic NLP research, from the adoption of statistical models in the 1980s Dorr et al. (1999) to neural sequence-to-sequence modeling Sutskever et al. (2014) and attention mechanisms Bahdanau et al. (2015).

Similarly, the rapid growth of the World Wide Web in the 1990s created an acute need for technologies to search the growing sea of information, leading to the development of NLP-based search engines such as Lycos Mauldin (1997), followed by PageRank Page et al. (1999) and the growth of Google. The need to index and monetize vast quantities of textual information led to an explosion in information retrieval research, and the NLP field and ever-growing web data continue to co-develop.

In a more recent example, IBM identified automated question answering (QA) as a new business opportunity in a high-information world, and developed the Watson project Ferrucci et al. (2010). Watson's early successes catapulted QA into the center of NLP research, where it has continued to drive both novel technology development and benchmark evaluation datasets used in hundreds of basic NLP studies Rajpurkar et al. (2016).

These and other examples illustrate the key role that application needs have played in driving innovation in NLP research. This reflects not only the history of the field but the role that integrating basic and applied research has in enriching scientific endeavor Stokes (1997); Branscomb (1999); Narayanamurti et al. (2013); Shneiderman (2016). An integrated approach has been cited by both Google Spector et al. (2012) and IBM McQueeney (2003) as central to their successes in both business and research. The aim of our paper is to facilitate this integration in NLP more broadly, through presenting a rubric for studying and facilitating the process of getting both to and back from application.

### A practical definition

For an operational definition of Translational NLP, it is instructive to consider four phases of a generic workflow for tackling a novel NLP problem using supervised machine learning.1 First, a team of NLP experts works with subject matter experts (SMEs) to identify appropriate corpora, define concepts to be extracted, and construct annotation guidelines for the target task. Second, SMEs use these guidelines to annotate natural language data, using iterative evaluation, revision of guidelines, and re-annotation to converge on a high-quality _gold standard_ set of annotations. Third, NLP experts use these annotations to train and evaluate candidate models of the task, joined with SMEs in a feedback loop to discuss results and needed revisions of goals, guidelines, and gold standards. Finally, buy-in is sought from SMEs and practitioners in the target domain, in a dialogue informed by empirical results and conceptual training. NLP adoption in practice identifies failure cases and new information needs, and the process begins again.

Footnote 1: While workflows will vary for different classes of NLP problems, dialogue between NLP experts and subject matter experts is at the heart of developing almost all NLP solutions.

This laborious process is needed because of the gaps between expertise in NLP technology and expertise in use cases where NLP is applied. NLP expertise is needed to properly formulate problems, and subsequently to develop sound and generalizable solutions to those problems. However, for uptake (and therefore impact) to occur, these solutions must be based in deep expertise in the use case domain, refiled in a computable manner through annotation or knowledge resource development. These distinct forms of expertise are generally found in different groups of individuals with complementary perspectives (see e.g. Kruschwitz and Hull (2017)).

Given this gap, we define **Translational NLP** as the development of theories, tools, and processes to enable the direct application of advanced NLP 

tools in specific use cases. Implementing these tools and processes, and engaging with basic NLP experts and SMEs in their use, is the role of the Translational NLP scientist. Although every use case has unique characteristics, there are shared principles in designing NLP solutions that undergird the whole of the research and application process. These shared translational principles can be adopted by basic researchers to increase the impact of NLP methods innovations, and guide the translational researcher in developing novel efforts targeting fundamental gaps between basic research and applications. The framework presented in this paper identifies common variables and asks specific questions that can drive this research.

For examples of this process in practice, it is valuable to examine NLP development in the medical domain. Use-inspired NLP research has a long history in medicine (Sager et al., 1982; Ranum, 1989), frequently with an eye towards practical applications in research and care. Chapman et al. (2011) highlight shared tasks as a key step towards addressing numerous barriers to application of NLP on clinical notes, including lack of shared datasets, insufficient conventions and standards, limited reproducibility, and lack of user-centered design (all factors presenting basic research opportunities, in addition to NLP task improvement). Several efforts have explored the development of graphical user interfaces for conducting NLP tasks, including creation and execution of pipelines (Cunningham, 2002; D'Avolio et al., 2010, 2011; Soysal et al., 2018), although these efforts generally do not report on evaluation of usability by non-NLP experts. Usability has been investigated by other studies involving more focused tools aimed at specific NLP tasks, including concept searching (Hultman et al., 2018), annotation (Gobbel et al., 2014b,a), and interactive review of and update of text classification models (Trivedi et al., 2018, 2019; Savelka et al., 2015). Recent research has utilized interactive NLP tools for processing cancer research (Deng et al., 2019) and care (Yala et al., 2017) documents. By constructing, designing, and evaluating tools designed to simplify specific NLP processes, these efforts present examples of Translational NLP.

## 3 The Translational NLP framework

We present a conceptual framework for Translational NLP, to formalize shared principles describing how basic and applied research interact to create NLP solutions. Our framework codifies fundamental variables in this process, providing a roadmap for negotiating the design of methodological innovations with an eye towards potential applications. Although it is certainly not the case that every basic research advance must be tied to a downstream application need, designing foundational technologies for potential application from the beginning produces more robust technologies that are easier to transfer to practical settings, increasing the impact of basic research. By defining common variables, our framework also provides a structure for aligning application needs to basic technologies, helping to identify potential failure points and new research needs early for faster adoption of basic NLP advances.

Our framework has two components:

1. A definition of broad classes of stakeholders in translating basic NLP innovations into applications, including the roles that each stakeholder plays in defining and guiding research;
2. A checklist of fundamental questions to structure the Translational NLP process, and to guide identification of basic research opportunities in specific application cases.

### Stakeholders

NLP applications involve three broad categories of stakeholders, illustrated in Figure 2. Each contributes differently to technology implementation and identifying new research challenges.

**NLP Experts NLP researchers bring key analytic skills to enable achieving the goals of an applied system. NLP experts provide _methodological sophistication_ in models and paradigms for analyzing language, and an understanding of the nature of language and how it captures information. NLP researchers provide much-needed _data expertise_, including skills in obtaining, cleaning, and formatting data for machine learning and evaluation, as well as _conceptual models for representing information needs_. NLP scientists identify research opportunities in modeling information needs, bringing linguistic knowledge into the equation, and developing appropriate tools for application and reuse.

**Subject Matter Experts** Subject matter experts (SMEs) provide the context that helps to determine what information is important to analyze and what the outputs of applied NLP systems mean for the application setting. SMEs, from medical practitioners to legal scholars and financial experts, bring 

an understanding of _where relevant information can be found_ (e.g., document sources Fisher et al. (2016) and sections Afzal et al. (2018)), which can help identify new types of language for basic researchers to study Burstein (2009); Crossley et al. (2014) and new challenges such as sparse complex information Newman-Griffis and Fosler-Lussier (2019) and higher-level structure in complex documents Naik et al. (2019). In addition, the context that domain experts offer in terms of the needs of target applications feeds back into _evaluation_ methods in the basic research setting Graham (2015).

SMEs are also the consumers of NLP solutions, as tools for their own research and applications. Thus, SMEs must also be consultants regarding the trustworthiness and reliability of proposed solutions, and can identify key application-specific concerns such as security requirements.

**End Users** The end users of NLP solutions span a range of roles, environmental contexts, and goals, each of which guides implementation factors of NLP applications. For example, collecting patient language in a lab setting, in a clinic, or at home will pose different challenges in each setting, which can inform the development of basic NLP methods. Application settings may have limited _computational resources_, motivating the development of efficient alternatives to high-resource models (e.g. Wang et al. (2020)), and have different _human factors_ affecting information collection and use.

End users have different constraints on _data availability_, in terms of how much data of what types can be obtained from whom; the extensive work funded by DARPA's Low Resource Languages for Emergent Incidents (LORELEI) initiative Christianson et al. (2018) is a testament to the basic research arising from these constraints.

Beyond the individual domain expert, end users use NLP technologies to address their own information needs according to the _priorities_ of their organizations. These organizational priorities may conflict with existing modeling assumptions, highlighting new opportunities for basic research to expand model capabilities. For example, Shah et al. (2019) highlight the conceptual gap between predictive model performance in medicine and clinical utility to call for new research on utility-driven model evaluation. Spector et al. (2012) make a similar point about Google's mission-driven research identifying unseen gaps for new basic research.

The role of the **Translational NLP** researcher is to interface with each of these stakeholders, to connect their goals, constraints, and contributions into a single applied system, and to identify new research opportunities where parts of this system conflict with one another. Notably, this creates an opportunity for valuable study of SME and end user research practices, and for participatory design of NLP research Lazar et al. (2017). Our checklist, introduced in the next section, provides a structured framework for this translational process.

### Translational NLP Checklist

The path between basic research and applications is often nebulous in NLP, limiting the downstream impact of modeling innovations and obscuring basic research challenges found in application settings. We present a general-purpose checklist covering fundamental variables in translating basic research into applications, which breaks down the translational process into discrete pieces for negotiation, measurement, and identification of new research opportunities. Our checklist, illustrated in Figure 3, is loosely ordered from initial design to application details. In practice, these items reflect different elements of the application process and are constantly

Figure 2: Attributes of key stakeholders in the translational process for NLP.

 

re-evaluated via a feedback loop between the application stakeholders. While many of these items will be familiar to NLP researchers, each represents potential points of failure in translation. Designing the research process with these variables in mind will produce basic innovations that are more easily adopted for application and more directly connected to the challenges of real-world use cases.

We illustrate our items for two example cases:

_Ex. 1:_ Analysis of multimodal clinical data (scanned text, tables, images) for patient diagnosis.

_Ex. 2:_ Comparison of medical observations to government treatment and billing guidelines.

**Information Need** The initial step that guides an application is defining inputs and outputs, at two levels: (1) the _overall problem_ to address with NLP (led by the subject matter expert), and (2) the _formal representation_ of that problem (led by the NLP expert). The overall goal (e.g., "extract information on cancer from clinical notes") determines the requirements of the solution, and is central to identifying a measurement of its effectiveness. Once the overall goal is determined, the next step is a formal representation of that goal in terms of text units (documents, spans) to analyze and what the analysis should produce (class labels, sequence annotations, document rankings, etc.). These requirements are tailored to specific applications and may not reflect standardized NLP tasks. For example, a clinician interested in the documented reasoning behind a series of laboratory test orders needs: (1) the orders themselves (text spans); (2) the temporal sequence of the orders; and (3) a text span containing the justification for each order.

_Ex. 1:_ type, severity, history of symptoms.

_Ex. 2:_ clinical findings, logical criteria.

**Data Characteristics** A clear description of the _language data to be analyzed_ is key to identifying appropriate NLP technologies. Data characteristics include the natural language(s) used (e.g., English, Chinese), the genre(s) of language to analyze (e.g., scientific abstracts, quarterly earnings reports, tweets, conversations), and the type(s) of linguistic community that produced them (e.g., medical practitioners, educators, policy experts). This information identifies the sublanguage(s) of interest Grishman and Kittredge (1986), which determine the availability and development of appropriate NLP tools Grishman (2001). Corporate disclosures, financial news reports, and tweets all require different processing strategies Xing et al. (2018), as do tweets written by different communities Blodgett et al. (2016); Groenwold et al. (2020).

_Ex. 1:_ clinical texts, lab reports.

_Ex. 2:_ clinical texts, legal guidelines.

**Task Paradigms** To address the overall goal with an NLP solution, it must be formulated in terms of one or more _well-defined NLP problems_. Many real-world application needs do not clearly correspond to a single benchmark task formulation. For example, our earlier example of the sequence of lab order justifications can be formulated as a sequence of: (1) Named Entity Recognition (treating the order types as named entities in a medical knowledge base); (2) time expression extraction and normalization; (3) event ordering; and (4) evidence identification. Breaking the application need into well-studied subproblems at design time enables faster identification and development of relevant NLP technologies, and highlights any portions of the goal that do _not_ correspond with a known problem, requiring novel basic research.

_Ex. 1:_ document type classification, OCR, information extraction (IE), patient classification.

_Ex. 2:_ IE, natural language inference.

**Available Resources** The question of resources to support an NLP solution includes two distinct concerns: (1) _knowledge sources_ available to represent salient aspects of the target task; and (2) _compute infrastructure_ for NLP system execution

Figure 3: The eight items in our Translational NLP checklist, with key questions for each. Items are loosely ordered from initial design to application details, but should be regularly revisited in a feedback loop between application stakeholders.

 

and deployment. Knowledge sources may be symbolic, such as knowledge graphs or gazetteers, or representational, such as representative corpora or pretrained language models. For some applications, powerful knowledge sources may be available (such as the UMLS Bodenreider (2004) for biomedical reasoning), while others are severely under-resourced (such as emerging geopolitical events, which may lack even relevant social media text). These resources in turn affect the kinds of technologies that are appropriate to use.

In terms of infrastructure, NLP technologies are deployed on a wide variety of systems, from commercial data centers to mobile devices. Each setting presents constraints of _limited resources_ and _throughput requirements_Nityasya et al. (2020). An application environment with a high maximum resource load but low median availability is amenable to batch processing architectures or approaches with high pretraining cost and low test-time cost. Pretrained word represenstions Mikolov et al. (2013); Pennington et al. (2014) and language models Peters et al. (2018); Devlin et al. (2019) are one example of fundamental technologies that address such a need. Throughput requirements, i.e., how much language input needs to be analyzed in a fixed amount of time, often require engineering optimization for specific environments Afshar et al. (2019), but the need for faster runtime computation has led to many advances in machine learning for NLP, such as variational autoencoders Kingma and Welling (2014) and the Transformer architecture Vaswani et al. (2017).

_Ex. 1:_ UMLS, high GPU compute.

_Ex. 2:_ UMLS, guideline criteria, low compute.

**NLP Technologies** The interaction between task paradigms, data characteristics, and available resources helps to determine what types of _implementations_ are appropriate to the task. Implementations can be further broken down into _representation technologies_, for mathematically representing the language units to be analyzed; _modeling architectures_, for capturing regularities within that language; and _optimization strategies_ (when using machine learning), for efficiently estimating model parameters from data. In low-resource settings, highly parameterized models such as BERT may not be appropriate, while large-scale GPU server farms enable highly complex model architectures. When the overall goal is factorized into multiple NLP tasks, optimization often involves joint or multi-task learning Caruana (1997).

_Ex. 1:_ large language models, dictionary matching, OCR, multi-task learning.

_Ex. 2:_ dictionary matching, small neural models.

**Evaluation** Once a solution has been designed, it must be evaluated in terms of both _the specific NLP problem(s)_ and _the overall goal_ of the application. Standardized NLP task formulations typically define benchmark metrics which can be used for evaluating the NLP components: F-1 and AUC for information extraction, MRR and NDCG for information retrieval, etc. The design of these metrics is its own extensive area of research Jones and Galliers (1996); Hirschman and Thompson (1997); Graham (2015), and even established evaluation methods may be constantly revised Grishman and Sundheim (1995). Critically for the translational researcher, some metrics may be preferred over others (e.g., precision over recall), and standardized evaluation metrics may not reflect the goals and needs of applications Friedman and Hripcsak (1998). Improvements on standardized evaluation metrics (such as increased AUC) may even obscure degradations in application-relevant performance measures (such as decreased process efficiency). Translational researchers thus have the opportunity to work with NLP experts and SMEs to identify or develop metrics that capture both the effectiveness of the NLP system and its contribution to the application's overall goal.

_Ex. 1:_ F-1, patient outcomes.

_Ex. 2:_ F-1, billing rates.

**Interpretation** Interpretability and analysis of NLP and other machine learning systems has been the focus of extensive research in recent years Gilpin et al. (2018); Belinkov and Glass (2019), with debate over what constitutes an interpretation Rudin (2019); Wiegreffe and Pinter (2019) and development of broad-coverage software packages for ease of use Nori et al. (2019). For the translational researcher, the first step is to engage with SMEs to determine _what constitutes an acceptable interpretation_ of an NLP system's output in the application domain (which may be subject to specific legal or ethical requirements around accountability in decision-making processes). This leads to an iterative process, working with SMEs and NLP experts to identify appropriately interpretable models, or to identify the need for new basic research on interpretability within the target domain.

_Ex. 1:_ Evidence identification, model audits 

_Ex. 2:_ Criteria visualization, model audits.

**Application Engineering** Last but not least, the translational process must also be concerned with the _implementation_ of NLP solutions, both in terms of the specific technologies used and how they can fit in to broader information processing pipelines. The development of general-purpose NLP architectures such as the Stanford CoreNLP Toolkit Manning et al. (2014), spaCy Honnibal and Montani (2017), and AllenNLP Gardner et al. (2018), as well as more targeted architectures such as the clinical NLP framework presented by Wen et al. (2019), provide well-engineered frameworks for implementing new technologies in a way that is easy for others to both adopt and adapt for use in their own pipelines. Standardized data exchange frameworks such as UIMA Ferrucci and Lally (2004) and JSON make implementations more modular and easier to wire together. Leveraging tools and frameworks like these, together with good software design principles, makes NLP tools both easier to apply downstream and easier for other researchers to incorporate into their own work.

_Ex. 1:_ Multiple interoperable technologies.

_Ex. 2:_ Single decision support tool.

#### 3.2.1 Translating methodology advances into existing applications

While the checklist items can guide initial design of a new NLP solution, they are equally applicable for incorporating new basic NLP innovations into existing solutions. Any new innovation can be reviewed in terms of our checklist items to identify new requirements or constraints (e.g., higher computational cost, more intuitive interpretability measures). The translational researcher can then work with NLP experts, SMEs, and the end users to determine how to incorporate the new innovation into the existing solution.

## 4 Case Study: NLP for Disability Review

We illustrate our Translational NLP framework using our recent line of research on developing NLP tools to assist US Social Security Administration (SSA) officials in reviewing applications for disability benefits Desmet et al. (2020). The goal of this effort was to help identify relevant pieces of medical evidence for making decisions about disability benefits, analyzing vast quantities of medical records collected during the review process.

The stakeholders in this setting included: NLP researchers (interested in developing generalizable methods); subject matter experts in disability and rehabilitation; and SSA end users (limited computing, large data but strictly controlled, overall priorities of efficiency and accuracy).

The Translational NLP checklist for this setting is shown in Table 1. This combination of factors has led to several translational studies, including:

* Newman-Griffis et al. (2018) developed a low-resource entity embedding method for domains with minimal knowledge sources (lack of Available Resources).
* Newman-Griffis and Zirikly (2018) analyzed the data size and representativeness tradeoff for information extraction in domains lacking large corpora (Available Resources).
* Newman-Griffis and Fosler-Lussier (2019) developed a flexible method for identifying sparse health information that is syntactically complex (challenging Data Characteristics).
* Newman-Griffis and Fosler-Lussier (2021) compared the Task Paradigms of classification and candidate selection paradigms for medical coding in a new domain.

While these studies do not systematically explore Evaluation, Interpretation, or Application Engineering, they illustrate how the characteristics of one application setting can lead to a line of Translational NLP research with broader implications. Several further challenges of this application area remain unstudied: for example, representing and modeling the complex timelines of persons with chronic health conditions and intermittent health care and adapting NLP systems to highly variable medical language from practitioners and patients around the US. These present intriguing challenges for basic NLP research that can inform many other applications beyond this case study.

Of course, these studies are far from the only examples of Translational NLP research. Many studies tackle translational questions, from domain adaptation (shifts in Data Characteristics) and low-resource learning (limited Available Resources), and the growing NLP literature in domain-specific venues such as medical research, law, finance, and more involves all aspects of the translational process. Rather, this case study is simply one illustration of how an explicitly translational perspective in study design can identify and connect broad opportunities for contributions to NLP research 



## 5 Discussion

Our paradigm of Translational NLP defines and gives structure to a valuable area of research not explicitly represented in the ACL community. We note that translational research is not meant to replace either basic or applied research, nor do we intend to say that all basic NLP studies must be tied to specific application needs. Rather we aim to highlight the value of studying the _processes_ of turning basic innovations into successful applications. These processes, from scaling model computation to redesigning tools to meet changing application needs, can inform new research in model design, domain adaptation, etc., and can help us understand why some tools succeed in application while others fail. In addition to helping more innovations successfully translate, the principles outlined in this paper can be of use to basic and applied NLP researchers as well as translational ones, in identifying common variables and concerns to connect new work to the broader community.

Translational research is equally at home in industry and academia, and already occurring in both. While resource disparities between industrial and academic research increasingly push large-scale modeling efforts out of reach of academic teams, a translational lens can help to identify rich areas of knowledge-driven study that do not require exascale data or computing resources. The general principles and interdisciplinary nature of translational research make it a natural fit for public knowledge-driven academic settings, while its applicability to commercial needs is highly relevant to industry.

Our framework provides a starting point for the translational process, which will evolve differently for every project. The specifics of different applications will expand our initial questions in different ways (e.g., "Data Characteristics" may involve multimodal data, or different language styles), and the dynamics of collaborations will shift answers over time (e.g., a change in evaluation criteria may motivate different model training approaches). Our checklist provides a minimal set of common questions, and can function as a touchstone for discussions throughout the research process, but it can and should be tailored to the nature of each project. Our framework is itself a preliminary characterization of Translational NLP research, and will evolve over time as the field continues to develop.

## 6 Conclusion

We have outlined a new model of NLP research, Translational NLP, which aims to bridge the gap between basic and applied NLP research with generalizable principles, tools, and processes. We identified key types of stakeholders in NLP applications and how they inform the translational process, and presented a checklist of common variables and translational principles to consider in basic, translational, or applied NLP research. The translational framework reflects the central role that integrating basic and applied research has played in the development of the NLP field, and is illustrated by both the broad successes of machine translation, speech processing, and web search, as well as many individual studies in the ACL community and beyond.

### Acknowledgments

This work was supported by the National Library of Medicine of the National Institutes of Health

\begin{table}
\begin{tabular}{l l} \hline \hline Information Need & _Overall goal:_ Improve disability benefits review process by highlighting relevant information \\  & _Formal representation:_ Spans of evidence, with attributes for activity type and level of limitation \\ \hline Data Characteristics & Medical records and administrative forms from USA, mostly English \\ \hline Task Paradigms & Information extraction (spans), information retrieval (documents), span classification (activity and limitations) \\ \hline Available Resources & Minimal knowledge sources for function and disability, no public corpora; US government computing systems; high throughput requirements (thousands of records/day) \\ \hline NLP Technologies & Low-latency, low-compute sequence models; rule-based systems \\ \hline Evaluation & Standard metrics (F-1, accuracy). Information retrieval metrics reported for use case prototypes. \\ \hline Interpretation & Interpretation needs primarily around human decision-making; NLP tools highlight and organize information in context. No ML interpretability reported in published results. \\ \hline Application Engineering & Open-source implementations using standardized frameworks for preprocessing. No data exchange reported. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Translational NLP checklist items for Disability Review case study, including notes on published results.



under award number T15 LM007059, and National Science Foundation grant 1822831.

## References

* M. Afshar, D. Diligach, B. Sharma, X. Cai, J. Boyda, S. Birch, D. Valdez, S. Zelisko, C. Joyce, F. Modave, and R. Price (2019)Development and application of a high throughput natural language processing architecture to convert all clinical documents in a clinical data warehouse into standardized medical vocabularies. Journal of the American Medical Informatics Association26 (11), pp. 1364-1369. Cited by: SS1.
* N. Afzal, V. P. Mallipeddi, S. Sohn, H. Liu, R. Chaudhry, C. G. Scott, I. J. Kullo, and A. M. Arruda-Olson (2018)Natural language processing of clinical notes for identification of critical limb ischemia.. International Journal of Medical Informatics111, pp. 83-89. Cited by: SS1.
* L. J. Anzaldi, A. Davison, C. M. Boyd, B. Leff, and H. Kharrazi (2017)Comparing clinician descriptions of frailty and geriatric syndromes using electronic health records: a retrospective cohort study. BMCGeriatrics17 (1), pp. 248. External Links: Document Cited by: SS1.
* D. Bahdanau, K. Cho, and Y. Bengio (2015)Neural machine translation by jointly learning to align and translate. In ICLR 2015, Cited by: SS1.
* Y. Belinkov and J. Glass (2019)Analysis methods in neural language processing: a survey. Transactions of the Association for Computational Linguistics7, pp. 49-72. Cited by: SS1.
* S. E. Blodgett, L. Green, and B. O'Connor (2016)Demographic dialectal variation in social media: a case study of African-American English. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 1119-1130. External Links: Link, Document Cited by: SS1.
* O. Bodenreider (2004)The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Research32 (90001), pp. D267-D270. Cited by: SS1.
* L. M. Branscomb (1999)The false dichotomy: scientific creativity and utility. Issues in Science and Technology16 (1), pp. 66-72. Cited by: SS1.
* J. Burstein (2009)Opportunities for natural language processing research in education. In Computational Linguistics and Intelligent Text Processing, GICLing 2009, Berlin, Heidelberg, pp. 6-27. Cited by: SS1.
* V. Bush (1945)Science, The endless frontier. U.S. Government Printing Office. Cited by: SS1.
* A. J. Butte (2008)Translational Bioinformatics: coming of age. Journal of the American Medical Informatics Association15 (6), pp. 709-714. Cited by: SS1.
* R. Caruana (1997)Multitask learning. Machine learning28 (1), pp. 41-75. Cited by: SS1.
* W. W. Chapman, P. M. Nadkarni, L. Hirschman, L. W. D'Avolio, G. K. Savova, and O. Uzuner (2011)Overcoming barriers to NLP for clinical text: the role of shared tasks and the need for additional creative solutions. Journal of the American Medical Information Association18 (5), pp. 540-543. Cited by: SS1.
* C. Christianson, J. Duncan, and B. Onyshkevych (2018)Overview of the DARPA LORELEI program. Machine Translation32 (1), pp. 3-9. Cited by: SS1.
* S. A. Crossley, L. K. Allen, K. Kyle, and D. S. McNamara (2014)Analyzing discourse processing using a simple natural language processing tool. Discourse Processes51 (5-6), pp. 511-534. Cited by: SS1.
* H. Cunningham (2002)GATE, a general architecture for text engineering. Computers and the Humanities36 (2), pp. 223-254. Cited by: SS1.
* Leonard W. D'Avolio, Thien M. Nguyen, W. R. Farwell, Y. Chen, F. Fitzmeyer, O. M. Harris, and L. D. Fiore (2010)Evaluation of a generalizable approach to clinical information retrieval using the automated retrieval console (ARC). Journal of the American Medical Informatics Association17 (4), pp. 375-382. Cited by: SS1.
* L. W. D'Avolio, T. M. Nguyen, S. Goryachev, and L. D. Fiore (2011)Automated concept-level information extraction to reduce the need for custom software and rules development. Journal of the American Medical Informatics Association18 (5), pp. 607-613. Cited by: SS1.
* Z. Deng, K. Yin, Y. Bao, V. Diego Armengol, C. Wang, A. Tiwari, R. Barzilay, G. Parmigiani, D. Braun, and K. S. Hughes (2019)Validation of a semiautomated natural language processing-based procedure for meta-analysis of cancer susceptibility gene pentemature. JCO Clinical Cancer Informatics (3), pp. 1-9. Cited by: SS1.
* S. Desai, G. Goh, A. Babu, and A. Ally (2020)Lightweight convolutional representations for on-device natural language processing. arXiv preprint arXiv:2002.01535. Cited by: SS1.
* B. Desmet, J. Porcino, A. Zirikly, D. Newman-Griffis, G. Divita, and E. Rasch (2020)Development of natural language processing tools to support determination of federal disability benefits in the U.S. In Proceedings of the 1st Workshop on Language Technologies for Government and Public Administration (LT4Gov), Marseille, France, pp. 1-6. Cited by: SS1.

 

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dorr et al. (1999) Bonnie J Dorr, Pamela W Jordan, and John W Benoit. 1999. A Survey of Current Paradigms in Machine Translation. volume 49, pages 1-68. Elsevier.
* Elsahar and Galle (2019) Hady Elsahar and Matthias Galle. 2019. To annotate or not? predicting performance drop under domain shift. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2163-2173, Hong Kong, China. Association for Computational Linguistics.
* Ferrucci et al. (2010) David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, and Others. 2010. Building Watson: An overview of the DeepQA project. _AI magazine_, 31(3):59-79.
* Ferrucci and Lally (2004) David Ferrucci and Adam Lally. 2004. UIMA: an architectural approach to unstructured information processing in the corporate research environment. _Natural Language Engineering_, 10:1-26.
* Fisher et al. (2016) Ingrid E Fisher, Margaret R Garnsey, and Mark E Hughes. 2016. Natural Language Processing in Accounting, Auditing and Finance: A Synthesis of the Literature with a Roadmap for Future Research. _Intelligent Systems in Accounting, Finance and Management_, 23(3):157-214.
* Friedman and Hripcsak (1998) C Friedman and G Hripcsak. 1998. Evaluating natural language processors in the clinical domain. _Methods of information in medicine_, 37(4-5):334.
* Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_, pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
* Gilpin et al. (2018) L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In _2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)_, pages 80-89.
* Gobbel et al. (2019) Glenn T. Gobbel, Jennifer Garvin, Ruth Reeves, Robert M. Cronin, Julia Heavirland, Jenifer Williams, Allison Weaver, Shrimalini Jayaramaraja, Dario Giuse, Theodore Speroff, Steven H. Brown, Hua Xu, and Michael E. Matheny. 2014a. Assisted annotation of medical free text using RapTAT. _Journal of the American Medical Informatics Association_, 21(5):833-841.
* Gobbel et al. (2014b) Glenn T. Gobbel, Ruth Reeves, Shrimalini Jayaramaraja, Dario Giuse, Theodore Speroff, Steven H. Brown, Peter L. Elkin, and Michael E. Matheny. 2014b. Development and evaluation of RapTAT: a machine learning system for concept mapping of phrases from medical narratives. _Journal of Biomedical Informatics_, 48:54-65.
* Graham (2015) Yvette Graham. 2015. Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 128-137, Lisbon, Portugal. Association for Computational Linguistics.
* Grishman (2001) Ralph Grishman. 2001. Adaptive information extraction and sublanguage analysis. In _Proceedings of the Workshop on Adaptive Text Extraction and Mining, Seventeenth International Joint Conference on Artificial Intelligence (IJCAI-2001)_, pages 1-4, Seattle, Washington, USA.
* Grishman and Kittregde (1986) Ralph Grishman and R Kittregde. 1986. _Analyzing language in restricted domains: Sublanguage description and processing_. Lawrence Erlbaum Associates.
* Grishman and Sundheim (1995) Ralph Grishman and Beth Sundheim. 1995. Design of the MUC-6 evaluation. In _Proceedings of the 6th Conference on Message Understanding_, pages 1-11. Association for Computational Linguistics.
* Groenwold et al. (2020) Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating African-American Vernacular English in Transformer-Based Text Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5877-5883, Online. Association for Computational Linguistics.
* Hirschman and Thompson (1997) Lynette Hirschman and Henry S. Thompson. 1997. Overview of evaluation in speech and natural language processing. In Ronald A. Cole, editor, _Survey of the State of the Art in Human Language Technology_. Cambridge University Press.
* Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing.
* Hultman et al. (2018) Gretchen Hultman, Reed McEwan, Serguei Pakhomov, Elizabeth Lindemann, Steven Skube, and Genevieve B. Melton. 2018. Usability Evaluation of an Unstructured Clinical Document Query Tool for Researchers. _AMIIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science_, 2017:84-93.

 

* Jones and Galliers (1996) Karen Sparck Jones and Julia R. Galliers. 1996. _Evaluating Natural Language Processing Systems: An Analysis and Review_. Springer-Verlag, Berlin, Heidelberg.
* Kingma and Welling (2014) Diederik P Kingma and Max Welling. 2014. Autonecoding variational bayes. _ICLR 2014_.
* Kouloumpis et al. (2011) Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The good the bad and the omg! In _Fifth International AAAI conference on weblogs and social media_. Citeseer.
* Kruschwitz and Hull (2017) Udo Kruschwitz and Charlie Hull. 2017. Searching the Enterprise. _Foundations and Trends in Information Retrieval_, 11(1):18.
* Lazar et al. (2017) Jonathan Lazar, Jinjuan Heidi Feng, and Harry Hochheiser. 2017. _Research methods in human-computer interaction_. Morgan Kaufmann. Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
* Mauldin (1997) M I Mauldin. 1997. Lycos: design choices in an Internet search service. _IEEE Expert_, 12(1):8-11.
* McQueeney (2003) David F McQueeney. 2003. IBM's evolving research strategy. _Research-Technology Management_, 46(4):20-27.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. _arXiv preprint arXiv:1301.3781_, pages 1-12.
* Naik et al. (2019) Aakanksha Naik, Luke Breitfeller, and Carolyn Rose. 2019. TDDiscourse: A Dataset for Discourse-Level Temporal Ordering of Events. In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_, pages 239-249, Stockholm, Sweden. Association for Computational Linguistics.
* Narayanamurti et al. (2013) Venkatesh Narayanamurti, Tolu Odumsou, and Lee Vinsel. 2013. RIP: The basic/applied research dichotomy. _Issues in Science and Technology_, 29(2):31-36.
* Neumann et al. (2019) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019. ScispaCy: Fast and robust models for biomedical natural language processing. In _Proceedings of the 18th BioNLP Workshop and Shared Task_, pages 319-327, Florence, Italy. Association for Computational Linguistics.
* Newman-Griffis and Fosler-Lussier (2019) Denis Newman-Griffis and Eric Fosler-Lussier. 2019. HARE: a Flexible Highlighting Annotator for Ranking and Exploration. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_, pages 85-90, Hong Kong, China. Association for Computational Linguistics.
* Newman-Griffis and Fosler-Lussier (2021) Denis Newman-Griffis and Eric Fosler-Lussier. 2021. Automated Coding of Under-Studied Medical Concept Domains: Linking Physical Activity Reports to the International Classification of Functioning, Disability, and Health. _Frontiers in Digital Health_, 3:620828.
* Newman-Griffis et al. (2018) Denis Newman-Griffis, Albert M Lai, and Eric Fosler-Lussier. 2018. Jointly embedding entities and text with distant supervision. In _Proceedings of The Third Workshop on Representation Learning for NLP_, pages 195-206, Melbourne, Australia. Association for Computational Linguistics.
* Newman-Griffis and Zirikly (2018) Denis Newman-Griffis and Ayah Zirikly. 2018. Embedding transfer for low-resource medical named entity recognition: A case study on patient mobility. In _Proceedings of the BioNLP 2018 workshop_, pages 1-11, Melbourne, Australia. Association for Computational Linguistics.
* Nityasya et al. (2020) Made Nindyatama Nityasaya, Haryo Akbarianto Wibowo, Radityo Eko Prasojojo, and Alham Fikri Aji. 2020. No Budget? Don't Flex! Cost Consideration when Planning to Adopt NLP for Your Business. _arXiv preprint arXiv:2012.08958_.
* Nori et al. (2019) Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Carunana. 2019. InterpretML: A unified framework for machine learning interpretability. _arXiv preprint arXiv:1909.09223_.
* Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank citation ranking: Bringing order to the web. Technical report, Stanford InfoLab.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Pober et al. (2001) Jordan S Pober, Crystal S Neuhauser, and Jeremy M Pober. 2001. Obstacles facing translational research in academic medical centers. _The FASEB Journal_, 15(13):2303-2313.
* Rahman et al. (2020) Protiva Rahman, Arnab Nandi, and Courtney Hebert. 2020. Amplifying domain expertise in clinical data pipelines. _JMIR Med Inform_, 8(11):e19612 

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392.
* Ranum (1989) David L Ranum. 1989. Knowledge-based understanding of radiology text. _Computer methods and programs in biomedicine_, 30(2-3):209-215.
* Recasens and Hovy (2009) Marta Recasens and Eduard Hovy. 2009. A deeper look into features for coreference resolution. In _Proceedings of the 7th Discourse Anaphora and Anaphor Resolution Colloquium on Anaphora Processing and Applications_, DAARC '09, page 29-42, Berlin, Heidelberg. Springer-Verlag.
* Reis et al. (2010) Steven E Reis, Lars Berglund, Gordon R Bernard, Robert M Califf, Garret A Fitzgerald, Peter C Johnson, National Clinical Consortium, and Translational Science Awards. 2010. Regengineering the national clinical and translational research enterprise: the strategic plan of the National Clinical and Translational Science Awards Consortium. _Academic Medicine_, 85(3):463-469.
* Rubio et al. (2010) Doris McGartland Rubio, Ellie E Schoenbaum, Linda S Lee, David E Schteingart, Paul R Marantz, Karl E Anderson, Lauren Dewey Platt, Adriana Baez, and Karim Esposito. 2010. Defining translational research: implications for training. _Academic Medicine_, 85(3):470-475.
* Rudin (2019) Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. _Nature Machine Intelligence_, 1(5):206-215.
* Sager et al. (1982) N Sager, IDJ Bross, G Story, P Bastedo, E Marsh, and D Shedd. 1982. Automatic encoding of clinical narrative. _Computers in Biology and Medicine_, 12(1):43-56.
* the 28th International Conference on Legal Knowledge and Information Systems_.
* Shah et al. (2019) Nigam H Shah, Arnold Milstein, and PhD Bagley Steven C. 2019. Making Machine Learning Models Clinically Useful. _JAMA_, 322(14):1351-1352.
* Shneiderman (2016) Ben Shneiderman. 2016. _The new ABCs of research: Achieving breakthrough collaborations_. Oxford University Press.
* Shneiderman (2018) Ben Shneiderman. 2018. Twin-Win Model: A human-centered approach to research success. _Proceedings of the National Academy of Sciences of the United States of America_, 115(50):12590-12594.
* Slater et al. (2017) Stefan Slater, Srecko Joksimovic, Vitomir Kovanovic, Ryan S Baker, and Dragan Gasevic. 2017. Tools for educational data mining: A review. _Journal of Educational and Behavioral Statistics_, 42(1):85-106.
* Slocum (1985) Jonathan Slocum. 1985. A Survey of Machine Translation: Its History, Current Status, and Future Prospects. _Comput. Linguist._, 11(1):1-17.
* a toolkit for efficiently building customized clinical natural language processing pipelines. _Journal of the American Medical Informatics Association_, 25(3):331-336. Publisher: Oxford Academic.
* Spector et al. (2012) Alfred Spector, Peter Norvig, and Slav Petrov. 2012. Google's hybrid approach to research. _Communications of the ACM_, 55(7):34-37.
* Stokes (1997) Donald E Stokes. 1997. _Pasteur's quadrant: Basic science and technological innovation_. Brookings Institution Press.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Trivedi et al. (2019) Gaurav Trivedi, Esmaeel R Dadashzadeh, Robert M Handzel, Wendy W Chapman, Shyam Visweswaran, and Harry Hochtheiser. 2019. Interactive NLP in Clinical Care: Identifying Incidental Findings in Radiology Reports. _Appl Clin Inform_, 10(04):655-669.
* Trivedi et al. (2018) Gaurav Trivedi, Phuong Pham, Wendy W. Chapman, Rebecca Hwa, Jayce Wiebe, and Harry Hochheiser. 2018. NLPReViz: an interactive tool for natural language processing on clinical text. _Journal of the American Medical Informatics Association_, 25(1):81-87.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Wang et al. (2020) Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020. Hat: Hardware-aware transformers for efficient natural language processing. _arXiv preprint arXiv:2005.14187_.
* Wen et al. (2019) Andrew Wen, Sunyang Fu, Sungrim Moon, Mohamed El Wazir, Andrew Rosenbaum, Vinod C Kaggal, Sijia Liu, Sunghwan Sohn, Hongfang Liu, and Jungwei Fan. 2019. Desiderata for delivering nlp to accelerate healthcare ai advancement and a mayo clinic nlp-as-a-service implementation. _NPJ Digital Medicine_, 2(1):1-7.
* Wiegreffe and Winter (2019) Sarah Wiegreffe and Yuval Winter. 2019. Attention is not not explanation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 11-20, Hong Kong, China. Association for Computational Linguistics.

 

* Xing et al. (2018) Frank Z Xing, Erik Cambria, and Roy E Welsch. 2018. Natural language based financial forecasting: a survey. _Artificial Intelligence Review_, 50(1):49-73.
* Yala et al. (2017) Adam Yala, Regina Barzilay, Laura Salama, Molly Griffin, Grace Sollender, Aditya Bardia, Constance Lehman, Julliette M Buckley, Suzanne B Coopey, Fernanda Polubriaginof, et al. 2017. Using machine learning to parse breast pathology reports. _Breast Cancer Research and Treatment_, 161(2):203-211.