Title: COST-EFFECTIVE INTERACTIVE NEURAL ATTENTION LEARNING

Abstract: We propose a novel interactive attention learning framework which we refer to as Interactive Attention Learning (IAL), in which the human annotators interactively manipulate the allocated attentions to correct the model's behavior, by updating only the attention-generating model without having to retrain the entire network. For efficient update of the attention generator without retraining, we propose a novel attention mechanism, Neural Attention Process (NAP), which can generate stochastic attentions based on scarce attention-level labels, and can incorporate new training instances without retraining. Further, to minimize human interaction cost, we propose a cost-effective algorithm that selects the most negative training instances that yield incorrect and non-intuitive interpretation with influence function and re-rank the attentions on the input features by their uncertainties, such that the annotators label the instances and attentions that are more influential to the prediction first. We validate IAL on various datasets from the healthcare and finance domains, on which it significantly outperforms baseline approaches with conventional attention mechanism and random selection of instances when using the same number of annotations, with significantly shorter annotation time per instance owing to attention reranking. Further qualitative analysis shows that IAL also yields interpretations that agree well with human interpretations.

1. INTRODUCTION:

Deep neural networks have been the most prevalent tools for predictive modeling tasks nowadays, as they are powerful and can learn complex functions with multiple layers of non-linear transformations without manual engineering of representations. However, the complex nature of the model at the same time makes it very difficult to interpret what they have learned, and brought a new challenge of interpretability. Interpreting deep neural networks is crucial to their applications to real-world application domains such as healthcare (Choi et al., 2016; Heo et al., 2018; Ahmad et al., 2018; Sankar et al., 2019) , finance (Grath et al., 2018; Wong, 2018) , and autonomous driving (Kim & Canny, 2017; Chi & Mu, 2017) . For such high-risk tasks, incorrectly learned correlation could result in severe consequences (e.g. mortality, large financial loss, or accidents), and the deployments of such unreliable models could be avoided if the model is interpretable.

Although many recent models proposed diverse solutions to interpretability (Choi et al., 2016; Ahmad et al., 2018; Lage et al., 2018) , we face yet another challenge: not all interpretations are correct. Interpretable models should provide human-understandable and intuitive interpretations (Gilpin et al., 2018; Lage et al., 2018) , that conform to the domain knowledge of human experts. For instance, if an interpretable model failed to provide correct and human-intuitive interpretation of why it arrived at its decision on diagnosing a patient of heart failure, the final decision maker, the human physician, may not be able to trust the model's decision, even if the actual decision is correct. Often, deep learning models tend to learn tricks that exploit dataset bias, which is another barrier in building trust between humans and machine learning algorithms.

In the cognitive neuroscience perspective, human learning and understanding are built upon two integral parts: interaction and explanation. Our brain's biological functions are constantly developed by internal reflection (Back-propagation) and external explanation (Human feedback) during social interactions (Clark et al., 2015) . In this sense, interactive learning frameworks could be effective Based on this motivation, we propose an interactive learning framework that allows the users to manipulate the model, by adjusting the provided model interpretations. However, there are several critical challenges that need to be addressed for such an interactive learning framework to be critical. First, to reflect human feedback, the model may need to be retrained, but this is very expensive for deep neural networks. Thus, we need an efficient approach to correct the model's behavior without retraining the entire network. Secondly, requesting and obtaining human feedbacks could be highly expensive (e.g. asking for clinician's annotation), and obtaining feedbacks on already correct interpretations is wasteful. Therefore, we need to selectively provide interpretations on samples and input features that can actually bring in performance improvements of the model. Finally, due to scarcity of human feedbacks, the updated model may overfit, which we need to prevent.

To overcome theses challenges, we propose a novel interactive learning framework which we refer to as Interactive Attention Learning (IAL) that can data-efficiently update the trained network without having to train the entire model or overfitting to scarce human feedbacks (See Figure 1 ). Specifically, we provide the model interpretations in the form of attention allocated on the input variables, and obtain feedbacks from domain experts to correct the model's interpretation by providing a mask on the attention, which is used as supervised labels to update the attention generator without retrainig the main network (Figure 1 (c)). However, since the retraining of the attention generator alone could be costly and is prone to overfitting, we propose a novel attention mechanism based on Neural Process, which we refer to as Neural Attention Process (NAP), that can data-efficiently generate attentions with scarce human labels, incorporate additional labeled instances without retraining, and output uncertainty (Figure 1 (a)). Further, since obtaining human attention labels for datasets with large number of instances and large number of input variables is costly, we select the examples that has the most negative effect on the generated interpretation using the influence function (Cook & Weisberg, 1980) , and then sort the attended input variables by the measured uncertainty in order to interact with the users in the most efficient way (Figure 1(b) ).

We validate our interactive attention learning framework on a variety of tasks, including excercise posture correction task, and cerebral infarction risk prediction from electronic health records (EHR), and real-estate price forecast. The experimental results show that our model outperforms the baseline network or naive interactive learning scheme by significant margins with much smaller annotation cost, in terms of number of instances and time to annotate each instance. Further quantitative and qualitative analysis of the learned attention weights shows that our model is able to generate interpretations that align well with the physician's evaluations on the same EHR records.

Our contribution in this paper is threefold:

• We propose a novel interactive attention learning framework with an efficient attention mechanism based on Neural Processes, which enables to efficiently correct the model's understanding with scarce human feedbacks without retraining of the entire network.

• We propose a cost-effective interactive learning algorithm to rank the examples and the attended input variables, in order to maximize the effect of each annotation and thus to minimize the human-machine interaction cost.

• We validate our model on five real-world tasks from three different domains (fitness, healthcare, and finance) for binary, multi-label classification, and regression tasks, and show that our model obtains significant improvements over baselines with less human feedbacks.

2. RELATED WORK:

Interpretable machine learning The literature in interpretable machine learning is vast, but we only discuss a few. A popular approach to obtain interpretable model is to build a simple proxy model that mimics the (local) behaviors of a complex model, using either simplified linear models (Ribeiro et al., 2016) or decision trees (Sato & Tsukimoto, 2001; Augasta & Kathirvalavakumar, 2012; Salzberg, 1994) . Another approach, specific for neural networks, is analyzing their learned representations (Sharif Razavian et al., 2014; Yosinski et al., 2014) at each unit via visualization. Bau et al. (2017) further consider interpretability of representations in light of their correspondence to semantic concepts, and utilize it for controlling the behavior of generative adversarial networks (Bau et al., 2019) . In tihs work, we propose a novel interactive learning framework that can make use of the model's interpretation to iteratively correct the model's understanding, while minimizing the interaction cost via cost-effective instance selection and reranking.

Attention Mechanism Attention mechanism is an effective approach to adaptively select a subset of features (or inputs) in an input-dependent manner, such that the model dynamically focuses on more relevant features for prediction. This mechanism works by input-adaptively generating coefficients for the input features to locate more weights to the features that are found to be relevant for the given input. Attention mechanisms have achieved success with various applications, including image translation (Xu et al., 2015) , machine translation (Bahdanau et al., 2015) , memory-augmented networks (Sukhbaatar et al., 2015) , and visual question answering (Das et al., 2017) , and healthcare (Choi et al., 2016; Heo et al., 2018) Train the network weights θ (1) by minimize θ (1) L(θ (1) ; Dtrain) + Ω(θ (1) )

4 else 5 (D (s) selected = {x (1:T ) k , y k } K k=1 , α) ← CESR(θ (t-1) ) Cost-Effective Selection 6 & Reranking-Algorithm 2 7 {m k } K k=1 ← Evaluate(D (s) selected , α) Evaluate & get feedback for attention α 8 φ (s) ← NAP(xs, {m k } K k=1 , φ)

Efficient model update with NAP. 9 end 10 end

3. APPROACH:

We now describe our interactive attention learning framework with neural attention process. While our method is generic enough to be applied to any types of prediction tasks, we focus on the case using time-series data. A training dataset is represented as ) i , . . . , x (T ) i ] and x (t) i ∈ R D is the D-dimensional observation vector for the t-th time step. y i is the corresponding label defined according to the task at hands (e.g., a real number for regression). We are interested in building a neural network with two components -an embedding network f v (•; ω) embeds x (1:T ) i into v (1:T ) i , and an attention generating network f α (•; φ) computes the attention weights. The attention generating networks accelerate the learning by attending to specific parts of the inputs, and more importantly, provides a way to interpret the decision of the model. Thus we are interested in improving the quality of attention by interacting with human experts. More specifically, when the model produces a prediction with an attention, a human supervisor evaluates 𝒍 (𝟏) LSTM Layer Input 𝒍 (𝟐) 𝒍 (𝟑) 𝒙 (𝟏) 𝒙 (𝟐) 𝒙 (𝟑) 𝒓 (𝟏) 𝒁 (𝟐) 𝒁 (𝟑) 𝒓 (𝟐) 𝒓 (𝟑) 𝜇

D train = {(x (1:T ) i , y i )} N i=1 , where x (1:T ) i = [x (1

Annotation:

𝒛 𝜇 𝒛 𝜎 𝒛 𝜎 𝒛 (1) (2) (2) (3) (3) 𝒍 (𝟏)

Concatenate:



Linear Transformation:

Nonlinear Function Attention 𝒍 (𝟐) 𝒛 (𝟏) 𝒛 (𝟐) 𝒍 (𝟑) 𝒛 (𝟑) (A-1) Embedding Network (A-2) Neural Process (A-3) NP Output Generation

Figure 2 : Neural Attention Process (NAP). Embeded outputs l and the global latent variable z are generated from the embedding network (A-1) and neural process (A-2), respectively, and the final attention output α is produced by the linear transformation (A-3).

the attention and gives feedback in the form of a binary attention mask m. This attention mask then could be used as supervision to further improve the model by updating the parameter φ for f α .

We describe the overall interactive attention learning (IAL) framework in Algorithm 1. We aim to improve the model over S interactions with human supervisors. At the first training stage (s = 1), we initially train the model parameter θ (1) without the help of human supervisors by minimizing the loss L(θ (1) ; D train ) + Ω(θ (1) ) where L is the loss function for the task and Ω is the regularization function. For each of the following iterations, we select instances and attended input variables, obtain the attention mask m from the human supervisor, and update the model parameters by minimizing the loss L(θ (s) ; θ (s-1) , m, D train ) + Ω(θ (s) ).

3.1. NEURAL ATTENTION PROCESS:

In this section, we describe how to effectively update the model with sparse annotations using the variant of neural process (Garnelo et al., 2018b) which we refer to as Neural Attention Process (NAP).

Before describing our approach, we briefly explain how attention is applied using RETAIN Choi et al. (2016) as an example. Given the input embedding v (1:T ) , attention coefficients for both timesteps and input variables are constructed as follows:

EQUATION

EQUATION

α (1:T ) time = Softmax(e (1) , . . . , e (T ) ),

EQUATION

where α (1:T ) time are attention weights applied for time-steps and α (1:T ) var are attention weights for the input variables. We may also consider the stochastic attention as in (Xu et al., 2015) . Having α = {α (1:T ) time , α (1:T ) var }, the model can make predictions as ŷi =

T t=1 α (t) time • (α (t) var v (t) i )

where is the element-wise multiplication. Now we describe the actual algorithm for NAP. Let {m (1:T ) k } K k=1 be a set of annotations represented as masks, given for the selected subsamples of the training data. The idea is that, instead of updating the parameter φ using these small number of examples, we let network take the summarization of the annotation set as an additional input. This approach, when trained properly, can automatically adapt without retraining when a new set of annotations is further given. The overall pipeline of neural attention process is depicted in Figure 2 .

Embedding the inputs (A-1) we first embed the input x (1:T ) using LSTM (Hochreiter & Schmidhuber, 1997) 

into l (1:T ) = [g (1:T ) , h (1:T ) ].

Embedding & summarizing the annotations (A-2) Given the set of annotation masks

{m (1:T ) k } K k=1 , we build an intermediate representation {r (1:T ) k } K k=1 via another LSTM.

Then, for each time step, we build a summarized representation r(t) by a permutation-invariant operation (for instance, average),

EQUATION

Having r(1:T) , we define a distribution for the summary variable z as Gaussian:

z (t) ∼ N (µ(r (t) ), σ 2 (r (t) )), µ(r (t) ) = W µ r(t) + b µ , σ(r (t) ) = softplus(W σ r(t) + b σ ). (5)

Generating attentions (A-3) Now we generate the attention by a similar procedure to (3), but instead of feeding only, l (1:T ) = (g (1:T ) , h (1:T ) ), we feed both l (1:T ) and the annotation summarization vector z (1:T ) by concatenation. This allows the network to naturally reflect the information obtained from the summarization z (1:T ) without having to retrain the whole attention network parameter φ.

Algorithm 2: Cost-Effective Selection & Reranking (CESR)

Input :Dtrain = {x (1:T ) i , yi} N i=1 , Dvalid = {x (1:T ) j , yj} M j=1 , P, K, θ (s-1) . Output :D (s) selected = {x (1:T ) k , y k } K k=1 , α. 1 Evaluate the network with Dvalid. 2 Sort valid points {(x (1:T ) j , yj)} M

j=1 in the descending order of L(x (1:T ) j , yj; θ (s-1) ). selected = {x k , y k } K k=1 w.r.t. the approximated influence. 8 Compute the attention weights α for D (s) selected . 9 Approximate uncertainty of attention weights Var(α) for D (s) selected via Monte-Carlo sampling.

Training NAP Even with NAP, we need at least one training procedure to update φ so that it can take z (1:T ) as an additional input. For this training, we use two strategies to make the NAP to readily generalize to the future annotations to be given. First, at each training step, we randomly subsample the annotations to comprise random task to train the model. This prevents the model from completely over-fitted to the entire annotation set {m (1:T ) k } K k=1 . Secondly, we regularize the summarization vector z (1:T ) by positing a prior distribution. We then train the ELBO in similar fashion to the original neural process objectvie (Garnelo et al., 2018b) .

3.2. COST-EFFECTIVE SELECTION AND RE-RANKING:

Acquiring human annotations is highly expensive and thus it is crucial to prioritize the most important instances and attentions (input variables) that negatively affect the model accuracy. The most native approach to identify important negative examples in the training set, would be to retrain the entire model parameter θ every time, while omitting a single targeted point. To avoid such costly retraining, we utilize influence functions (Cook & Weisberg, 1980) and uncertainty to perform negative instance selection and feature re-ranking respectively in a cost-effective manner.

Influence functions efficiently estimate the effect of removing particular train points on a model without retraining (Koh & Liang, 2017) . For notational simplicity, we set train points s 1 , ..., s N , where s i = (x i , y i ) ∈ D train 1 . Given a scenario where we approximate the change of the model's predictions by removing a data point s, we can formalize the change as θ-sθ, where θ = argmin θ

1 n n i=1 L(θ, s i ) and θ-s = argmin θ 1 n-1 si =s L(θ, s i ).

That is, θ is the parameter that minimizes the empirical risk of the full dataset and θ-s is the empirical minimizer of the dataset without the train point s. Since upweighting s i with δ = -1 n has the same effect as removing s i from the train set (Koh & Liang, 2017), we approximate the influence by upweighting s i on the loss at all valid points as follows:

EQUATION

where

H θ = 1 n n i=1 ∇ 2 θ L( θ, s i

) is the Hessian matrix to compute the second partial derivative of the function as a quadratic approximation with respect to the empirical loss around θ and we set the validation data points

s valid p = {(x p , y p )|x p ∈ R d , y p ∈ R 1 } P p=1

and Ĩup, loss (s i , s valid ) represents the influence of a train point s i on the the sum of loss at all valid points {s p } P p=1 . Cost-Effective Selection & Reranking (CESR) We now describe the procedure illustrated in (B)-Figure 1 , in detail. (1) We train the entire network with a training set, sort valid points in descending order of its loss, and select top P -valid points. (2) We then compute influence scores for training points on the loss at P -valid points D valid = {x p , y p } P p=1 , and (3) sort the influence scores of training points in descending order of Ĩup, loss (s i , s valid ) and select top K-train points denoted as

D selected = {x k , y k } K k=1 .

(4) Finally, using the approximate uncertainty of each variable's attention Var(α(x d s )) with Monte Carlo dropout (Gal & Ghahramani, 2016) , we sort the set of variables in descending order of the approximated uncertainty. One important aspect of CESR is that the identified set of valid points with the highest losses at evaluation can be effectively used as means for detecting negative train points via influence functions. Algorithm 2 describes the entire procedure. 

3.3. ATTENTION EVALUATION AND EFFICIENT MODEL UPDATE:



4. EXPERIMENTS:

We validate the performance and cost-effectiveness of our interactive neural attention learning, on five datasets from three domains.

1) Fitness -Squat Pose Correction This dataset contains 4,000 video frames of human subject performing squats with 11 multi-labels classification task (e.g., 0: Correct posture, 1: Incorrectexaggerated knees-forward movement, 2: Incorrect-sitting on the thighs instead of between them). We extract 14 pairs of key points from joints (e.g., left shoulder or right ankle) to have a clear picture of which body joints an attentional network attends to for a given instance. The task is to classify whether a person performs the correct posture or 10 types of incorrect posture.

2) Medical Check-ups These datasets are subsets of the electronic health records database of a major hospital, consisting of medical check-up records from 2009 to 2012 (4 time-steps) for patients over age 15 in out-patient units, including around 1.5 million records. We extracted 245,000 patient records from this database, in which each record contains 34 variables including general patient information (e.g., sex and height), vital signs (e.g., systolic pressure and hemoglobin level), and risk-inducing behaviors(e.g., alcohol consumption). The task is to predict the onset of the following disease in one year: 1) Heart Failure, 2) Cerebral Infarction, 3) Cardiovascular Disease (CVD).

3) Realestate Sales Transactions This datasets is a subset of residential sales transaction database from New York City Department of Finance consisting of approximately 70,700 house records with 27,000 sales transaction records over 15 years from 2011 to 2018 (8 time-steps). The dataset includes 3100 housing transactions and each record includes 47 variables (e.g., housing characteristics (number of bathrooms or bed rooms) and macro economic index (interest rates, GDP price index). The task is to forecast a residential property price in NYC in one year.

For all datasets, we generate train/valid/test with the ratio of 70%:10%:20%. For more details on the datasets, network configurations, and hyperparameters used, please see supplementary file.

Baselines We now describe the baselines and our models.

1) [RETAIN]::

The attentional recurrent neural network model (RETAIN) in (Choi et al., 2016) .

2) [RETAIN] Random Selection: RETAIN, but newly trained from a train set with k-train points omitted: k stands for the number of randomly selected train points from the original train set.

3) [RETAIN] IF Selection: RETAIN, but newly trained from a train set with top k-negative points omitted: k is the number of negative points selected via influence function (Koh & Liang, 2017) . 4) [IAL] UA-Random: Uncertainty-Aware attentional network (UA) in the interactive attention learning setting. k-data points are randomly selected, delivered to annotators for attention evaluation, and, during retraining, the set of parameters in the attention network are selectively fine-tuned by element-wise multiplication of binary annotation masks (m k ) with attention parameters (m k α). 5) [IAL] NAP-Random: Our Neural Attention Process model (NAP), in the IAL-Random setting (4). At each iteration, NAP is updated with binary annotation masks without retraining. 6) [IAL+Cost-Effective] Attention Cross Entropy (ACE): Retrain the attention network using a binary cross entropy loss function between the attention weight vector α and the attention annotation m k in generating attention weights, which adopts a similar approach with (Choi et al., 2019) . 7) [IAL+Cost-Effective] NAP-Selective Our IAL framework with Neural Attention Process (NPA).

4.1. EVALUATION OF THE FINAL MODEL PERFORMANCE:

We first examine the prediction accuracy of baselines and our model. Table 1 contains the accuracies of baselines and our model measured with Area Under the ROC curve (AUROC) on the risk prediction tasks, accuracy on squat posture task with multi-labels, and percentage error on real-estate price forecast. Note that IF Selection, which uses influence functions to remove instances with negative influence scores, performs relatively better on most tasks than other RETAIN baselines, but fails to improve on CVD and squat posture task. We observe that UA-Random, whose newly learned knowledge of the human annotations from randomly selected data points, performs worse than NAP-Random on all tasks, which is caused by overfitting to a particular example with retraining. Note that NAP-Selective works significantly better than NAP-Random for all tasks, which shows that the effect of attention annotation process cannot have much effect on the model when the instances are randomly selected. The Attention Cross Entropy model, which retrained with the binary cross entropy loss function between attention weights and annotation masks, performs worse than NAP-Selective, caused by severe overfitting with scarce annotation samples. failed to predict the label at s=1 (a), but make a correct prediction at s=1 (c). We visualized five variables that have clinically meaningful changes. Across the change of attentions from (a) to (c), the physicians consider that attentions on age, HDL, and LDL in (a) are false positive and smoking as false negative, except SysBP as true positive. Noting that the patient's age ( 30) is younger than the median age (50 years-old) of female CVD patient (Garcia et al., 2016) , initial NAP-Selective (a) allocated too much weights on age, which led to an overconfident attention model and in turn resulted in the incorrect prediction. However, our model gradually allocated less weights on age over iteration, as it started to learn what to attend to from physicians on a disjoint set of training data. Note that attention on smoking highly increased at s=3 (c), which is also clinically guided by a physician for the reason that CVD risk increases by 25% for women who smoke cigarettes (Huxley & Woodward, 2011) .

Interpretability and Accuracy of Generated Attentions:

In Figure 5 , each bar graph corresponds to top 5 feature variables that are selected most often by each method on the CVD task with 810 EHR records. Interestingly, all variables that NAP-Selective attends to the most are interpreted by physicians as key risk factors for accessing a CVD patient.

Although NP-Random failed to ignore GGT which is a relatively a less important variable, it accesses the key variable better than other models with IF-Selection. For broad clinical descriptions for figure 5 , please see supplementary file. The line graphs in Figure 6 (top) shows the change in model accuracy over iterations, with NP-Random and NP-Selective. On the risk prediction and posture estimation tasks, the accuracy of NP-Selective increases over the rounds of interaction, while NP-Random achieves marginal increases only on heart failures and CVD tasks, and degnerates accuracy on others. We further measure the average response time of the annotators with and without reranking of the attented inputs. The bar graphs (bottom) show that annotators spend less time with annotation if variables are prioritized by its uncertainty using our uncertainty-based reranking method (red bars), compared to presenting them in the original order (green bars), on all tasks. We further analyze the effect of the re-ranking algorithm by comparing top-10 input variables selected by our re-ranking algorithm, with the top-10 variables whose attentions are corrected by the annotators without variable re-ranking. In Figure 7 , We see that 7 out of 10 input variables that are most often selected by our algorithm (colored in red) corresponded to those selected by human annotators, which further demonstrates the effectiveness of the method.

4.2. EVALUATION OF THE COST-EFFECTIVE SELECTION AND RE-RANKING:



Hemoglobin:



5. CONCLUSION:

We proposed an interactive learning framework through the attention generated by the model, using a novel attention mechanism based on Neural Process that efficiently enables correction of a model's interpretability with scarce human feedback without retraining the entire network. Further, we propose a cost-effective instance selection and attention re-ranking algorithm to minimize the human-machine interaction cost while maximizing its effect. We validated our model on five real-world tasks from the healthcare, fitness, and finance domains and showed that our model significantly outperforms the baselines with fewer annotation cost in terms of the number of training instances and annotation time, while generating more human-interpretable attentions.

6. DETAILED DESCRIPTION OF DATASETS AND EXPERIMENTAL SETUP:



6.1. DATASETS:

Squat Posture Correction This datset consists of 4000 video frames of squat posture our research team collected over 6 months. Professional trainers performed the conventional squat, filed by three Kinect V-2 devices placed at three different angles. Going through discussions with trainers, we determined one correct squat postures and 10 types of typical incorrect posture that non-experience people mistakenly make, which makes posture correction task as multi-label classification task (e.g., 1) Exaggerated knees-forward movement or 2) sitting on the tighs instead of betweeen them). The average runtime for one video is 5.8 seconds with around 60 frames. For cost-efficiency, we set the frame skip as 3, such that each instance has 14 timesteps. Instead of using raw pixels as input, we extracted 14 pairs of body joints from a human object in frames by using the famous Openpose model (Cao et al., 2017) . Extracted body points consist of 14 pairs of x and y coordinates, which is expected highly useful when determining attention labels due to its autonomical locality. All data examples have two labels: 1) Label for class, 2) Label for attention. For example, the data example, labeled as sitting on the tighs instead of betweeen them has attention labels: Left hip, Right hip, Left knee, Right Knee. Information about the extracted 14 pairs of body joints and 11 classes are shown in the table. We performed additional experiment on this dataset with respect to interactive attention learning in the next chapter. Three annotators participated in the annotation evaluation procedures. In the case that the same set of negative train points is delievered to multiple annotators, the accumulated sets are aggregated into one annotation matrix by averaging:

m k = 1 I I i=1 m (i) k .

Electronic Health Records This datasets is a subset of electronic healthcare records-based database from healthcare organization, consisting of around 1.5 million records. The database contains demographic information including medical aid beneficiaries, treatmenet information, disease histories, and drug prescription records. In total, 34 features regarding vital signs, social and behavioral factors, medical history, and general information, were extracted from the database over 12 years. Total cholesterol level and fasting glucose level were sampled after overnight fasting and systolic blood pressure and diastolic blood pressure were checked through medical examinations. Also, there were several questionnaires that are designed to identify social and behavioral risk factors, such as smoking habit, alcohol consumption, and time spent on excercise. Individual medical history was followed with drug perscription history and clinical codes of the 10th revision of the International Classification of Diseases (ICD-10). We determined patients with pancreatic cancer by identifying ICD code, C25, on examination and treatment records. On the labeling process, we exclude those who had previous pancreatic cancer-related treatment records as well as pre-existing medical history of pancreatic cancer. Two physicians participated in the experiments with CVD, cerebral infarction, and heart failure tasks, as an annotator.

Real-estate Price Forecast in New York City The datasets are the subset of residential sales transaction database from the Department of Finance's Rolling sales files list properties, sold in the last 17 years from 2003 to 2018. We combine the subset from the rolling sales files with another subset extracted from Final Property Assessment Data from all NYC properties. The dataset we processed has a very hetero geneous set of homes spread over five boroughs in New York City: 1) The Bronx, 2) Queens, 3) Brooklyn, 4) Manhattan, 5) Staten Island. Each house is described by a total of 182 attribute variables. These attributes specific to 1) the house-related profiles (Number of bed rooms and bethrooms, square footage, and the year built), 2) Realestate owner-related information (Tax information or Salary), 3) Geographical information (e,g, distance from a hospital, school score in that neighborhood, and the number of hospital facilities within 5 miles), 4) Global economic indicators (e.g., Global copper price, interest rates, total vehicle sales, and Russell 2000). Two real-estate business managers in New York City annotated attentions for real-estate price forecast tasks.

6.2. CONFIGURATION AND PARAMETERS FOR THE RISK PREDICTION TASKS:

We trained all the models using Adam (Kingma & Ba, 2014) optimizer with dropout regularization. We set the maximum iteration for Adam optimizer as 10, 000, and for other hyperparameters, we searched for the optimal values by cross-validation, within predefined ranges as fol-lows: Mini batch size: {32, 64, 128, 256}, annotation subsampling batch size: {8, 16, 32} learning rate: {0.01, 0.001, 0.0001}, L-2 regularization: {0.02, 0.002, 0.0002, 0.0004}, and dropout rate {0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5}. Exercise Posture Correction Task In the interactive attention learning framework, professional trainers interactively evaluate visualized attentions generated from the attentional network via annotation attention masks. In figure 8 , the size of white circles represents the size of attention weights.

7. BENEFITS OF INTERACTIVE ATTENTION LEARNING FRAMEWORK:

For the given the instance (Label 2. Rounding back like c, Attention label: R-Knee, L-Knee, R-Hip, L-Hip), (c) shows that the network evenly generates weights on both left and right knees by allocating more weights on R-Knee over three iterations, compared to the initial iteration (b). An attention network relearned how to attend for a given input, as a human annotator guided (d) shows that attention answer for correcting posture. Further Interpretation in the cerebral infarction and fatty liver tasks We further quantitatively compared the accuracy of attentions, using variables selected meaningful by the physicians as ground truth labels (avg. 134 variables per patient). We randomly selected 10 age groups from 40s to 80s for cerebral infarction and fatty liver risk prediction tasks. In table 3, we observe that Type 1 error significantly decreases only with three iterations (from t=1 to t=3).

8. DETAILED DESCRIPTION OF ATTENTION ANNOTATION INTERFACE:

Attention Annotation Interface in the CVD and cerebral infarction tasks Clinical features with high correlation to each task (CVD, Cerebral) were highly annotated according to the value of each features. For example, when the attention is relatively high on LDL value with 90 compared to other features, physician annotated LDL to further change its attention value over the loop. Furthermore, clinical features with low correlation to each task were highly annotated when model weighs them with relatively high attention. Detailed explanations on each features are summarized in Annotation Rules: Cardiovascular Disease and annotation Rules: Cerebral Infarction.

Annotation Rules -Cardiovascular Disease Risk stratification and detailed critea on when the model should attend to each features are summarized below: First of all, hypertension(systolic blood pressure(SBP) > 140mmHg and diastolic blood pressure(DBP) > 90 mmHg) is quantitatively the most important risk factor of cardiovascular disease (CVD) (Stanaway et al., 2018) . Insulin resistance, hyperinsulinemia, diabetic dyslipidemia, and elevated blood glucose are associated with atherosclerotic CVD (Kannel & McGee, 1979; Almdal et al., 2004; Zavaroni et al., 1989) . Dyslipidemia, hypercholesterolemia with serum cholesterol ≥ 200 mg/dL can be accounted for the attributable risk of CVD (Yusuf et al., 2004; Lowe et al., 1998) . Reductions in low-density lipoprotein (LDL) cholesterol levels with the use of statin reduce the risk of CVD (Downs et al., 1998) . Low HDL level (<40mg/dL) raises risk for developing CVD, while high HDL level(>60mg/dL) acts as a protective factor of CVD (Ridker et al., 1998) . Obesity(BMI > 30) is associated with a number of risk factors for atherosclerosis, CVD, and cardiovascular mortality. Risk factors for CVD includes diabetic condition of a patient, such as insulin resistance and glucose intolerance (Eckel et al., 2004; Calle et al., 1999) .

Among social history of a patient, exposure to tobacco is independent major risk factor, dosedependently increasing the risk for total atherosclerotic CVD, coronary heart disease(CHD), cerebrovascular disease, heart failure, and mortality (Jee et al., 1999; Qiao et al., 2000; Foody et al., 2001) . Smoking cessation is known to be beneficial for preventing CVD; smoking cessation is associated with the reduction in cardiac event rate (Rose et al., 1982) , where the risk further decreases with elongation of time since quitting (Novello, 1990) . While epidemiologic data indicate that moderate alcohol intake has a protective effect on CHD (Gemes et al., 2016) , binge drinking increases the risk for CVD (Roerecke & Rehm, 2010; Ruidavets et al., 2010) . Moderate exercise has a protective effect against CHD and all-cause mortality (Powell et al., 1987) .

Next, among non-modifiable risk factors, CVD risk increases with aging (over age 45 for men, over age 55 for women), and for the same age patient group, men are more prone to develop cardiovascular disease than women (Jousilahti et al., 1999) . Family history of CVD is an independent risk factor for CHD; high risk for the individuals with first-degree relatives who developed atherosclerotic CVD or death from CVD (male relative prior to age 55 and female relative prior to age 65) (Patel et al., 2018; Stone et al., 2014) . A wider definition of this significant family history of CVD might also include CVD-related death, stroke, or transient ischemic attack (Patel et al., 2018) . History of stroke can also be risk factor of CVD as they both have similar pathophysiology (Anderson et al., 1991) . Family history of hypertension(systolic blood pressure > 140mmHg and diastolic pressure > 90 mmHg) (Anderson et al., 1991 ), diabetes (Anderson et al., 1991) can indirectly be a risk factor of CVD. Additionally, for other features like hemoblogin, urine protein, AST, ALT, GGT, Creatinine and history of pulmonary tuberculosis, there is no proven evidence on the effect of these values with cardiovascular disorder (Pencina et al., 2019) .

Annotation Rules -Cerebral Infarction Risk stratification and detailed critea on when the model should attend to each features are summarized below:

History of stroke and transient ischemic attack in the same territory strongly predicts future stroke occurrence (Society, 2016) . Hypertension(SBP > 140mmHg and DBP > 90 mmHg) is quantitatively the most common and most important risk factor for stroke with estimated relative risk of 4.0-5.0 and estimated prevalence of 25-40% (Society, 2016; Ezekowitz et al., 2003; Jorgensen et al., 1994) .

A cardiac evaluation (e.g. echocardiogram) to find out whether patient has cardiac disease, such as atrial fibrillation or other embolic conditions, is important in managing risk factors for stroke (Jorgensen et al., 1994; Ezekowitz et al., 2003) . Diabetes itself and diabetic conditions such as insulin resistance, elevated blood glucose increase the likelihood of large and small artery occlusive disease (Jorgensen et al., 1994) . Risks for stroke stem not only from increased likelihood of atherogenesis but also from aggravation of other risk factors including hypertension and hyperlipidemia (Najarian et al., 2006) . Preventing dyslipidemia by lowering LDL cholesterol and elevating HDL may prevent strokes (Society, 2016; Hindy et al., 2018) . Also, compared to those with normal BMI, obese and overweight patients have significantly better early and long-term survival rates, which is called the paradox of obesity (Vemmos et al., 2011; Banack & Kaufman, 2013) .

Among social history of a patient, smoking increases the likelihood of CVD, more than doubling the risk of stroke with relative risk of 1.5-2.9 and estimated prevalence of 4-8% (Shah & Cole, 2010; Ezekowitz et al., 2003) which decreases with the cessation of smoking proportional to the period after cessation (Kawachi et al., 1993) . Moderate and high level of exercise is associated with reduced risk of stroke (Society, 2016; Potempa et al., 1995; Lee et al., 2003) . Epidemiologic data indicate that moderate alcohol intake has a protective effect on stroke. However, binge drinking increases the risk for stroke (Society, 2016; Hillbom et al., 1999) .

Aging is nonmodifiable risk factor for ischemic stroke, and also for mortality and morbidity (Society, 2016; Lee et al., 2003; Roy-O Reilly & McCullough, 2018) . Also, older subjects are more prone to develop CVD and emblic, thrombotic stroke compared to their younger counterparts (Roy-O Reilly & McCullough, 2018; Nakayama et al., 1994) . Individuals with family history of stroke (Liao et al., 1997; Jerrard-Dunne et al., 2003) , cardiac conditions(especially atrial fibrillation) (Fox et al., 2004) , and hypertension (Staessen et al., 2003; Wang et al., 2008) possess genetic susceptibility, thus high risk of developing stroke compared to the individuals without family history. Furthermore, family history of type II diabetes in any first degree relative have a two to three-fold increased risk of developing diabetes thus indirectly increasing risk of stroke, compared to individuals without family history (Consortium et al., 2013; Meigs et al., 2000) . Additionally, for other features like hemoblogin, urine protein, AST, ALT, GGT, Creatinine and history of pulmonary tuberculosis, there is no proven evidence on the effect of these values with cardiovascular disorder (Society, 2016).

:

We omit the time index in this section.

Figures and Tables:

Figure 1: Overview of our Interactive Attention Learning Framework.

Select top P -valid points D valid = {x (1train points (x (1:T ) i , yi) on loss at D valid . 6 end 7 Select top K-training points D (s)

Figure 3: Attention Annotation Interface (Healthcare).Given the provided set of selected negative train instances from CESR algorithm, (1) we present the visualizations of attentions on our online interactive user interface in Figure3.3, where the annotators can easily turn on/off the binary mask m k = {0, 1}. The interface visually emphasizes features that the annotator should pay attention to the bar plots or attention map depeding on the given task (see the appendix for more information). (2) Annotators examine attention weights to determine whether attention weights are incorrectly allocated, simultaneously checking for the corresponding input values and label. With time-series data, annotators evaluate the delivered attentions α (1:T ) time and α (1:T ) var via an annotation matrix, which are basically matrices of binarized attentions. (3) Accumulated annotations are efficiently updated to NAP without retraining.

Figure 4: Visualization of attention for a selected patient on the Cardio Vascular Disease (CVD) prediction task. Contribution indicates the extent to which each individual feature affects the onset of CVD in 1 year. Age -Age, Smoking -Whether the patient currently smokes, SysBP -Systolic blood pressure, HDL -High-density lipoproteins cholesterol, LDL -Low-density lipoprotein cholesterol. Bars correspond to attention weights.

Figure 6: (Top) Change of accuracy with 40 annotations for each iterations(s) on the all tasks between NAP-Selective (Red) vs NAP-Random (Green). (Bottom) Mean Response Time (mean-RT) of annotators to evaluate one data sample (Being prioritized by uncertainty vs Randomly ordered).

Figure 7: Top 10 variables (Green) ranked by the total number of being checked by physicians and top 10 variables (Blue) ranked by attention uncertainty in the CVD task.

Figure 8: Attention map for 14 pairs of body joints, generated from RNNs trained for squat pose correction task (Fitness).

Figure 9: Interactive Attention learning Annotation Interface for Squat Pose Correction Task.

Figure 10: Interactive Attention learning Interface for EHR datsets, on which physicians interactively guide the attention network to re-learn how to properly attend to features of a given input. Attentions from the attention network are visualized as above and physicians evaluate them on the web-based attention annotation interface.

The multi-class classification performance on the four electronic health records datasets and one fitness dataset. The reported numbers are mean-AUROC for EHR and mean-Accuracy for squat. In the realestate price forecast, the number indicates mean-percentage error, meaning a lower error indicates better performance.

Table content: <table><tr><td/><td/><td/><td/><td>EHR</td><td>Squat</td><td>Realestate</td></tr><tr><td/><td/><td/><td colspan="2">Heart Failure Cerebral Infarction</td><td>CVD</td><td>Posture</td><td>Forecasting</td></tr><tr><td>Baselines (RETAIN)</td><td colspan="3">RETAIN Random Selection 0.7852 ± 0.02 0.7921 ± 0.01 IF Selection 0.7984 ± 0.03</td><td>0.6168 ± 0.01 0.6116 ± 0.02 0.6182 ± 0.02</td><td>0.6164 ± 0.02 0.8425 ± 0.03 0.2522 ± 0.01 0.5671 ± 0.01 0.8221 ± 0.05 0.2540 ± 0.01 0.5882 ± 0.02 0.8363 ± 0.03 0.2434 ± 0.01</td></tr><tr><td>IAL (Random</td><td/><td>UA-Random</td><td>0.7824 ± 0.01</td><td>0.6191 ± 0.01</td><td>0.6012 ± 0.02 0.8512 ± 0.00 0.2632 ± 0.02</td></tr><tr><td>Selection)</td><td/><td>NAP-Random</td><td>0.8015 ± 0.02</td><td>0.6287 ± 0.03</td><td>0.6132 ± 0.02 0.8525 ± 0.01 0.2511 ± 0.01</td></tr><tr><td>IAL (Cost</td><td/><td>ACE</td><td>0.7982 ± 0.04</td><td>0.5992 ± 0.03</td><td>0.6193 ± 0.02 0.8450 ± 0.03 0.2519 ± 0.01</td></tr><tr><td>-effective)</td><td colspan="2">NAP-Selective</td><td>0.8157 ± 0.01</td><td>0.6374 ± 0.01</td><td>0.6304 ± 0.02 0.8562 ± 0.01 0.2381 ± 0.01</td></tr><tr><td/><td/><td/><td/><td>EHR</td><td>Squat</td></tr><tr><td/><td/><td/><td colspan="2">Heart Failure Cerebral Infarction</td><td>CVD</td><td>Posture</td></tr><tr><td/><td>NPA</td><td colspan="2">Random Order 0.8082 ± 0.01 Selective 0.8159 ± 0.01</td><td>0.6274 ± 0.01 0.6311 ± 0.02</td><td>0.6224 ± 0.02 0.8519 ± 0.01 0.6302 ± 0.01 0.8538 ± 0.01</td></tr></table>

Accuracy and percentage error on NPA-Selective and NPA-Random-order, in which veriables are randomly ordered. Only top 30% of variables were selected to the annotators.

Table content: <table><tr><td/><td/><td/><td/><td/><td>Age</td><td/><td colspan="2">Smoking</td><td colspan="2">SysBP</td><td/><td>HDL</td><td colspan="2">LDL</td></tr><tr><td/><td/><td colspan="2">2009</td><td/><td>31</td><td/><td>Yes</td><td/><td>139</td><td/><td/><td>54</td><td>97</td><td/></tr><tr><td/><td/><td colspan="2">2010</td><td/><td>32</td><td/><td>Yes</td><td/><td>134</td><td/><td/><td>55</td><td>97</td><td/></tr><tr><td/><td/><td colspan="5">Current State 33 yrs</td><td>Yes</td><td colspan="7">141 mmHg 55 mg/dL 102 mg/dL</td></tr><tr><td/><td>4 5</td><td/><td/><td colspan="2">False Positive False Negative True Positive</td><td>4 5</td><td/><td/><td colspan="2">False Positive False Negative True Positive</td><td>4 5</td><td/><td colspan="3">False Positive False Negative True Positive</td></tr><tr><td>Attention</td><td>2 3</td><td/><td/><td/><td>Attention</td><td>2 3</td><td/><td/><td/><td>Attention</td><td>2 3</td><td/><td/><td/></tr><tr><td/><td>1</td><td/><td/><td/><td/><td>1</td><td/><td/><td/><td/><td>1</td><td/><td/><td/></tr><tr><td/><td>0</td><td>Age Smoking (Amount)</td><td>Systolic BP Feature</td><td>HDL</td><td>LDL</td><td>0</td><td>Age Smoking (Amount)</td><td>Systolic BP Feature</td><td>HDL</td><td>LDL</td><td>0</td><td>Age Smoking (Amount)</td><td>Systolic Feature BP</td><td>HDL</td><td>LDL</td></tr></table>

Type 1 Error over three iterations, which shows percentage of features selected from the model that do not match the features selected by the clinicians.

Table content: <table><tr><td/><td colspan="2">Cerebral Infarction Fatty Liver</td></tr><tr><td>t=1</td><td>0.3242</td><td>0.2881</td></tr><tr><td>t=2</td><td>0.2844</td><td>0.2752</td></tr><tr><td>t=3</td><td>0.2485</td><td>0.2639</td></tr></table>

Peer review (iclr2020):
The authors introduce an interactive attention learning framework called Neural Attention Process (NAP) which produces weight predictions over the input variables in addition to the usual target predictions. The motivation behind this type of model is to obtain an interpretable explanation of the model’s output which can be shaped by a small set of human expert annotations. At a higher level NAP seems to be an extension of a previously introduced model called RETAIN with an additional component inspired by Neural Processes (NPs) which incorporates the annotations of the experts. The authors test their model on five different tasks from 3 datasets (health records, fitness and real estate) and compare to a number of related baselines. While I think the idea is interesting, in my opinion this paper is not ready for publication and I would argue for it to be rejected, for the following reasons: * Presentation: The paper is over badly written, sloppy and not well structured. * Badly written: there are lots of grammatical errors in the paper to the point that it severely affects its clarity (for example, the subject of a sentence often changes from plural to singular or vice versa within the same sentence and it’s unclear what the authors are referring to). * Sloppy: many variables are not defined in the paper, and some are called different names depending on the figure or part of the text (i.e. the use of ‘s’ or ‘t’ interchangeably to refer to the iterations). At some point even the name of the model (referred to as NAP for the majority of the paper) is just referred to as NP-Selective in the text and some figures (compare nomenclature in table 1 vs figure 5). This lack of attention to detail (although the name of the main model should hardly count as a detail) makes me worry about the scientific rigorousness of the work. * Not well structured: The paper is not very easy to follow as there are jumps between different models (RETAIN, NPs, NAPs) which are not always evident. On that note, I would have liked to see an objective function to know what exactly is being used to optimise theta. * Significance: Overall the results don’t seem incredibly significant (e.g. figure 6). If one of the main claims is that the number or annotated examples is smaller with NAP the authors should provide experimental evidence that this is the case (as this would be one of the most significant benefits of the model). * Motivation: a key motivation for this paper, as stated by the authors, is the desire to obtain interpretable representations, which in itself is an important research problem. However, I am not sure that the suggested solution is addressing this issue. If I understand correctly, what the model learns is, in essence, a weight mask over the inputs that determines how much each of the dimensions of the input contribute to the output. This could be achieved with any attention model in and is in itself not novel. I think a more interesting motivation for this model might be its ability to incorporate some human annotations that teach the model where to attend. However if this was the main story I think the authors should focus on making this a stronger argument and proving that narrowing down the attention with said annotations actually leads to better/faster/more robust learning, rather than focussing on the interpretability narrative. Some more specific comments and recommendations for the authors (I think there are some positive aspects of this paper and I hope that I can give a few points of advice to help improve its quality.) I will go through them in the order of the paper: 1) The name Neural attention process might be a bit confusing given that there are already Attentive Neural Processes. I would strongly recommend changing the name. 2) Check the language and make sure it’s correct, the introduction is particularly full of errors and you don’t want to lose the reader from the beginning. 3) If you are talking about interpretability you would have to mention as well all the work on interpretable representations (e.g. disentangled representations etc) and if you talk about attention, it is probably worth mentioning transformers. 4) Algorithm 1: this is not very clear. I would define the variables at the top that are not self-explanatory (such as S). Also, from the look of the algorithm theta is not updated in the else loop. 5) The change in notation is very confusing. It’s not always clear when you use ‘v’ vs ‘l’ or ‘s’ vs ‘t’. Sometimes it seems it depends on which of the different models you are talking about, but either way you should try to make it all uniform. 6) I would strongly recommend to specify the loss function and the regularisation term as this will make the task much clearer to the reader. 7) I would make the difference to RETAIN very clear and argue why you think your addition improves on the original (from the text at the moment it’s not even clear when you are describing RETAIN and when you talk about your model, maybe making a subsection would help?) 8) Figure 3 is too small. 9) You claim to have 5 datasets, but the way I would interpret your setup, you have 5 tasks from 3 datasets. 10) I would say a sentence or two about why you chose the baselines you chose and why you are hoping that they prove your different points. 11) Figure 6 has no axis labels on the plot itself, which are definitely necessary. I also recommend not using red and green as the two colours as it will be hard for colour blind to distinguish between those two.