dataesr/scientific-paragraphs-categorization huggingface dataset:
Rule learning approaches have been extensively used for knowledge representation in expert systems  [192]. However, a central problem with rule generation approaches is the coverage (amount) and the specificity (length) of the rules generated. This problem relates directly to the intention for their use in the first place. When building a rule database, a typical design goal sought by the user is to be able to analyze and understand the model. The amount of rules in a model will clearly improve the performance of the model at the stake of compromising its intepretability. Similarly, the specificity of the rules plays also against interpretability, since a rule with a high number of antecedents an/or consequences might become difficult to interpret. In this same line of reasoning, these two features of a rule based learner play along with the classes of transparent models presented in Section 2.
XAI and Theory-guided Data Science We envision an exciting synergy between the XAI realm and Theory-guided Data Science, a paradigm exposed in  [374]  that merges both Data Science and the classic theoretical principles underlying the application/context where data are produced. The rationale behind this rising paradigm is the need for databased models to generate knowledge that is the prior knowledge brought by the field in which it operates. This means that the model type should be chosen according to the type of relations we intend to encounter. The structure should also follow what is previously known. Similarly, the training approach should not allow for the optimization process to enter regions that are not plausible. Accordingly, regularization terms should stand the prior premises of the field, avoiding the elimination of badly represented true relations for spurious and deceptive false relations.
Fairness and Discrimination It is in the identification of implicit correlations between protected and unprotected features where XAI techniques find their place within discrimination-aware data mining methods. By analyzing how the output of the model behaves with respect to the input feature, the model designer may unveil hidden correlations between the input variables amenable to cause discrimination. XAI techniques such as SHAP  [224]  could be used to generate counterfactual outcomes explaining the decisions of a ML model when fed with protected and unprotected variables. Recalling the Fair AI principle introduced in the previous section,  [386]  reminds that fairness is a discipline that generally includes proposals for bias detection within datasets regarding sensitive data that affect protected groups (through variables like gender, race.
• Proxy features: there may be correlated features with sensitive ones that can induce bias even when the sensitive features are not present in the dataset. The next question that can be asked is what criteria could be used to define when AI is not biased. For supervised ML,  [393]  presents a framework that uses three criteria to evaluate group fairness when there is a sensitive feature present within the dataset: • Independence: this criterion is fulfilled when the model predictions are independent of the sensitive feature. Thus, the proportion of positive samples (namely, those ones belonging to the class of interest) given by the model is the same for all the subgroups within the sensitive feature. • Separation: it is met when the model predictions are independent of the sensitive feature given the target variable.
The proposal is based in self-learning locally generative models that use only a small part of the whole dataset available (weak supervision). It first finds recursively relevant prototypes within the dataset, and extracts the empirical distribution and density of the points around them. Then it generates rules in an IF/THEN format that explain that a data point is classified within a specific category because it is similar to some prototypes. The proposal then includes an algorithm that both generates explanations and reduces bias, as it is demonstrated for the use case of recidivism using the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) dataset  [400]. The same goal has been recently pursued in  [401] , showing that post-hoc XAI techniques can forge fairer explanations from truly unfair black-box models.
g. a reasoning engine operating on an open semantic database). All produced information is then fused to further ensure the quality, correctness and manageability of the produced knowledge about the items in the data set. Other data fusion approaches exist beyonds the ones represented in Figure  13. As such, data-level fusion can be performed either by a technique specifically devoted to this end (as depicted in Figure  13. b) or, instead, performed along the learning process of the ML model (as done in e. g. DL models). Similarly, model-level data fusion can be made by combining the decisions of different models (as done in tree ensembles).
Conclusions and Outlook This overview has revolved around eXplainable Artificial Intelligence (XAI), which has been identified in recent times as an utmost need for the adoption of ML methods in real-life applications. Our study Fundación BBVA a Equipos de Investigación Científica 2018 call (DeepSCOP project). This work was also funded in part by the European Union's Horizon 2020 research and innovation programme AI4EU under grant agreement 825619. We also thank Chris Olah, Alexander Mordvintsev and Ludwig Schubert for borrowing images for illustration purposes. Part of this overview is inspired by a preliminary work of the concept of Responsible AI: R. Benjamins, A. Barbado, D. Sierra, "Responsible AI by Design", to appear in the Proceedings of the Human-Centered AI: Trustworthiness of AI Models & Data (HAI) track at AAAI Fall Symposium, DC, November 7-9, 2019  [386].
Figure1: Evolution of the number of total publications whose title, abstract and/or keywords refer to the field of XAI during the last years. Data retrieved from Scopus R (December 10th, 2019) by using the search terms indicated in the legend when querying this database. It is interesting to note the latent need for interpretable AI models over time (which conforms to intuition, as interpretability is a requirement in many scenarios), yet it has not been until 2017 when the interest in techniques to explain AI models has permeated throughout the research community.
Acknowledgments Alejandro Barredo-Arrieta ,  Javier Del Ser  and  Sergio Gil-Lopez  would like to thank the  Basque Government  for the funding support received through the  EMAITEK and ELKARTEK programs. Javier Del Ser  also acknowledges funding support from the  Consolidated Research Group MATHMODE  ( IT1294-19 ) granted by the  Department of Education of the Basque Government. Siham Tabik ,  Salvador Garcia ,  Daniel Molina  and  Francisco Herrera  would like to thank the  Spanish Government  for its funding support ( SMART-DaSCI  project,  TIN2017-89517-P ), as well as the  BBVA Foundation  through its Ayudas.
By using the seasonal SVR model, it can deal with the seasonal cyclic nature of electric load for accurate and better prediction. However, the computational time is increased due to a large number of iterations. Fan et al. [31]  proposed a novel electricity load forecasting model by hybridizing the phase space reconstruction (PSR) algorithm with the bi-square kernel (BSK) regression model, namely the PSR-BSK model. The authors investigated the performance of the model using an hourly dataset of NYISO, USA, and New South Wales market. Hong et al. [32]  proposed an electric load prediction model, namely the H-EMD-SVR-PSO model, which combines the empirical mode decomposition (EMD) method, particle swarm optimization (PSO) algorithm, and SVR, to improve predictive accuracy. Based on electrical load data from the Australian electricity market, experimental results showed that the proposed H-EMD-SVR-PSO model received more satisfactory prediction performance than other comparable models.
Our Previous Works In this section, we briefly describe several STLF models that we proposed in our previous studies and their differences from the proposed model. In  [33] , we built two STLF models using the ANN and SVR for four building clusters of a private university in South Korea. For the prediction, we considered not only weather information and time data but also university events, office hours, and class hours. Subsequently, we evaluated the prediction performance of each model by using 5-fold cross-validation. The comparison showed that the ANN-based forecasting model had better performance than the SVR-based model. In  [34] , we proposed another STLF model based on an auto-encoder (AE) and RF. The AE was used to extract weather information features and time factors effectively.
We constructed an RF-based forecasting model using feature extraction values and historical electric loads for day-ahead electric load forecasting. The model was evaluated using the electric energy consumption data of university campuses and the results showed that it gave a better performance than the proposed model in  [33]. In  [35] , we proposed a recurrent inception convolution neural network (RICNN) that combines recurrent neural networks (RNN) and 1-dimensional convolutional neural networks (CNN) to forecast multiple short-term electric loads (48 time steps with an interval of 30 min). A 1-D convolution inception module was used to calibrate the prediction time and hidden state vector values calculated from nearby time steps. By doing so, the inception module could generate an optimized network via the prediction time generated in the RNN and nearby hidden state vectors.
The proposed RICNN model was verified using the electric energy consumption data of three large distribution complexes in South Korea. In  [36] , we constructed diverse ANN models using different numbers of hidden layers and diverse activation functions and compared their performances in a 30 min STLF resolution. To compare the prediction performance, we considered electric load data collected for two years from five different types of buildings (including the dataset used in this study). The comparison showed that a scaled exponential linear unit (SELU)-based ANN model with five hidden layers had a better average performance than other ANN-based STLF models. In  [37] , we proposed a two-stage electric load forecasting model that combined XGBoost and RF using MLR for the efficient operation of CCHP.
In  [36] , the SELU-based ANN model with five hidden layers showed that the dataset we used in this study exhibited insufficient prediction accuracy compared to the other building types because its electric loads are close to zero. In  [37] , we proposed a two-stage electric load forecasting model to combine XGBoost and RF using MLR. However, to use the forecasted values from one-stage more efficiently, we have to consider existing input variables. Eventually, we further develop our research and propose integrated applications with CCHP operation scheduling and electric rate recommendations, not just ending with forecasts.
Thus, we used the short-term weather forecast data that have 3 h resolution and used linear interpolation to calculate 1 h weather forecast data from them. The shortterm weather forecast data consists of values such as daily minimum temperature, daily average temperature, daily maximum temperature, temperature, humidity, wind speed, and precipitation, as shown in Figure  2. In addition, to establish a more direct correlation between weather data and electric energy consumption, we considered the discomfort index (DI)  [40]  and wind chill (WC)  [41]. DI and WC are defined using Equations (  1 ) and (  2   As a result, we use nine types of weather data (i. e. , daily maximum temperature, daily average temperature, daily minimum temperature, temperature, humidity, wind speed, precipitation, In addition, to establish a more direct correlation between weather data and electric energy consumption, we considered the discomfort index (DI)  [40]  and wind chill (WC)  [41].
Historical Electric Energy Consumption Historical electric energy consumption is a good indicator of electric usage forecasts as it shows electricity usage patterns and trends  [42]. We consider a specific time for 10 different days to reflect historical electric energy consumption. Herein, the historical electric energy consumption we considered included the last six days and the same day of the previous four weeks. For instance, if the forecast time is Saturday, 29 June 2019, 4 p. m. , we use historical load data measured at 4 p. m. from 1, 8, 15, 22, 23, 24, 25, 26, 27, and 28 June. However, the weekday and weekend load patterns could be different. To reflect trends through historical load data more accurately, we use additional data that indicate whether the ten days used as input variables were holidays or not.
The First Stage: Constructing Two STLF Models In the first stage, we build two STLF models based on XGBoost and RF, which are well-known tree-based ensemble models in time series prediction  [47] , by using various input variables. They are based on boosting and bootstrap aggregating (bagging) algorithms, respectively. Compared to other boosting algorithms and bagging algorithms, the XGBoost and RF models show better predictive accuracy and have the highest correlation with actual power consumption. In addition, as XGBoost supports various loss functions, we can choose an appropriate loss function depending on the characteristics of the data. On the contrary, it suffers from overfitting during training  [48]. RF can handle high dimensional data well, but it cannot give precise value for the regression model because the final prediction is the average of all the predictions from the subset trees  [49].
Random Forest RF is a flexible machine learning algorithm that produces excellent results even without hyper parameter tuning. It has become one of the most commonly used machine learning algorithms because it can be easily used for classification and regression. Moreover, it can work efficiently on a large amount of data and handle thousands of input variables without deleting them, which is why it performs well. The basic principle of RF is called the bagging algorithm  [53]. Bagging is an ensemble algorithm designed to improve the stability and accuracy of individual forecasting models such as decision trees. It selects a random sample of size n from the training set, fits it in the individual forecasting models, and produces a result that is averaged or voted on all individual forecasting models.
Comparison of Prediction Performance with Various STLF Models In this paper, we compare popular machine learning algorithms such as decision tree (DT), gradient boosting machine (GBM), bagging algorithm, and so on, to explain why we chose XGBoost and RF models in the first stage. Besides, we compare the performance with the prediction model (Persistence) which is actually using in the data collection environment. Persistence model uses the previous day (or the corresponding day in the previous week) as a prediction. Persistence implies that future values of the time series are calculated on the assumption that conditions remain unchanged between the current time and future time. As the second stage of our proposed model uses the predicted values of these two models, we divide the dataset into training set 1 (training the first-stage model), training set 2 (training the second-stage model), and test set (forecasting electric energy consumption and economic analysis), at a ratio of 50:25:25.
Specifically, data collected from January 2015 to December 2016 was used as training set 1, data collected from January 2017 to December 2017 was used as training set 2, and data collected from January 2018 to December 2018 was used as test set. The performance of each machine learning algorithm was measured using the training set 2. Figure  3  shows monthly energy consumption and divided dataset. In addition, we compare our proposed model with various STLF models composed of different machine learning algorithms in the first-stage, and several forecasting models from our previous studies in the secondstage. To do this, we divided the dataset into training and test sets, at a ratio of 75:25.
Specifically, data collected from January 2015 to December 2016 was used as training set 1, data collected from January 2017 to December 2017 was used as training set 2, and data collected from January 2018 to December 2018 was used as test set. The performance of each machine learning algorithm was measured using the training set 2. Figure  3  shows monthly energy consumption and divided dataset. In addition, we compare our proposed model with various STLF models composed of different machine learning algorithms in the first-stage, and several forecasting models from our previous studies in the second-stage. To do this, we divided the dataset into training and test sets, at a ratio of 75:25. Additionally, we selected a coefficient of variation of the root mean square error (CVRMSE) and mean absolute percentage error (MAPE) because they are easier to understand than other performance indicators such as the root mean square error (RMSE) or mean squared error (MSE)  [64].
Y i and Ŷi are the actual and predicted values, respectively. Figure  4  exhibits the comparison of CVRMSE and MAPE results for each machine learning algorithm. As shown in Figure  4 , XGBoost and RF models show better prediction performance in training set 2 compared with other machine learning algorithms. The performance of machine learning techniques is better than the persistence model which is a statistical model. In addition, the performance of the XGBoost and RF models was better than the other machine learning algorithms. XGBoost performed well because it allows users to choose an appropriate loss function depending on the characteristics of the data. RF performed well because it can handle high-dimensional data well. Table  8  summarizes the Pearson correlation coefficients between the forecasted values of machine learning algorithms and actual electric energy consumptions.
We found that the forecasted values of XGBoost and RF present higher correlation coefficients than those of other machine learning algorithms. Therefore, we used the forecasted values of XGBoost and RF as new input variables for the second stage. As shown in Figure  4 , XGBoost and RF models show better prediction performance in training set 2 compared with other machine learning algorithms. The performance of machine learning techniques is better than the persistence model which is a statistical model. In addition, the performance of the XGBoost and RF models was better than the other machine learning algorithms. XGBoost performed well because it allows users to choose an appropriate loss function depending on the characteristics of the data. RF performed well because it can handle high-dimensional data well.
Economic Analysis Based CCHP Operation Scheduling In this section, we describe how CCHP operation scheduling is made based on economic analysis. To maximize the annual economic benefits, it is also essential to determine the electric rate and amount of contract demand at the same time. We perform an experiment to find the optimal electric rate and contract demand to maximize on economic benefits. A monthly economic analysis using the test set confirms that the economic benefits are similar to the monthly energy consumption, as shown in Figure  5. We can see that high economic benefits can be obtained in summer and winter when energy consumption is high. Because the industrial building where the electric energy consumption data was collected is equipped with advanced meters, the electric rate of industrial service (A) II and industrial service (B) can be chosen.
In addition, since the building's supply voltage is between 3300 V and 66,000 V, we choose the high voltage A as the electric rate of the building. Industrial service (A) II has two options, and industrial service (B) has three options. As a result, five different electric rates are compared in the experiment. Figure  6  shows the annual economic benefit of each electric rate based on contract demand. Economic benefits ($) Because the industrial building where the electric energy consumption data was collected is equipped with advanced meters, the electric rate of industrial service (A) II and industrial service (B) can be chosen. In addition, since the building's supply voltage is between 3300 V and 66,000 V, we choose the high voltage A as the electric rate of the building. Industrial service (A) II has two options, and industrial service (B) has three options.
Conclusions In this study, we proposed a novel 2-stage STLF model that combines popular STLF models by using a DNN to further expand the domain of applicability. In the first stage, we used XGBoost and RF algorithms to predict day-ahead electric energy consumption. In the second stage, we built a load forecasting model based on DNN by using the forecasted results of XGBoost and RF and other external data as new input variables. To verify the forecasting performance of our proposed model, we performed day-ahead forecasting using actual factory electric energy consumption data and compared its accuracy with several machine learning methods and our previous forecasting models. The comparison showed that our proposed model showed the best prediction performance in terms of CVRMSE and MAPE. Additionally, to show the applicability of our model, we performed CCHP operation scheduling based on forecasting and economic analysis, decided the best electric rate and contract demand, and showed how much could be saved by the decision.
Polychromatic Flow Cytometry The antibodies used for cytometric analyses are listed in Supplemental Table  1. A total of 2310  6  frozen PBMCs were surface stained with antibodies specific for CD3, CD8, CD45RA, CCR7, CD27, CD28, and CD57. In addition to this cocktail, intracellular staining was performed using antibodies directed against T-bet, GZMB, PERF-1, and EOMES after fixation and permeabilization (Intracellular Fixation & Permeabilization Buffer Set; Thermo Fisher). A Yellow LIVE/DEAD Fixable Dead Cell Stain Kit was used to exclude dead cells from the analysis. BD CompBeads stained separately with individual mAbs were used to define the compensation matrix. Cells were analyzed with an LSRII flow cytometer (BD Immunocytometry Systems). Data were analyzed using FlowJo version 9. 7. 6 (TreeStar).
Sequencing was carried out as paired-end 2325-bp reads with an additional eight cycles per index. Data were separated by barcode and aligned using TopHat version 2. 0. 10 with the default settings. Transcripts were quantified by the Broad Technology Labs computational pipeline using Cuffquant version 2. 2. 1. 31 Briefly, data were processed through Cuffnorm if 50% of the reads aligned and if at least 100,000 pairs were aligned per sample. The default settings, including geometric normalization, were used for normalization and expression level data in the form of log2-transformed fragments per kilobase of transcript per million mapped fragment values were used for subsequent analyses. The selection of the most discriminative genes to classify the three CD8 subsets was performed using sparse partial least squares discriminant analysis (sPLS-DA)  32  with the mixOmics package in R.
33  A total of 100 repeats were used to establish the sPLS-DA model. Two components within each set of 100 genes were optimal to discriminate the three CD8 subsets. For gene expression representation, sample plots of the sPLS-DA and clustering analysis results were generated in R version 3. 5. 1 using the ade4  34, 35  and heatmap3 (https:// CRAN. R-project. org/package5heatmap3) packages, respectively. The biologic significance of selected genes was assessed using High-Throughput GoMiner. 36 Gene-ontology categories enriched with a false discovery rate of ,5% and containing at least five represented genes were selected. RNA-seq data were deposited in the Gene Expression Omnibus under the accession number GSE129356.