dataesr/scientific-paragraphs-categorization huggingface dataset:
Optimization Strategy REGENN operates on a tridimensional space shared between samples, timestamps, and variables. In such a space, it carries out a time-based optimization strategy. The training process iterates over the time-axis of the dataset, showing to the network how the variables within a subset of time-series behave as time goes by and later repeating the process through subsets of different samples in distinct batches of preset window-size. However, the shared graph A is built during the first epoch of the training phase. Such a process happens in parallel to the optimization process, not impacting training stability nor convergence. After that point, A will remain as it is during the training and testing. The network's weights are shared among the entire dataset and optimized towards best generalization simultaneously across samples, timestamps, and variables.
We used Adam  [50] , a gradient descent-based algorithm, to optimize the model and, as the optimization criterion, the Mean Absolute Error (MAE), which is a generalization of the Support Vector Regression  [51]  with soft-margin criterion where Ω is the set of internal parameters of REGENN, Y is the network's output, and T the ground truth: minimize Ω s i=1 w j=1 Y i,j -T i,j (6) Due to the SARS-CoV-2 data behave as a streaming dataset, we adopted a transfer learning approach to train the network on that dataset. Transfer learning shares knowledge across different domains by using pre-trained weights of another neural network. The approach we adopted, although different, resembles Online Deep Learning  [52]. The main idea is to train the network on incremental slices of the time-axis, such that the pre-trained weights of a previous slice are used to initialize the weights of the network in the next slice.
Datasets The results are based on three datasets, all of which are multi-sample, multivariate, and vary over time. The first dataset describes the Coronavirus Pandemic, referred to as SARS-CoV-2, made available by John Hopkins University  [53]. It describes 3 variables through 120 days for 188 countries and varies from the first day of the pandemic to the day it completed four months of duration. The second one is the Brazilian Weather dataset collected from 253 sensors during 1,095 days regarding 4 variables. The third dataset is from the 2012 PhysioNet Computing in Cardiology Challenge  [54] , from which we are using 9 variables across 48 hours recorded from 11,988 ICU patients. The variables within the datasets are: The number of datasets that can be used for multiple multivariate time-series forecasting is still limited.
Although the ones we experimented with have a small number of variables, there is no theoretical upper bound for the number of samples, timestamps, and variables REGENN can handle. However, as REGENN deals with dense tridimensional tensors, increasing any of the axes (samples×time×variables) in size will make the tensor grow at an almost-cubic rate, quickly overflowing the GPU Memory. Accordingly, RE-GENN ends up suffering from a trade-off between the dataset's size and the available hardware.
Data Pre-processing The datasets were individually min-max normalized into a zero-one scale on the variable-axis to avoid gradient spikes and speed up training. We used the training data to define the min-max parameters required for normalization. Such parameters were applied for normalizing the test data as well. As a consequence of normalizing the entire dataset, the network's output will follow a similar scale. Therefore, the output was inversely transformed to what it was before the normalization for evaluating the results. A simplistic yet effective approach to train time-series algorithms is through the Sliding Window technique  [55] , also referred to as Rolling Window. The window size is well known to be a highly sensitive hyperparameter  [56] ,  [57]. Consequently, we followed a non-tunable approach, in which we set the window size before the experiments, just taking into consideration the context and domain of the datasets.
These values were used across all windowbased experiments, including the baselines and ablation tests. It is noteworthy that most of the machine-learning algorithms are not meant to handle time-variant data, such that no sliding window was used in those cases. Conversely, we considered training timestamps as features and those reserved for testing as multi-task regression labels. On the deep learning algorithms, we used a window size of 7 days for training and reserved 7 days for validation (between the training and test sets) to predict the last 14 days of the SARS-CoV-2 dataset. The 7-7-14 split idea comes from the disease incubation period, which is of 14 days. On the other hand, we used a window size of 84 days and reserved 28 days for validation to predict the last 56 days in the Brazilian Weather dataset.
The 84-28-56 split is based on the seasonality of the weather data, such that we will look to the previous 3 months (a weather-season window) to predict the last 2 months of the upcoming season. Finally, we used a window size of 12 hours for training and 6 hours for validation to predict the last 6 hours of the PhysioNet dataset. The 12-6-6 split comes from the fact that patients in ICUs are in a critical state, such that predictions within 24 hours are more useful than long-term predictions.
The tuning process follows by updating the hyperparameters whenever observing better results on the validation set, leading us to a set of optimized but no optimum hyperparameters. We used the set of optimized hyperparameters to evaluate REGENN on the test set and the default values for all the other algorithms  [49] ,  [58] ,  [59]  unless explicitly required for working with a particular dataset, as was the case of LSTNet  [39] , DSANet  [40] , and MLCNN  [42]. The complete list of hyperparameters is in the Supplementary Material.
Computer Environment The experiments related to machine-learning and timeseries algorithms were carried out on a Linux-based system with 64 CPUs and 750 GB of RAM. The experiments related to deep-learning on the SARS-CoV-2 dataset were carried out on a Linux-based system with 56 CPUs, 256 GB of RAM, and 8 GPUs (Titan X -Pascal). The Brazilian Weather and PhysioNet datasets were tested on a different system with 32 CPUs, 512 GB of RAM, and 8 GPUs (Titan X -Maxwell). While CPU-based experiments are even across all CPU architectures, the same does not hold for GPUs, such that the GPU model and architecture must match to guarantee reproducibility. Aiming at complete reproducibility, we disclose not only the source code of REGENN on GitHub 2 , but also the scripts, pre-processed data, and snapshots of all trained networks on a public folder at Google Drive 3.
Results Subsequently, we go through the experimental results in each one of the benchmark datasets. Due to the fact that not all the algorithms perform evenly across them, we display the 34 most prominent ones out of the 49 tested algorithms; for the extended results, please refer to the Supplementary Material. We also discuss the ablation experiments, which were carried out with REGENN's hyperparameters; in the Supplementary Material, we provide two other rounds of this same experiment using as hyperparameters PYTORCH's defaults 4   the GSE layers, by using the cosine similarity on the adjacency matrices of co-occurring variables (see Section 2. 2).
Experiment on SARS-CoV-2 dataset The SARS-CoV-2 has being updated daily since the beginning of the Coronavirus Pandemic. We used a self-to-self transfer-learning approach to train the network in slices of time due to such a dataset's streaming nature. In short, the network was re-trained every 15 days with new incoming data, using as starting weights, the pre-trained weights from the network trained in the past 15 days so to evaluate REGENN's performance over the pandemic's progression. In such a case, when the pandemic completed 45 days, the first time-slice in which REGENN was trained, it outperformed the second-placed algorithm, the Orthogonal Matching Pursuit, by 27. 27% on the Mean Absolute Error (MAE), 16. 50% on the Root Mean Square Error (RMSE), and 38. 87% on the Mean Squared Logarithmic Error (MSLE).
For all subsequent time-slices (i. e. , 60, 75, 90, 105, and 120 days), the second-placed algorithm was the Exponential Smoothing, which was outperformed with improvement no lower than 47. 40% on the MAE, 17. 19% on the RMSE, and 37. 39% on the MSLE. We further detail the results on the time-slices mentioned above in the Supplementary Material. However, in Fig. 5 , we detail the results on the complete dataset of 120 days, in which REGENN surpassed the Exponential Smoothing, the second-best algorithm, by 75. 21% on the MAE, 64. 87% on the RMSE, and 79. 61% on the MSLE. As a result of the analysis of the dataset in time-slices, we noticed that, as time goes by and more information is available on the SARS-CoV-2 dataset, the problem becomes more challenging to solve by looking individually at each country and more natural when looking at all of them together.
The problem did not go away, but more infected people were discovered as more tests were made, and also because the disease spread worldwide. A similar scenario can be drawn from the number of recovered people and the number of cases, as infected people with mild or no symptoms were unaware of being infected. Contrarily, we can see that the similarity between recovered and deaths 0 5 10 15 20 25 Fig. 7 : Baseline results for the Brazilian Weather dataset presented in descending order of MAE. In this experiment, REGENN once more outperformed all the competing algorithms, demonstrating versatility by performing well even on a highly-seasonal dataset with improvement no lower than 11. 95%. In the face of seasonality, the Elman RNN surpassed the Exponential Smoothing, the previously second-best algorithm.
Experiment on Brazilian Weather dataset The Brazilian Weather dataset is a highly seasonal dataset with a higher number of samples, variables, and timestamps than the previous one. For simplicity's sake, in this experiment, REGENN was trained on the whole training set at once. The results are in Fig. 7 , in which REGENN was the first-placed algorithm, followed by the Elman RNN in second. REGENN overcame the Elman RNN by 11. 95% on the MAE, 11. 96% on the RMSE, and 25. 84% on the MSLE. We noticed that all the algorithms perform quite similarly for this dataset. The major downside for most algorithms comes from predicting small values close to zero, as noted by the MSLE results. In such a case, the ensembles showed a high variance when compared to REGENN. We believe this is why the Elman RNN shows performance closer to REGENN rather than to Exponential Smoothing, the third-placed algorithm, as REGENN has a single estimator, while the Exponential Smoothing is an ensemble of estimators.
Another understanding of why some algorithms underperform on the MSLE is related to their difficulty to track temporal dependencies, such as weather seasonality. The ablation results are in Tab. 3, in which we observed again that the network without the GSE layers already surpasses the baselines. When decommissioning the GSE layers of REGENN and using GRU instead of LSTM on the decoder, we observed a 3. 86% improvement on the MAE, 10. 02% on the RMSE, and 25. 34% on the MSLE when compared to the Elman RNN results. Using REGENN instead, we achieve a further performance gain of 8. 42% on the MAE and 2. 16% on the RMSE over the ablation experiment. Fig. 8  depicts the Evolution Weights for the current dataset, in which we can observe a consistent similarity between pairs of variables in the input graph, which does not repeat in the evolved graph, implying different relationships.
Experiment on PhysioNet dataset The PhysioNet dataset presents a large number of samples and an increased number of variables, but little information on the time-axis, a setting in which ensembles still struggle to perform accurate predictions, as depicted in Fig. 9. Once again, REGENN keeps steady as the first-placed algorithm in performance, showing solid improvement over the Linear SVR, the second-placed algorithm. The improvement was 7. 33% on the MAE and 35. 13% on the MSLE, while the RMSE achieved by REGENN laid within the standard deviation of the Linear SVR, pointing out an equivalent performance between them. The Linear SVR is an ensemble of multiple estimators, while REGENN uses only one, making it better accurate for dealing with the current dataset. As in Tab. 4, the ablation results reveal that a neural network architecture without the GSE layers can achieve a better performance than the baseline algorithms.
In this specific case, we see that by using a bidirectional LSTM instead of unidirectional on the decoder module of the neural network, we can achieve a performance almost as good as REGENN, but not enough to surpass it, as REGENN still shows an improvement of 1. 05% on the MAE and 0. 98% on the RMSE over the experiment with bidirectional LSTM. In this specific case, REGENN learns by observing multiple ICU patients. However, one cannot say that an ICU patient's state is somehow connected to another patient's state. Contrarily, the idea holds as in the first experiment, 10 20  30   Fig. 9 : Baseline results for the PhysioNet Computing in Cardiology Challenge 2012 dataset presented in descending order of MAE, in which REGENN was the algorithm with the best performance followed by the Linear SVR.
Consequently, training might take long sessions, even with consistently reduced learning rates on plateaus or simulated annealing techniques; this is influenced by the fact that the second GSE layer has two dynamic inputs, which arise from the graph-evolution process. However, we observed that throughout the epochs, the Evolution Weights reach a stable point with no major updates. As a result, the network demonstrates a remarkable improvement in its final iterations when the remaining weights intensely converge to a near-optimal configuration. Even though REGENN has a particular drawback, it demonstrates excellent versatility, which comes from its superior performance in the task of epidemiology modeling on the SARS-CoV-2 dataset, climate forecasting on the Brazilian Weather, and patient monitoring on intensive care units on the PhysioNet dataset.
Because of that, REGENN is a precursor in multiple multivariate time-series forecasting and, even though this is a challenging problem, REGENN surpassed all the baselines and remained effective after three rounds of 30 ablation tests through distinct hyperparameters. The experiments were carried out over the SARS-CoV-2, Brazilian Weather, and PhysioNet datasets with improvements, respectively, of at least 64. 87%, 11. 96%, and 7. 33%. As a consequence of the results, REGENN shows a new range of possibilities in time-series forecasting, starting by demonstrating that ensembles poorly perform if compared to a single model able to learn the entanglement between different variables by looking at how they interact as time goes by and how multiple multivariate time-series synchronously evolve. Notes.
Nevertheless, the window size of the Sliding Window is well known to be a highly sensitive hyperparameter  [8, 9] ; to avoid an increased number of hyperparameter, we followed a non-tunable approach, in which we set the window size before the experiments taking into consideration the context of the datasets; such values were even across all window-based trials, including the baselines and ablation. Optimization Strategy. ReGENN operates on a three-dimensional space shared between samples, time, and variables. In such a space, it carries out a time-based optimization strategy. The training process iterates over the time-axis of the dataset, showing to the network how the variables within a subset of time-series behave as time goes by, and later repeating the process through subsets of different samples.
The network's weights are shared among the entire dataset and optimized towards best generalization simultaneously across samples, time, and variables. The dataset T ∈ R s×t×v is sliced into training T ∈ R s×w×v and testing T ∈ R s×z×v as follows: T =        T 1,1,v T 1,2,v T 1,2,v. T 1,w,v T 2,1,v T 2,2,v T 2,2,v. T 2,w,v T 3,1,v T 3,2,v T 3,2,v. T 3,w,v. T b,1,v T b,2,v T b,2,v. T b,w,v        T =        T 1,1+z,v T 1,2+z,v T 1,2+z,v. T 1,w+z,v T 2,1+z,v T 2,2+z,v T 2,2+z,v. T 2,w+z,v T 3,1+z,v T 3,2+z,v T 3,2+z,v. T 3,w+z,v. T b,1+z,v T b,2+z,v T b,2+z,v. T b,w+z,v        Once the data is sliced, we follow by using a gradient descent-based algorithm to optimize the model. In this work's scope, we used Adam  [10]  as the optimizer, as it is the most common optimizer among time-series forecasting problems.
As the optimization criterion, we used the Mean Absolute Error (MAE), which is a generalization of the Support Vector Regression  [11]  with soft-margin criterion formalized as it follows: minimize w 1 2 w 2 F + C × n i=1 (ξ i + ξ * i ) subject to y i -(w • x i ) -b ≤ ρ + ξ i , (w • x i ) + b -y i ≤ ρ + ξ * i , ξ i , ξ * i ≥ 0. where w is the set of optimizable parameters, • F is the Frobenius norm, and both C and ρ are hyperparameters. The idea, then, is to find w that better fit y i , x i ∀i ∈ [1, n] so that all values are in [ρ + ξ i , ρ + ξ * i ], where ξ i and ξ * i are the two farther opposite points in the dataset. A similar formulation on the Linear SVR implementation for horizon forecasting was presented by Lai et al. [3]. Due to the higher-dimensionality among the multiple multivariate time-series used in this study, in which we consider the time to be continuous, the problem becomes: minimize Ω 1 2 Ω 2 F + C × s i=1 w j=1 ξ i subject to Y i,j -T i,j ≤ ρ + ξ i,j , ξ i,j ≥ 0.
where Ω is the set of internal parameters of ReGENN, Y is the network's output and T the ground truth. When disregarding C and setting ρ as zero, we can reduce the problem to the MAE loss formulation: minimize Ω s i=1 w j=1 Y i,j -T i,j Pay Attention to Evolution: Time Series forecasting with Deep Graph-Evolution Learning Square-and logarithm-based criteria can also be used with ReGENN. We avoid doing so, as this is a decision that should be made based on each dataset. Contrarily, we follow the SVR path towards the evaluation of absolute values, which is less sensitive to outliers and enables ReGENN to be applied to a range of applications. Transfer-Learning Approach. We adopted a Transfer Learning approach to train the network on the SARS-CoV-2 dataset that, although different, resembles Online Deep Learning  [12].
Time Axis Baselines Algorithms. Open-source Python libraries provided the time series and machine learning algorithms used along with the experiments. Time series algorithms came from the statsmodels 1  , while the machine learning ones majorly from the Scikit-Learn 2. Further algorithms, such as XGBoost 3  , LGBM  4  , and CatBoost 5 , Pay Attention to Evolution: Time Series forecasting with Deep Graph-Evolution Learning have a proprietary open-source implementation, which was preferred over the others. We used the default hyperparameters over all the experiments, performing no fine-tuning. However, because all the datasets we tested are strictly positive, we forced all the negative output to become zero, such as made by a ReLU activation function. A list with names and algorithms tested along with the experiments is provided in Tab 1, which contains more algorithms than we reported in the main paper.
90 and 105 days, and also on the complete SARS-CoV-2 dataset. 21 11 Ablation on the first 45 days of the SARS-CoV-2 dataset. 22 12 Ablation on the first 60 days of the SARS-CoV-2 dataset. 23 13 Ablation on the first 75 days of the SARS-CoV-2 dataset. 24 14 Ablation on the first 90 days of the SARS-CoV-2 dataset. 25 15 Ablation on the first 105 days of the SARS-CoV-2 dataset. 26 16 Ablation on the complete SARS-CoV-2 dataset. 27.
ACKNOWLEDGMENTS This work was partially supported by the  Coordenação de Aperfeiçoamento de Pessoal de Nível Superior -Brazil  ( CAPES) -Finance Code 001 ;  Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) , through grants  2014/25337-0 ,  2016/17078-0 ,  2017/08376-0 ,  2018/17620-5 ,  2019/04461-9 , and  2020/07200-9 ;  Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)  through grants  167967/2017-7 ,  305580/2017-5 , and  406550/2018-2 ;  National Science Foundation  awards IIS-2014438, PPoSS-2028839, and IIS-1838042; and, the  National Institute of Health awards NIH   R01 1R01NS107291-01 , and  R56HL138415. We thank  Jeffrey Valdez  for his aid with Sunlab's computer infrastructure,  Lucas Scabora , for his careful review of the paper, and  Gustavo Merchan, M.
88 5. 25 2. 83 1. 92 1. 95 2. 00 2. 69 5. 04 5. 42 5. 75 7. 42 8. 19 5. 54 5. 77 10. 24 10. 69 6. 17 4. 86 4. 85 4. 95 4. 68 0. 28 0. 29 0. 41 0. 80 0. 96 0. 30 0. 41 1. 61 1. 75 0. 50 0. 28 0. 27 0. 27 0. 37 OVERALL IMPROVEMENT 8. 42% 2. 16% 0. 00% Ablation results from experimenting over the Brazilian Weather dataset that was conducted by varying the Recurrent Unit (RU) and its directional flag between Bidirectional (B) and Unidirectional (U). We use E to designate the non-conventional Transformer Encoder and AR the feed-forward Autoregressive linear component.
49 18. 88 18. 42 18. 46 18. 54 19. 78 18. 55 18. 63 18. 88 24. 85 18. 22 18. 23 18. 48 24. 44 50. 46 48. 55 47. 78 48. 33 47. 35 51. 29 48. 28 48. 93 50. 31 56. 41 47. 31 47. 45 49. 72 53. 92 1. 40 1. 40 1. 37 1. 36 1. 38 1. 44 1. 38 1. 37 1. 39 2. 92 1. 37 1. 37 1. 37 2. 08 OVERALL IMPROVEMENT 1. 05% 0. 98% 0. 0% Ablation results over the PhysioNet Computing in Cardiology Challenge 2012 dataset, in which the Recurrent Unit (RU) is changed together with its directional flag, varying between Bidirectional (B) and Unidirectional (U). We use E as a shortening for the Transformer Encoder and AR for the Autoregressive linear component.
In this work, we moved one step forward by coupling for the first time the newly developed TOUGH3  [21]  with versions 6. 0 and 7. 0 of FLAC3D  [29] , hence implementing a parallelized version of the well-known coupled simulator. The coupled simulator integrates all the new functionalities of TOUGH3, including the use of PETSc solvers, together with the improved solver performance in FLAC3Dv6/7 as well as the possibility of using Python scripting compared to the FISH programming embedded in previous versions of FLAC3D. After verifying the correctness of the approach comparing the simulation results with an analytical solution and with the previous version, we evaluate the performance of the newly developed approach. Finally, we present results of two case studies, aimed at understanding the potential for fault reactivation during CO 2 sequestration and the evolution of stress and strain during nuclear waste disposal in a deep geological repository.
Single codes parallel computing performance TOUGH3 is parallelized by means of MPI, while FLAC3D uses threaded processes. Up to very recently, FLAC3D only ran on Windows operating system, but can now be run on Linux -based systems. For a Windows-based system, to make use of the MPI parallelization provided in TOUGH3, the code can be compiled in Cygwin or Windows Subsystem for Linux (Windows 10 and 11). For the current manuscript, we used WSL1. Each of the two codes was evaluated individually. The speed increase was calculated with respect to the single MPI process/thread calculation. Results are shown in Fig. 1. We evaluated TOUGH3 on two different operating systems (OS) but on an identical hardware configuration: a 32-core virtual workstation with AMD EPYC 7742 at 2. 25 GHz and equipped with 64 GB RAM.