dataesr/scientific-paragraphs-categorization huggingface dataset:
In Fig. 3 (b) the network tries to reproduce a path seen during training but we first let the material rest at zero strain for five time steps before loading starts and for another five time steps at the end of the path. With purely data-driven latent dynamics, the initial rest disturbs the memory structure of the network and causes large deviations for a large portion of the path. For the rest at the end of the path, we see that the surrogate fails to predict the characteristic that the stress does not change upon constant deformation. Training data-driven models to accurately reproduce path dependency is therefore not straightforward: their latent representations of material state are not interpretable and even phenomena as trivial as resting at zero strain must be learned from data. At the core of successful applications of RNNs to this task are either extensive datasets obtained with carefully crafted sampling strategies  (Wu et al.
Neural network encoder For simplicity, we opt for modeling the evolution of 𝜽 using classical feedforward neural networks with fully-connected layers. As both architectures in Fig. 5  ultimately compute macroscopic stresses given macroscopic strains, we can use supervised learning to train the model with a straightforward Maximum Likelihood approach. Gathering the complete set of network weights in a vector 𝐰 and seeing the complete surrogate as a monolithic model that computes an approximation σ for stresses, we adopt the following observation model for the snapshot stresses 𝝈: 𝝈 = σ (𝜺, 𝐰) + 𝜉, 𝜉 ∼  ( 𝜉|𝟎, 𝛽 -1 𝐈 ) (13) where the superscript 𝛺 is dropped for convenience, 𝐈 is an identity matrix, and 𝜉 is an additive Gaussian noise. 4  Under the assumption of a squared loss, maximizing the likelihood of a training dataset with 𝑁 observations amounts to minimizing the loss function  (Bishop, 2006) : 𝐿 = 1 2 𝑁 ∑ 𝑛=1 ‖ ‖ ‖ 𝝈 𝑛 - σ ( 𝜺 𝑛 , 𝐰 ) ‖ ‖ ‖ 2 (14) with the variance of the noise that explains data misfit being simply 𝛽 = 𝑁∕2𝐿.
If  is implemented in a code base that allows for automatic differentiation (e. g. in PyTorch), these time dependencies are naturally taken into account as long as a persistent gradient tape is used within each strain path. 5  In this work we instead implement network training directly into an existing FE code, and therefore opt for the pragmatic approach of computing all partial derivatives of quantities derived from  using finite differences. Finally, in order to enforce upper and lower bounds for 𝜽 and avoid unphysical parameter values (e. g. negative elasticity moduli), we apply sigmoid activation to the final layer of the network and scale the parameters back from a [0, 1] range using predefined bounds: 𝜃 𝑖 = 𝜃 low 𝑖 + 𝜃 𝜎 𝑖 ( 𝜃 upp 𝑖 -𝜃 low 𝑖 ) (16).
Numerical examples The proposed model was implemented in an in-house Finite Element code developed using the open-source C++ numerical analysis library Jem/Jive  (Nguyen-Thanh et al. , 2020). In order to allow for seamless online retraining, network training was also implemented within the same code. We start this section by describing the datasets and model selection strategies used to build the surrogates. We then investigate the performance of the approach under several choices of encoders and decoders. Finally, we use the model within an FE 2 simulation and demonstrate the online stabilization procedure of Section 3. 5. All simulations are performed on cluster nodes equipped with Xeon E5-2630V4 processors and 128 GB RAM running CentOS 7.
We therefore limit ourselves to monotonic paths for training. For consistency, we also employ exclusively monotonic data to perform model selection. For efficiency, we limit the present investigation to 2D simulations (i. e. three strain components) in the plane perpendicular to the fibers, but nevertheless expect the discussion and conclusions to generalize to 3D simulations as long as appropriate orthotropic decoders are employed. Datasets with 2000 monotonic strain paths are generated under both plane strain and plane stress assumptions. Fig. 7  shows the complete plane strain dataset, with a similar one also being generated for plane stress. Each path is generated with an FE 2 simulation of a single macroscopic element under displacement control along a fixed direction in strain space sampled from a uniform distribution.
To circumvent convergence issues, we employ an adaptive time stepping technique that progressively reduces time step size when the simulation does not converge and gradually increases it back for subsequent increments. The simulations are stopped once a strain norm of 10 % is reached. As the adaptive scheme leads to paths with different numbers of time increments, we balance the dataset by ensuring every path is composed of 30 steps with strain norms as equally spaced as possible. To keep model selection straightforward and avoid the need for cumbersome k-fold cross validation or bootstrapping, we train a preliminary model with enough flexibility and an extensive training dataset and gradually increase the size of the validation set until the validation error converges to a good estimate of the expected prediction error  (Hastie et al.
, 2009). This results in validation sets with 500 paths selected at random from the original datasets, leaving 1500 paths to be used for training. We then perform model selection by gradually increasing the complexity of our FNN encoders until the validation error stabilizes. From experimenting with different architectures, we find that encoders with 5 hidden layers of 50 units each with Scaled Exponential Linear Unit (SELU)  (Klambauer et al. , 2017)  activation provide enough flexibility for all the examples treated here. To ensure enough regularization when computing learning curves with small datasets, we employ Bernoulli dropout layers with a rate of 1 % after every hidden layer. Networks are trained for 20 000 epochs using the Adam optimizer with recommended parameters  (Kingma and Ba, 2017)  and 32-path minibatches.
The model with lowest historical validation error is kept after training, further reducing the risk of overfitting on small datasets. To assess the capabilities of the trained surrogates, we compute an additional test dataset comprising 50 monotonic, 50 unloadingreloading and 50 slow cycling paths, examples of which are shown in Fig. 8. To keep the comparisons fair, none of these paths are used to perform model selection and are therefore only considered after the surrogates are trained. We will use example curves like those from Fig. 8  for visual inspection of the model performance, but also the complete sets of 50 curves each for more rigorous statistical analysis. A summary of the models considered in the following sections can be seen in Table  1. 𝜺 → 𝐼 𝜀 1 , 𝐼 𝜀 2 𝜺 → 𝐼 𝜀 1 , 𝐽 𝜀 2 𝜺 →  → εp 𝑥𝑥 𝜺 →  → 𝐼 σ 1 , 𝐽 σ 2.
Elastic decoder It is interesting to first consider the simple linear-elastic decoder of Eq. (  17 ), as it has no internal variables and therefore leads to a surrogate model comparable in nature to a conventional FNN trained on stress-strain pairs. As we will demonstrate, however, the limited physical bias provided by such simple model already proves advantageous. Here we let both elastic properties be controlled by the learned encoder: 𝜽 = [ 𝐸 𝜈 ] ( 23 ) where the bounds 10 1 < 𝐸 < 10 5 and 0 < 𝜈 < 0. 5 are enforced as described in Eq. (  16 ). We first perform a feature selection study and investigate how efficiently the model learns as the size of the dataset is increased. From the original plane strain training dataset of 1500 monotonic strain paths, we draw datasets with sizes ranging between 1 and 150 paths without replacement and use them to train networks with different encoder features.
To get a reliable estimate of the expected prediction error, we repeat this process 50 times for each dataset size and encoder type, and for comparison we also do the same for conventional FNNs trained directly on stress targets (keeping the same architecture but going directly from the final hidden layer to stresses). This amounts to a total of 3400 trained networks from which we can compute an estimate of the prediction error by averaging ‖ ‖ 𝝈 -σ‖ ‖ over the 500 paths left for validation. Fig. 9 (a) plots averages of the validation error over the 50 training datasets used for each size. Although the hybrid architecture does not show an advantage over the FNN when the encoder is trained on strain features, there is a clear gain in learning speed when using only the two first strain invariants as features.
Apart from accelerating learning and resulting in lossless dimensionality reduction, using invariants also results in a surrogate which is frame invariant under small strains. For comparison, we also train a conventional FNN on the same set of features, but those are unsurprisingly not enough to describe general strain states and much of the material response is interpreted by the FNN as observation noise. We zoom into the first part of the learning curves in Fig. 9  Training the invariant-based hybrid network with the complete dataset of 1500 curves leads to surrogates with validation errors of about 4 MPa, accurately representing the monotonic behavior of the original micromodel. Fig. 10  shows representative predictions of this model for paths from the test set. It is also interesting to plot how the latent space of the hybrid model evolves within a single strain path.
Here we are therefore concerned with the effect of trading the flexibility of an elastic decoder for significantly more physical bias from a lower-fidelity representation of material behavior. 𝜽 = [ 𝜎 y ] ( At this point it is interesting to compare the performance of the hybrid surrogate with predictions coming from the state-of-the-art mesoscale material model for polymer composites proposed by  Vogler et al. (2013). It is an orthotropic elastoplastic model with pressuredependent non-associative flow precalibrated with a small number of monotonic uniaxial and biaxial stress-strain curves obtained from simulations on the exact same micromodel of Fig. 6  (see  Van der Meer, 2016  for details on the calibration procedure). For this section, we switch to a dataset in plane stress, allowing the 𝐽 2 model to describe richer nonlinear behavior under strain states.
This result is not entirely surprising, as the elastic decoder introduces much less bias into the model and therefore allows for a greater degree of flexibility when fitting monotonic data. On the other hand, what cannot be directly gleaned from Fig. 14  is that the 𝐽 2 decoder benefits from having physics-based memory coming from its internal variables that allows for making predictions of non-monotonic behavior based solely on our assumption that nonlinearities come from plastic strain and therefore without ever having to see it during training. In Fig. 16  we plot predictions of the 𝐽 2 surrogate for two different unloading-reloading paths from the test dataset. The model predicts unloading very well without being trained for it. Nevertheless, as Fig. 14  suggests, the model struggles to predict monotonic behavior under a number of different scenarios, from which it follows that any non-monotonic predictions along the same directions will also be inaccurate.
[ 𝐼 𝜎 1 𝐽 𝜎 2 ] → Melro). We also include predictions from the precalibrated mesomodel by  Vogler et al. (2013)  and selected curves from Fig. 9 (a) for comparison purposes. As before, we train 50 networks of each type for each size of dataset ranging from 1 to 150 paths drawn from the original dataset with 1500 paths. Each trained network is then used to compute the validation error over the 500 monotonic validation paths and the 150 test paths (50 extra monotonic paths, 50 paths with unloading-reloading and 50 slow cycle paths). This results in an extensive study comprising 6800 trained networks and over one million test set simulations. are summarized in Fig. 18 , with each point in a curve being the average over 50 networks. Once again using invariants as features proves beneficial, leading to lossless dimensionality reduction and frame invariant surrogates.
All tested models perform better than the precalibrated mesomodel, with a gap of more than one order of  magnitude for the best performing surrogates. Interestingly, models with Melro-based decoders seem to learn as fast and be as flexible as models with elastic decoders, already for the monotonic curves in the validation dataset. This suggests that the new decoder does not impose extra undesirable bias in learning the specific material behavior treated here other than the assumptions that had already been introduced by elasticity (e. g. symmetries and couplings encoded by the elastic stiffness tensor). Any benefits reaped when extrapolating to non-monotonic paths, as we will see in the following, are therefore obtained at a negligible price in terms of monotonic behavior accuracy. This stands in contrast with the discussion on the 𝐽 2 decoder of the previous section.
Although  Fig. 18  is not enough to discern between several of our encoder choices, it is interesting to take a closer look at the two clearly underperforming options. Fig. 19   → Melro for the same monotonic test path. The model with a single feature struggles to predict the entirety of the path, indicating that further reducing the dimensionality of the feature space is not possible for this dataset. The oscillatory stress predictions make this model unsuitable for online stress evaluation in a multiscale setting. For the model with plastic strain features, the feature extractor shows no plastic strains until high stress levels while in the micromodel plasticity starts much earlier, forcing the surrogate to remain in the elastic regime until a sudden jump brings it back to the expected path.
Moving to unloading-reloading paths, we compare the performance of different feature sets by plotting the average test error over the 50 unloading-reloading paths in Fig. 20(a). Here an interesting observation can be made: even the surrogate [ 𝐼 𝜀 1 𝐼 𝜀 2 ] → Elastic -which cannot predict unloading at all -attains a lower test error than the precalibrated mesomodel. This apparent contradiction can be explained by plotting in Fig. 20(b ) the average error computed only at unloading or reloading time steps: use of an elastic decoder -and therefore of a conventional FNN or an RNN trained with insufficient data -excels at predicting monotonic response but is consistently inaccurate for nonmonotonic paths and shows little improvement when more monotonic paths are added to the training dataset. In contrast, the best-performing Melro models are consistently more accurate than the precalibrated mesomodel even when trained on very little data.
We plot in Fig. 21  selected representative unloading paths from the test dataset for four of the surrogates. Unloading is once again well captured without having been seen during training, and since it emerges from a purely physical mechanism, it is reasonable to expect unloading at different points along the path to yield comparable results (c. f. Fig. 3(a) ). Nevertheless, relatively small differences in unloading slope can still lead to large differences in stress at the end of the unloading branches. Furthermore, the model can struggle with tension-compression switches and predict spurious hysteresis loops. Indeed, we observe a consistent inability by the models to properly predict switches between tension and compression within the same path. This becomes clear when looking at slow cycling test paths composed of several of these switches (Fig.
8(c) ). We plot learning curves for the test on slow cycling paths in Fig. 22 , for complete paths as well as exclusively for the non-monotonic branches of the paths. In contrast with results up until now, here we see larger differences in performance for different feature sets. As expected, elastic decoders are once again shown to be unsuitable to predict non-monotonic paths, and the difference here is even more pronounced than in for singleunloading paths (c. f. Fig. 20 ) as most of the path is composed of unloading/reloading branches. The model encoded with stress invariants coming from an elastoplastic feature extractor performs best among the models we test. But crucially, none of the surrogates manages to surpass the precalibrated mesomodel in this case. As a demonstration, we select a representative path from the test dataset and plot predictions made with four different feature sets in Fig.
26(a). In contrast, the stabilization procedure of Section 3. 5 allows for a complete path to be obtained. For this first result, we stabilize the network for 5 epochs with a learning rate of 1 × 10 -5 for the stabilization loss (Eq. (  22 )) and 1 × 10 -9 for retraining on a single monotonic training path selected at random. We also consider a model with an unloading/reloading switch after the onset of macroscopic plasticity. Results are shown in Fig. 26(b). The surrogate approximates the full-order behavior fairly accurately 9  and several orders of magnitude faster than the full-order model. It is worthwhile to further comment on the obtained levels of acceleration. On average, the computation of a single monotonic training path takes approximately 250 s, and training a surrogate for 20 000 epochs with the complete 1500-path dataset takes approximately 51 h on a single computing core.
This would amount to a total offline computation time of 155 h, higher than the required for the FE 2 simulations of Fig. 26. This would however not be a consistent comparison. The hybrid nature of the proposed approach allows for training accurate surrogates with only a small fraction of this complete dataset. Indeed, Fig. 18  suggests a surrogate trained on 20 paths would offer comparable performance, which would amount to a total offline computation time (dataset generation and training) of approximately 2 h. Furthermore, this offline cost does not scale with the density of the macroscopic mesh being considered or with the number of macroscopic simulations being performede. g. in a parametric study or design optimization procedure. We now look closer on the performance of the proposed online stabilization approach.
We empirically find that retraining the network until every violating material point is fully stabilized is not strictly necessary in order to achieve convergence, and therefore opting for a small number of stabilization epochs proves to be an efficient approach. It is nevertheless interesting to investigate the impact of the number of stabilization epochs and of the subsequent retraining minibatch on the original dataset. We solve the monotonic example of Fig. 26(a ) with different numbers of stabilization/retraining epochs ranging from 2 to 100 and compute the validation loss (on the 500-path validation set used for model selection) at the end of every macroscopic time increment in order to keep track of how much the stabilized network deviates from its original pretrained state. Results are shown together with the corresponding load -displacement curves in Fig.
In order to accurately separate the different groups, one should feed the clustering algorithms with the correct objective number of clusters; nonetheless, there is no true solution to estimate that value without upstream manual processing. Additionally, the grain of the desired partitioning (i. e. , comprising only the regular states, or also the anomalies within smaller groups) will affect that value: it is difficult to answer the question "how many behaviors and anomalies are contained within this N-month long dataset?" with absolute certainty, and especially in a fully automatic and blind fashion. For that reason, this paper introduces a new unsupervised clustering methodology, specifically crafted to answer the above-mentioned difficulty of setting the objective number of clusters. This method consists of recursively splitting a space according to a split criterion: the space is partitioned a first time, then each cluster is analyzed using an integrity criterion, and, if it obtained a bad score, the cluster is itself split, and so on until the integrity of every issued cluster is acceptable.
0 solutions. The manuscript is composed as follows: Section 2 is a detailed review of the literature on anomaly detection, the techniques to estimate the number of clusters of a dataset and the clustering techniques which do not require it. Then, the tools necessary for the proposed methodology are presented in Section 3, and the proposed SPRADA method is introduced and detailed in Section 4, and it is applied to a real industrial dataset in Section 5 to assess its accuracy and representativeness, and the findings are discussed in Section 6. Finally, Section 7 concludes this study.
Eventually, some works used the real-time data collected by a Supervisory Control and Data Acquisition (SCADA) system to detect possible measurement intrusion, by using a classical Long Short-Term Memory (LSTM)  [38] , or by using supervised classification to identify both normal and abnormal activities, and finally learn the possible intrusions using supervised machine learning-based approaches  [39]. All these techniques perform well, achieving promising results in the identification and prediction of anomalies, but they are computationally demanding, require a significant amount of data (a typical limitation in deep learning), and are supervised. Therefore, they are not suited to the fully-automatic and blind identification of the distinct anomalies-and behaviors-of industrial systems. That being said, the proposed SPRADA methodology may serve as a first fully-automated and non-parametric pre-processing tool to identify the historical anomalies, which may later on be fed to the works mentioned in the previous paragraph, whence the necessity of such a tool, and the correct estimation of the true number of groups (both anomalies and behaviors) in real industrial processes' datasets.
Among others, some greatly popular criteria are the silhouette coefficients, which are integrity indicators providing, for every datapoint, a score indicating how close it is from its same cluster's neighbors and how distant it is from the other clusters' data. This approach gave birth to a dedicated clustering method, named silhouette clustering  [44] , which proposes to converge toward an optimal clustering (in the sense of the silhouette coefficients) by exhaustively and iteratively swapping the clusters' barycenters and any other data to see if any improvement is obtained by this swap  [45]. An example of application can be found in  [43] , who simulated a dataset composed of a certain number of noisy Gaussian distributions, and endeavored to retrieve the number of groups using the silhouettes-based version of the Elbow method; they obtained very good results, but the main problem with this method is its computational heaviness  [46] , since the silhouette coefficients themselves take a long time to compute (O(N 2 ), with N the number of data), and the comparisons to every pair barycenter-data (O(kN), with k the number of clusters) are numerous.
Therefore, the complexity of the algorithm is in O(kN 3 i), with i the number of iterations before convergence, which does not make it well-suited to large datasets such as industrial ones. In addition, another well-known method is the Gap statistic  [47]. This exploratory analysis technique relies on a simulated ground truth, obtained by generating a reference dataset under the null hypothesis, i. e. , with the same statistical characteristics than the original one. Then, both the original and simulated datasets are clustered with a same algorithm, with the same value set for the number of clusters, and the average dispersion (for instance the mean distance between any possible pair of points) of the clusters for both partitions are compared. This procedure is repeated for several values as number of clusters, and that leading to the lowest difference between the mean dispersion of both partitions is chosen as the optimal number of clusters to build.