dataesr/scientific-paragraphs-categorization huggingface dataset:
5. RSGPLS. An approach developed by  Durif et al. (2015). It consists of adapting the SGPLS method by introducing a ridge penalty to ensure convergence of parameter estimations, and stability in hyper-parameter tuning. They also propose an adjustment of the L 1 constraint in order to further penalize less significant predictors. Hyperparameters are tuned using CV MSE. 6. Lasso. The adapted lasso regression for logistic framework, available in the R package glmnet, as a benchmark. Concerning bootstrap-based approaches, the incorporation of PLS methodology into GLM, developed by  Bastien et al. (2005) , was used. Due to non-convergence issues for parameter estimations, some samples were excluded using a threshold for parameter estimations. Indeed, a model built using a bootstrap sample with at least one parameter estimate that is higher in absolute value than 10 4 times the one (in absolute value as well) estimated on the original dataset, leads to the exclusion of this bootstrap sample.
Each method was performed one hundred times in order to obtain relevant results. However, due to the high observed stability of results extracted with the BootYTdyn approach, and in order to save computational time, this was only performed twenty times instead of a hundred. Stability results for tuning parameter selection, except for the bootstrap-based methods, are shown in Fig. 7     Based on results summarized in Table  9 , the lasso methodology, using CV-based MSE or deviance values for the selection of its hyper-parameter, is the most stable. This can be In the following, two additional independent public datasets, stated by  Marisa et al. (2013)  as being comparable with our original dataset, were included for comparison of predictive abilities. These datasets are named GSE18088 (n=53)  (Gröne et al.
, 2011)  and GSE14333 (n=247)  (Jorissen et al. , 2009). Both the MSE and number of misclassified (MC) value of the selected models were computed on both the training and test parts of the original dataset, as well as on the two additional datasets. These results are summarized in Table  12. Since independent datasets were available, we decided to follow the suggestion of  Van Wieringen et al. (2009 , p. 1596) : "the true evaluation of a predictor's performance is to be done on independent data". In this real data study, BootYTdyn retains only one predictor, which is also retained by all other methods. Thus, as expected, this most sparse support induced the highest values of both the MSE and number of misclassified observations, based on the training subset of the original dataset. It also provided the highest values based on the test subset of the original dataset, which is to be expected since, as explained in Section 3.
2. 1, these two parts are well-balanced in terms of anatomo-clinical characteristics. Thus, this causes a bias in evaluation of model's predictive abilities, by making comparable MSE and misclassified values based on both the training and testing subsets. Results in Table  12  confirm this property, and also highlight the usefulness of additional independent datasets for reliably comparing predictive abilities. While a well-designed model for predictive purposes will provide similar PMSE values on comparable independent additional datasets compared to MSE obtained on the training dataset, an over-fitted one would be related to higher PMSE values, due to its dependance on training data. Thus, the differences, noted ∆ MSE, between the MSE obtained on the training subset of the original dataset, and those obtained on the three test datasets, are shown in Fig.
9. Our new dynamic bootstrap-based approach is the only one that exhibits this MSE stability property, while all others provided higher PMSE values on the two additional datasets than on the original one. This led BootYTdyn to have only 48 (16%) misclassified values on these two additional datasets, which represents the best result among all studied approaches. Thus, we can reasonably assume that our new method has helped to remove non-informative predictors, in the sense of not being relevant for improving predictive ability. Lastly, the unique extracted probe set, named 230784 at, is already known to be related to the original location in the distal colon. The sign of the regression coefficient obtained with BootYTdyn is coherent with this state-of-the-art result, and thus strengthens our conclusion.
In the PLS framework, conclusions based on our simulations are twofold. First, for datasets with negligible random noise in y, SPLS BootYT is recommended. Indeed, both in terms of accuracy and predictive abilities, it outperforms all other techniques compared here. Second, for datasets with non-negligible random noise in y, which represents the more realistic case, our new dynamic bootstrap-based method is recommended. As for SPLS BootYT for the negligible random noise variability framework, BootYTdyn outperform all other methods for each property studied. Furthermore, it is the only method which concludes as to a decreasing number of significant predictors, while the random noise variability increases, which was expected here. Results obtained from our classification study using real datasets match with previous conclusions.
Indeed, BootYTdyn is the only one which leads to the expected PMSE values on two additional independent datasets, which allows us to suppose that over-fitting issues were avoided, and that these PMSE are induced by noise or information which cannot be modeled using only gene expression. Furthermore, the extracted probe set is already known to be linked to the relevant location in the distal colon, which strengthens our confidence in this new dynamic approach. Lastly, our new bootstrap-based SPLS implementation improves the stability of this method. Indeed, in all cases studied, both the SPLS CV and SGPLS CV conclude in a higher number of different sets of hyper-parameters than our bootstrap-based versions do, leading to higher numbers of different supports and models too. In the future, simulations need to be done to booster the results obtained on real datasets for the logistic framework (Section 4).
Cyber-Physical Systems (CPSs) are a mature research technology topic that deals with Artificial Intelligence (AI) and Embedded Systems (ES). They interact with the physical world via sensors/actuators to solve problems in several applications (robotics, transportation, health, etc. ). These CPSs deal with data analysis, which need powerful algorithms combined with robust hardware architectures. On one hand, Deep Learning (DL) is proposed as the main solution algorithm. On the other hand, the standard design and prototyping methodologies for ES are not adapted to modern DL-based CPS. In this paper, we investigate AI design for CPS around embedded DL. The main contribution of this work is threefold: (1) We define an embedded DL methodology based on a Multi-CPU/FPGA platform. (2) We propose a new hardware design architecture of a Neural Network Processor (NNP) for DL algorithms.
Proposed Methodology In this section, we propose an embedded DL-based methodology for a FPGA-based CPS platform design using a hardware NNP. The first step of our methodology was the definition of the system with its requirements and architecture. Then, different software algorithms were designed for data processing and DL. Those algorithms were hardware accelerated using High Level Synthesis (HLS) software tools or designed from scratch with a Hardware Description Language (HDL). Finally, the hardware accelerators were synthesized and uploaded on a hardware prototype to be tested. Considering those steps, several hidden tasks were present from data processing to data management, and the configuration of a hardware prototype. The goal of our methodology was to mitigate those hidden tasks, either with simplification or automation.
Hardware Prototype One contribution of this work is to develop a real prototype by acquiring physical data from the real world and by processing it in order to obtain an accurate analysis. This analysis was conducted by transforming physical data into specific features that were used with a DL approach to be classified. The hardware prototype needed to host an embedded processing application to transform physical data as features for an NN so that data could be classified by our DL application (NNP). Hence, the hardware prototype hosted an FPGA bitstream containing hardware-accelerated embedded processing to calculate features from data as well as the NNP IP classifying those data. We also needed to set up the hardware prototype with an OS and a devicetree in order to execute the FPGA control software.
The hardware prototype setup was automated with an automation tool that we developed for this case  [20]. This tool deployed a bootloader (U-Boot) and a First Stage Bootloader (FSBL) to help obtain the first stages of the platform. The tool also deployed a Unix kernel with its initial ramdisk, preconfigured system files, and a generic devicetree to gain access to all components on the hardware prototype. Once set up, the hardware prototype needed software to control the FPGA processing as well as to transmit data between the different processing elements, since most data are available inside the platform DRAM to act as an interface between the different processing components, such as CPU and FPGA.
Hardware Acceleration The embedded processing was built as an FPGA hardware IP. The hardware accelerator development was simplified using HLS software tools or HDL, but data management was still a sensitive part of the development because of the FPGA constraints. In this methodology, we considered that data is received and transmitted as a FIFO (First In, First Out) queue in order to simplify data management, even if it may mean extra calculation for processing tasks. This led to the embedded processing application receiving data from sensors and directly transmitting the processed information to the NNP. It also meant that the NNP received its data (input vector and weight matrices) as a FIFO queue and needed to compute the classification as data transmission progress. The embedded processing algorithms needed to be tweaked in order to compute FIFO transmitted data and used the smallest amount of internal cache (BRAM) possible, since FPGA does not have infinite internal memory.
Neural Network Processor Implementation The NNP was designed to simplify the integration of DL in embedded CPS applications. To keep this processor simple, some constraints were defined: process the simplest NN architecture (fully connected NN without bias) with as few activation functions as possible, process any number of layers independently of their depth and width, and be re-configurable at runtime. A no-bias architecture was chosen here because bias calculation needs more computational power and time. Computing a fully connected NN, also called Dense Neural Network (DNN), is mainly about matrix calculation. The main problem that arises from the implementation is not related to computation but data management. Weight matrices need to be loaded from external memory to be transmitted afterwards to the FPGA in order to allow for different configurations and because of the limited FPGA memory cache (BRAM) compared to the size of nowaday weight matrices that can reach dozens of megabits when using 32-bit floating-points.
However, to optimize data transmission, each hidden layers' output needs to be kept in FPGA cache. Moreover, weights need to be sorted correctly depending on the scheduling for neuron processing units. In this implementation, inputs and weights are floating-point numbers and no compression is currently done. In order to correctly control and configure this NNP, a configuration software is made which has for its main tasks loading binary files containing weight matrices to the DRAM, preparing the instructions for the NNP depending on each weight matrix dimension, and initializing DMA transmissions.
Algorithm 2 describes in a shorter way to process the engine module process. As said before, the activation functions are limited to four activation functions: relu, linear, sigmoid, and softmax. Relu, linear, and sigmoid activation functions are computed directly inside the neuron processing unit. However, in the case of the softmax activation function, the exponential part is performed in the neuron processing unit and the division by the sum of the output vector is performed inside the scheduler module because only the scheduler has access to the whole output vector. All the SystemC source code we wrote for the NNP is available online  [21]. A testbench is available to load datasets and neural network models. The synthesis for each of those components is done using Vivado HLS  [22]. The resources used for the hardware neuron processing component is shown in Table  2.
Regarding weight sorting, since data were a FIFO queue and to reduce data memory cache usage inside the FPGA, the weights were transmitted in the same order as the transmissions to their associated neuron processing unit. The scheduling algorithm loaded all pairs of inputs and weights to each core until all neurons were processed. This means that weights must be sorted depending on layer size and the number of available cores. The basic process is about dividing the weight matrix into sub-matrices with a size depending of the number of available cores (Figure  5 ). Each sub-matrix represents the weights for one set of K neuron processing units. Those weights needed to be sorted by processing unit, and this was done with a transposition. Then, each transposed matrix was vectorized for memory writing purposes.
Experimentation and Results In this section, two experiments are presented: the test and validation of our NNP and the smart LIDAR for pedestrian detection case study. The NNP test and validation present the time performance and accuracy of our processor. The smart LIDAR case study is made to test our proposed methodology alongside the NNP. We propose to share our experiments on a smart LIDAR for object classification application and the results of this experiment. We describe in detail the workflow and each component of the application as well as performance and resource utilization. It is noted that there is no real LIDAR involved in this work; we instead used the 3D Point Cloud People Dataset acquired from a Velodyne HDL 64E S2 sensor  [23, 24]. This dataset recorded real-world data in downtown Zurich (Switzerland) and recorded mostly pedestrians.
NNP: Test and Validation Tests were done on a Zedboard development kit  [25]  using the Xilinx Zynq-7000 SoC (XC7Z020)  [26]. Each test corresponds to the usage of a specific known dataset with a specific NN architecture for each dataset. NN model training and testing were done with Tensorflow  [27]. NN topologies are shown in Figure  6. Table  3  compares the accuracy results for each dataset achieved by our NNP depending on its number of cores and the Tensorflow software executed on a CPU. These results mean that our NNP correctly computes floating-point numbers without error. It is noted that the maximum number of cores possible on this FPGA platform is 4 because of the DSP (Digital Signal Processor) limitation (220 DSP available on our FPGA platform with each core using 48 DSP; see Table  2 ).
In this work, the execution time is our main concern, and it is obviously related to the number of parameters in the NN and the number of cores in the processor. Figure  7a  shows the execution time of three datasets depending on the number of cores. The execution time seems to be close to linear, with a same architecture and a different number of cores, except for when there is only one core, which shows a bottleneck. Moreover, the execution time per parameter seems to be the same between the different datasets, as shown in Figure  7b. With Vivado HLS transforming the SystemC models to HDL, our hardware threads run in parallel (the scheduler and neuron processing units are independent finite state machines using the same clock). The clock for the hardware threads runs at 100 MHz, which is 150 MHz lower than the maximal frequency on our hardware.
Deep Learning Software The first step when working on this prototype was to define how to classify objects from 3D data, such as the point cloud received from the LIDAR. One way to classify objects is to convert point cloud to voxels and then to use deep learning to determine the category  [15]. The dataset used in our case study was the Sydney Urban Objects Dataset (SUOD)  [30, 31] , but we converted point clouds into a 32 × 32 × 32 voxel grid using a volumetric binary occupancy grid approach. The training was performed with the architecture represented in Figure  9  using Tensorflow software  [27]. The hardware used for the training was 16 GB of RAM with an Intel Core i7-8550U CPU with 4 cores, 8 threads, a base frequency of 1. 8GHz up to a turbo frequency of 4 GHz, as well as a 8 MB cache.
Embedded Processing Three-dimensional cloud data were the input of the smart LIDAR. Each object in the point cloud needed to be extracted and transformed to voxels as an input for the NNP. To achieve this, four tasks were needed, as shown in Figure  8. The first step was to make an occupancy grid to detect objects inside the point cloud. The second step was to remove all points that were not considered objects. The third step was to isolate objects with a "sliding box". The results for those three steps were presented in a previous paper  [33]. The fourth step was to convert extracted objects into a volumetric binary occupancy grid. Once the object was transformed into a 32 × 32 × 32 voxel grid, it was sent to the NNP. The embedded processing was written in SystemC and synthesized to RTL with an HLS software (Xilinx Vivado HLS  [22] ).
The algorithm for the "points to voxels" module is presented in Algorithm 4. The module received all the points from a box two times: the first time to calculate the bounding box and the second time to calculate the volumetric binary occupancy grid. The input and ouput of the "points to voxels" module are shown in Figure  10. The pedestrians were extracted in boxes of 2 × 2 × 2 meters and then converted to a 32 × 32 × 32 voxel grid. The FPGA synthesized results are shown in Table  4. After the hardware IPs implewere mented, we tested all extracted pedestrian boxes to find the mean execution time per point (Table  5 ). voxel size ←  (24, 24, 24) ; padding size ←  (32, 32, 32)     The embedded processing was synthesized with the NNP. The bitstream was then ported on top of the platform. The hardware platform used was a ZedBoard Zynq-7000.
The SD card generation was automated using our script  [20]. This script deploys a UNIX operating system (OS) and all other required resources to boot this OS. The weight matrices were also integrated within the SD card along with the configuration software. In order to evaluate the system, two tests were performed. The first test was related to the SUOD. The accuracy was evaluated with all classes contained in the dataset. The second test was related to the 3D Point Cloud People Dataset. We extracted all pedestrian boxes from the Polyterrasse set to test if they were correctly classified, which means 599 fully visible pedestrians. Thus, the accuracy is related to the number of boxes that were correctly classified as pedestrians. With 4,204,160 parameters, the processing time of this network topology is shown in Figure  11.
Even if the NNP integration step is not automated yet, it is portable for several applications and platforms. In addition, the current NNP can be improved at several levels: (1) data management needs to be refactored and optimized in order to speed up computation, (2) a flexible architecture is needed to integrate more activation functions and topology types. Moreover, we think that the use of the Vivado HLS software tool to implement the NNP might slow down the final design compared to a from scratch HDL model. Currently, our NNP is adapted for lightweight neural networks since the execution time might be enough to reach real-time constraints in some applications while using low power. Although the limited topology and activation functions choice might be a constraint for some applications, NNP design reuse, in the context of platform based design for CPS systems, is a motivation for us to investigate the possibility of automated generation of this NNP with the needed number of cores and then the automation of the whole methodology, since several steps are already automated using our automation software.
Conclusions In this paper, a new methodology for Cyber-Physical Systems (CPS) design and prototyping is presented. It is defined around an embedded Deep Learning (DL) approach. In order to compute this embedded DL algorithm, we propose a new hardware Neural Network Processor (NNP) architecture. We also share our experience of the design/implementation and the porting of the DL-based NNP on a real hardware Multi-CPU/FPGA platform (Zynq). Our DL-NNP model used real data coming from a LIDAR sensor. Hardware threads were used to transform data from a 3D Point Cloud (LIDAR) to a voxel grid, which is considered the input of our NNP. Results related to the NNP performances are presented, and the whole methodology is validated with a smart LIDAR for pedestrian detection case study. The prediction of the NNP execution time is dependent on the number of parameters (weight matrices) in the Neural Network (NN) and the number of NNP cores.
The new economical hybrid composite material is made of an organic matrix containing methyl methacrylate, a woven reinforcement including a reinforcing glass fiber and a fabric perlon having an absorbing role. The textile reinforcement made up of several folds reinforcing laid out according to the orientations (90, 45 2 , and 0). A numerical simulation with the ANSYS Workbench software is carried out to study the behavior of the HDPE pipe with surface defect and with defect repaired by the new hybrid composite material in the form of rings to consolidate the cracked area of the tube. The numerical results will allow us to decide on a real practical use of the new hybrid composite.
Therefore, the evaluation of the residual strength of pipes with defects should be as accurate as possible and based on confirmed experimental and numerical methods  [11] [12] [13] [14] [15] [16] [17] [18]. The objective of our work is to assess cracked HDPE pipes considering the crack size effect on the burst pressure. For this purpose, a finite element model is developed in ANSYS software. The model is validated with experimental data  [19]. The maximum service pressure that an HDPE pipe with surface defect could be computed from the numerical model and used for comparison. Then, we studied the behavior of the tube with the new hybrid composite material. The hybrid composite that served as a ring strengthening on the weakened area by the corrosion. The numerical results obtained are important for practical use of the new hybrid composite for rehabilitating drinking water supply pipes in HDPE.
NUMERICAL MODELING o model the pipeline we used the ANSYS Workbench 2020R1 software. A model has been developed that simulates the behavior of a HDPE pipe following an increase in internal pressure while taking into account the presence of a longitudinal crack in its outer wall. Given the symmetry of the HDPE 100 tube, we modeled a half cylinder. The dimensions of the structure are: length, radius and thickness. Thus we were able to model the surface defect of the HDPE tube by taking into account the geometry of the defect, the boundary conditions and the mesh  [20]. For the simulation, we used the experimental results presented in  [19]. Initially, we modeled a pipe in PEHD100 without defect presenting the same characteristics as those presented in  [19]  and we calculated the rupture pressure which corresponds to the value of the Von Mises stress when this one reaches the tensile strength of the material.