dataesr/scientific-paragraphs-categorization huggingface dataset:
The trained deep neural network (topology and weights) can be automatically converted to a format understandable by the inference library, but there are no tools to generate and deploy the application code. Moreover, the test application must be written by hand. Nevertheless, a template source code for a few development boards (e. g. , the SparkFun Edge) as well as a few demo applications (e. g. , keyword spotting) are available. Finally, TFLite Micro does not come with tools to measure metrics such as the inference time or the RAM and ROM usage. TFLite Micro supports computation in both floating point in binary32 format and fixed point on 8-bit integers. The quantization technique uses a non-power-of-two scale factor, a symmetric range for the weights and an asymmetric range for the activations.
STM32Cube. AI STM32Cube. AI is a software suite from STMicroelectronics that enables the deployment of deep neural networks onto their STM32 family of microcontrollers. STM32Cube. AI supports deployment of trained deep neural network models from several frameworks including Keras and TensorFlow Lite. A wide range of operations are supported  [45] , allowing the deployment of several deep neural network architectures, such as multi-layer perceptron and convolutional neural networks, including residual neural networks. STM32Cube. AI is a software suite fully integrated with other STMicroelectronics development and deployment tools such as STM32CubeMX, STM32CubeIDE and STM32CubeProgrammer. This provides a very straightforward and easy to use flow. Moreover, a test application is included to evaluate the model on target with a real test dataset, providing metrics on inference time, and ROM and RAM usage, without having to write a single line of code.
Like TFLite Micro, STM32Cube. AI supports computations in floating-point binary32 format and fixed point on 8-bit integers. In fact, the quantization on 8-bit integers comes from TFLite. There is no support for fixed point on 16-bit integers. STM32Cube. AI also has an optimized inference engine that seems to be partially based on CMSIS-NN. However, as the source code of the inference engine is not freely available, it is not clear what is optimized and how. The inference library is entirely proprietary/closed-source, therefore it is not possible to manipulate and extend this library. This represents a major drawback in a research environment. It is also not possible to use STM32Cube. AI on microcontrollers which are not part of the STMicroelectronics portfolio. The inference process and optimizations are not detailed, but unlike TFLite Micro, the network topology is compiled into a set of function calls to the closed-source library rather than being interpreted at runtime.
MicroAI: Training For the training phase, MicroAI is simply a wrapper around Keras or PyTorch. A dataset requires an importation module to be loaded into an appropriate data model. The training process expects a RawDataModel instance, which gathers the training and test sets. This instance contains numpy arrays for the data and the labels. A higher-level data model HARDataModel is also available for human activity recognition to process subjects and activities more easily. This model is then converted to a RawDataModel using the DatamodelConverter in the preprocessing phase. The preprocessing phase also includes features such as normalization. Dataset importation modules for UCI-HAR, SMNIST and GTSRB (described in Section 6) are included and can easily be extended to new datasets. To make use of a deep neural network architecture in the model configuration, it must be first described according to the training framework API in use.
The description of the model is a template where parameters can be set by the user in the configuration file. MicroAI provides the following neural network architectures for both Keras and PyTorch: All architectures use ReLU activation units. In the configuration file, several model settings can be described, each inside their own [[model]] block. Each model will be trained sequentially. A common configuration for all the models can be specified in a [model_template] block. Model configuration also includes optimizer configuration and other parameters such as the batch size and the number of epochs. Once the model is trained, some post-processing can be applied. It is for instance possible to remove the SoftMax layer for Keras models with the RemoveKerasSoftmax module. This layer is indeed useless when only inference is performed.
Even though it also performs model training, the quantization-aware training described in Section 4. 3 is also included for PyTorch as a post-processing step in the QuantizationAwareTraining module. The actual training step before post-processing is seen as a general training, before optionally performing post-training quantization or quantizationaware training. The quantization-aware training can be seen as a fine-tuning on top of the more general training (which can also be skipped if necessary). The quantization-aware training does not actually convert the weights from a floating-point data type to an integer data type with fixed-point representation. This conversion is instead performed by the KerasCNN2C conversion tool. Support for additional learning frameworks can be added by creating a new class implementing the LearningFramework interface and by supplying compatible model templates.
Similarly, KerasCNN2C produces a portable library that can be included in any project. So far, only integration with the Nucleo-L452RE-P and the SparkFun Edge boards has been performed. Support for other platforms can be added by providing project files that call the inference code and a module that interfaces with the build and deployment tools for that platform. Please note that none of these tools can take a trained PyTorch model as an input to deploy onto a microcontroller. The trained PyTorch model must therefore be converted to a Keras model prior to the deployment. Our framework provides a module to perform semi-automatic conversion from a PyTorch model to a Keras model. A Keras model that matches the structure of the PyTorch model must be programmed and the matching between the PyTorch model and Keras model layer names also must be specified.
KerasCNN2C: Conversion Tool from Trained Keras Model to Portable C Code KerasCNN2C is a tool that we developed to automatically generate, from a trained Keras model exported as an HDF5 file, a C library for inference. It can also be used independently of the MicroAI framework. In this work, only 1D models are evaluated on target. Work is underway for full support of 2D-model deployment. Training and quantization are already supported, therefore 2D models are evaluated offline. Here are the supported layers so far: • Add • AveragePooling1D • BatchNormalization • Conv1D • Dense • Flatten • MaxPooling1D • ReLU • SoftMax • ZeroPadding1D Layers can have multiple inputs such as the Add layer, thus allowing residual neural networks (ResNet) to be built. Sequential convolutional neural networks or multi-layer perceptron models are also supported.
KerasCNN2C: Conversion Process This tool first parses the model using the Keras API from TensorFlow 2. 4 and generates an internal representation of the topology (i. e. , a graph), with each node corresponding to a layer. Then, a series of transformations is performed to produce a graph better suited for deployment on a microcontroller: convert BatchNorm  [51]  weights from the mean µ, the variance V, the scale γ, the offsets β and to a multiplicand w and an addend b using the following formula: w = γ σ (5) σ = √ V + (6) b = β - γ × µ σ (7) so that the output of the BatchNorm layer can be computed as y = w × x + b. It could be folded in the previous convolutional layer, but this is not implemented yet. Then, for each node in the graph, the weights of the layer go through the quantization and conversion module if the conversion to fixed-point representation is enabled.
The C inference function is generated from a Jinja2 [52] template file using the layer's configuration. Similarly, the layer's weights are converted into a C array from a Jinja2 template file. Code generation is used to avoid runtime overhead of an interpreter such as the one used in TensorFlow Lite for Microcontrollers. Additionally, it allows the compiler to perform better optimizations. In fact, the layer's configuration is generated as constants or literals in the code, allowing the compiler to perform appropriate optimizations such as loop unrolling, using immediates when relevant and doing better register allocation. By default, GCC's -Ofast optimization level is enabled. Moreover, the code is written in a simple and straightforward way. So far, no special effort has been made to further optimize the source code for faster execution.
Results All the results presented in this section rely on the same model architecture, a ResNetv1-6 network with the layers shown in Figure  4. The number of filters per layer f is the same for all layers, but is modified to adjust the number of parameters of the model. The convolutional and pooling layers are one-dimensional except when handling the GTSRB dataset, for which they are two-dimensional. For each experiment, the residual neural network is initially trained using 32-bit floatingpoint numbers (i. e. , without quantization), and then evaluated over the testing set. This baseline version is depicted as float32 in the figures shown in the following. The float32 neural network is quantized for inference with fixed-point on 16-bit integers and is then evaluated without additional training.
This version is depicted as int16 in the figures shown hereafter. Quantization is performed using the Q7. 9 format for the whole network, meaning the number of bits n for the fractional part is fixed to 9. The float32 neural network is also trained and evaluated for inference with fixed-point on 8-bit integers using quantization-aware training. This version is indicated as int8 in the figures. In this case the fixed-point precision can vary from layer to layer and is determined using the method introduced in Section 4. 1. 4. The SGD optimizer is used for all experiments. The stability of the SGD optimizer motivated this choice, especially for the quantization-aware training. Training parameters are described below for each dataset. Additionally, training and testing sets are normalized using the z-score of the training set.
It is worth noting that Mixup  [53]  is also used during training. Accuracy is not evaluated directly on the target due to the amount of time it would require. Only inference time for the UCI-HAR dataset is measured on the target. In the figures, each point represents an average over 15 runs. The University of California Irvine's hosted Human Activity Recognition dataset (UCI-HAR)  [54]  is a dataset of activities of daily living recorded using the accelerometer and gyroscope sensors of a smartphone. In this experiment, we use the raw data from the sensors divided into fixed time windows, rather than the precomputed features. The reason is that we want to perform real-time embedded recognition. To do so, it is necessary to avoid the overhead of computing the features for each inference before entering the deep neural network.
Instead, the features are extracted by the convolutional neural network itself. The dataset is divided into a training set of 7352 vectors and a testing set of 2947 vectors. Each vector is a one-dimensional time series of 2. 56 s composed of 128 samples sampled at 50 Hz, with 50% overlap between vectors. Each sample has 9 channels: 3 axes of total acceleration, 3 axes of angular velocity and 3 axes of body acceleration. Six different classes are available in the dataset: walking, walking upstairs, walking downstairs, sitting, standing and lying. The initial training without quantization is performed using a batch size of 64 over 300 epochs. The initial learning rate is set to 0. 05, the momentum is set to 0. 9 and the weight decay is set to 5 × 10 -4. The learning rate is multiplied by 0. 13 at epochs 100, 200 and 250.
The quantization-aware training for fixed-point on 8-bit integers uses the same parameters. As can be seen in Figure  5 , for the UCI-HAR dataset, the same accuracy is obtained using a 16-bit quantization (UCI-HAR int16) or 32-bit floating-point (i. e. , the baseline UCI-HAR float32), whatever the number of filters per convolution. On the other hand, we observe that the 8-bit quantization causes a drop in accuracy that increases in magnitude up to 0. 81% when the number of filters per convolution grows, even though quantization-aware training is used to mitigate this issue. In Figure  6 , we observe that the accuracy obtained using 8-bit and 16-bit quantization is similar only for deep neural networks exhibiting a reduced number of parameters, in other words a low memory footprint. As an example, for 16 filters per convolution, an 8-bit quantization leads to an accuracy of 92.
Spoken digits dataset (SMNIST) Spoken MNIST is the spoken digits part of the written and spoken digits database for multi-modal learning  [55]. This dataset is made of spoken digits extracted from the Google Speech Commands  [56]  dataset. The audio signal is preprocessed to obtain 12 MFCC plus an energy coefficient using a window of 50 ms with 50% overlap over the audio files of approximately 1 s each, generating one-dimensional series of 39 samples with 13 channels. The dataset is divided into training and testing sets of 34,801 and 4107 vectors, respectively. Some samples are duplicated to obtain 60,000 training vectors and 10,000 testing vectors. There are 10 different classes for each digit, from 0 to 9. The initial training, without quantization, uses a batch size of 256 over 120 epochs.
The German Traffic Sign Recognition Benchmark (GTSRB) The German Traffic Sign Recognition Benchmark (GTSRB  [57] ) is a dataset containing various color pictures of road signs. Image sizes vary between 15 × 15 to 250 × 250 pixels. In this experiment, the two-dimensional images were scaled to 32 × 32 pixels using bilinear interpolation and anti-aliasing, while keeping the 3 color channels (red, green, blue). The dataset is divided into training and testing sets of 39,209 and 12,630 vectors, respectively. There are 43 different classes, one for each type of road sign in the dataset. The initial training without quantization uses a batch size of 128 over 120 epochs. The initial learning rate is set to 0. 01, the momentum is set to 0. 9 and the weight decay is set to 5 × 10 -4. The learning rate is multiplied by 0.
1 at epochs 40, 80 and 100. The quantization-aware training for fixed-point on 8-bit integers uses a batch size of 512 over 120 epochs. The initial learning rate, momentum and weight decay are the same as for the initial training. The learning rate is multiplied by 0. 1 at epochs 20, 60, 80 and 100. The accuracy results obtained for 8-and 16-bit quantization and the 32-bit floating-point versions are shown in Figure  9  for different numbers of filters. As can be seen, the 16-bit quantization (GTSRB int16) provides an accuracy similar to the one obtained with the baseline (GTSRB float32). In the meantime, a drop in accuracy of up to 1. 1% can be observed when the 8-bit quantization is used with this GTSRB dataset. However, as it was observed with the SMNIST dataset, the accuracy approaches that of the baseline when the network has more filters (a drop of only 0.
This network has been trained on the UCI-HAR dataset presented in Section 6. 1. 1. Inference time is measured from 50 test vectors from the testing set of UCI-HAR on the microcontrollers. TensorFlow Lite for Microcontrollers version 2. 4. 1 has been used to deploy the deep neural network on the SparkFun Edge board, while STM32Cube. AI version 5. 2. 0 has been used to deploy it on the Nucleo-L452RE-P board, for the 32-bit floating-point and fixed-point on 8-bit integers inference. Our framework is used to deploy the deep neural network on both platforms for 32-bit floating-point, fixed-point on 16-bit integers and fixed-point on 8-bit integer inference. It is worth noting that optimizations for the Cortex-M4F provided by CMSIS-NN are enabled for both TensorFlow Lite for Microcontrollers and STM32Cube.
AI tools. Our framework does not make use of these optimizations yet. The main characteristics of the frameworks are summarized in Table  4. To compare software and hardware platforms, only the results with 80 filters per convolution are analyzed below. Nevertheless, results with less than 80 filters are still available in the tables of Appendix E to highlight how fast and efficient a small deep neural network can be when deployed on a constrained embedded target. They also highlight a higher overhead for very small neural networks especially for TensorFlow Lite for Microcontrollers compared to our framework. In Figure  11 , we can observe that TFLite Micro has a higher overhead than STM32Cube. AI, while MicroAI exhibits a slightly lower overhead than STM32Cube. AI. As outlined in Table  A4  of Appendix E, when the number of filters per convolution increases, most of the ROM is used by the model's weights.
The inference time obtained for both platforms and the different deployment tools is illustrated in Figure  12. As can be seen, the STM32Cube. AI with the 8-bit inference provides the best solution as it requires only 352 ms for one inference. In the same configuration, TensorFlow Lite for Microcontrollers requires 592 ms for one inference. Finally, 1034 ms and 1003 ms are required for one inference using our framework on the Nucleo-L452RE-P board and the SparkFun Edge board, respectively. When using fixed-point on 16-bit integers for the inference, our framework provides approximately the same performance as with 8 bits. The reason is that the inference code is the same: similar instructions are generated, and computations are performed using 32-bit registers. On the Nucleo-L452RE-P, we can observe that the inference time for one input is 1223 ms, while it is only 1042 ms on the SparkFun Edge board.
Discussion First, a high variance is observable when we compare the accuracy results obtained on the three datasets versus the model size. This variability makes it difficult to draw any definitive conclusions. However, there is a trend in our results that provides some insights into performance for each experiment. As has been shown, execution using fixed-point on 8-bit and 16-bit integers provides a significant decrease in the inference time, thus also reducing the average power consumption. As power consumption is a key parameter in embedded systems, shorter inference times are interesting as they make it possible either to reduce the microcontroller's operating frequency or to put the microcontroller in sleep mode for a longer period between two inferences. In addition, execution using 8-bit and 16-bit integers also provides a significant reduction in memory footprint.
We also observed that using 9 bits instead of 8 bits during the post-training quantization allows us to outperform the TensorFlow Lite quantization performance. Some results showing this improvement are available in Appendix C for the UCI-HAR dataset. From these results, we can conclude that the slight additional precision brought by the combination of per-filter quantization, asymmetric range and non-power of two scale factor does in fact matter. Implementing these methods in our framework seems therefore required to reduce the accuracy loss of our 8-bit quantization. Another benefit of 8-bit quantization is that SIMD instructions can be used (with some classes of microcontrollers) to improve the inference time and thus further reduce the power consumption. Such instructions allow performing in a single cycle either 2 multiply-accumulate operations with 16-bit operands and a common accumulator (SMLAD), or 2 additions of 16-bit operands (QADD16), or a shift and saturation operation (SSAT).
The SMLAD, QADD16 and SSAT instructions are not yet used in our framework, but this work is in progress. Nonetheless, a 16-bit quantization scheme can be used with our framework, which is not the case with either TensorFlow Lite for Microcontrollers or STM32Cube. AI. As presented in the results, the 16-bit quantization from our framework provides a good compromise between accuracy, inference time and memory footprint, without requiring additional work on quantization-aware training. The results obtained on inference time clearly show that both the software and hardware platforms have a substantial impact on energy efficiency. STM32Cube. AI offers the most optimized inference engine in terms of execution time, both in floating-point and fixedpoint on integers. Our results show that TensorFlow Lite for Microcontrollers is slower than STM32Cube.
AI in both conditions. For the floating-point inference, our framework is in between these two software platforms and is only slightly slower than STM32Cube. AI. However, as optimizations using SIMD instructions have not been implemented yet in our framework, inference using 8-bit integers still provides lower performance than TensorFlow Lite for Microcontrollers and STM32Cube. AI. Regardless of the software performance, running STM32Cube. AI on a Nucleo-L452RE-P board is only competitive with inference using 8-bit integers when compared to TensorFlow Lite for Microcontrollers in 32-bit floating-point inference on the SparkFun Edge board. The reason is that the Ambiq Apollo3 microcontroller on the SparkFun Edge board is much more energy efficient. In all remaining cases, running TensorFlow Lite for Microcontrollers or our framework on the SparkFun Edge board provides much better energy efficiency figures than running STM32Cube.
We compared our NGS data to different machine learning algorithms using the k-fold cross-validation method and to neural networks (deep learning) to measure the performance of the different ML algorithms and determine which one is a valid model for confirming NGS variant calls in cancer diagnosis. We trained our machine learning with 70% of our data samples, extracted from our local database (our data structure had 7 parameters: chromosome, position, exon, variant allele frequency, minor allele frequency, coverage and protein description) and validated it with the 30% remaining data. The model offering the best accuracy was chosen and implemented in the NGS analysis routine. Artificial intelligence was developed with the R script language version 3. 6. 0. We trained our model on 70% of 102,011 variants.
Our best error rate (0. 22%) was found with random forest machine learning (ntree = 500 and mtry = 4), with an AUC of 0. 99. Neural networks achieved some good scores. The final trained model with the neural network achieved an accuracy of 98% and an ROC-AUC of 0. 99 with validation data. We tested our RF model to interpret more than 2000 variants from our NGS database: 20 variants were misclassified (error rate < 1%). The errors were nomenclature problems and false positives. After adding false positives to our training database and implementing our RF model routinely, our error rate was always < 0. 5%. The RF model shows excellent results for oncosomatic NGS interpretation and can easily be implemented in other molecular biology laboratories. AI is becoming increasingly important in molecular biomedical analysis and can be very helpful in processing medical data.