dataesr/scientific-paragraphs-categorization huggingface dataset:
Model Training. Using the dataset split and the model definition we can turn our attention to training a regressor. In Listings 3 we create a model instance (line 1), transfer the model parameters (line 3) to the Adam optimizer  [29]  which uses a learning rate of 0. 01 and set the model to be trainable (line 5). In each epoch we set the accumulated cost to be zero (line 8) iterate over the temporal snapshots in the training data (line 9), make forward passes with the model on each temporal snapshot and accumulate the spatial unit specific mean squared errors (lines 10-13). We normalize the cost, backpropagate and update the model parameters (lines 14-17). Listings 4: Evaluating the recurrent graph convolutional neural network on the test portion of the spatiotemporal dataset using the time unit averaged mean squared error.
Model Evaluation. The scoring of the trained recurrent graph neural network in Listings 4 uses the snapshots in the test dataset. We set the model to be non trainable and the accumulated squared error as zero (lines 1-2). We iterate over the test spatiotemporal snapshots, make forward passes to predict the number of chickenpox cases and accumulate the squared error (lines 3-7). The accumulated errors are normalized and we can print the mean squared error calculated on the whole test horizon (lines 8-10). The optimizer registers the model parameters and the model parameters are set to be trainable (lines 5-6). We iterate over the temporal snapshot iterator 200 times and the iterator returns a temporal snapshot in each step. Importantly the snapshots which are PyTorch Geometric Data objects are transferred to the GPU (lines 8-10).
The use of PyTorch Geometric Data objects as temporal snapshots allows the transfer of the time period specific edges, node features and target vector with a single command. Using the input data a forward pass is made, loss is accumulated and weight updates happen using the optimizer in each time period (lines  [11] [12] [13] [14] [15] [16] [17]. Compared to the cumulative backpropagation based training approach discussed in Subsection 3. 3 this backpropagation strategy is slower as weight updates happen at each time step, not just at the end of training epochs.
Documentation. The source-code of PyTorch Geometric Temporal and Sphinx  [8]  are used to generate a publicly available documentation of the library at https://pytorch-geometric-temporal. readthedocs. io/. This documentation is automatically created every time when the code-base changes in the public repository. The documentation covers the constructors and public methods of neural network layers, temporal signal iterators, public dataset loaders and splitters. It also includes a list of relevant research papers, an in-depth installation guide, a detailed getting-started tutorial and a list of integrated benchmark datasets.
CONCLUSIONS AND FUTURE DIRECTIONS In this paper we discussed PyTorch Geometric Temporal the first deep learning library designed for neural spatiotemporal signal processing. We reviewed the existing geometric deep learning and machine learning techniques implemented in the framework. We gave an overview of the general machine learning framework design principles, the newly introduced input and output data structures, long-term project viability and discussed a case study with source-code which utilized the library. Our empirical evaluation focused on (a) the predictive performance of the models available in the library on real world datasets which we released with the framework; (b) the scalability of the methods under various input sizes and structures. Our work could be extended and it also opens up opportunities for novel geometric deep learning and applied machine learning research.
Practice Case Study: Incremental Model Training with GPU AccelerationExploiting the power of GPU based acceleration of computations happens at the training and evaluation steps of the PyTorch Geometric Temporal pipelines. In this case study we assume that the Hungarian Chickenpox cases dataset is already loaded in memory, the temporal split happened and a model class was defined by the code snippets in Listings 1 and 2. Moreover, we assume that the machine used for training the neural network can access a single CUDA compatible GPU device[48].
[5,6]rties and granularity of the spatiotemporal datasets introduced in the paper with information about the number of time periods ( ) and spatial units (| |). Wikipedia Math. Contains Wikipedia pages about popular mathematics topics and edges describe the links from one page to another. Features describe the number of daily visits between 2019 and 2021 March. •Twitter Tennis RG and UO. Twitter mention graphs of major tennis tournaments from 2017. Each snapshot contains the graph of popular player or sport news accounts and mentions between them[5,6]. Node labels encode the number of mentions received and vertex features are structural properties. • Covid19 England. A dataset about mass mobility between regions in England and the number of confirmed COVID-19 cases from March to May 2020 [38].
Each day contains a different mobility graph and node features corresponding to the number of cases in the previous days. Mobility stems from Facebook Data For Good 1 and cases from gov. uk 2. • Montevideo Buses. A dataset about the hourly passenger inflow at bus stop level for eleven bus lines from the city of Montevideo. Nodes are bus stops and edges represent connections between the stops; the dataset covers a whole month of traffic patterns. • MTM-1 Hand Motions. A temporal dataset of Methods-Time Measurement-1 [36] motions, signalled as consecutive graph frames of 21 3D hand key points that were acquired via MediaPipe Hands [64] from original RGB-Video material. Node features encode the normalized xyz-coordinates of each finger joint and the vertices are connected according to the human hand structure.
[29]predictive performance of spatiotemporal neural networks evaluated by average mean squared error. We report average performances calculated from 10 experimental repetitions with standard deviations around the average mean squared error calculated on 10% forecasting horizons. We use the incremental and cumulative backpropagation strategies. ±0. 010 1. 116 ± 0. 017 2. 039 ± 0. 022 2. 010 ± 0. 009 1. 469 ± 0. 030 1. 465 ± 0. 026 0. 788 ± 0. 011 0. 832 ± 0. 020 were trained for 100 epochs with the Adam optimizer[29]which used a learning rate of 10 -2 to minimize the mean squared error. 4. 2. 2 Experimental findings. Results are presented in Table5where we also report standard deviations around the test set mean squared error and bold numbers denote the best performing model under each training regime on a dataset.
Practice Case Study: Incremental Model Training with GPU AccelerationExploiting the power of GPU based acceleration of computations happens at the training and evaluation steps of the PyTorch Geometric Temporal pipelines. In this case study we assume that the Hungarian Chickenpox cases dataset is already loaded in memory, the temporal split happened and a model class was defined by the code snippets in Listings 1 and 2. Moreover, we assume that the machine used for training the neural network can access a single CUDA compatible GPU device[48]. 1 model = RecurrentGCN(node_features=8, filters=32) 2 device = torch. device('cuda')3 model = model. to(device)4 5 optimizer = torch. optim. Adam(model. parameters(), lr=0. 01) 6 model. train() 7 8 for epoch in range(200):.
Dataset Signal Graph Frequency 𝑇 |𝑉 | Chickenpox Hungary Temporal Static Weekly 522 20 Windmill Large Temporal Static Hourly 17,472 319 Windmill Medium Temporal Static Hourly 17,472 26 Windmill Small Temporal Static Hourly 17,472 11 Pedal Me Deliveries Temporal Static Weekly 36 15 Wikipedia Math Temporal Static Daily 731 1,068 Twitter Tennis RG Static Dynamic Hourly 120 1000 Twitter Tennis UO Static Dynamic Hourly 112 1000 Covid19 England Temporal Dynamic Daily 61 129 Montevideo Buses Temporal Static Hourly 744 675 MTM-1 Hand Motions Temporal Static 1/24 Seconds 14,469 21 These newly released datasets are the following: • Chickenpox Hungary. A spatiotemporal dataset about the officially reported cases of chickenpox in Hungary. The nodes are counties and edges describe direct neighbourhood relationships.
The dataset covers the weeks between 2005 and 2015 without missingness. • Windmill Output Datasets. An hourly windfarm energy output dataset covering 2 years from a European country. Edge weights are calculated from the proximity of the wind- mills -high weights imply that two windmill stations are in close vicinity. The size of the dataset relates to the group- ping of windfarms considered; the smaller datasets are more localized to a single region. • Pedal Me Deliveries. A dataset about the number of weekly bicycle package deliveries by Pedal Me in London during 2020 and 2021. Nodes in the graph represent geographical units and edges are proximity based mutual adjacency rela- tionships.
Digital holography is a very efficient technique for 3D imaging and the characterization of changes at the surfaces of objects. However, during the process of holographic interferometry, the reconstructed phase images suffer from speckle noise. In this paper, de-noising is addressed with phase images corrupted with speckle noise. To do so, DnCNN residual networks with different depths were built and trained with various holographic noisy phase data. The possibility of using a network pre-trained on natural images with Gaussian noise is also investigated. All models are evaluated in terms of phase error with HOLODEEP benchmark data and with three unseen images corresponding to different experimental conditions. The best results are obtained using a network with only four convolutional blocks and trained with a wide range of noisy phase patterns.
In particular, neural networks are able to learn very complex functions from databases. In contrast with these traditional approaches, machine learning-based solutions such as convolutional neural networks (CNNs) use dataset examples and are able to learn how to invert very complex degradation functions  [9]. They have been used to simulate wavelets and multiresolution analysis, shrinking and thresholding algorithms, sparse representations, block matching, and dictionary learning  [10, 11]. Many neural architectures have been developed for Gaussian noise such as residual learning for image recognition  [12]  and generative adversarial networks (GANs)  [13]. Note that, in the field of digital holography and digital holography microscopy, several papers related to applications of CNN were published  [14] [15] [16].
While the amount and the diversity of natural images are huge and thus allow us to train deep networks with many parameters, when moving to phase data processing in digital holography, the quantity and the diversity are clearly reduced. Indeed, there is currently no way to obtain experimental phase data with speckle noise together with its clean version. That is the reason why simulated data is required. Image de-speckling ground-truth clean images have been generated from outputs of commercial optical coherent tomography scanners  [22]. In  [23] , a database including 25 fringe patterns divided into 5 patterns and 5 different signal-to-noise ratios was generated with a realistic noise simulator  [24]  to foster the diversity of phase fringe patterns. To improve de-noising performances, one solution is to go deeper, i.
HOLODEEP Database This database consists of five different types of noise-free phase fringe patterns and was used to train the models and for development purposes. Each pattern was degraded with realistic speckle decorrelation noise with statistics described in  [23]. From each noisefree fringe pattern, five noisy fringe patterns controlled with a parameter, namely ∆, were generated with the simulator presented in  [23] , corresponding to different signal-to-noiseratios (SNR) in the range [3 dB-12 dB]. The parameter ∆ was used to mimic strongly degraded experimental phase data. The higher ∆, the smaller the SNR. In real conditions, there are several degradation sources that may induce more decorrelation noise than expected if all is perfect. As examples, the reconstruction of holographic data might not be perfectly in focus  [30] , the pixels could have a large active surface  [3] , the recording could have a low number of pixels or saturated pixels  [31] , the number of useful quantization bits could be insufficient  [32] , or there also could be wavelength changes between exposures  [33].
As a consequence, all of these degradation sources have an increase in speckle decorrelation and then an increase in noise. Thus, using ∆ is a useful way to obtain data with more noise in order to mimic possible experimental conditions. In the simulator described in  [23] , ∆ corresponds to small changes in the wavelength between the two exposures. Therefore, adjusting ∆ is useful to increase speckle decorrelation and thus to decrease the SNR in phase data. The simulated images, sized 1024 × 1024 pixels, were generated using Matlab and are available in the Matlab mat format or as tiff images. The 25 images used for training the models are shown in Figure  1.
DATAEVAL Database This validation database consists of three images used for testing the model with images that have not been seen during the training or development processes. Two phase images, namely Test1 and Test2, were simulated using the simulator in Reference  [23] , similar to that for simulating the HOLODEEP database. The SNR of the two phases are respectively 3. 05 dB (see Figure  2b ) and 1. 26 dB (see Figure  2e ). These phase maps are not included in the HOLODEEP database. The last phase is an experimental noisy phase from vibration measured at 17 512 Hz, named Test3 with an SNR = 2. 52 dB. The clean phase is shown in Figure  2g , the noisy phase is shown in Figure  2h , and the noisy phase obtained is shown in Figure  2i. The experimental setup and methodology to obtain such phase images is described in References  [3, 4].
NATURAL Database This database is generally used for natural gray-level image Gaussian de-noising. It consists of 400 images of size 180 × 180. The RGB images are available at the link http://www. eecs. berkeley. edu/Research/Projects/CS/vision/grouping/BSR/BSR_bsds500. tgz (accessed on 2 June 2021). Noisy images were obtained by adding Gaussian noise with different SNR values (over 13 dB) directly to the clean images.
Signal Processing Approaches for Speckle De-Noising Following the protocol described in  [23] , three algorithms from signal processing were tested: WFT2F, BM3D, and DtDWT. The results are given in terms of the standard deviation ∆φ of the phase error e ij defined in Equation (  1 ), where N is the total number of pixels and e ij = φ denoised (i, j)φ noise f ree (i, j) is the difference between the de-noised phase φ denoised and the noise-free phase φ noise f ree at pixel (i, j), ∆φ 2 = 1 N ∑ i,j e ij -m e 2 , ( 1 ) where m e is the average of e(i, j) over the set of pixels. Note that, since φ denoised and φ noise f ree are calculated modulo 2π, the difference e ij has to also be computed modulo 2π according to e ij = arg[exp(i e ij )]. The baseline results are given in terms of the average of ∆φ over the whole HOLODEEP database (i.
e. , 25 images sized 1024 × 1024) and with the three images of the DATAEVAL database. The results for the phase error ∆φ are summarized in Table  1. The iteration number corresponds to how many times the noisy image has been processed by the de-noiser. From Table  1 , one can be observed that only one iteration is required using WFT2F to obtain the best error at ∆φ = 0. 026 rad with HOLODEEP because WTF2F uses a threshold on the decomposition 2D waveforms and the process ends after one iteration. Even with three iterations, the two other methods only reach ∆φ = 0. 046 rad (DtDWT) and ∆φ = 0. 068 rad (BM3D), thus confirming the best performance for WFT2F.
Baseline Implementation The starting network considered in this section is the one proposed in  [17] , called DnCNN. It includes 59 layers organized upon a first input layer (3 × 3 convolutional layer and rectified linear units ReLU), 16 intermediate convolutional blocks (ConvBlocks: 3 × 3 × 64 convolutional layer, batch normalization and ReLU), and one output layer (3 × 3 × 64 convolutional layer), which is used to reconstruct the output noise. The denoised image is the subtraction of the noisy image and the ouput noise. The loss function is an L2 loss between the reference and the predicted pixel values. The parameters of the training process are summarized in Table  2. DnCNN network was pre-trained with 400 grey natural images sized 80 × 80 from the NATURAL database and optimized with the Adam algorithm.
Data Pre-Processing and Implementation The following experiments consider two independent parameters: the type of phase pattern (five patterns in the HOLODEEP database) and the level of speckle noise. For each original image sized 1024 × 1024, candidate patches are extracted. These patches are sized 50 × 50 without any overlap. A random selection aims at extracting 384 patches per image. The seed is fixed once for all experiments in order to have reproducible patch selection. The whole patches are then shuffled in order to remove their dependency to a specific image. The cosine and sine input patches are normalized between 0 and 1. A Tensorflow implementation was used as the starting point https://github. com/ wbhu/DnCNN-tensorflow (accessed on 2 June 2021) and adapted with Matlab matrices as inputs https://git-lium.
univ-lemans. fr/tahon/dncnn-tensorflow-holography/ (accessed on 2 June 2021). DL-Py is the Python implementation used in this paper. The architecture is described in Figure  4 , where tf denotes the tensorflow library and D is the number of ConvBlocks. During the training step, the convergence is very fast in the first 10 epochs and then the loss function decreases continuously and slowly. The maximum number of epochs was fixed to 200 as the performances do not increase significantly with more epochs. However, due to cluster usage constraints, the training has to be stopped before the computing time overpasses a limit of 20 days. The number of epochs corresponding to the best phase error is included in Table  3. The final model is the one that reaches the best results with the development set.
Evaluation of a Pre-Trained Network In a second step, how the network pre-trained on natural images with additional Gaussian noise can be better is estimated. Then, it is adapted to holographic phase images or to the direct use of a network trained entirely with holographic phase images. Four hundred images of the NATURAL database are used to pre-train the network with the best architecture obtained in the previous section, i. e. , four ConvBlocks (see  Section 5). Once the network is pre-trained, a second fine-tuning stage is carried out using holographic images following the aforementioned protocol. The DL-nat-pt model corresponds to the model trained with natural images during 75 epochs, which seems reasonable regarding the 50 epochs used to train the original DnCNN  [10]. Without finetuning, this model reaches ∆φ = 0.
Pre-Training Table  3  shows that the pre-trained model outperforms the initial models only when a small level of noise (∆ = 0) is used for fine-tuning. This leads to the conclusion that pre-training the network on natural images helps to compensate for the lack of diversity in the specific training data and the relatively small amount of training data. Thse results confirm the advantage of using pre-trained models when the amount of specific target data is low  [35]. Two hypotheses may explain the poor performances reached by the pre-trained model. The NATURAL and HOLODEEP databases differ on many points: additive Gaussian vs. multiplicative speckle noise and natural vs. wrapped phase images. Such a data difference could explain the poor performances obtained with pre-training: training a network with phase images using an initialization obtained on NATURAL database does not seem worthy in the present case.
Therefore, training a network with phase data corrupted with speckle noise requires deeper investigated. The second hypothesis concerns the performance of the model trained on NATURAL data. Due to cluster usage constraints, the total number of epochs to train this model is 75 epochs. It aims to obtain a model performed on natural images. However, this number is higher than the 50 epochs used to train the original DnCNN model mentioned in  [17]  and the model might be too specific for natural images. As such models require a lot of resources to be trained, we did not have the opportunity to train it on a higher number of epochs. However, it is worth considering this aspect.
Evaluation on Target Images Table  4  summarizes the performances obtained with the development and validation images. DL-Py-2. 5-4 performs better on the training data HOLODEEP (∆φ = 0. 035 rad) and on Test1 (∆φ = 0. 072 rad). However, the performance is degraded when testing with Test2, which has a high level of noise, and with Test3, which is the phase image from vibration experiments. No clear answer can be given here. DL-Py-2. 5-4 model is trained on a large number of data and noise; thus, it should be able to deal with a high level of noise. However, from the construction of the HOLODEEP database, there are a few redundancies in the phase images, and Test1 appears relatively similar to those in HOLODEEP while Test2 and Test3 are not. Therefore, the model might not be easily generalizeable to unseen images.
The poor performance of DL-Py models with phase images with a high level of noise may be related to the additive hypothesis implemented in the network itself. A multiplicative model will be investigated in the future. Further work intends to improve speckle de-noising by combining the advantages of the two approaches following preliminary works on the addition of a noise estimator  [34]. Other data augmentation functions will be implemented in order to increase the amount of training data. In addition, the construction of a new database with an increased diversity of fringe images would be of interest to train the networks with a high diversity of patterns. Author Contributions: M. T. prepared the neural networks, S. M. and P. P. prepared the database and evaluation process. M. T. , S. M. and P.
Inherited Metabolic Diseases (IMDs) are rare diseases caused by genetic defects in biochemical pathways. Earlier diagnosis and advances in treatment have improved the life expectancy of IMD patients over the last decades, with the majority of patients now surviving beyond the age of 20. This has created a new challenge: as they grow up, the care of IMD patients' needs to be transferred from metabolic pediatricians to metabolic physicians specialized in treating adults, through a process called "transition. " The purpose of this study was to assess how this transition is managed in Europe: a survey was sent to all 77 centers of the European Reference Network for Hereditary Metabolic Disorders (MetabERN) to collect information and to identify unmet needs regarding the transition process. Data was collected from 63/77 (81%) healthcare providers (HCPs) from 20 EU countries.
The collection of these important data can serve as a starting point for common European transition best practice recommendations. The European Reference Network for Hereditary Metabolic Disorders (MetabERN), established in 2017, connects centers specialized in rare metabolic diseases at EU level; it represents 77 healthcare providers (HCPs) from 23 EU Member States and 44 patient organizations. It is also endorsed by the Society for the Inborn Errors of Metabolism (SSIEM). Overall, the network follows almost 33,000 IMD patients (16,586 adults and 16,277 pediatric patients as of November 2020). MetabERN is organized in nine work packages (WP) and in seven subnetworks (SNW), each SNW being specific for the metabolic defects and/or pathways involved in the disease (for details please refer to https://metab.