dataesr/scientific-paragraphs-categorization huggingface dataset:
Results Our experiments were conducted on four real-world datasets with varied parameters for n, d, and k, which allowed evaluating our solutions more practically. Fig. 4  (Nursery), Fig. 5  (Adult), Fig. 6  (MS-FIMU), and Fig. 7  (Census-Income) illustrate for all the evaluated protocols, the averaged MS E avg (y-axis) according to the longitudinal privacy parameter ∞ (x-axis) with 1 = 0. 3 ∞ (plot on the top) and with 1 = 0. 6 ∞ (plot on the bottom), respectively. As one can notice in the results, for all datasets, ALLOMFREE consistently and considerably outperforms the state-of-the-art protocols, namely, L-SUE (a. k. a. Basic-RAPPOR)  [11]  and L-OUE (that uses OUE  [14]  twice). Indeed, the difference between the performances of ALLOMFREE and the other longitudinal LDP protocols increases proportionally according to the privacy guarantees, i.
e. , for high ∞ and 1 values, the gap is bigger. This is first because in all datasets there are attribute(s) with a small domain size (e. g. , k j = 2 or k j = 3), in which L-GRR can provide smaller variance values than the L-UE protocols (cf. subsection 4. 4). Secondly, by adequately selecting the probabilities p 1 , q 1 , p 2 , q 2 for the L-UE protocol (i. e. , L-OSUE) also optimizes data utility. Thus, since there is a way to measure the approximate variance of the extended protocols (i. e. , Eq. (  11 )), given the sampled attribute, ALLOMFREE adaptively selects one of the optimized protocol (i. e. , L-GRR or L-OSUE) whose smaller variance improves the data utility. In addition, among the L-UE protocols applied individually, the experimental results with multidimensional data approximate the numerical results with a single attribute from subsection 4.
4. For instance, the proposed L-OSUE provides similar or better performance than L-SUE while always outperforming L-OUE. Besides, L-SOUE always outperforms L-OUE too, achieving performance similar to those of L-OSUE and L-SUE in low privacy regimes (i. e. , high values). As we have already shown in subsection 4. 4, even though OUE has better utility than SUE for onetime collection  [14] , applying OUE twice does not provide higher utility. To complement the results of Figs. 4 5 6 7 , Table  3  ( 1 = 0. 3 ∞ ) and Table  4  ( 1 = 0. 6 ∞ ) exhibit all datasets and ∞ guarantees the following utility metrics: U L-SUE = MS E avg (L-SUE) -MS E avg (ALLOMFREE) MS E avg (L-SUE) U L-OUE = MS E avg (L-OUE) -MS E avg (ALLOMFREE) MS E avg (L-OUE) (17) in which U L-SUE and U L-OUE represent the accuracy gain of ALLOMFREE over the state-of-the-art L-SUE and L-OUE protocols, respectively.
From Tables  3  and 4 , one can notice that ALLOM-FREE considerably improves the quality of the fre-     quency estimates in comparison with the state-of-theart L-SUE and L-OUE protocols. On average, AL-LOMFREE improves the results of L-SUE at least 10% with the MS-FIMU dataset in Table  3  and at most 30. 38% with the Nursery dataset in Table  4  for the privacy guarantees ∞ and 1 analyzed. Similarly, on average, ALLOMFREE improves the results of at least 19. 32% with the MS-FIMU dataset in Table  3  and at most 54. 96% with the Nursery dataset in Table 4. The highest gain of accuracy was about ∼ 71%, achieved with the Nursery dataset when ∞ = 4 in Table 4 in comparison with the L-OUE protocol. Finally, as one can note, with higher values of 1 , ALLOM-FREE will provide much higher utility than the other protocols.
Conclusion This paper investigates the problem of collecting multidimensional data throughout time for the fundamental task of frequency estimation under LDP guarantees. We extend and analyze three state-of-the-art LDP protocols, namely, GRR  [18] , OUE  [14] , and SUE  [11] , and propose an optimized solution, namely, ALLOMFREE, which randomly samples one attribute per user and adaptively selects a protocol with a lower variance (i. e. , L-GRR or L-OSUE) in order to improve data utility. Through experimental validations, we demonstrate the advantages of ALLOMFREE over the state-of-the-art protocols L-SUE  [11]  and L-OUE  [14]  by using four real-world datasets, with the gain of accuracy on average ranging from 10% up to 55% for the analyzed range of ∞ and 1 privacy guarantees. For future work, we suggest and intend to improve the frequency estimates through post-processing tech-niques  [56, 43]  and to design LDP protocols for longitudinal and multidimensional studies considering both numerical and categorical data.
Acknowledgements This work was supported by the  EIPHI-BFC Graduate School  (contract " ANR-17-EURE-0002 ") and by the  Region of Bourgogne Franche-Comté   CADRAN  Project. The work of  Héber H. Arcolezi  was partially supported by the  European Research Council (ERC)  project  HYPATIA  under the European Union's  Horizon 2020 research and innovation programme. Grant agreement n. 835294. All computations have been performed on the " Mésocentre de Calcul de Franche-Comté ".
ANR-20-CE28-0007-01  (to B. M),  ANR-21-CE28-0010  (to D. S),  ANR-CONV-0002  ( ILCB ),  ANR-11-LABX-0036  ( BLRI ),  ANR-17-EURE-0029  (NeuroMarseille), the  French government  under the  Programme "Investissements d'Avenir" , the  Initiative d'Excellence d'Aix-Marseille Université  ( A*MIDEX ,  AMX-19-IET-004 ).
In medical imaging, accurate segmentation is crucial to improving diagnosis, treatment, or both. However, navigating the multitude of available architectures for automatic segmentation can be overwhelming, making it challenging to determine the appropriate type of architecture and tune the most crucial parameters during dataset optimisation. To address this problem, we examined and refined seven distinct architectures for segmenting the liver, as well as liver tumours, with a restricted training collection of 60 3D contrast-enhanced magnetic resonance images (CE-MRI) from the ATLAS dataset. Included in these architectures are convolutional neural networks (CNNs), transformers, and hybrid CNN/transformer architectures. Bayesian search techniques were used for hyperparameter tuning to hasten convergence to the optimal parameter mixes while also minimising the number of trained models.
It was unexpected that hybrid models, which typically exhibit superior performance on larger datasets, would exhibit comparable performance to CNNs. The optimisation of parameters contributed to better segmentations, resulting in an average increase of 1. 7% and 5. 0% in liver and tumour segmentation Dice coefficients, respectively. In conclusion, the findings of this study indicate that hybrid CNN/transformer architectures may serve as a practical substitute for CNNs even in small datasets. This underscores the significance of hyperparameter optimisation.
However, there has been a recent application of transformer-based structures  13  , originating from natural language processing, to the field of computer vision, resulting in a new group of architectures exhibiting superior performance as compared to CNN architectures in certain tasks  14, 15. Transformer-based architectures have been applied in medical imaging  [16] [17] [18]. Nonetheless, the superiority of transformers compared to CNNs has not been established yet in small datasets, where CNNs can still outperform transformer-based architectures due to the requirement of a large amount of data to exploit the full capacity of this architecture type. Thus, the circumstances in which transformers can outperform CNNs are unclear. The proliferation of models  [16] [17] [18]  complicates the selection of a candidate for a specific application.
The sub-field of pre-training known as Self-supervised Learning  19  is highly beneficial in medical imaging. Unlike pre-training, it can be implemented on images without annotations  20. Handling 3D medical images that potentially exhibit noteworthy differences between images necessitates data preprocessing in order to standardise the dataset. Such preprocessing usually includes procedures such as normalising intensity levels and re-sampling to guarantee that the input data is of high quality and homogeneous throughout the entire dataset. 21. Data augmentation methods  [22] [23] [24] [25]  including random rotation, scaling, and flipping can be used to artificially enhance the size of the training dataset and enhance the generalisation capabilities of models to new data. Learning paradigms are utilised to regulate model progression in training.
Typically when deep learning pipelines are published, they usually entail the optimisation and design of training strategies and architecture for specific tasks, often leading to reduced performance when applied to other tasks or datasets. In an effort to address this issue, frameworks like nnUNet  33  have been introduced to enhance generalisation across heterogeneous datasets. Building on this idea, the objective of this study is to combine and compare the training strategies of seven promising deep learning pipelines for 3D medical image segmentation specifically within the context of SIRT treatment planning, with a focus on liver and tumour segmentation. To this end, this study combines the different elements of the seven pipelines into a single one, and optimises the performance of the seven corresponding architectures by varying and adapting the state of each hyperparameter using Bayesian search.
With this approach, the study aims to identify the most effective strategies for 3D liver tumour segmentation. This study focuses on network selection, pre-processing, data augmentation and training. In this paper the combinations of the proposed architecture and the proposed training strategy will be referred to as a pipeline. All models are trained and optimised on the A Tumor and Liver Automatic Segmentation (ATLAS)  34  dataset, which consists of 3D contrast-enhanced magnetic resonance images (CE-MRI) of the liver with annotations. Several studies using comparable datasets have been documented in the scientific literature but on private dataset. Christ et al. 35  performs Diffusion-Weighted MRI (DW-MRI) segmentation over 31 patients using cascaded fully convolutional neural networks, Zhao et al.
36  proposed liver tumour detection through the using of generative adversarial networks, and similarly, Kim et al. 37  performed region-of-interest detection of hepatocellular carcinoma (HCC) using a multi-centre CE-MRI dataset. Xiao et al  38  introduced a liver tumour segmentation solution employing radiomic features from T2 delay-phase CE-MRI over This study distinguishes itself from the existing literature by providing a comprehensive and unbiased comparison of the most promising architectures for 3D medical image segmentation. In addition, it explores the impact of different learning strategies. In particular, this work goes further than previous publications by examining the impact of each hyperparameter in relation to both the other hyperparameters and the backbone network. This approach provides a more nuanced understanding of the factors that contribute to the performance of deep learning models for medical image segmentation.
The main contribution of this work can be summarised as follows: • A study of the impact of learning strategies on seven state-of-the-art architectures for 3D medical image segmentation. • A first segmentation comparison between the liver and the tumour using publicly available CE-MRI data. • A fair and rigorous comparison between CNN, transformer, and hybrid architectures on the ATLAS dataset. • An evaluation of the most commonly studied hyperparameters in the literature. • A set of guidelines for future optimisation work on small, multi-structural datasets, especially involving different sizes of structures. • A GitHub repository containing code that facilitates the training and tuning of the seven deep learning architectures for 3D segmentation tasks and reproduce the results: github link to be defined.
Dataset The ATLAS  34  dataset was used in this study to design and evaluate the complete optimisation process. The dataset is composed of 90 contrast-enhanced magnetic resonance images (CE-MRI), collected from 90 patients with hepatocellular carcinoma (HCC). Alongside the CE-MRI, label images of the liver and tumours were provided. The labels were manually delineated by an experienced MRI radiologist using the MIM SurePlan LiverY90 software  41  from the transversal view CE-MRIs. As shown in Figure  2 , there are three classes in this dataset: background, liver and tumour. The ATLAS dataset contains 3D CE-MRI of the whole liver and tumour, consisting of 44 to 136 transverse slices of the thorax and abdomen. The pixel spacing of each slice in the dataset ranges from 0. 68 × 0. 68 mm² to 1.
41 × 1. 41 mm², with a slice thickness of 2 mm to 4 mm. During the reconstruction of the images composing the ATLAS dataset, bias field correction was applied to every image. The ATLAS dataset is composed of two distinct sets. The training set which contains 60 images and the testing set with 30 images. To evaluate the performance of each model in this study, we randomly divided the training dataset into two sets: 48 images were used to train the models, and 12 images were reserved to validate them. The test set was used to measure the optimisation process's impact.
Validation metrics The model's performance was measured on the validation set at each epoch during training, and the best results were recorded. Due to the incapacity of the models to process large 3D images at once on a 32 Gb GPU, a sliding window approach was adopted to validate by patching over the images with a 0. 5 overlap. To evaluate the validation segmentation quality, we utilised an exponential moving average on the generalised dice (GD)  54  as our validation criterion. The aim of this metric is to give priority to models that exhibit superior generalisation abilities on a testing dataset, instead of simply selecting the model with the highest GD value from the validation set. The GD is defined as follows: GD(A, B) = 2 ∑ N i=1 w i |A i ∩ B i | ∑ N i=1 w i (|A i | + |B i |) (1) where N is the number of classes (three in our case), A i and B i (∀i ∈ 1, N ) are respectively the predictions and labels for class i, and |.
Data pre-processing and data augmentation influence Data pre-processing and data augmentation are among the most important aspects of the learning process in deep learning algorithms, as they provide robustness and generalisation capacity to the models  [21] [22] [23] [24] [25]. The main objective of data pre-processing is to standardise a dataset  21. Therefore, a range of pre-processing techniques could be employed such as image re-sampling, image cropping, and image normalisation. The large part of the studied pipelines uses median re-sampling since it appears as a fair compromise between image quality and memory consumption. Thus, all the images were resampled to the median spacing of the dataset: 1. 04 × 1. 04 × 3. 00 mm³ leading to images after re-sampling between 320 × 250 × 44 pixels (320 × 250 × 132 mm) to 512 × 512 × 136 pixels (512 × 512 × 408 mm).
The ATLAS dataset presents a significant class imbalance among the different structures (background >> liver >> tumour). To address this, an oversampling strategy  59  was also fixed for every training iteration leading to an equal distribution among classes in the centre of the chosen patch. During the optimisation process, the only studied hyperparameter that affected the pre-processing was the image normalisation strategy. Pixel intensities vary significantly across different images due to the various machines and sequences used to acquire the ATLAS dataset. Consequently, inter-image normalisation strategies were not taken into consideration. However, two intra-normalisation strategies, namely, min-max normalisation and Z-score normalisation, were analysed. Min-max normalisation consists of re-scaling the pixel intensities of an image between 0 and 1 and is defined as: x ′ = x -min(X) max(X) -min(X) , ( 9 ) where x is the original pixel intensity value, x ′ is the normalised pixel intensity value, and X the ensemble of the pixels value in the image.
Z-score normalisation, is another widely-used technique to re-scale the pixel intensities of images. Z-score normalisation involves re-scaling the pixel intensities to have zero mean and unit variance and is defined as: x ′ = x -µ(X) σ (X) , ( 10 ) where x is the original pixel intensity value, x ′ is the normalised pixel intensity value, µ(X) is the mean pixel intensity value in the image, and σ (X) is the standard deviation of pixel intensity values in the image. Data augmentation enables the artificial expansion of a training dataset, resulting in more resilient algorithms with superior generalisation capabilities. This is particularly crucial when working with medical image datasets, which usually have limited data. Data augmentation can be categorised into two forms, image transformation augmentation  23  and image generation-based augmentation  60.
Overall progression on the test set After completing the entire optimisation process, the most effective hyperparameter combinations for each model were identified and the models performance was measured on the test set. The subsequent sections present a detailed assessment of the models' performance on tasks related to liver and tumour segmentation, providing insights into the effectiveness of the chosen parameters. In Table  10 , we present the progression of the performance throughout the entire optimisation process when testing on the  liver dataset. The table highlights the differences between the initial baseline results and the highest performance achieved during training, based on the validation criteria. Improvements can be observed across all models, with an average increase of 1.
Literature comparison In the field of liver and liver tumour segmentation, there is a dearth of studies providing results on mono-modal MRI. Comparisons with other datasets are not consistently applicable due to distinctive imaging protocols and patient groups. Nevertheless, the work by Christ et al. 35  , as the only study conducted within a similar imaging modality (DW-MRI), remains relevant. They reported a Dice coefficient of 87% for liver and 69. 7% for tumour segmentation across 31 patients. On the liver, the CHAOS segmentation challenge  65  saw a dice score of 95. 2% achieved.
Limitations The primary objective of this study was to identify the best hyperparameter configuration without utilising external data. However, due to the need to limit the computational expenses, several hyperparameters were intentionally not examined in this research. Here are the primary limitations of our study: • Mono-centre dataset: all images in the ATLAS dataset were obtained from patients at the University Hospital of Dijon, France. As a result, conclusions reached in this study may have varied with a more diverse dataset. • Cross-validation omission: cross-validation was not utilised in our study due to the extended training period of the deep learning architectures examined. This choice could influence the reliability of our performance metrics and their applicability to diverse data samples.
The majority of these architectures were developed with datasets of different complexities in mind. Consequently, deviations to these architectures could result in diverse outcomes, particularly when applied to datasets of simpler or more complex natures than originally anticipated. Although Bayesian search offers an elegant strategy for fine-tuning hyperparameters by using past insights to guide the search towards areas of interest, it has inherent limitations when it comes to training models in parallel. Unlike grid or random search methods, which can train numerous hyperparameter combinations simultaneously, the Bayesian approach works sequentially while determining the next evaluation based on the findings of previous ones. In our research context, evaluation necessitated testing 19 unique combinations.
Nevertheless, hybrid transformer architectures, typically associated with large datasets, have shown their capability to match CNN performance even with limited datasets. Thus, for future directions, it appears promising to expand the dataset size through self-supervised and mixed supervised learning. This is especially true given the potential of transformers to benefit from such expansions. This study encourages a re-evaluation of the role of transformers in scenarios with limited data and highlights their emerging relevance in medical image segmentation. Such an improvement in automatic segmentation represents a significant step towards the potential automation of this process, reducing the reliance on manual segmentation performed by radiologists. This advancement not only streamlines the workflow but also promises to improve patient care by ensuring more precise and timely diagnoses and treatment planning.
Validation interviews To validate the final outcomes of the studies and to gain deeper insights from the results regarding the state of practice, we performed interviews with selected practitioners after we collected all the survey data. Protocol. The interview protocol was designed following the guidelines by  Hove and Anda (2005). All interviews were conducted online, through video calls. To enable participants to become familiar with the interview objectives  (Hove and Anda, 2005) , we asked participants to answer the survey prior to the interview. The interviews were structured in four sections. First, during the introduction, we discussed the research goals and background. Second, we asked participants to share data about their background and about the projects (involving ML components) they are working on.
Data analysis. To process the interviews, we used thematic analysis, as recommended by  Cruzes and Dyba (2011). The process consisted of (i) data extraction -the interviews were transcribed, read, and key points were extracted, (ii) data coding -the answers were categorised and linked to the practices, (iii) code to themes translation -for each transcript the codes were combined into potential themes (e. g. , testing), (iv) high-order theme modelling -the themes were compared and merged, (v) synthesis assessment -arguments for the extracted data were established, e. g. , in terms of agreement.
We also analysed the adoption rates based on the types of data used by respondents and based on the practice type. Since the practices are general and apply to any context (e. g. , to any project constraint, such as data type), we do not expect the adoption rates to change significantly between data types. However, the type of data directly influences the choice of algorithms and can indirectly influence practice adoption; for example, when using deep neural networks to process images or text, training is not preceded by a feature extraction step. The distribution of data types and the adoption of practices grouped by data type is illustrated in Fig. 7. We note that tabular data, text and images have similar representation in our set of survey responses, while audio, time series and graphs have a lower representation (which implicitly makes their adoption rates less reliable).
We convert the categorical effects to numbers and perform regression instead of classification because the outcomes have a natural order. In the attached replication package we also experiment with classification, which entails similar results  (Serban et al. , 2020a). Training was performed using 5-fold cross-validation. For all experiments, we used under-sampling on the training data to remove class imbalance. We also experimented with over-sampling algorithms (in particular with SMOTE  (Chawla et al. , 2002; Torgo et al. , 2013) ), but the increase in performance was not significant. The grid search RF used 384 candidate configurations for each of the five folds. For AutoML, we employed auto-sklearn  (Feurer et al. , 2015) , where we tested both a modest time budget of 1 h and a larger budget of 5 h.