dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements. This work is supported in Japan by  ISAS/JAXA  for Pre-Phase A2 studies, by the acceleration program of JAXA research and development directorate, by the  World Premier International Research Center Initiative (WPI)  of  MEXT , by the  JSPS Core-to-Core Program of A. Advanced Research Networks , and by  JSPS KAKENHI  Grant Numbers  JP15H05891 ,  JP17H01115 , and  JP17H01125. The Canadian contribution is supported by the  Canadian Space Agency. The French LiteBIRD phase A contribution is supported by the  Centre National d'Études Spatiales (CNES) , by the  Centre National de la Recherche Scientifique (CNRS) , and by the  Commissariat à l'Énergie Atomique (CEA). The German participation in LiteBIRD is supported in part by the  Excellence Cluster ORIGINS , which is funded by the  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)  under  Germany's Excellence Strategy  (Grant No.
The analysis of data coming from massively deployed Internet of Things (IoT) devices pave the way to a myriad of intelligent applications in several vertical domains. Federated Learning (FL) has been recently proposed as a prominent solution to train Machine Learning (ML) models directly on top of devices (FL clients) generating data, instead of moving them to centralized servers in charge of training procedures. FL provides inherent benefits, mainly in terms of privacy preservation and reduction of network congestion for datasets exchange. Despite the huge recent research efforts, it still faces challenges for a practical implementation effectively targeting low communication footprint, robustness and interoperability. To fill this gap, in this work we propose a novel comprehensive framework built upon the Message Queue Telemetry Transport (MQTT) publish/subscribe messaging protocol and the Open Mobile Alliance (OMA) Lightweight Machine-to-Machine (LwM2M) semantics to facilitate FL operations and make them more suited to handle IoT devices acting as FL clients.
Introduction In the recent years Internet of Things (IoT) devices proliferated at a dramatic pace. According to Ericsson  [1] , there will be 24 billion interconnected devices by 2050. Through the application of Machine Learning (ML) techniques, the big amount of data generated by such devices can unlock intelligence into different domains, spanning from transportation to industries, from healthcare to agriculture. Centralized ML functions typically placed in a cloud server may incur critical scalability limitations given the IoT data explosion. Moreover they suffer from users' privacy leakage because of the need to transfer the end-devices data to a third party server for training purposes. Indeed, data to be collected exist in the form of isolated islands and may contain confidential information, e.
g. , in the case of healthcare applications, and hence, data sharing may not be possible  [2]. Such threat may be posed even within the same enterprise in the case of Industrial IoT (IIoT) applications  [3]. The concept of Federated Learning (FL) has been recently proposed to address the aforementioned issues  [4, 5]. Unlike traditional ML techniques, FL enables on-device training of an ML model, by coordinating multiple devices, acting as FL clients, with a central server, referred to as FL aggregator, without sharing actual datasets. The population of FL clients involved in the training procedures may encompass heterogeneous devices, with different processing capabilities and experiencing different channel conditions when exchanging model/parameters with the FL aggregator. Randomly selecting the FL clients may result in poor performance in terms of training convergence time and/or model accuracy  [6].
Devices either constrained in terms of computing/battery resources and/or slow in exchanging data with the FL aggregator may slow down the overall training. Instead, devices owning poor datasets may compromise the accuracy of the model. Nonetheless the huge recent literature proposing smart client selection schemes, e. g. ,  [6] [7] [8] [9] [10] [11] , a few works have focused on the procedures needed to discover the FL clients capabilities and to enable judiciously selecting them. Retrieving information about the status of the potential FL clients is even more relevant when IoT devices, such as smartphones, vehicles, laptops or tablets, are involved  [12, 13]. An IoT client may not be always able to perform on-device computation to reach the target convergence within an expected time. Moreover, more exacerbated challenges need to be faced in terms of communication, computation, storage, power, and energy utilization, e.
g. , straggler issue  [13]. https://doi. org/10. 1016/j. comnet. 2023. 109576 Received 31 October 2022; Received in revised form 6 January 2023; Accepted 11 January 2023 In  [7]  potential clients notify about their resources (e. g. , processing capabilities, dataset, etc. ) the FL aggregator. The latter one leverages the retrieved information to estimate the time needed to build the global model, starting from updates of locally trained models, and to update it. Then, clients are selected according to the performed estimates. Similarly in  [11] , the FL aggregator transmits a participant request message to all the potential clients. Among them, those wishing to participate in the training send a response message including their data distribution and the quantity of private data. However, the aforementioned works do not discuss the protocol leveraged to exchange such information.
In  [14]  we proposed to leverage: (i) the Message Queue Telemetry Transport (MQTT) protocol  [15]  for FL clients-FL aggregator interactions aimed to retrieve the capabilities of the potential clients and (ii) the Open Mobile Alliance (OMA) Lightweight Machine-to-Machine (LwM2M) data models  [16]  for the semantic description of such capabilities. The solution has been early implemented confirming its viability. Besides interactions for the sake of discovering clients capabilities, which are neglected in the majority of works targeting client selection, FL requires extensive communication between the selected FL clients and the FL aggregator. As early argued in  [17] , the impact of the communication protocol leveraged for such purpose on the FL performance (e. g. , in terms of the overall time taken to train the model) and on the resulting communication footprint, has not been adequately investigated.
• We design communication primitives based on MQTT, with relevant pre-defined topics and messages payloads, augmented with the OMA LwM2M semantics, to target interoperability and specifically cope with IoT challenges. • We assess the performance of the proposal through a Proof-of-Concept (PoC) with virtualized and physical FL clients, when compared to a benchmark FL implementation, under different link settings and for different datasets. The remainder of this work is organized as follows. Section 2 provides background and motivations for the work. The proposal is discussed in Section 3. The main procedures foreseen by the conceived framework are presented in Section 4, Section 5, and Section 6. Section 7 discusses the main benefits of the conceived framework. Achieved results are reported in Section 8, before concluding in Section 9 with hints on future work.
Federated learning: an overview The FL paradigm has been proposed in 2016  [4]  to boost Artificial Intelligence (AI) solutions thanks to the distributed and privacyenhancing nature. In FL, a shared global model is trained by iteratively aggregating model updates from multiple devices over multiple rounds. In each round, the FL clients download the latest global model from the FL aggregator, train the model on their local datasets, and report their respective model updates to the FL aggregator. The latter one combines all local model updates and builds a new improved global model. Model aggregation can be performed either at the edge or at the cloud  [12]. The process is iterated until the global training is complete. By leveraging the computing resources of distributed learners, the FL aggregator can enhance the training quality while minimizing user privacy leakage, since the raw data are not required for the training at the FL aggregator.
FL limitations The aforementioned advantages make FL a suitable candidate to build a myriad of intelligent and privacy-enhanced IoT applications, in domains ranging from smart healthcare to smart transportation  [20]. Notwithstanding, FL has to face several challenges, among which: device heterogeneity (e. g. , in terms of computational resources, available power), wireless channel uncertainties, and data disparity, due to unbalanced local datasets and non-independent and identical distribution (non-IID) data among devices  [12]. The random selection of the FL clients participating in local training procedures, as in the original FL implementation  [4] , might provide long convergence time. This can be detrimental for synchronous federated learning where the FL aggregator waits for the updated local weight parameters uploaded by selected FL clients at each round.
Instead, a protocol is needed to enable potential FL clients and the FL aggregator to exchange such data with a low communication overhead and in a uniform manner to cope with the inherent heterogeneity of involved devices. The communication overhead incurred during the client discovery procedure represents only a small amount of the overall data exchange necessary for the distributed model training. Indeed, after the client selection, the FL aggregator and FL clients need to exchange initially the original model and then, the updated local and global model parameters over multiple rounds  [4]. The choice of the communication protocol leveraged during these stages may affect the FL performance and, in turn, impact the network  [17] , especially for large model sizes, hence deserving a proper investigation which motivates our study.
The proposed FL-MQTT framework: basics In this work, we propose a comprehensive lightweight framework, we refer to as FL-MQTT, to support interactions between an FL aggregator, implemented at the edge, and distributed IoT devices acting as FL clients. The framework builds upon our preliminary work in  [14] , which specifically targeted the support of FL client discovery for the sake of implementing smart client selection procedures. There, MQTT is leveraged to enable the discovery of FL client capabilities. Moreover, the OMA LwM2M semantics is exploited for the description of the capabilities of the devices wishing to act as FL clients. In this work we leverage both MQTT and OMA LwM2M, thus adding to the scalability of MQTT the interoperability offered by the semanticrich OMA LwM2M data model.
The latter component taskid is needed since there could be multiple instances of the same task type on the same aggregator. This MQTT publish message will carry in the payload the client identifier and its capabilities described through the OMA LwM2M semantics using the ClientFL object, Table  3. At this stage, without loss of generality, we assume that information about the devices's battery, processing and memory capabilities, as well as the main characteristics of the owned dataset, is included in the ClientFL object. The overall workflow for the MQTT/OMA LwM2M-supported FL client discovery phase is reported in Fig. 2.
They are distinguished by the Round ID  resource, which is zero for the original model and greater than zero for the model updates. Other parameters related to the training procedure that the FL aggregator could send to FL clients round by round could be the learning rate or the size of the mini-batch  [30]. These parameters can be added to the object defined in Table  6  simply by defining new OMA LwM2M resources. After locally performing the training on their own dataset, the FL clients publish on topic#2,trai the model parameters. The same object as in Table  6  is used to this purpose. In this case, the object also carries the FL Entity ID resource to track the FL client from which the model update has been received. The resource is not necessary in the publish message of topic#1,trai.
Considering two domains A and B, where there is an overlap in the space of the characteristics related to the datasets and/or a similarity between the FL clients, a model learned on A is transferred to B by exploiting these small overlaps. Thanks to the envisioned MQTT-based procedures that have been defined, a second FL aggregator belonging to domain B can easily obtain information on the model trained by an FL aggregator belonging to domain A. This is possible at every stage of the training procedure, through a simple MQTT subscription. ML model-agnosticism. To compute the local learning models using local datasets, various schemes can be used at end-devices. They could be, for instance, feed-forward neural networks (FNN), convolutional neural networks (CNN), support vector machines (SVM), and long-short term memory (LSTM).
FL-MQTT implementation A hybrid population of FL clients is considered. Some of them are deployed through Docker containers  [34] , whereas one is implemented on a RaspberryPi4. Also in this latter case, the application that implements the FL client has been containerized. Another container hosts the FL aggregator. The Docker containers hosting the FL clients and the FL aggregator are connected through the use of the Docker bridge network. A MikroTik switch connects the RaspberryPi4, hosting one of the FL clients, to the machine hosting the virtualized FL aggregator. Both the FL clients and the FL aggregator include tensorflow  [35]  libraries for python. For the MQTT client implementation on both the FL aggregator and the FL clients we leverage the Mosquitto software  [26]. The MQTT broker, also deployed through Mosquitto, is co-located with the FL aggregator.
Dataset We used two different datasets: the Canadian Institute for Advanced Research, 10 classes (CIFAR-10)  [38]  and the modified National Institute of Standards and Technology database (MNIST) handwritten digits  [39]. They are both widely used in the FL research arena  [4, 40, 41]. Moreover, we refer to them because they exhibit complementary characteristics in terms of total size of the datasets (163 MB for CIFAR-10 and 57 MB for MNIST) and size of the individual samples (2. 5 kB for CIFAR-10 and 0. 6 kB for MNIST). The first dataset is made up of 60,000 color images with a size of 32 × 32 pixels. The images are labeled with one of the 10 classes available, which are mutually exclusive. The ten classes are: airplane, car, bird, cat, deer, dog, frog, horse, ship, and truck. The second dataset instead contains 70,000 grayscale images of hand-drawn figures, from zero to nine.
Client discovery and notification The client discovery stage is crucial to allow the retrieval of information about the capabilities of the potential FL clients and drive a smart selection procedure. The workflow conceived in our proposal incurs a negligible overhead both in terms of exchanged messages and in terms of latency before the distributed training starts. The amount of data exchanged during the discovery stage can be calculated as a function of the parameters 𝑁, Ñ, as follows: 𝐸𝑥𝑐ℎ𝑎𝑛𝑔𝑒𝑑𝐷𝑎𝑡𝑎 𝑑𝑖𝑠𝑐 (𝑁, Ñ) = (𝑎 + 𝑏) ⋅ 𝑁 + 𝑐 ⋅ Ñ, (1) where the parameters 𝑎, 𝑏, 𝑐 are defined in Table  7  for the two different datasets. They refer respectively to the traffic (in kB) generated: by each candidate FL client to subscribe to 𝑡𝑜𝑝𝑖𝑐#1, 𝑑𝑖𝑠𝑐; by the FL aggregator when it publishes (on the topic 𝑡𝑜𝑝𝑖𝑐#1, 𝑑𝑖𝑠𝑐) the DiscoveryFL object (forwarded by the MQTT broker to each candidate FL client); and by an FL client passing the self-eligibility check to publish the ClientFL object (on the topic 𝑡𝑜𝑝𝑖𝑐#2, 𝑑𝑖𝑠𝑐).
2  The results can be generalized as  𝐸𝑥𝑐ℎ𝑎𝑛𝑔𝑒𝑑𝐷𝑎𝑡𝑎 𝑠𝑒𝑙 ( Ñ) = 𝑑 ⋅ Ñ + 𝑒 ⋅ Ñ. ( ) 2 The parameters 𝑑 and 𝑒 respectively represent the traffic generated by each FL client to subscribe to topic#1,sel and to receive the notification from the FL aggregator via the broker on the same topic. Their values are reported in Table  7. This metric is independent of the considered dataset. Fig. 5  reports the overall amount of data exchanged during the client discovery and notification stages, according to Eqs. (  1 ) and (  2 ), whose parameters are derived from experimental results. For 𝑁 = Ñ=100 the amount of data exchanged during the client discovery and notification stages is equal to 102. 8 kB in the worst case under the considered settings (i. e. , CIFAR-10 dataset). This confirms the small overhead incurred by the procedure.
Model training We first compare the FL approach against a conventional centralized learning, in which a set of clients share their datasets with a server in charge of training procedures. In particular, results have been reported in terms of accuracy (see Fig. 7 ) for both the MNIST and CIFAR-10 datasets and compared with the accuracy achieved by the FL approach, when fixing the number of clients to 5 and when varying the number of iterations. 3 First of all, we can observe that there are no remarkable differences between the centralized learning and FL approach. This confirms the FL effectiveness. In particular, the maximum training accuracy achieved after 200 rounds  [4]  by the FL global model is 95% for CIFAR-10 and 95. 8% for MNIST. A training loss of 0. 001 is reached for both datasets.
We then compare the amount of data exchanged by our proposal during the training stage, once FL clients are discovered, selected and notified, against the gRPC protocol. 4 To achieve a fair comparison, the training procedure is set so to provide the same results in terms of accuracy and loss for both  solutions, under the settings reported in Table 8  [4]. Measurements were carried out with 50% of the number of samples for both datasets. 5 Fig. 8  shows the amount of exchanged data during the training procedure, when varying the loss probability over the links interconnecting the FL aggregator and 5 FL clients. The 𝑡𝑐 utility has been leveraged to set the packet loss probability over the aforementioned links to resemble realistic network conditions. The performance of FL-MQTT is reported both when the QoS level is set to 0 and when it is set to 2.
Results are reported for the CIFAR-10 dataset only to avoid the cluttering of the plot. However, no significant differences are noticed when varying the dataset. This is expected since the size of the original models for the two datasets are comparable: (i) 137 kB for the one used on the MNIST dataset and (ii) 175 kB for the one used on the CIFAR-10 dataset. More in detail, the size of publish messages carrying the structure of the NN sent on topic#1,trai is: (i) 143 kB for MNIST and (ii) 182 kB for CIFAR-10. The publish message containing the updated weight values, sent on topic#2,trai, has a size of: (i) 102 kB for MNIST and (ii) 122 kB for CIFAR-10. Whatever the QoS level, the proposal allows to reduce the amount of exchanged data compared to the considered benchmark for both datasets. This is mainly because MQTT uses smaller headers compared to the gRPC-based solution which is based on the HTTP protocol  [49].
Not surprisingly, the most efficient solution in absence of losses is FL-MQTT with QoS level 0, because it incurs the lower overhead. Interestingly, as the link loss increases the improvements of FL-MQTT over the gRPC solution in terms of saved network resources for data exchange get more significant, for both datasets. This is true also when the QoS level 2 is enabled for FL-MQTT. In the latter case, for link loss equal to 20%, it is possible to exchange the lowest amount of data. This trend has to be ascribed to the fact that, thanks to the handshake enforced at the application layer, there is no need to retransmit the long publish messages in case of losses. Retransmissions are instead enforced at the transport layer by FL-MQTT (QoS 0) and gRPC. Moreover, for the sake of completeness, to understand the benefits of FL compared to the centralized approach, the differences in terms of exchanged data should be also considered.
Processing and RAM footprint To further understand the differences between the two compared messaging schemes, results are also reported in terms of average CPU and memory usage during the data exchange. Results only refer on the FL client side, since it could be more resource-constrained. Measurements have been made over a virtual machine hosted on a physical device equipped with an Intel Core i7-9750H processor, a 16 GB DDR4 RAM and an NVIDIA GeForce GTX1050 Ti graphics card. Instead, 4 GB of RAM and one processor thread of the physical device have been dedicated to the virtual machine. Table  9  shows that the FL-MQTT approach exhibits a significantly lower impact on the CPU usage compared to gRPC, whereas a slightly higher RAM consumption is incurred. The achieved trends confirm the better suitability of the MQTT protocol for constrained devices compared to gRPC.
Interoperability test As discussed in Section 7, the proposal can facilitate transfer learning among different FL domains. The last set of results is aimed to quantify the benefits in terms of training time when enabling FTL. We consider a primary domain, 𝑑1, with an FL aggregator, denoted as 𝐴1, and a set of FL clients, as described before, and a secondary target domain, 𝑑2, with another FL aggregator, denoted as 𝐴2. The latter one works with a set of FL clients having similar but different local datasets compared to the primary scenario. In particular, we used 50% of the unused samples in the previous measurements campaign. 6 During the training procedure in 𝑑1, 𝐴2 tries to benefit from the trained model of the primary domain and shares it with its set of clients as a (partially trained) original global model.
The conceived publish/subscribe framework and the selfexplanatory topics allow 𝐴2 to subscribe to topic#1,trai and topic#2,trai on the broker in order to obtain information on the structure of the NN model and on the global weights in 𝑑1, from round to round. Fig. 9  shows the time saved by 𝐴2 to train the global model in its domain. Results are reported when varying the time (in rounds) after which 𝐴2 takes the model from 𝑑1, compared to the case in which it starts with an original untrained global model. The higher the number of rounds, the better trained is the model in 𝑑1, hence the larger the benefits for 𝐴2, for both the considered datasets. Indeed, 𝐴2 saves more time to train the model with its set of clients, because the latter ones, by starting with a partially trained model, perform more quickly the local training.
Main findings Results collected in a realistic PoC show the superiority of the proposal compared to a popular FL implementation solution, i. e. , gRPC, in terms of lower communication and computation footprint, under different datasets. This makes the solution particularly viable for IoT scenarios. Interoperability is also proven, by assessing the benefits of letting a different set of FL aggregator and FL clients reuse partially trained models in a primary domain. Since typically constrained IoT devices are involved, saving time in the overall training procedures, even in a small percentage, means that less resources are wasted by FL clients to perform the local training.
ABSTRACT: The link between the cortical and cardiovascular systems is garnering increased attention due to its potential to offer valuable insights into brain and heart function coupling. Current joint analysis methodologies largely involve invasive or high-cost neuroimaging methods. EEG and ECG/PPG, however, provide non-invasive, cost-effective, and portable alternatives enabling broader data collection in both laboratory and clinical settings. However, the analysis of these biosignals is challenging for scalable applications due to their complex nature. Existing research and tools often lack consensus in processing and statistical methodologies, easy-to-use user interface, or batch processing capacity of large datasets, impeding reproducibility. A further void exists in standardized methods for EEG and heart-rate variability (HRV) feature extraction, undermining clinical diagnostics or the robustness of machine learning models.