dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2019). When using neural-based models, the conventional way of integrating further knowledge about specialised domains into models pre-trained on general corpora is to leverage pre-training by doing transfer learning and fine-tuning the model, tweaking its original weights to suit the tasks at hand better. If fine-tuning BERT is the most used adaptation method, it will rapidly hit a performance ceiling if the domain is too specialised or small, and some methods  (Schick and Schütze, 2020)  exist to tackle this problem. BERT will heavily rely on the context to build representations of too specialised or rare words. Each layer of the encoder's attention mechanism enriches the new representation of the input data with contextual information by paying attention to different parts of the text.
GCN is typically built using an adjacency matrix (based on a given relationship graph). The GCN's central idea is to take the weighted average of a node and all its neighbours during the convolution operation and uses both node features and the structure for the training. Text GCN  (Yao et al. , 2019)  is a particular version of GCN, jointly learning both words and documents embeddings, and suited for situations with less training data. Graph methods do not encode positional information, and when information about position or context is needed, it might not be enough. Hence, pairing a GCN with a model that can grasp contextual information, like BERT, seems necessary. There has not been much work trying to combine BERT and GNN. Since experts make inferences with relevant domain knowledge when performing domain-specific tasks, previous work has been conducted to integrate this knowledge into language models using knowledge graphs.
We rely on the GCN to fetch these missing pieces of information, connect them and integrate them back into the BERT model. In this work, we wanted to examine if we can improve results for languages and domains for which there are no BioBERT and Clinical BERT, e. g. there are no WindBERT or PoliticalBERT for hypothetical wind energy or politics related tasks. This step of adaptation to the domain would make sense and even be necessary. Using a graph neural network constructed over the corpus' vocabulary exclusively follows the same logic. It comes from our decision to find a way to improve the model intrinsically without using external resources. Even if many resources exist for the English language, it is still crucial to explore ways to adapt existing models to our downstream specialised tasks where the volume of data is often insufficient.
Conclusion The quality of word representations obtained through language models is often correlated to the volume of data available. In the case of specialised domains, these methods might be less efficient due to the usually modest size of the corpora. This is particularly exacerbated in the case of specialised and low-resource domains, and the models might need to go through an adaptation phase. This work seeks to understand how these methods can be adapted on the fly by using additional features from GCN. The achieved results outperform the baselines across the board. It shows that it is beneficial to merge local and global information, and those resulting representations yield some additional advantages and are more worthwhile for the downstream task. Future work will cover analysis of attention layers before and after adding more global information.
The numerical stability of the present formulation is limited by the explicit discretization of the convection and diffusion terms. The Courant-Friedrichs-Lewy (CFL) number is specified according to the original definition given by Kim and Choi  [21]  as follows: CFL = ∆t 2 S |U| dS. (6) In this work, the numerical simulations are performed by the in-house FORTRAN 90 code NSMP3D. We use the solver SOR, previously shown by  [22] , with relaxation parameter as 1. 0 and tolerance as 10 -5. The numerical discretization in the code is given in the next section.
Despite the growing demand for software implementations of ECDSA secure against attackers with full control of the execution environment, scientific literature on ECDSA white-box design is scarce. The CHES 2021 WhibOx contest was thus held to assess the state-of-the-art and encourage relevant practical research, inviting developers to submit ECDSA white-box implementations and attackers to break the corresponding submissions. In this work, attackers (team TheRealIdefix) and designers (team zerokey) join to describe several attack techniques and designs used during this contest. We explain the methods used by the team TheRealIdefix, which broke the most challenges, and we show the efficiency of each of these methods against all the submitted implementations. Moreover, we describe the designs of the two winning challenges submitted by the team zerokey; these designs represent the ECDSA signature algorithm by a sequence of systems of low-degree equations, which are obfuscated with affine encodings and extra random variables and equations.
In particular, we added two variables and two equations in the implicit round functions of E (1) , and two variables and one equation in those of E (2). We also used Tigress  [Col]  for both challenges to obfuscate the C source code. Tigress is an obfuscator for C language that protects programs against dynamic and static reverse engineering attacks. We used the transformations 5  Flatten (flattens the code to remove structured flow), AntiTaintAnalysis (disrupts tools that make use of dynamic taint analysis), AddOpaque (adds opaque predicates), EncodeLiterals (replaces integers and strings with run-time expressions) and CleanUp (renames variables and functions).
Acknowledgment The authors would like to thank the other members of the TheRealIdefix team:  Yannick Bequer ,  Luk Bettale ,  Laurent Castelnovi ,  Thomas Chabrier ,  Nicolas Debande ,  Roch Lescuyer ,  Sarah Lopez  and  Nathan Reboud. Adrián Ranea  is supported by a PhD Fellowship from the  Research Foundation -Flanders (FWO)  under grant No. 11E1921N. Chaoyun Li  is an FWO post-doctoral fellow under grant No. 1283121N. Ward Beullens  is an FWO post-doctoral fellow under grant No. 1S95620N.
This challenge has been tackled in the literature. For example, DeXMS  [5]  enables IoT devices to interact by synthesizing software mediators that translate between different communication protocols. However, issues such as runtime detection of protocol incompatibility and the on-fly creation of mediators are yet to be addressed. The above challenges together highlight that composing systems that make use of a variety of technologies is difficult to design, maintain and adapt. Therefore, system engineering efforts to support post-deployment composition is needed  [12, 30]. This calls for approaches that address these challenges and are able to (i) identify independent systems with their unique characteristics and contexts; (ii) recognize the need to compose with other systems; (iii) detect the need for mediation logistics; and (iv) set in action any support roles needed to actuate mediation.
• A mixed-methods evaluation of Hetero-Genius using a realworld Internet of Vehicles (IoV) scenario. Specifically, we assess (i) effectiveness and added value through a user study with a group of developers, and (ii) scalability through extensive simulation of representative contexts. Overall, Hetero-Genius cuts the amount of effort and, consequently, costs required from system developers to develop integrated systems almost by half compared to current development practices. Furthermore, it reduces the room for error resulting in code with 55% more correctness scores, on average.
Holon ontology. Automatic IoT system composition requires devices to discover each other along with the functionalities they provide. This would in-turn require devices to advertise themselves. However, the enabler for device advertising and discovery is a definition using a uniform language. The definition needs to include all the concepts and properties of a device in order to provide its comprehensive description. For this, we build on our previous work in which we provided an ontological model in the literature that was designed to describe systems  [4, 12, 21]. This holonic ontology is based on the CoDAMOS standard ontology  [22] , which is flexible and extensible for describing context-aware computing infrastructures. Concepts defined in the CoDAMOS ontology are based on the concept of Thing, which is an abstract concept to represent anything.
Hence, a holon is a programmatic first-class entity that describes any unitary or composite system. Our system model includes an ontology for each device in the system. The holon description will then be utilized to automate and reason about system construction at runtime. 3. 2. 2 Holon discovery. The holon ontology described above is the enabler for automatic discovery of systems and their functionalities. The different types of systems leverage the ontology semantics, which enables different system components to understand each other. This in turn enables all constituent systems to adopt the same mechanism to publish their services and properties, and to discover services and properties of other systems. For this purpose, the Holon Registry component provides a database where each system publishes their holon description in.
Hence, automated solutions are required to enable data exchange in such IoT systems. We represent each (e. g. , a car, a mobile app, a restaurant, etc. ) as a holon that provides services defined based on (possibly) heterogeneous profiles, models, and grounding concept. In this work, we rely on the Data eXchange Mediator Synthesizer (DeXMS)  [5]  to enable heterogeneous IoT devices to interact with each other. DeXMS addresses the heterogeneity of IoT devices and services by synthesizing software mediators. As depicted in Figure  4 , the grounding of services is given as input to DeXMS. This includes information such as IoT protocol, data model, data format and operation (or topic) semantics. DeXMS accepts these descriptions, and provides back a concrete mediator for bridging purposes. 3. 2.
Finally, the Mobile App is a holon that exposes an interface for a user to interact with the system. The Mobile App holon is attached to a Platform concept that defines the mobile operating system. The Mobile App exposes a search service which searches for a required service (e. g. , food or parking) and order service to submit orders to those service. As depicted in Figure  4 , DeXMS uses the Data eXchange (DeX) API, which implements post and get primitives for sending and receiving messages using existing IoT protocols such as CoAP, MQTT, XMPP, etc. In Figure  4 , the mediator converts temperature data coming from a package (in JSON format through the HTTP protocol) to be received from a system's dashboard (in XML format through the MQTT protocol). Considering a set of heterogeneous IoT devices that have to interconnect with devices deployed in an IoT systems, DeXMS accepts as input their input/output data representation models and synthesizes the required mediators.
As a result, we designed and implemented the Hetero-Genius architecture to leverage ontological descriptions of independent IoT systems to discover their APIs and protocols; as well as to automatically integrate independent systems into an integrated one using an abstract workflow. Interoperability between independent systems is achieved using mediators that translate different data exchange protocols. Experimental outcomes. We implemented the Hetero-Genius architecture in Java and evaluated it using subjective user experiments that involved 26 developers of varying levels of experience. The evaluation results show that our approach reduces development overhead by 47%, on average. Along with improved productivity, correctness of the developed code improved by 55% on average. In addition, we also assessed the feasibility of Hetero-Genius at scale using controlled simulations.
(2012). The advent of highly-integrated silicon probes forced development in another direction. The large volumes of data required above all increasing automation. Several very capable software suites have been developed, including the Klusta suite  (Rossant et al. 2016) , Kilosort  (Pachitariu et al. 2016) , MountainSort  (Chung et al. 2017 ), Spyking Circus  (Yger et al. 2018 ), Tridesclous (Pouzat and Garcia n. d. ) and YASS  (Lee et al. 2020 ). Typically, these must exploit the locality of extracellular signals and project spike waveforms in a reduced dimensional space, where the dimensions are the principal components of spike waveforms. In most suites, spikes are then redetected and sorted using template matching of the cluster waveforms. The software presented here arose from the requirements of spike-sorting cerebellar cortical activity.
To achieve this, we automated cluster fusions within a decision framework ensuring that cluster quality could only be increased. This in turn required jitter correction both within and between clusters and elimination of duplicate spikes. This resulted in a pipeline in which all post-processing was fully automated. We verified the efficacy of our optimisations by injecting spike waveforms into the traces and quantifying their recovery. For cerebellar recordings our package improves the spikesorting analyses of two standard packages. Having established the utility of our software for analysing cerebellar activity, we tested it on the synthetic SpikeForest datasets approximating activity from other brain regions  (Magland et al. 2020 ). Here, too, our software outperformed the constituent Kilosort and MountainSort analyses.
Data The data presented here were obtained from chronic recordings in experiments studying longitudinal changes of cerebellar cortical and particularly Purkinje cell activity during eyeblink conditioning. The data and recording methods are described in detail in the corresponding paper (Llobet et al, in preparation). In brief, 64-site multielectrodes (Cambridge Neurotech) were implanted in mouse cerebellum and advanced towards the putative eyeblink region until cells responding strongly to the air puff directed at the ipsilateral eye were recorded. Continuous recordings throughout learning and extinction sessions (per day, 3 sessions of 30 minutes and 75 trials) were obtained. The mice were walking continuously on a wheel; the possible influence of the locomotion on the cerebellar activity should be borne in mind.
However, the overall statistics of firing are unlikely to be strongly influenced by the brief eyeblink conditioning trials. The recordings were band-pass filtered (0. 1 Hz-7. 5 kHz) at acquisition and before spike sorting the instantaneous median of all 64 channels was subtracted point-wise from each channel; this attenuated common local field potentials, interference and movement artefacts. The data will be made available as part of Llobet et al (in preparation). We also validated our software using the synthetic 'cortical' dataset described in  Jun et al. (2017)  and made available as part of the SpikeForest project  (Magland et al. 2020 ).
Software We used Kilsort 2 (which performed better than the newer versions) and MountainSort 3 and 4. We implemented a small patch in MountainSort 4 enabling the use of an asymmetric window around detected spikes in order better to accommodate the long waveform of complex spikes without introducing unnecessary noise through signal-free portions of the trace preceding detected spikes. Our software package, implemented in Python 3. 8, accepts multiple input files in the 'PHY' format (https://phy. readthedocs. io/en/latest/) or can schedule analyses via SpikeInterface  (Buccino et al. 2020 ). The operation of the package is controlled by parameters that are read from a simple configuration file edited by the user. The package, called Lussac 1  , is publicly available under a free-software licence at https://github.
4 ms and a single complex spike in any interval of 1. 5 ms. (During intra-analysis fusions, the spike that best fitted the cluster mean was retained; during inter-analysis fusions, the first spike of the coincident pair was kept. ) This procedure of testing multiple fusions seeking the maximal quality score is susceptible to overestimating the quality because of noise in its estimation. Below we validate the procedure using datasets for which a ground truth is available, showing that the procedure is nevertheless able to improve the quality of the results.
We first counted the number of clusters of each type detected. We also counted the number of Purkinje cell clusters with both simple and complex spikes. The results are graphed in Fig. 8. Lussac provided an approximately 50 % improvement in the number of simple/complex spike pairs over the single analyses. We also verified the overall accuracy of the spike sorting for native and combined analyses by injecting spikes into the existing traces and observing how they were detected. We injected unfiltered mean spike waveforms from existing clusters generated by the Lussac pipeline, avoiding known existing spike times from the same cluster and their refractory periods, and reanalysed the data. We compared the results with the Lussac package with those from the MountainSort and Kilosort analyses.
The score tends to be lower for clusters with high signal-to-noise ratios, as can be seen in Fig. 9C -F, and was consistently higher for the Lussac package than the individual sorting softwares. It is worth recalling that for complex spikes the contamination is overestimated because of their correlated firing, causing the quality score of all complex spike clusters to be underestimated. We carried out a more detailed analysis of the performance of the different packages on the injected spikes (Fig. 10 ). This shows that Lussac recovered more spikes from more clusters, with on average less contamination and an overall better quality score than both Mountainsort and Kilosort, for both complex and non-complex spikes.
Validation using SpikeForest simulated datasets The successful validation of our package using injected spikes is subject to some circularity, both because we chose to inject spikes that had been correctly detected by our package, which would tend to confer an advantage for recovering the same spikes, and because we evaluated performance using the same quality score guiding cluster fusions and retention. In order to validate our package in a way that avoided this issue, we sought a dataset including a ground truth. We decided to use a synthetic data set generated by simulation of cortical pyramidal cells and internrueones  (Jun et al. 2017 ) and made available through the SpikeForest project  (Magland et al. 2020 ). An additional advantage was that these simulations aimed to reproduce activity from a brain region other than the cerebellum.
Because every spike time is known in the simulated dataset, we were able to determine the absolute quality according to the score of (9). As shown in Fig. 11 , our package outperformed both MountainSort and Kilosort in terms of the number of clusters detected (Fig. 11A ) and cluster scores (Fig. 11B ). The breakdown of the score results depending on signal-to-noise ratio (Fig. 11C ) shows that some of this performance arises, as expected, through better treatment of units with lower signal-to-noise ratios.
Discussion From a starting point of trying to improve the treatment of cerebellar complex spikes in standard spike-sorting analysis packages, we have developed a principled method for combining analyses from different packages and using different parameters. As part of this development, we have fully automated the basic post-processing procedures of combining over-split clusters and applying a quality threshold to the results. A comparative evaluation of our procedures with single analyses from MountainSort and Kilosort demonstrated that the performance of our package was never worse and, in particular for complex spike clusters with lower signal-to-noise ratios, sometimes markedly better. The strictly positive improvement is guaranteed by the use of the final quality metric to guide cluster selection and fusion during the merges-actions are only undertaken if they improve quality.
Thus, insofar as our quality metric is valid, our method should never make things worse. There was, however, no guarantee that merging analyses would improve results. The improvement observed unsurprisingly depended on the signalto-noise ratio of the unit. All analyses methods yielded similar good results for units with a strong signal, although Lussac was generally still able to eke out a small improvement. At lower signal-to-noise ratios, however, the difference was more marked. We believe several different mechanisms contribute. The most obvious is that the ability to merge analyses offers the freedom to optimise parameters for different spike types. The cerebellar complex spike is a fairly extreme example of an unusual spike waveform for which current software has not been explicitly optimised.
This may arise from the (very reasonable) acceleration technique of performing some steps of the analysis on random spike samples of controlled size. However, we can observe that this randomness manifests in stochastic detection of entire clusters at lower signal-to-noise ratios. The procedure of combining multiple analyses enables these intermittent detections to be accumulated, resulting in a higher probability of detection. The use of multiple analyses forced us to automate what were traditionally manual post-processing steps. Depending on the (ever-increasing) size of the dataset, this automation could represent a significant gain in researcher efficiency, but also in objectivity and reproducibility. For our initial objective of improving treatment of cerebellar complex spikes, our package is beneficial, a finding validated by spike injection.
When interpreting the absolute quality metrics of the complex spike clusters, it should be borne in mind that their contamination tends to be overestimated (because of their correlated firing). For non-complex spikes, the gains from merging analyses were more modest but still appeared worthwhile. Moreover, as shown by the validation using the simulated datasets from SpikeForest, combining multiple analyses also seems to offer a significant improvement for activity recorded from other brain regions. Merging analyses offers a simple method for improving spike sorting that does not require the use of a specific sorting algorithm. The approach is especially beneficial if the recordings include units with different waveform characteristics. Using one simple configuration file, the package offers a turnkey approach to automating both post-processing and the merging of multiple analyses.
Acknowledgements This work was supported by the  Fondation pour la Recherche Médicale  through grant ' Equipe   FRM DEQ20160334927 '. This work has received support under the  program 'Investissements d'Avenir ' launched by the  French Government  and implemented by the  ANR  with the references  ANR-10-LABX-54   MEMOLIFE  and  ANR-10-IDEX-0001-02 PSL* Université Paris. We thank the  SpikeInterface team  for very helpful discussions and for implementing suggested extensions. We are also grateful to  Guillaume Dugué  for helpful discussions.
The development of highly efficient, robust and scalable numerical algorithms lags behind the rapid increase in massive parallelism of modern hardware. We address this challenge with the accelerated pseudo-transient (PT) iterative method and present a physically motivated derivation. We analytically determine optimal iteration parameters for a variety of basic physical processes and confirm the validity of theoretical predictions with numerical experiments. We provide an efficient numerical implementation of PT solvers on graphical processing units (GPUs) using the Julia language. We achieve a parallel efficiency of more than 96 % on 2197 GPUs in distributed-memory parallelisation weak-scaling benchmarks. The 2197 GPUs allow for unprecedented tera-scale solutions of 3D variable viscosity Stokes flow on 4995 3 grid cells involving over 1.