dataesr/scientific-paragraphs-categorization huggingface dataset:
Metric We use Accuracy (Acc) and Balanced Accuracy (BalAcc) as evaluation metrics, defined as Acc = 1 | | i∈ 1 ŷi =y i , BalAcc = 1 |C| i∈C Recall i , where is the set of test nodes indexes, y i is the ground truth label for node v i , ŷi is the predicted label for node v i , C is the set of classes, and Recall i is the recall score for class i. Accuracy (Acc) is used in the challenge to rank participants and Balanced Accuracy (BalAcc) is applied for additional analyses since it takes into account the imbalanced label distribution of datasets.
Datasets A total of 15 graph datasets were used during the competition: Five public datasets were directly downloadable by the participants so they could develop their solutions offline. Five feedback datasets were made available on the platform during the feedback phase to evaluate AutoGraph algorithms on the public leaderboard. Finally, the AutoGraph algorithms were evaluated with 5 private datasets, without human intervention. These datasets are quite diverse in domains, shapes, density, and other graph properties because we expect AutoGraph solutions Two teams tie in the 6th and 10th place. We list them both. to have good generalization ability. On the other hand, we intentionally keep the characteristics of 5 feedback datasets and 5 private datasets similar to enable transferability. We summarize dataset statistics in Table  2.
First Placed Winner The 1st winner is from team aister. Their code is open source here 2. The authors use four GNN models, two spatial ones: GraphSage  (Hamilton et al. , 2017)  and GAT  (Veličković et al. , 2018) , and two spectral ones: GCN  (Kipf and Welling, 2017)  and TAGConv  (Du et al. , 2017)  to process node features collectively. For each GNN model, a heavy search is applied offline to determine the important hyperparameters as well as the boundaries. In the online stage, they use a smaller search space to determine the hyperparameters. In order to accelerate the search, they do not fully train each configuration but instead early stop at 16 epochs if the validation loss is not satisfactory. Additional features are used: node degrees, distribution of 1-hop and 2-hop neighbor nodes' features, etc.
Second Place Winner The 2nd winner is from team PASA_NJU. Their code is open source here 3. They also split the solution into two stages: offline stage and online stage. In the offline stage, the authors train a decision tree based on public data and other self collected datasets to classify graph types into one of three classes. Then they use GraphNAS  (Gao et al. , 2020)  to search massively optimal GNN architectures including aggregation function, activation, number of heads in attention, and hidden units. In the online stage, the authors rapidly classify the dataset and fine tune the offline searched model.
Third Place Winner The 3rd winner is from team qqerret. Their code is open source here 4. The core model is a variant of spatial based GNN, which aggregates 2-hop neighbors of a node with additional linear parts for the node itself. Basically, the new embedding of node i is ĥ(i) = j∈N 2 (i) a j h(j) + α(wh(i) + b). Additionally, in the GNN output layer, a few features per node are concatenated for the final fully connected layer, including the number of edges, whether this node connects to a central node that has a lot of edges, the label distribution of 1-hop neighbor nodes, and the label distribution of 2-hop neighboring nodes.
RESULTS We conduct additional experiments after the AutoGraph challenge to further analyze the results. We first reproduce winning solutions and then we compare them with academic solutions. Three gaps are concluded. The first gap is presented as follows and two other gaps are concluded in section 4. 2. Gap #1: Modeling Scope Is the First Gap of AutoGraph Between Academia and Industry. In academia, researchers focus mainly on Neural Architecture Search methods to find better GNN architectures. Their contributions differ in their search space, search strategy, and evaluation methods. However, 3 https://github. com/Unkrible/AutoGraph2020 4 https://github. com/white-bird/kdd2020_GCN industrial solutions, e. g. , 1st solution, focus more on feature engineering and model ensemble. For GNN architectures, they prefer the existing ones with little modification.
Reproducing Winning Solutions We reproduce all winning methods on all the datasets and include their results in Table  3. We observe that all three winning solutions are close in performance and all significantly beat the GCN baseline. On the other hand, in the AutoGraph challenge, due to the nature of the competition, we rank methods based on their accuracy. We state that this is not sufficient to evaluate solutions comprehensively from the scientific perspective. We add the balanced accuracy here just to show that for some methods that show close performance in accuracy, they could diverge a lot in balanced accuracy. Regarding both accuracy and balanced accuracy, we conclude that 1st solution, which comes from Meituan Dianping Company, is indeed the best among the top winners. Thus, we will later use their solutions for comparison with academic solutions.
Neural Architecture Search for GNN We further adopt NAS methods for GNN and compare with the baseline and 1st solution coming from the industry. We choose the recent F 2 GNN  (Wei et al. , 2022)  in our experiment, which searches for data-specific GNN topology. To compare fairly with GCN baselines, we fix the aggregation to GCN and search only the GNN topology, which we call F 2 GCN. Since F 2 GCN requires at least 4 layers, we also run a 4 layer GCN baseline for better comparison. The results are given in Table  4. Gap #2: Effectiveness Is the Second Gap of AutoGraph Between Academia and Industry. We observe from Figure  2  and Table 4 that all baselines and F 2 GCN methods are not as good as 1st winning solution. However, for many datasets,  L2, L4 mean 2, and 4 layers for the GNN architecture.
Numbers in parentheses are the relative accuracy with respect to 1st solution. We regard 1st solution as 100%. The last line is the average percentage. e. g. , e, f, g, i, j, F 2 GCN is very close to the best industrial solution. On average, F 2 GCN which focuses only on architecture search reaches 97. 3% of the best solution. Note that the 1st solution constructs additional node features and uses multiple GNN architectures for ensemble while F 2 GCN does not use any feature engineering or model ensemble. This shows the effectiveness of the winner's engineering practices and F 2 GCN's adaptive architecture search. The winning teams also have access to public datasets and the public leaderboard to iteratively fine tune their methods. F 2 GCN does not assume any prior knowledge of the datasets, which shows further its effectiveness.
Numbers in parentheses are the relative number of parameters with respect to 1st solution. We regard 1st solution as 100%. The last line is the average percentage. To better understand the solutions, we calculate the number of parameters of the baseline, F 2 GCN, and the 1st solution, as shown in Table  5. Gap #3: Efficiency Is the Third Gap of AutoGraph Between Academia and Industry. From Figure  3  and Table  5 , F 2 GCN uses significantly fewer parameters than the best industrial solution on most datasets (13 out of 15). On average, F 2 GCN consumes 45. 1% of the 1st solution in terms of parameter size, which is quite resource-efficient. Note that feature engineering and ensemble do not contain additional parameters and basically, F 2 GCN searches one GNN model to compete with the ensemble of 4 types of GNN models in the 1st solution.
CONCLUSION We organized the first Automated Graph Learning (AutoGraph) Challenge at KDD Cup 2020. We presented in this article its settings, dataset, and solutions, which are all open sourced. Furthermore, we ran additional post-challenge experiments to compare the baseline [Graph Convolution Network (GCN)], the winning solution (feature engineering-based ensemble of various Graph Neural Networks), and a recent and efficient Neural Architecture Search (NAS) for the GNN method called F 2 GCN. This article provides results that could bridge the gap between academic research and industrial practices, by correcting the bias favoring certain approaches. This gap is currently at 3 aspects: Gap #1 modeling scope. (academia focuses more on model-centric approaches, emphasizing NAS; industry emphasizes data centric approaches and feature engineering); Gap #2 effectiveness.
(academic solutions are perceived by the industry to be less effective than their own counterpart); Gap #3 efficiency. (academic solutions are perceived to be parsimonious or slower than industrial solutions). Our results indicate that the "academic" NAS-based approach that we applied attains performances closely matching those of the winning industrial solution while being both faster and more parsimonious in the number of parameters, therefore, closing Gap #2 and #3. Moreover, we hope that these results will help reduce Gap #1, by encouraging industrial practitioners to apply NAS methods (and particularly F 2 GCN), eventually combining the best of both approaches. We believe the results we obtained are significant since they involve a benchmark on 15 datasets. KDD Cup 2020, contributed to manuscript revision, read, and approved the submitted version.
ACKNOWLEDGMENTS We acknowledge the following people for helping organize the AutoGraph challenge:  Xiawei Guo  and  Shouxiang Liu. We also appreciate the following people and institutes for open sourcing datasets that are used in our use cases:  Andrew McCallum ,  C. Lee Giles ,  Ken Lang ,  Tom Mitchell ,  William L. Hamilton ,  Maximilian Mumme ,  Oleksandr Shchur ,  David D. Lewis ,  William Hersh ,  Just Research , and  Carnegie Mellon University ,  NEC Research Institute, Carnegie Mellon University, Stanford University, Technical University of Munich, AT&T Labs , and  Oregon Health Sciences University.
Then a two layer vanilla GCN is applied for learning node embeddings. Another MLP with softmax outputs the final classification. Dropout is used. All the hyperparameters are fixed by our experience. There is no time management since the model is simple and one full training will not cost more time than the allowed budget. table. During model training, an MLP is first used to map node features to the same embedding 1 https://github. com/AutoML-Research/AutoGraph-KDDCup2020 dimension.
Implementation We used an open PyTorch implementation for both unpaired and paired image-to-image translation, providing source codes to train and test both CycleGAN and Pix2Pix models  (Isola et al. , 2016; Zhu et al. , 2017). 4 Our code was run on Google Colab, including data retrieval. The step of pairing data was conducted with a QGIS script. All sets of samples (around 80% for training, and around 20% for tests), the images produced by the tests, as well as the trained model, the loss functions logs, and if possible, the intermediary images and models, are retrieved and stored. The experimental computational time implies around 200 epochs, knowing that an epoch is between 130 and 170 seconds on Google Colab, making around 8 hours in total. 5.
Old map ST to ortho-images (CycleGAN) First trials have been made on a very small dataset to test our second hypothesis (H2), e. g. CycleGAN could be more powerful to manage complex-styled spatial arrangements. Extensive tests have been conducted with the all sets of samples, to approach and refine both our protocol and the generated results. Figure  5  presents our best results of three old map style transfers to an ortho-image. First, an 'Etat-Major' style transfer, with roads-enhancement, effectively visually augments the content of the ortho-image with this particular old map style: a mixed style is created between the ortho-image and the 'Etat-Major' style, i. e. mixing colors, textures and contrasts, as well as emphasizing parcels, outlines and urban spots very nicely (Figure  5  -1).
Discussion About the task design and the control over the map ST. We propose a task of map ST which could be now refined, as soon as we better understand the structure of the expected style through the learning process. According to our expectations about the map style transfer, the task and the protocol could be adapted and refined, in order to specifically target some map styles, from very simple to complex ones, with adapted additional controls over the datasets, the task and the process, to drive the process. In this paper, we have been considering augmented road networks only, with particularly efficient and nice effects on the map style transfer and the quality of the results, in terms of the preservation of the spatial topology of those networks, and also on neighboring objects. This validates the choice of this control on the task.
Notes 1. https://www. tensorflow. org/tutorials/generative/style_transfer 2. https://www. geoportail. gouv. fr/ 3. A subtle work has been done here to select suitable, complete, sufficiently homogeneous and typical maps, according to their specific assembly table, in order to avoid some pitfalls between some map types: problem of sheets connection, missing or incomplete data in some area, uncolored sheets, etc. 4. https://github. com/junyanz/pytorch-CycleGAN-and-pix2pix 5. Important to note that technical choices (sampling plan, size samples, tools) on such topic are quite related to computational time issues: this has been strengthened by the pandemics context and the requirement to work remotely on laptops.
27±0. 11. For water percentage estimates we used the plagioclase hygrometer calibration of  Waters and Lange (2015). Calculations were made using a spreadsheet that we downloaded from the American Mineralogist data repository (http://www. minsocam. org). Here, we also assumed that plagioclase crystals crystallized in equilibrium with a liquid having a bulk rock composition. In addition, we used temperatures previously obtained with the plagioclase-liquid thermometer and a pressure value of 150 MPa estimated from MELTS simulations with Popocatépetl magmas  (Witter et al. , 2005). The estimate of the standard error in the hygrometer model is 0. 35 wt% H 2 O, while the estimated accuracy of the thermobarometer is ±36ºC.
Processing delay measurement We assessed the processing delay of hearing aid apps using three click sounds (centered at 500 Hz, with an interval of 1 second) The electroacoustic performance of most applications (apps) was poorer than that of a hearing aid. Many hearing aid apps exceeded the recommended level for processing delay. High-end apps had a better ability to match amplification targets than low-end apps in simulated real-ear measurements. Only a few apps improved the signal-to-noise ratio in a headand-torso simulator. The second was first recorded by the microphone of the smartphone, processed by the app, and then presented by the earbud to the headphones' microphone. The processing delay was calculated in MATLAB (R2019a; MathWorks, Natick, MA, USA) using a correlation function, yielding the time between the first and second recordings of the click.
Hearing instrument testing The hearing instrument testing (HIT) and simulated real ear measurement (S-REM) were performed using Affinity 2. 0 software (ver. 2. 6. 0; Interacoustics, Middelfart, Denmark). One of the earbuds connected to the smartphone was linked with adhesive putty (Blu Tack; Bostik, Colombes, France) to a 2-mL coupler (HA-1). To ensure reliability of the results, measurements were performed twice in succession. The high-frequency average (HFA) of each hearing aid app was measured at three frequencies, i. e. , 1, 1. 6, and 2. 5 kHz. We measured the amplification characteristics through the HFA full on gain (FOG) and output sound pressure level (OSPL)90, which correspond to the outputs of 50 dB SPL and 90 dB SPL inputs, respectively. Then, we explored the sound quality with high-frequency peaks and other parameters, such as equivalent input noise (EIN) and total harmonic distortion (THD).
ACKNOWLEDGMENTS We are grateful to  Ms. Ye Jin Byeon  for her help in collecting data. This research was supported by a grant from the  Patient-Centered Clinical Research Coordinating Center  funded by the  Ministry of Health & Welfare, Republic of Korea  (grant number:  HI19C0481 ,  HC19C0128 ). We thank  Dr. Cho Wan-Ho  and  Dr. Chang Jiho  from the  Korea Research Institute of Standards and Science (KRISS; Daejeon, Korea)  for their help and thoughtful advice regarding signal-to-noise ratio measurements using their material.
e. where the threshold of 'Reference conditions' is not reached but still good). This minimal light threshold could be selected according to the most sensitive species (which might be always low) or depending on conservation goals and ecosystems. Indeed, this threshold will determine which areas will have to be preserved and which areas will have to be restored. In France, 'Pyrenees National Park' determined a threshold of sensitivity to light, using data previously collected on Rhinolophus and Myotis bats. The results show that, whatever the bat species observed, from a level of light pollution of about 19. 8-20. 5 mag/arcsec 2 , the number of contacts established with each species decreases  (Fresse, 2018). Then, to design their dark infrastructure, three classes of darkness quality were retained: poor (15.
The input data were obtained from the field, scientific literature, or expert opinion on 5 species. These examples show that dark infrastructure is a tool that various stakeholders adopt to preserve and restore darkness, in both urban and natural contexts. The spatial scales are also variable, ranging from a single municipality or even a neighbourhood to vast territories. These pioneering projects will be a valuable aid for subsequent initiatives. For this purpose, a further work will be necessary to review the methods applied in these projects to characterize dark infrastructure, to identify the technical levers for implementing better lighting management in these spaces, and to list any difficulties encountered.
, 2017). More than 400 AOPs have currently been proposed, although only a small fraction  (Carvaillo et al. , 2019)  as of the April 2022) have fully cleared the OECDs review process  (AOP, 2021). Their aim is to support the assessment of health risks by describing evidence that links KEs to an apical AO. They can then be used in human toxicology, ecotoxicology, and to support the development of Integrated Approach to Testing and Assessment (IATA) that can be used for regulatory purposes. AOPs are stressors-agnostic. Building an AOP is complex and must ideally follow the OECD handbook guidelines with several quality assessments  (OECD, 2018). Based on the AOP-wiki database (https://aopwiki. org/, last accessed March 2022), the central repository for AOPs, the AhR has already been proposed in several AOPs, but never in one characterized by the AO breast cancer related to death.
Data input, automatic identification, and extraction of the available knowledge An artificial intelligence based method, named AOP-helpFinder (htt ps://aop-helpfinder. u-paris-sciences. fr/index. php), was used to scope the available relevant literature  (Carvaillo et al. , 2019; Zgheib et al. , 2021; Rugard et al. , 2020 Jan 1; Jornod et al. , 2020; Jornod et al. , 2022). This tool screens the contents of abstracts from the PubMed database using lists of key words, including stressors, MIEs, KEs and AOs. It then automatically finds, and extracts links co-mentioned in abstracts between pairs of events (MIE/KE/AO) and ranks them using graph theory. In the proposed study, the key words included MIEs (AhR …), KEs (migration, hypoxia, colony…) and an AO (breast cancer, mammary tumor…). All biological events used for the screening were defined using AOP-wiki database and personal keywords.
The newly established version of the AOP-helpFinder tool was applied on the more than 30 million available abstracts from PubMed database, which allowed us to use the two proposed options: i) the refinement filter based on machine learning method (lemmatization process of text) combined with deletion of words of context in order to improve the accuracy of the outputs, and ii) the reduced search option which allow to screen only a part of the abstracts  (Jornod et al. , 2022). This allows to not screen the introductory part of an abstract which usually refer to a study hypothesis, with the aim to decrease false positives. In order to cross-check the selected concepts, and to support the evidence in favor of the proposed AOP, all selected articles (after text mining and manual curation) were run through the PubTator Central tool (https://www.
Third, the OECD guidelines were used to evaluate the KERs: i) biological plausibility (mechanistic approach), ii) empirical support (dependent change and the experimental data present), iii) evidence and finally iv) quantitative evidence (closely linked to the tailored Bradford-Hill criteria). These were evaluated using the system proposed by the OECD (high, moderate, low) and detailed in the guidebook. Finally, the AOP was overall assessed using the three primary considerations: the biological plausibility, the essentiality of each KE and the empirical support of KERs. Since the detail of the evaluation of the overall weight assessment of the AOP is not described in the OECD guidebook, this evaluation was subjective and decided by our team using all the different elements studied before.
AOP-helpFinder text mining tool A total of 17,760 articles related to the MIE, AhR or aryl hydrocarbon receptor, were identified in the PubMed database (as of December 2021). This list of AhR-related abstracts was used to screen the ones concerning the putative AOP. All abstracts had to mention at least one event from the AO list (breast cancer or synonym) and one event from the KE list (KEs leading to metastasis: invasion, stemness…). The list of key words used for searching AOs and KEs can be found in Supplementary Table  1. After merging the identified abstract using their PubMed identifications (PMID), only 407 abstracts were kept from the AhR list with both a key event from our AO list and our KE list (Fig. 1 ). Four steps were used for the manual curation, that was performed by several experts.
The 113 articles were also run through the PubTator Central tool to screen for biomedical concepts. The most frequently found disease was breast cancer, followed by toxicity and concepts concerning metastasis and aggressivity. TCDD (or dioxin, or 2,3,7,8-tetrachlorodibenzo-pdioxin) was by far the most used chemical followed by estrogen, indole-3-carbinol, alpha-naphthoflavone, polycyclic aromatic hydrocarbons and hexachlorobenzene, mainly ligands of the AhR. Most Fig. 1. Workflow of the text mining procedure to identify all abstracts related to the events searched for implementation of our AOP. First, the AOP-helpFinder tool screens the available literature in the Pubmed database using the key word "AhR" (our MIE). It then selects the abstracts containing both an AO (breast cancer) and a KE (aggressiveness).
Conclusion In conclusion, we propose a simple and robust AOP associating activation of the AhR and breast cancer related death through migration, invasion, inflammation, and neo-angiogenesis. The main limitation of our AOP is the variability of the pathways activated by the AhR ligands and the possibility of alternative pathways. To overcome these limitations, our KE had to be present no matter the ligand and in a consistent way. Moreover, the originality of our work lies in the use of artificial intelligence too such as AOP-helpfinder, which enables a thoroughly search of existing knowledge in the PubMed database and PubTator  (Carvaillo et al. , 2019; Zgheib et al. , 2021; Rugard et al. , 2020 Jan 1). Therefore, our literature review was complete and evidence in favor of our proposed AOP was overwhelming.