dataesr/scientific-paragraphs-categorization huggingface dataset:
Recruitment We recruited participants on Amazon Mechanical Turk for all five studies using the services of CloudResearch (previously TurkPrime,  46  ). For Study 1 we recruited about 300 participants (we started with 50 participants to ensure that the demographic filters worked correctly, and then followed up with 249 additional participants). For Study 2 the sample size was determined based on ref. 23; consequently, about 300 participants in each role who showed adequate comprehension of the rules of the third-party punishment and trust games were recruited, yielding a total of 3761 individuals whose choices we analyze (note that these also cover the human only conditions, i. e. , all data presented in this paper have been part of our data collection). In Study 3 the sample size was determined based on Study 2, consequently, about 300 participants were recruited in the role of Trustors in each condition, yielding a total of 2514 individuals whose guesses we analyze.
No participants in Study 2 were allowed to participate in Study 3; otherwise their experience in Study 2 may have influenced their guesses in Study 3. In Study 4 we invited the same participants who took part in Study 2 in the role of Trustor who were paired with Helpers. Therefore, all our analyses presented for Study 4 follow a within-person design; 458 individuals (48. 67% of the sample of Trustors in Study 2) participated in Study 4. As detailed above, we analyze data from individuals who believed the norm-manipulation. Sample composition is described in Supplementary Table  S7. In Study 5, we recruited on the same platform (to be able to track repeat participation, as many MTurk workers are also present on other similar platforms, like Prolific), but given the data quality restrictions applied, we did not eliminate prior participants from the pool of those who could take part, yielding an additional 1343 new individuals, and 734 returnees.
First, we created a survey group within CloudResearch for all HITs associated with this study. The same MTurk worker cannot take multiple surveys in the same CloudResearch survey group. Second, we activated the "ballot box stuffing" option in Qualtrics, to prevent multiple entries from the same IP address. In addition, we sequentially compiled a list of the MTurk IDs of workers who entered the study using an external SQL database and we automatically verified that each new worker was not already on this list. Our sample is limited to unique participants who took the study once (unless explicitly stated otherwise). Chronologically, Study 1 was conducted after Studies 2-5, and repeat entry in this study was possible to capitalize on data quality filters. In Studies 1, 3, and 5 in addition to these filters, we showed the HIT only to MTurk workers who were "CloudResearch Approved Participants" to further enhance data quality using the services of Clou-dResearch.
These filters alleviate concerns of data quality which were especially paramount at the time of data collection for Study 3  47, 48  , however they were not yet available at the time of conducting Study 2, which is why they were not employed. In addition to these filters, in Study 1, we collected data from Twitter users, which is possible via prescreening filters managed by CloudResearch. Indeed, 98% of our sample are self-reported Twitter users as intended, and 93% of these participants use Twitter at least weekly. Last, but not least, all five studies contain eight comprehension check questions: four related to the third-party punishment game, and four related to the trust game. In Study 1, participants were only allowed to proceed to the questions that form our main focus when they have answered all questions correctly.
In addition to comprehension checks, we used manipulation checks in Studies 2-5 to make sure that participants were able to recall the treatment they participated in (i. e. , participants were asked to identify the bot(s) if any in their experimental condition), and to gauge in Studies 4 and 5 if they believed the treatment, as people cannot be randomized to beliefs, and therefore, we analyze the data of participants who were treated as intended. Our robustness analyses investigate if results on the full sample, and the restricted sample to participants with full comprehension and passing manipulation checks are consistent. The manipulation checks affected the sample in Studies 4 and 5 the most. In Study 4, interestingly, participants were less likely to accept the consensus information as true (52%) when presented with cases of humans sharing with bots, compared to the other cases.
Sample composition and compensation Data collection for Study 1 took place between 7-15 of June 2022, for Study 2 between 8-13 of August, 2019; for Study 3 between 14-19 of July, 2020; for Study 4 between 3-17 of August, 2020; and for Study 5 between 11-28 of September, 2021. Across the five studies we recruited 7917 individuals who completed one or more of the five studies. In Study 2, 2287 people failed to answer at least two comprehension check questions in either of the two sets on two attempts, whom we do not refer to as participants, and whose data we do not analyze (note that they have never made decisions in the game). In this study we analyzed data from 3761 individuals who showed adequate comprehension. In Study 3, 2514 individuals participated, and 2066 people (82% of the sample) showed the same level of comprehension as those in Study 2.
042, 0. 248] 90% window of the trust-gain. Note however, that these combined results should be interpreted with caution due to lack of evidence that the homogeneity assumption holds. Prior research identified selective attrition as a threat to causal identification when effect-heterogeneity is present  50. Therefore, we calculated the percentage of participants who left the study after passing the comprehension check questions across experimental conditions, i. e. , after they saw the treatment they were assigned to, yielding an average attrition rate across treatments of 2. 74% in Study 2, 0. 36% in Study 3, 0. 23% in Study 4, and 0. 67% in Study 5, respectively. We found no systematic differences across the conditions in their demographic composition within study (Supplementary Note 10), and therefore it is unlikely that selective attrition compromises data quality and our inferences.
Pre-registration The pre-registration of Study 2, took place on Open Science Framework (previously Experiments in Governance and Politics, egap. org) prior to the analysis of the outcome data (https://osf. io/cx5yv/, February 2, 2020). We pre-registered Study 3 and 4 prior to data collection (https://osf. io/3fhzn/, July 14, 2020 and https://osf. io/x53a4/, August 3, 2020, respectively). For Study 2 we registered an amendment after the analysis of the outcome data (https://osf. io/6r98y/, September 12, 2020). Here, we discuss the amendment in detail (for all hypotheses, see Supplementary Fig. S1 ). The pre-analysis plan (PAP) originally contained 13 clearly stated hypotheses. In the amendment, we made two modifications to the PAP: on the one hand, we added four new hypotheses, all of which were direct consequences of hypotheses already registered in the original PAP.
Acknowledgements We would like to thank  Peter Bearman ,  Byungkyu Lee , and  Mario Molina  for their valuable comments, and the participants of the  Indiana University Network Science Institute  workshop for their insights and suggestions. We thank  Katharina Klaunig ,  Irene Lin ,  Hannah Kasak-Gliboff , and  Yao Xu  for research support coding the qualitative responses. K. M. acknowledges the support of the  Research Enhancement Funds received from NYUAD , funding from the  NYUAD Center for Interacting Urban Networks (CITIES) , funded by  Tamkeen  under the  NYUAD Research Institute  Award  CG001 ; K. M. and T. R. acknowledge discretionary research support received from  NYUAD ; J. F. B acknowledges support from the  Agence Nationale de la Recherche  ( ANR-19-PI3A-0004  and  ANR-17-EURE-0010 ), and the research foundation  TSE-Partnership.
The proposed architectures were tested on four datasets with two evaluation scenarios (intra and cross datasets). Experimental results showed that both PAtt-Unet and DAtt-Unet improve the performance of Att-Unet in segmenting Covid-19 infections. Moreover, the combination architecture PDAtt-Unet led to further improvement. To Compare with other methods, three baseline segmentation architectures (Unet, Unet++, and Att-Unet) and three state-of-the-art architectures (InfNet, SCOATNet, and nCoVSegNet) were tested. The comparison showed the superiority of the proposed PDAtt-Unet trained with the proposed hybrid loss (PDEAtt-Unet) over all other methods. Moreover, PDEAtt-Unet is able to overcome various challenges in segmenting Covid-19 infections in four datasets and two evaluation scenarios.
Due to data limitation, most existing works have used small data to train their models  (Fan et al. , 2020; Abdel-Basset et al. , 2021; Liu et al. , 2021; Elharrouss et al. , 2021). On the other hand, these works have only used infected slices to train and test their models  (Fan et al. , 2020; Wang et al. , 2020; Abdel-Basset et al. , 2021; Liu et al. , 2021; Kumar Singh et al. , 2021; Elharrouss et al. , 2021; Wang et al. , 2022). However, this does not correspond to the real scenario where the infections in CT-scan usually occur only in a few slices. Therefore, it is interesting to investigate the performance of the segmentation approaches for both infected and uninfected slices corresponding to a realistic configuration. In the primary datasets used for Covid-19 segmentation, only the middle slices of the CT scans were used.
However, infection can occur in any area of the lung. In recent months, additional datasets for Covid-19 segmentation have been made publicly available  (Radiologists, 2019; Ma et al. , 2021; Zhang et al. , 2020). The aim of this work is to segment Covid-19 infections in a real scenario (all slices of 3D CTs are used for training and testing). To this end, inspired by the Att-Unet architecture, we propose three different architectures. Moreover, we compare our results with three CNN-based segmentation architectures and three published architectures for Covid-19 segmentation whose implementation codes have been made publicly available. Training segmentation models with the Cross-Entropy (CE) loss function usually have problems in predicting object boundaries, resulting in fuzzy boundaries  (Qin et al.
, 2019). In addition, a Covid-19 infection may have multiple small infection regions. To address these issues, we proposed an additional BCE loss function that focuses on the infection boundaries. The small infection regions have more boundary pixels (relative to the area of the infection) than the larger ones. Therefore, the proposed hybrid loss function weighs the small infection regions more than the large infections. For pulmonary and Covid-19 infections, there may be high variability in screening and changes in intensity and illumination from one CT-scanner to another and from one imaging environment to another. Patient age, ethnicity, and severity of Covid-19 infection may also affect the appearance of the CT scan. Therefore, it is interesting to investigate the effectiveness of segmentation methods in cross-datasets experiments.
The cross-datasets experiments investigate the generalization ability of the proposed machine learning solutions and can provide a clear idea of their applicability in the real world. In order to provide an efficient Covid-19 infection segmentation solution that can overcome the above challenges from different aspects, our main contributions can be summarized as follows: Decoder Att-Unet (PDAtt-Unet) architecture using the pyramid and attention gates to preserve the global spatial awareness in all of the encoder layers. In the decoding phase, PDAtt-Unet has two independent decoders that use the Attention Gates to segment infection and lung simultaneously. • To address the shortcomings of the binary cross entropy loss function in distinguishing the infection boundaries and the small infection regions, we propose the   loss that focuses on the edges of the infection regions.
• To evaluate the performance of our proposed architectures, we use four public datasets with two evaluation scenarios (intra and cross datasets), all slices from CT scans are used for the training and testing phases. • To compare the performance of our approach with other CNNbased segmentation architectures, we use three baseline architectures (Unet  Ronneberger et al. , 2015 , Att-Unet Oktay et al. , 2018 and Unet++ Zhou et al. , 2018)  and three state-of-the-art architectures for Covid-19 segmentation  (InfNet Fan et al. , 2020 , SCOATNet Zhao et al. , 2021a , and nCoVSegNet Liu et al. , 2021). The experimental results show the superiority of our proposed architecture compared to the basic segmentation architectures as well as to the three state-of-the-art architectures in both intradatabase and inter-database evaluation scenarios.
The codes of the proposed architectures and the loss function have been made publicly available at. https://github. com/faresbougourzi/PDAtt-Unet. (Last accessed April, 20 ℎ 2022). The remainder of this paper is organized as follows: Section 2 presents some related work on segmentation of Covid-19 infections from CTscans. In Section 3, we describe our proposed approach. Section 4 consists of the description of the datasets used and the evaluation scenarios and metrics. Section 5 presents and discusses the experiments and results. Section 6 shows some segmentation examples. Finally, Section 7 concludes the paper.
proposed dilated dual attention U-Net (D2 A U-Net) to automatically segment lung infection in Covid-19 CT slices. To evaluate the performance of their approach, they took only the infected slices from Segmentation dataset nr. 2 (Radiologists, 2019) and the COVID-19-CT-Seg dataset  (Ma et al. , 2021)  and used them as training data. As test data, they used the 100 slices of the dataset presented in Radiologists (2019). Liu et al. (2021)  proposed a two-stage cross-domain transfer learning system. Their framework consists of two main components. First, they proposed nCoVSegNet, a deep learning-based approach that uses attention-driven feature fusion and large receptive fields. Second, they trained nCoVSegNet with a cross-domain transfer learning strategy that leverages knowledge from natural images (i.
e. , ImageNet) and medical images (i. e. , LIDC-IDRI) to reinforce final training on CT images with Covid-19 infections. They used MosMedData dataset  (Morozov et al. , 2020)  for the final stage of transfer learning (40 slices for training and 10 slices for testing), then they tested the obtained model on the COVID-19-CT-Seg dataset  (Ma et al. , 2021). In  Wang et al. (2022) , Ruxin Wang et al. proposed an encoderdecoder CNN-based system for segmenting Covid-19 infections from the CT scans. Their approach is based on aggregation of peer and crosslevel contextual learning. To capture the complex structure, they used an autofocus module to detect and incorporate multiscale contextual information at the peer level. They also proposed a panoramic module to capture complementary fine details and semantic information.
To evaluate the performance of their approach, Ruxin Wang et al. combined different datasets, MosMedData dataset  (Morozov et al. , 2020; Ma et al. , 2021) , COVID-19 CT segmentation (Radiologists, 2019), and Segmentation dataset nr. 2 (Radiologists, 2019), and removed the slices that have no infection. Then, the obtained dataset was randomly divided into a training dataset and a test dataset (it is not clear whether the patient-independent protocol was followed or not). Deng-Ping Fan et al. proposed a deep network (Inf-Net) for segmentation of Covid-19 lung infections  (Fan et al. , 2020). Their Inf-Net approach uses parallel partial decoders to aggregate high-level features and create a global map. They also used implicit reverse attention and explicit edge attention to model the boundaries and improve the representations.
, 2020; Abdel-Basset et al. , 2021; Liu et al. , 2021; Elharrouss et al. , 2021). This reduces the reliability of these approaches in real-world scenarios, where Covid-19 infection exhibits high case-to-case variability in shape, type, location, and intensity  (Sun et al. , 2020; Kumar Singh et al. , 2021; Laradji et al. , 2021). On the other hand, many state-of-the-art approaches were trained and evaluated using only the infected slices  (Fan et al. , 2020; Wang et al. , 2020; Abdel-Basset et al. , 2021; Liu et al. , 2021; Kumar Singh et al. , 2021; Elharrouss et al. , 2021; Wang et al. , 2022). However, this does not correspond to the real scenario, where the infections in CT-scan usually occur only in a few slices with different degrees. In addition to the above two evaluation limitations, most of the state-of-the-art approaches were evaluated only for the intradataset scenario.
However, the cross-dataset scenario investigates the generalization capability for real-world application scenarios. This is a very important evaluation scenario, especially in the case of CTscanning data where the intensity and illumination vary greatly from one CT-scanner to another and from one acquisition setting to another. The goal of this work is to develop an efficient CNN-based approach for segmenting Covid-19 infections from CT-scans in real clinical situations, using all slices of 3D CT-scans for training and testing. In addition, extensive evaluations are performed on four datasets for intradatasets and cross-datasets to investigate the generalization ability of the proposed approach in different real-world scenarios.
The ground-truth edge pixels are obtained by applying the morphological gradient to the ground-truth infection regions. After obtaining the ground-truth edge pixels, the BCE loss function is applied between the ground-truth pixels and the corresponding pixels in the mask map. The weight of the edge loss  is experimentally determined for each dataset. For simplicity, the training of the proposed PDAtt-Unet using the hybrid loss function is referred to as PDEAtt-Unet. In the proposed edge loss, the pixels at the infection boundary, which cause more confusion in the prediction than the other pixels, are weighted more heavily. Unlike other targets where the edges in the original images have significant structure, the edges of the Covid-19 infection have extremely few pixels, so adding a head or a branch to predict the edges is useless for solving multiple tasks, as found experimentally.
Datasets To evaluate the performance of our proposed architectures, we used four publicly available datasets, which are COVID-19 CT segmentation (Radiologists, 2019), Segmentation dataset nr. 2 (Radiologists,   (Ma et al. , 2021)  consists of 20 Covid-19 CT scans. All cases contain Covid -19 infections. This dataset has been labeled by many experienced radiologists and the labeling is consistent. The entire lung mask, containing both normal and pathologic regions, was labeled. There are a total of 1844 infected slices out of 3520 slices. For CC-CCII Segmentation Dataset  (Zhang et al. , 2020) , lesions were segmented from CT slice images from the China Consortium of Chest CT Image Investigation (CC-CCII). A total of 750 CT slices from 150 Covid-19 patients were manually segmented into background, lung field, ground glass opacity (GGO), and consolidation.
The used datasets have long and similar names. To avoid confusion between names, we refer to them as Dataset_1, Dataset_2, Dataset_3, and Dataset_4, as shown in Table  2. In our experiments, we explored both intra-dataset and cross-dataset scenarios. For intra-dataset experiments, Dataset_2 (Radiologists, 2019), Dataset_3  (Ma et al. , 2021)  and Dataset_4  (Zhang et al. , 2020)  are randomly split into 70%-30% as training and test splits, respecting the patient-independent splitting scenario in which all slices of a patient belong to only one split (training or validation). For the cross-dataset evaluation scenario, we used the trained models on Dataset_2 (trained on the training data 70%) and tested them on Dataset_1 (Radiologists, 2019), Dataset_3  (Ma et al. , 2021) , and Dataset_4  (Zhang et al.
For these types of metrics,   ,  ,   , and   were first calculated for all test images, and then these metrics were calculated using the Eqs. (  7 ), (  8 ), (  9 ),  (10), and (11). However, the Dice score is the macro version of the F1 score (the F1 score is calculated for each test image, then the average over all test images is the Dice score). For  test images, the Dice score is defined as follows: Dice score = 100 ⋅ 1 𝑁 𝑁 ∑ 𝑖=1 2 ⋅ 𝑇 𝑃 𝑖 2 ⋅ 𝑇 𝑃 𝑖 + 𝐹 𝑃 𝑖 + 𝐹 𝑁 𝑖 ( 12 ) where    ,    ,    and    are True Positives, True Negatives, False Positives and False Negative for the th image, respectively. In the experimental results, it is normal to have low dice score average since in the three datasets there is a large number of uninfected slices, where totally correct prediction of the masks (black mask) will give 0 dice score for these slices which will be included in calculating the average.
Experimental setup For deep learning training and testing, we used the Pytorch  (Paszke et al. , 2019)  library with NVIDIA GPU Device GeForce TITAN RTX 24 GB. The batch size used consists of 6 images. We trained the networks for 60 epochs. The initial learning rate is 0. 01, which decays by 0. 1 after 30 epochs, followed by another decay of 0. 1 after 50 epochs. Furthermore, we used active data augmentation techniques which are: rotation using random angle between -35 degrees to 35 degrees, and random horizontal and vertical flipping. The values of  for the Edge loss in the proposed hybrid loss function (illustrated in Section 3. 4) is experimentally set to 0. 7, 2 and 0. 9 for Dataset_2, Dataset_3 and Dataset_4, respectively.
Experimental results on intra-dataset scenario In this section, we evaluate the performance of the proposed PDEAtt-Unet and compare it with Unet  (Ronneberger et al. , 2015) , Att-Unet  (Oktay et al. , 2018) , Unet++  (Zhou et al. , 2018) , InfNet  (Fan et al. , 2020) , SCOATNet  (Zhao et al. , 2021a)  and nCoVSegNet  (Liu et al. , 2021). Tables  3 4 5 show the experimental results using Dataset_2 (Radiologists, 2019), Dataset_3  (Ma et al. , 2021)  and Dataset_4  (Zhang et al. , 2020) , respectively. From the results of Tables  3, 4 , and 5, our proposed approach (PDEAtt-Unet) outperforms three baseline segmentation architectures and three state-of-the-art approaches. Specifically, the experimental results of Dataset_2 show that our approach achieves 77. 60%, 41. 68%, and 63. 40% for F1 score, Dice score, and IoU, respectively.
In this dataset, the best competing approach is SCOAT-Net  (Zhao et al. , 2021a) , which achieved 69. 39%, 31. 56%, and 53. 13% for F1 score, Dice score, and IoU, respectively. Therefore, our approach outperformed the SCOATNet approach by 8. 2%, 10. 1%, and 10. 3% for F1 score, Dice score, and IoU, respectively. Using the intra-dataset experiment results for Dataset_3 ( Table  4 ), we find that InfNet  (Fan et al. , 2020)  outperforms the other competing methods, achieving 72. 91%, 37. 02%, and 57. 37% for F1 score, Dice score, and IoU, respectively. On the other hand, our proposed PDEAtt-Unet approach outperforms all methods, especially InfNet by 2. 60%, 4. 25%, and 3. 3% for F1 score, Dice score, and IoU, respectively. Table  5  shows that nCoVSegNet  (Liu et al. , 2021)  outperforms the other state-of-the-art comparison approaches (InfNet and SCOAT-Net) and the baseline architectures (Unet, Unet++, and Att-Unet).
On the other hand, our proposed PDEAtt-Unet approach outperforms nCoVSegNet by 1. 3%, 2. 3%, and 1. 8% for F1 score, Dice score, and IoU, respectively. The intra-dataset experiment results show that our proposed PDEAtt-Unet approach outperforms all competing methods and the second best approach differs from one dataset to another. This shows that our approach is efficient and consistent under different evaluation conditions.