dataesr/scientific-paragraphs-categorization huggingface dataset:
PyREnArA-Trait-Based Recordings of Tools to Capture Variability The artifacts are recorded using the half-automated system PyREnArA (Python-R-Environment for Artifact Analysis; Suppl. 2), which has been developed over several semesters in seminars with students from the Friedrich-Alexander-University Erlangen-Nürnberg and the University of Cologne. The coding (RJ, FL, and GR) also has been part of student projects. PyREnArA can be accessed following this link: https:// doi. org/ 10. 5281/ zenodo. 77780 71  (John et al. 2023 ). In the first step, artifact drawings and photos are scanned and oriented according to a standard protocol to minimize measurement errors. At this point, the program extracts traditional as well as new traits of object morphology and size, such as dimensions, the geometric center, the outline-to-area ratio, angles, symmetry, the position of the longest extension (e.
These bars are used by the researcher to color-code information on retouch. Subsequently, the thus-prepared images are again uploaded, and information of the length, position, and kind of retouch are extracted (Suppl. 2). Third, information on site location (in decimal degrees), dating (in ka cal BP), and traditional typological categorization of the artifacts are recorded. Eventually, each artifact is recorded with quantitative information on its chronological and spatial position as well as its individual morphological configurations. This means that the present variability in morphology and tool design is recorded at the level of individual artifacts, instead of the level of types. Eventually, the obtained data can be analyzed using descriptive as well as multivariate statistics.
Shape Analysis-Assessing Diversity The recorded contours are analyzed with regard to their shape. We use an elliptic Fourier analysis (EFA), which is a well-established procedure in geometric morphometrics  (Claude, 2008 , Bonhomme et al. , 2014)  (Suppl. 2. 5) and has proven its potential for lithic analysis (e. g. ,  Leplongeon et al. , 2020; Matzig et al. , 2021). The resulting outline EFA data is then submitted to a betweengroup PCA  (Cardini & Polly, 2020). The PCA scores are used to distinguish an optimal number of clusters, determined by a post hoc analysis using the silhouette criterion-i. e. , an internal cluster validity criterion  (Schmidt et al. , 2022) in a hierarchical agglomerative clustering algorithm  (Murtagh & Legendre, 2014). Besides classical applications of geometric morphometrics, the classes separable by their shape are used in order to assess the diversity present within the dataset.
Redundancy Analysis For the redundancy analysis (RDA), we use the ratio scaled (metric) variables: length, width, length-width ratio, area, percent area, contour length, geometrical center (MP) -center of mass (CM)-y-offset, MP-CM-x-offset, and fractal dimension as dependent variables. These exhibit a reasonable correlation structure as can be seen in the pairsplot (Suppl. 3). As the independent (canonical) variable of the RDA, we selected time as captured in the 14 C-dates of the assemblages. The RDA was performed using the function "rda()" of R package vegan  (Oksanen et al. , 2020)  in R 4. 2 (R Core Team, 2022).
Entire Investigated Period In a first step, we analyzed the whole dataset as well as all three clusters for the entire investigated period, before dividing them chronologically in a second step (see below). The permutation test of the RDA for the whole dataset (p = 0. 057) is slightly above the threshold for significance, and only cluster 2 returned significant results (p = 0. 011) (Suppl. 5, Tab. 5. 1). However, the explanatory value of the factor time for the analyzed features is with just 2. 7% rather low (Suppl. 5, Tab. 5. 2). In simple words, all differences between the artifacts regarding length, width, fractal dimension, etc. can only be related to a small amount to the differences between the assemblage dates. Hence, morphological differences related to the chronological position of the assemblages exist, but other differences between the artifacts, not related to time, are much more pronounced.
This is highlighted by the fact that around 47% of the observed variance in the entire dataset is explained by the first PCA axis (Suppl. 5. Tab. 5. 2), which runs perpendicular to the RDAaxis and is thus unrelated to time. Figure  2  shows the RDA results for cluster 2 over the entire investigated period. The younger the projectiles, the lower are their percent area values, indicating that the objects fill less space in the bounding box by becoming pointier, and the lower the MP-CM-x-offset, indicating a decrease in asymmetric lateralization. In parallel, the projectiles show higher values for the length-width-ratio, thus becoming slenderer. To see if the comparably weak correlation of change with time is a result of trends and countertrends, canceling each Fig. 1  Results of the shape analysis.
Maybe a bit more surprising is the absence of such evidence for the location of retouch at the tip, base, or medial part of the projectile and the mean absolute length of the retouch. The shaping of the hafting part (usually base and lateral parts) can be considered a crucial element of projectile design, as is the tip, where retouch can alter the ability to penetrate prey skin. In these cases, it might be inferred that all available options observable in our dataset performed rather similarly so that no preference was given to any of these options in the long run. The only traits related to retouch that show identifiable trends through time are the percentage values of lateral retouch and-alongside-the mean relative length of retouched edges as well as the percentage values of backing retouch (Fig.
The latter trend is still observable in the younger specimens of cluster 2 together with a tendency to become pointier. It thus seems that, despite some divergences within the subsets of clusters 1, 2, and 3, there is an overarching long-term development visible in the entire dataset. This might indicate a certain regression to a mean shape, which-after a pronounced diversification in projectiles at around 33 ka cal BP  (Maier et al. 2021 )-is approximated independently and complementary in all three clusters. This finding might be interpreted as a process of more or less conscious standardization. However, given that the same trend is visible in all clusters and occurs over the course of several millennia, it seems more likely that we are observing the results of selective pressure towards an optimization of physical ballistic properties.
Acknowledgements We are grateful to all colleagues and students with whom we could discuss the different aspects of PyREnArA and the case study throughout the work on both the recording system and the manuscript. In particular, we thank  Marcel Weiß  for input on the PyREnArA code. We also thank our anonymous reviewers for their thorough reading and valuable comments to improve our manuscript. V. C. Schmid  was supported by the  Austrian Science Fund  ( FWF  project number  T 1321-G ).
Many of these expressions (but not all) are already published in the literature. However, some of the expressions presented in this paper are very new to our knowledge, especially the testing for median of time-to-event endpoint in single-arm trial (Section 4. 5). In the same Section 4. 5, we have shown that frequently used 1/(no. of events) (e. g. see  [18] ) consistently under-estimates the variance of log(median) (see Figure  3 ), and we have derived alternative expression for variance. Expressions of PPoS for the binary endpoint with beta prior are presented in Section 5 with example. Implementation in R through LongCART package  [19]  and R shiny app (https://ppos. shinyapps. io/public/) are discussed in Section 6 and illustrated in Appendix 2.
e. , ξ = (log 2) -1 = 1. 443) if m d is the plain sample median (see Appendix 1b). Further, var(log m d ) = (log 2) -2 • β -2 /d (i. e. , ξ = (log 2) -1 • β -1 = 1. 443 • β -1 ) if m d is the plain sample median from a Weibull time-to-event distribution with β shape parameter (see Appendix 1b). For other estimator (e. g. , smallest time with at least 50% Kaplan-Meier (KM) estimate of survival probability), ξ 2 is simply the ratio of the variance of the log of estimated median to that of MLE. We have plotted the empirical SE of log(KM estimate of median) for various event sizes (ranging between 20 and 60) and sample sizes (0%, 30% and 50% more than event size) in Figure  3. Empirical SE were obtained as follows: (a) 5000 datasets were simulated, (b) in each of the simulated datasets, event times were generated from exponential distribution with median of 12 months and KM estimates for median were obtained, and (c) SD of log(KM estimate of median) were obtained.
Organised data is easy to use but the rapid developments in the field of bioimaging, with improvements in instrumentation, detectors, software and experimental techniques, have resulted in an explosion of the volumes of data being generated, making well-organised data an elusive goal. This guide offers a handful of recommendations for bioimage depositors, analysts and microscope and software developers, whose implementation would contribute towards better organised data in preparation for archival. Based on our experience archiving large image datasets in EMPIAR, the BioImage Archive and BioStudies, we propose a number of strategies that we believe would improve the usability (clarity, orderliness, learnability, navigability, self-documentation, coherence and consistency of identifiers, accessibility, succinctness) of future data depositions more useful to the bioimaging community (data authors and analysts, researchers, clinicians, funders, collaborators, industry partners, hardware/software producers, journals, archive developers as well as interested but non-specialist users of bioimaging data).
Introduction Scientific data archival has a long history of providing publicly accessible storage of experimental data that typically involves manual and automated curation and annotation with appropriate metadata for reuse by others  (Whitlock et al. 2010; Rausher et al. 2010; Berman et al. 2014 ). In the area of bioimaging, global and public resources such as EMPIAR  (Iudin et al. , 2016 (Iudin et al. , , 2023) )  and the BioImage Archive  (Ellenberg et al. , 2018; Hartley et al. , 2022)  provide a valuable service to the life-science community by supporting the archival and reuse of imaging data, often acquired at considerable cost, in line with the aspirations of the FAIR Guiding Principles  (Wilkinson et al. , 2016). There are numerous advantages and benefits to reusing bioimaging data, including more economical use of limited resources such as instrumentation and highly skilled technical staff.
Moreover, specimens may be unique, costly to acquire, or difficult to reproduce, meaning that such data may only be accessible via archives. Archived data can be mined for reanalysis, verification and validation, and for development of new analytical techniques and software tools, such as machine learning model training. Reuse of such data may also lead to improvements in how it is produced, both technologically and methodologically. As practitioners in bioimaging data archiving, it is our experience that handling large datasets presents several data-management challenges, particularly in recent years with the rapidly increasing volumes of bioimaging data  (Ellenberg et al. , 2018). For example, it took eight years for EMPIAR to archive a total of one petabyte of data, but the second petabyte took only 14 months  (Iudin et al.
, 2023). Bioimaging datasets may comprise numerous and sometimes very large files in a variety of, sometimes proprietary, formats. Individual files may include multiple channels and time points and data and metadata from several specimens. Besides the raw image data, there may also be a need to archive processed data, reconstructed 3D volumes, segmentations, particle stacks and other derived or related data. There are two related but distinct avenues for organising data: labelling (metadata) and arranging data items (order). Metadata are essential to make the data useful even though metadata standards are difficult to enforce. Therefore, metadata standardisation has received a lot of attention with initiatives such as Bioschemas, an effort to improve findability of datasets via standardised textual annotations, MIAME  (Brazma et al.
, 2001) , recommendations for minimal metadata describing a microarray experiment, and the overarching FAIR Guiding Principles  (Wilkinson et al. , 2016). For bioimaging, REMBI  (Sarkans et al. , 2021)  provides community-supported recommendations on how to describe all aspects of bioimaging experiments including sample preparation, data processing and analysis. Whereas there are several ongoing efforts towards standardising bioimaging data formats (OME-NGFF  (Moore et al. , 2021) , DVID  (Katz & Plaza, 2019) , BigDataViewer  (Pietzsch et al. , 2015) , etc. ), we know of no efforts towards harmonising how to organise datasets for maximum usefulness for archival in mind. The organisation (order) of data is usually taken for granted and it falls upon refinements of the metadata to bear the burden of meaningfully describing the data.
g. , on the intended audience and the usefulness of archived data) or examples (e. g. , of personal identifiers) where requested • Making changes to the phrasing to add context or to make it more explicit, less ambiguous or clearer • Implementing corrections suggested by the reviewers (including adding a few references) • Making a few changes to the open-source bandbox software, e. g. to limit the amount of output • Changing the figures to have white backgrounds to improve their readability and to reduce the amount of black ink required for printing • Applying a number of minor corrections to interpunction, choice of words, etc. • Acknowledging the reviewers by name • Adding and correcting a few references Any further responses from the reviewers can be found at the end of the article meaningful domain insights (bioimage analysis).
Organising a dataset to minimally convey a structure in line with the actual experimental output can improve its usability while the bulk of meaningful attributes can be expressed in the metadata. The degree of usefulness depends directly on the quality of organisation, and thoughtful consideration of the needs of users (i. e. , those who consume the archived data) improves that usefulness. Good organisation also gives a dataset transparency and understandability: users can immediately distinguish the various experimental categories as well as plan how to analyse the data  (Petek et al. , 2022). Therefore, it helps to have a clear perspective of the various types of users. In general, we consider three types of users: intra-domain scientists, inter-domain scientists and extra-domain scientists  (Datta et al.
, 2021). (For the purpose of this article, we will refer to any such user of a dataset as a 'scientist', interested in extracting some knowledge from the archived data. ) Intra-domain scientists are familiar with key attributes of the data and may be able to quickly assess the usefulness of a dataset. An example would be a structural biologist mining an electron cryo-tomogram to extract sub-volumes that have not been previously studied. Inter-domain scientists may want to mine the data for purposes relevant to some other domain. For instance, on the genomics side, using spatial transcriptomics imaging data for fine-grained localization of individual transcripts would be a possible scenario for an inter-domain scientist. Extra-domain scientists are only interested in data for its technical properties, i.
Very long file names, potentially problematic characters, and deep nesting of folders are examples of how injudicious use of these resources can result in unusable data. A simple example of how this is useful is the way most operating systems apply ordering of a directory's contents either lexicographically or by other attributes such as date-time stamps. These take advantage of the familiarity that users have with the conventional ordering of these attributes. In non-trivial organisational tasks, we may need to express complex relationships between the entities at hand. For instance, a dataset that consists of the experimental measurements resulting from a sequence of treatments applied on a set of specimens measured at various points in time requires the use of specimen, treatment and time-point identifiers as well as other experimental attributes (data formats, alternative perspectives, transformations of the data such as changes in units, etc.
) to be captured in such a way as to preserve the main experimental relationships. In that case, we can expand our set of organisational resources to include file formats in addition to the set of symbols (letters, numerals, punctuation, uppercase and lowercase) used to create the various identifiers. Ideally, we would like to keep repetition to a minimum so that the nature of the experiment can be readily discerned. The way organisational resources are used affects the usability of the resulting organisation: using too few of them will obscure the meaning of the organisation while using too many will overwhelm potential users. For example, including redundant folders along any part of the hierarchy (folders that contain only a single folder which in turn contains the actual data) makes it tedious to navigate through a dataset.
On the other hand, dumping all files into one folder will make it difficult for the end user to distinguish between groups of semantically related files, especially when thousands of files are present. Similarly, naming files and folders by referring to entities inaccessible to their intended users (e. g. , local machine names or private accession codes that external users will not have access to or even fathom) consumes precious 'name space' without conveying any useful information. Organising data is thus an investment of time and effort with the aim of improving the usefulness of the data. We can therefore formulate the organisation task as follows: given a set of related data items associated with an experiment, how may they be organised to best convey their relationships using as few organisational resources as possible while maximising their usability? To achieve this, we define the term facet to refer to the various attributes germane to the experiment which may be included in the folder and file names.
A non-exhaustive list of facets is: specimen names (organism, tissue, cell type/line), experimental roles (treatments vs. controls), time (developmental status, date, elapsed time), processing status (raw data, by algorithm, procedure), commonly available experimental equipment (microscopes, detectors, preparation equipment model names), replicates, file types (3D volumes, particle stacks), names of software used for processing. This guide attempts to solve the organisation task by providing 10 recommendations that arise from our experience of handling hundreds of large image datasets in the public archives EMPIAR, BioImage Archive and BioStudies  (Sarkans et al. , 2018). Ideally, we would like to organise potentially numerous and voluminous data to maximise ease of use and hence facilitate the user's ability to: 1.
Recommendations We will motivate our guide by referring to a fictitious EMPIAR dataset. This dataset has a clear structure, but we propose that it can be further improved following the recommendations in the guide below. Our goal is to improve the file/folder structure shown in Figure  1  to better convey the relationships between the experimental facets while economising on the organisational resources available. For clarity, we have refrained from listing several thousand uncompressed TIFF files in the folders designated 'Raw'. The example dataset illustrates several properties of its organisation that undermine the goal of being usable: • Verbosity/redundancy typically manifested in repetition of references which may be resolved using the file hierarchy, such as: ○ Folders containing only a single folder which in turn contains the folder with the actual data.
g. , spaces vs hyphens. See text for more details. ○ 'Topology' and 'topology' ○ 'Treatment' vs 'Treatement' ○ 'Tr1-323' and 'Tr1323' ○ Inconsistency may also be observed in folder structure. For example, only one of the treatment folders (the one with '3738932082' in the name) has an extra child folder, breaking the pattern of the others. • Obscurity tends to occur by identifiers with no obvious meaning, e. g. , references to external resources such as figure numbers in a related paper, machine identifiers, script names, etc. ○ The numerical identifiers such as '0923480928' have no obvious meaning in the context of the dataset. ○ 'Tr1-323' may be an external reference but its meaning is unclear. Understandably, in certain cases such obscurity may be useful to keep identifiers which convey additional information.
For example, in cryogenic-specimen electron microscopy (cryoEM) pipelines, the dataset may consist of multiple subsets obtained with different open-source software, e. g. particle picking by EMAN2  (Tang et al. , 2007) , beam-induced motioncorrection by MotionCorr  (Li et al. , 2013) , contrast-transfer function (CTF) correction by gCTF  (Zhang, 2016) , classification by RELION (Scheres, 2012), reconstruction by cryoSPARC  (Punjani et al. , 2017) , etc. The 10 recommendations we present below are divided into four groups: planning (recommendation 1), structure (recommendations 2-4), naming (recommendations 5-7) and miscellaneous (recommendations 8-10). We have provided further guidance within each group for related concepts.
Structure This section contains recommendations to address the hierarchical organisation of files and folders only. (2) Top-level folder. Have one parent folder into which all sub-datasets are located. Such a top-level folder is also a good location to include auxiliary data that apply to the collection such as README or integrity verification files (see recommendation 10), which provide users with the context of the data organisation. (3) Filename length, path length and folder depth. a. Impose an upper limit on the length of file and folder names. We propose a working upper limit of 50 characters. Even though modern operating systems have no limitations on the lengths of names, end users will still struggle typing very long names which increases the likelihood of transcription errors. In some cases, older software that is still widely used by the bioimaging community imposes limits on the number of characters for file paths, e.
g. , IMOD  (Mastronarde, 2006)  imposes a file-path length limit of 320 characters. It is useful to bear in mind that, increasingly, users interact with datasets via a web browser, which also has a practical limit (based on the device's memory) on the number of files that can be selected in the browser's select dialog. b. Limit the folder depth to a reasonable maximum. As a rule of thumb, three to four directory levels should be adequate for most applications but the fewer the better. This is in line with the ISA framework  (Sansone et al. , 2008) , which organises metadata in three levels (investigation, study, assay). In contrast, both shallow folder depth, with many and varied file types that are difficult to distinguish, and deep nesting of folders make navigation and selection a challenge.