dataesr/scientific-paragraphs-categorization huggingface dataset:
1). By visual comparison, the EZSEG algorithm was found to be the best for our circumstances, particularly proving useful at contouring the polar coronal holes that form a large fraction of the global coronal hole area at solar minimum. The software is available as part of the EUV2CHM Matlab package from the PSI website. 19 We converted the EZSEG algorithm from FORTRAN to IDL to plug-in to our software pipeline. This algorithm uses an initial intensity threshold to acquire coronal hole locations in an EUV image, and then uses an area growing technique to define connected regions. This continues until a second intensity threshold is reached, or the condition for connectivity is not met. The dual-thresholds and connectivity conditions (essentially the number of consecutive pixels) are defined on input.
Wagner et al. (2022)  also make comparisons between EUV images and open field models (specifically from the EUHFORIA model). In their work, they use a simple threshold condition after  Krista & Gallagher (2009)  to obtain observed coronal holes, and use a similar set of binary classification metrics (see  Wagner et al. 2022 for details). This work addresses the difficulty with polar coronal holes by truncating their binary classification to between ±60°latitude, and therefore score the model based on mid-and low-latitude coronal holes only. In the present work, our choice is to use a more complex coronal hole extraction software (EZSEG) and therefore a slightly more convoluted analysis pipeline with the advantage of having a higher degree of confidence in our extracted polar coronal holes.
The overall shape of the streamer belt is not expected to vary significantly, but a slight narrowing of the streamer belt could be observed still. The outer boundary of the WSA runs is located at 5 R e , at the exact same height at which the SMB line is extracted, allowing a direct comparison. The MAS MHD simulations from PSI extend much further, and the NL is extracted at 30 R e. In principle, this means the HCS could be shifted up to 8°due to Parker spiral curvature, Algorithm 2. Process to Generate White Light Streamer Belt Score  (Poirier et al. 2021)  1) Prepare (or download from Zenodo, doi:10. 5281/zenodo. 6342187) WL Carrington map for time interval of interest. 1a)(recommended) Clean up the WL map with a 2D Gaussian filtering (e. g. , using the medfilt2() function in MATLAB). 2) Prepare neutral line from model output (2D map of polarity at model outer boundary, pixel values = ±1).
3) Extract (or download) the SMB line. If downloaded then jump directly to step 4. 3a) Find longitudes and latitudes of max brightness points in the WL Carrington map. 3b)(optional) Smooth out the SMB line with a 1D Gaussian filtering (e. g. , using the smoothdata() function in MATLAB). 4) Measure (or download) the streamer belt (half) width (T i ) (provided in the SMB. ascii files in the Zenodo repository). One could also simply assume a constant streamer belt width and jump to step 5. If you choose the download option or want to perform your own measurements, please carefully read the footnote. a Follow steps 4a and 4b if you decide to perform your own measurements or else jump to step 5. 4a) Draw streamer envelope with a simple contour threshold. 4b) Measure the angular distances (T i ) between each point of the SMB line and the extracted contour of the streamer envelope.
Notes. a In this work and in the SMB files provided online (Zenodo doi:10. 5281/ zenodo. 6342187), all angular distances are measured in the direction perpendicular to the SMB line, which requires to compute normal vectors beforehand. For simplicity one could choose to measure these angular distances over latitude (i. e. , along the vertical axis) or just retain the closest distance to the streamer belt contour. Two streamer widths are given in the SMB files, for the northern and southern portion of the streamer belt, of which only one is used depending on the relative position of the modeled neutral line with the SMB line. One could also simplify the procedure by computing an average of the northern and southern (half) widths. but in practice this is a small correction compared to the dispersion of NL topology among our data set of model runs.
5 random chance threshold. Staying with the in situ polarity metric, the result for encounter 2 is significantly different: the ordering of the spacecraft scores is reversed between the MAS and WSA models compared to PFSS models; PSP continues to give the best score for MAS and WSA, but is the worst for the ADAPT PFSS models (middle scoring for GONGz). We also note the variation in the PFSS scores as a function of source surface is often comparable to the difference between different spacecraft at the same source surface height, indicating that the location in the heliosphere at which the in situ data is taken can be as important as the specifics of the model. We note that, since the boundary condition for the WSA model in encounter 2 is a single magnetogram chosen specifically to match PSP polarity measurements, it is not surprising that this specific value of metric 3 is one of the highest observed.
Following these time series, we can see how the coronal hole representation and streamer belt representation of each model evolved with each encounter. Examining the coronal hole representation (top row), we see for encounter 1 (top-left panel) of the ADAPT-driven PFSS models and the WSA model (also driven by ADAPT magnetograms) follow each other closely, showing a periodic variation peaking twice, with an amplitude of approximately 0. 1 corresponding to about 10% variation in the score. We note that the "observation" EUV Carrington map is static in each time interval, utilizing data combined over the course of a Carrington rotation; therefore all evolution shown derives from the evolution of the PFSS model driven by updating its magnetogram boundary condition each day. The PFSS-GONGz score (the only daily updated model driven by a non-ADAPT magnetogram) is quite different for the same source surface height (2.
The top panel shows the coronal hole score (metric 1), the bottom row shows the streamer belt score (metric 2). Note the time series for the MAS model (cyan) is a single value as there is only 1 magnetogram used for each encounter owing to the intractability of running 3D MHD models with time-dependent boundary conditions. For encounter 2, WSA also only uses a single magnetogram and so only has a single value for metric 1 (see main text, Section 4. 3). encounter, and given heliospheric spacecraft. For times when the model and data agree, the time series has a colored marker (with color indicating differing models) and is blank if they disagree. For times when there is not in situ data available, the markers are colored gray. Therefore, the times when the measured and modeled polarities disagree (or equivalently, when the positions of predicted HCS crossings are misaligned with the data) may be identified as gaps in the colored lines on the plot.
For the PFSS models, the lines for different source surface heights are laid out consecutively (from low to high source surface height). The growth or decay of the white gaps on these consecutive lines indicates if a given HCS crossing improves or worsens with changing source surface height. Overall, it is clear that most spacecraft data and model HCS combinations agree very well for the vast majority of time stamps, which accounts for the high metric scores. Notable exceptions are with PSP in encounter 2 and OMNI data for encounter 3 for which there are much larger gaps in agreement indicating a severe misalignment between data and modelHCS. For the PFSS models, we see that typically the disagreements (blank spaces) occur at the same time for a given spacecraft in all models and also usually follow the same qualitative trend as a function of source surface height.
This indicates that small misalignments between HCS warps and the empirical data correspond to the same heliographic locations independent of the input magnetogram. The MAS and WSA models sometimes have gaps aligned in time with the PFSS models (for example PSP Encounter 1 10/29, top-left panel), but often do not, suggesting the HCS output of these models can be qualitatively different to the PFSS models and sometimes in better agreement with spacecraft data. A very clear example of this is in the top middle panel (E2, PSP) where several large outages in agreement for PFSS models at all source surface heights occur while the MAS and WSA models both correctly predict the polarity of PSP throughout. The time series are usually very different between different spacecraft. The gaps in agreement can occur at different times for a given model.
Therefore, gaps in the lines indicate intervals of disagreement between model and measurement. The larger the fraction of the whole time series that is blank, the lower the resulting metric score reported in Figures  4  and 5. spacecraft is that for PSP the agreement overall is much more continuous, while for the 1 au data sets it is more intermittent with agreement switching on and off from one hour to another. This is caused by the relatively larger fluctuation amplitudes and Parker spiral angle in the B R data at 1 au (OMNI and STEREO A), meaning that the measured time series include fluctuations in polarity that do not correspond to current sheet crossings. This concludes our exposition of the metric scoring results in this work. We next discuss the implications of these results, both on the specific time intervals, models, and magnetograms studied here, as well as what they show about this framework for evaluating coronal models.
Discussion and Conclusions In this work, we have defined a model-agnostic framework to evaluate and rank coronal models based on their magnetic field topology using three independent sources of observational data. A given coronal model needs only to provide 2D maps of (a) open magnetic field at the photosphere and (b) magnetic polarity at the model's outer boundary. Models are given a metric score based on (1) how well they represent coronal holes as observed in EUV data for the relevant Carrington rotation, (2) how well they predict streamer belt topology observed by WL coronagraph observations, (3) how well they predict HCS crossing timing measured in situ by various heliospheric spacecraft. We applied our framework to a data set of coronal models for the first three encounters of PSP using potential field source surface (PFSS) models for a range of source surface heights and input magnetograms, as well as concurrent Wang-Sheeley-Arge (WSA) models and the Magnetohydrodynamic Algorithm outside a Sphere (MAS) models.
We carried out three investigations. The first focused on PFSS models only and explored the metric scores as a function of source surface height and choice of magnetogram inner boundary condition. The second compared the PFSS model scores to WSA and MAS scores, evaluating the impact of more physics in the model. The third examined the time evolution of the metric scores in a 60 day interval for each PSP encounter period to look at the stability of the results. For the PFSS-only investigations, we found the following: (A1) Quality of representation of coronal holes at the photosphere is a function of source surface height. Values above 2. 5 R e lead to observed coronal holes in EUV not being captured by the model (low precision) regardless of input boundary condition. Values too low can also lead to opening fields in the model that are not observed in EUV (low recall), although for models driven by the GONGz data product, the best scores occurred for the lowest source surface height.
2021). This meant a strong photospheric field concentration rotated on disk twice during this time interval and at this time produced sudden and significant changes in magnetograms. This produced simultaneous sharp changes both in the accuracy of representation of coronal holes, and in representation of the global streamer belt topology (both reductions in accuracy as implied by the metrics defined in this work). This change was also observed as a loss of accuracy in metric 3 (middle column Figure  7 ), most dramatically in PSP data but also to some extent in 1 au data, indicating severe distortion to the model HCS topology by this flux emergence. (C3) The agreement between PSP in situ data and the model predictions was typically more continuous than that of STEREO A and OMNI. This was illustrated in Figure  7  and demonstrates a simple advantage of sampling the solar wind closer to the Sun for the application of evaluating coronal models: the amplitude of magnetic vector fluctuations to the base value is reduced, and this means a more reliable and continuous measurement of which side of the HCS a given spacecraft is located.
This tool could be used by any coronal modeler by submitting the coronal hole and NL maps in the format prescribed here and are then presented with the resulting metric scores and supporting plots. While such an integrated software tool is beyond the scope of the current project, we presented in Algorithm Boxes 1, 2, and 3 descriptions of the algorithm to produce the respective metric score. We also supply the base observational data products to perform the comparisons for the time intervals studied in this work as a Zenodo repository (doi:10. 5281/ zenodo. 6342187) in an effort to allow other modelers to try evaluating this framework on their own models. As well as the specific observational data products, Python scripts are included for each metric as examples for loading and plotting the different file types.
Algorithm 3. Process to Generate In Situ Polarity Score 1) Prepare (or download from Zenodo, doi:10. 5281/zenodo. 6342187) measured in situ polarity time series. If preparing yourself, recommended to follow processing routine here where for each hour of data, generate histogram of B R , and take polarity of most probable value as the polarity "measurement" for that time. 2) Produce model-predicted polarity time series by following these steps: 2a) Use spice kernels to generate spacecraft position in the heliographic frame at an hourly cadence over the time interval of interest. 2b) Ballistically propagate this position inwards to.
Algorithm 3. Process to Generate In Situ Polarity Score 1) Prepare (or download from Zenodo, doi:10. 5281/zenodo. 6342187) measured in situ polarity time series. If preparing yourself, recommended to follow processing routine here where for each hour of data, generate histogram of B R , and take polarity of most probable value as the polarity "measurement" for that time. 2) Produce model-predicted polarity time series by following these steps: 2a) Use spice kernels to generate spacecraft position in the heliographic frame at an hourly cadence over the time interval of interest. 2b) Ballistically propagate this position inwards to 2. 5  R using the equation for the Parker spiral (Equation (1) of Badman et al. 2020). 2c) Sample the modeled 2D map of the neutral line using this projected orbit (via interpolation or nearest neighbor) to generate a time series of predicted polarity.
Current tools and limitations In the scientific Python ecosystem, it is well known that Docutils [docutils] and Sphinx [sphinx] are major cornerstones for publishing HTML documentation for Python. In fact, they are used by all the libraries in this ecosystem. While a few alternatives exist, most tools and services have some internal knowledge of Sphinx. For instance, Read the Docs [RTD] provides a specific Sphinx theme [RTD-theme] users can opt-in to, Jupyter-book [JPYBOOK] is built on top of Sphinx, and MyST parser  [MYST]  (which is made to allow markdown in documentation) targets Sphinx as a backend, to name a few. All of the above provide an "ahead-of-time" documentation compilation and rendering, which is slow and computationally intensive. When a project needs its specific plugins, extensions and configurations to properly build (which is almost always the case), it is relatively difficult to build documentation for a single object (like a single function, module or class).
• Example files are a non-ordered collection of files. • Assets files are untouched binary resource archive files that can be referenced by any of the above three ones. They are the only ones that contain backward references, and no forward references. In addition to the four categories above, metadata about the current package is stored: this includes library name, current version, PyPi name, GitHub repository slug 4 , maintainers' names, logo, issue tracker and others. In particular, metadata allows us to auto-generate links to issue trackers, and to source files when rendering. In order to properly resolve some references and normalize links convention, we also store a mapping from fully qualified names to canonical ones. Let us make some remarks about the current stage of IRD format. The exact structure of package metadata has not been defined yet.
At the moment it is reduced to the minimum functionality. While formats such as codemeta [CODEMETA] could be adopted, in order to avoid information duplication we rely on metadata either present in the published packages already or extracted from Github repository sources. Also, IRD files must be standardized in order to achieve a uniform syntax structure (Objective (b)). In this paper, we do not discuss IRD files distribution. Last, the final specification of IRD files is still in progress and regularly undergoes major changes (even now). Thus, we invite contributors to consult the current state of implementation on the GitHub repository  [Papyri]. Once the IRD format is more stable, this will be published as a JSON schema, with full specification and more in-depth description. 4. "slug" is the common term that refers to the various combinations of organization name/user name/repository name, that uniquely identifies a repository on a platform like GitHub.
For example, building SciPy & NumPy documentation IRD files on a 2021 Macbook Pro M1 (base model), including executing examples in most docstrings and type inferring most examples (with most variables semantically inferred) can take several minutes. End-users are responsible for installing desired IRD bundles. In most cases, it will consist of IRD bundles from already installed libraries. While Papyri is not currently integrated with 5. One could have IRD bundles not attached to a particular library. For example, this can be done if an author wishes to provide only a set of examples or tutorials. We will not discuss this case further here. package managers or IDEs, one could imagine this process being automatic, or on demand. This step should be fairly efficient as it mostly requires downloading and unpacking IRD files.
IRD files generation The current implementation of Papyri only targets some compatibility with Sphinx (a website and PDF documentation builder), reStructuredText (RST) as narrative documentation syntax and Numpydoc (both a project and standard for docstring formatting). These are widely used by a majority of the core scientific Python ecosystem, and thus having Papyri and IRD bundles compatible with existing projects is critical. We estimate that about 85%-90% of current documentation pages being built with Sphinx, RST and Numpydoc can be built with Papyri. Future work includes extensions to be compatible with MyST (a project to bring markdown syntax to Sphinx), but this is not a priority. To understand RST Syntax in narrative documentation, RST documents need to be parsed. To do so, Papyri uses tree-sitter [TS] and tree-sitter-rst [TSRST] projects, allowing us to extract an "Abstract Syntax Tree" (AST) from the text files.
Instead of storing the directive with the raw text, we parse the full document (potentially finding invalid syntax), and unparse to the raw text only if the directive requires it. Serialisation of data structure into IRD files is currently using a custom serialiser. Future work includes maybe swapping to msgspec [msgspec]. The AST objects are completely typed, however they contain a number of unions and sequences of unions. It turns out, many frameworks like pydantic [pydantic] do not support sequences of unions where each item in the union may be of a different type. To our knowledge, there are just few other documentation related projects that treat AST as an intermediate object with a stable format that can be manipulated by external tools. In particular, the most popular one is Pandoc [pandoc], a project meant to convert from many document types to plenty of other ones.
2) Render a single page. 3) Access raw data (e. g. images). We also assume that IRD files may be infrequently updated, that disk space is limited, and that installing or running services (like a database server) are not necessary available. This provides an adapted framework to test Papyri on an end-user machine. With those requirements we decided to use a combination of SQLite (an in-process database engine), Concise Binary Object Representation (CBOR) and raw storage to better reflect the access pattern (see Figure  3 ). SQLite allows us to easily query for object existence, and graph information (relationship between objects) at runtime. It is optimized for infrequent reading access. Currently many queries are done at runtime, when rendering documentation. The goal is to move most of SQLite information resolving step at the installation time (such as looking for inter-libraries links) once the codebase and IRD format have stabilized.
SQLite is less strongly typed than other relational or graph database and needs custom logic, but is ubiquitous on all systems and does not need a separate server process, making it an easy choice of database. CBOR is a more space efficient alternative to JSON. In particular, keys in IRD are often highly redundant, and can be highly optimized when using CBOR. Storing IRD in CBOR thus reduces disk usage and can also allow faster deserialization without requiring potentially CPU intensive compression/decompression. This is a good compromise for potentially low performance users' machines. Raw storage is used for binary blobs which need to be accessed without further processing. This typically refers to images, and raw storage can be accessed with standard tools like image viewers. Finally, access to all of these resources is provided via an internal GraphStore API which is agnostic of the backend, but ensures consistency of operations like adding/removing/replacing documents.
The project aims at replacing the CLI IPython question mark operator (obj?) interface (which currently only shows raw docstrings) in urwid with a new one written with Rich/Textual. For this interface, having images stored raw on disk is useful as it allows us to directly call into a system image viewer to display them. This option contains the most features, and therefore is the main one used for development. This environment lets us iterate over the rendering engine rapidly. When exploring the User Interface design and navigation, we found that a list of back references has limited uses. Indeed, it is can be challenging to judge the relevance of back references, as well as their relationship to each other. By playing with a network graph visualisation (see Figure  5 )), we can identify clusters of similar information within back references.
Of course, this identification has limits especially when pages have a large number of back references (where the graph becomes too busy). This illustrate as well a strength of the Papyri architecture: creating this network visualization did not require any regeneration of the documentation, one simply updates the template and re-renders the current page as needed. • A static AOT rendering of all the existing pages that can be rendered ahead of time uses the same class as the JIT rendering. Basically, this loops through all entries in the SQLite database and renders each item independently. This renderer is mostly used for exhaustive testing and performance measures for Papyri. This can render most of the API documentation of IPython, Astropy  [astropy] , Dask and distributed  [Dask] , Matplotlib  [MPL] ,  [MPL-DOI] , Networkx  [NX] , NumPy [NP], Pandas, Papyri, SciPy, Scikit-image and others.