dataesr/scientific-paragraphs-categorization huggingface dataset:
0 and backward-compatibility-breaking changes that necessitate the new major release, version 2. 0. Substantial changes include refactoring the grid, improving the component standard interface, dropping Python 2 support, and creating 31 new components -for a total of 58 components in the Landlab package. We describe reasons why many changes were made in order to provide insight for designers of future packages. We conclude by discussing lessons about the dynamics of scientific software development gained from the experience of using, developing, maintaining, and teaching with Landlab.
g. ,  Bandaragoda et al. , 2019). Development and maintenance of Landlab follows modern software development standards such as version control, integrated testing and documentation, continuous integration, and multiplatform binary distribution (e. g. ,  Adorf et al. , 2019; Hwang et al. , 2017; Mandli et al. , 2016; Poisot, 2015; Taschuk and Wilson, 2017; Wilson et al. , 2014). Our open-source development and use of semantic versioning (SemVer 2. 0. 0, https://semver. org, last access: 12 May 2020) provides a necessary but not sufficient tool for reproducible research in Earth surface dynamics (e. g. ,  Chen et al. , 2018). Landlab was designed as a key element in the Community Surface Dynamics Modeling System (CSDMS) suite of tools  (Peckham et al. , 2013). Initial development of Landlab began in 2012 and culminated in a version 1.
0 release (referred to as v1. 0) described by  Hobley et al. (2017). Figure  1  provides examples of the breadth of modeling efforts implemented with Landlab. Subsequent to the release of v1. 0, the core development team and many community members have contributed additional features and bug fixes to the software. Based on experience using and developing with Landlab, the development team identified changes to Landlab that were not backwards compatible, indicating a major release was necessary to convey to Landlab users to expect substantial changes. This motivated the creation of Landlab v2. 0, the focus of this contribution. A new major version was needed to support (a) backward-compatibility-breaking changes associated with refactoring core data structures and (b) removal of support below Python version 3.
The scope of this contribution is to review the core concepts that underpin Landlab's design (Sect. 3), describe the changes and new features added since v1. 0 (Sect. 4), discuss citation of software (Sect. 5), and document lessons we have learned about community software development from developing and maintaining Landlab (Sect. 6). Before concluding we provide recommendations for those interested in being involved with Landlab (Sect. 7). We note that while much of the contribution discusses general issues of scientific software development, Sect. 4 is inherently specific to Landlab and describes technical details of changes between v1. 0 and v2. 0. For a comprehensive description of the design and theory behind Landlab v1. 0 the reader is referred to  Hobley et al. (2017). Additionally, we will not present detailed description of the use of the software, discuss numerical methods, or review the literature that supports each process implemented in Landlab.
The three Landlab audiences The design of the Landlab package, its development practices, and the changes made in v2. 0 are best understood in light of the three audiences who interact with the package. Unlike software that is developed by dedicated software engineers who may not use the software themselves, Landlab developers also use the software for their research and teaching. Thus, the first audience is user-developers, people who extend, modify, or otherwise contribute to the source code in order to accomplish their goals. Notably, most Landlab userdevelopers have little to no background in software engineering. The second audience is users: people who use Landlab to write their own programs but do not modify or contribute to Landlab's source code. Among this group, it is natural for some to transition to becoming user-developers, who contribute new components or utilities to the main Landlab code base.
The final audience is teachers-students, people who use Landlab in an instructional classroom setting as part of a course. In creating the source code, writing the documentation, determining the development practices, and maintaining the package, the needs, abilities, and time constraints of all three audiences must be balanced. This is particularly important for packages like Landlab with a small active developer community (n < 20) and a research-scale user community (e. g. , tens to hundreds of researchers and perhaps a few thousand students over the lifetime of the software, rather than millions of users). Our approach is to adopt many of the key design principles underlying modern academic software design best practice (e. g. ,  Wilson et al. , 2017; Turing Way Community et al. , 2019).
These include an extensive automatic test suite, integrated documentation, version control, continuous integration, lint checking, and releasing binary packages for users. These design choices were made to ensure that Landlab is sustainable into the future to support the userdeveloper-learner communities (see  Hobley et al. , 2017). Community contributors play an important role in developing community open-source software. Two of their most important roles are improving and refining documentation when   (Zebari et al. , 2019 , reproduction of their Fig. 8 ), (c) cellular automaton simulation of normal fault facets  (Tucker et al. , 2020 , reproduction of their Fig. 9 ), (d) the evolution of post-glacial drainage networks  (Lai and Anders, 2018 , reproduction of their Fig. 4 ), (e) estimates of landslide probability  (Strauch et al.
Landlab core concepts A core design principle behind the Landlab package is modularity. Separating the elements of a numerical model into reusable parts decreases the human time associated with creating a new model or extending a current one. The design of Landlab is discussed extensively in  Hobley et al. (2017). Here we briefly summarize the key points to provide context to the changes and new features that are discussed further in Sect. 4. The modular design of Landlab comprises the following categories of software infrastructure: 1. Model grids: data structures implemented as Python classes that represent the computational domain, connectivity between parts of the domain, and provide a centralized location to store state variables. 2. Utilities: functions that provide solutions to common problems (e.
g. , numerical functions for gradients, mapping, and flux divergence; basic plotting; watershed delineation; and file input/output). 3. Components: representation of core surface processes (e. g. , stream power, flow accumulation, and precipitation) as a Python class with a common interface. The grid represents a 2-D domain as a dual graph. Each graph in the dual graph is a set of points, connected by lines, and outlining polygons. The two graphs are offset from one another such that the points of one graph are located inside of the polygons of the other graph. Each graph is a planar graph, meaning that the lines connecting points do not cross. In Landlab, we refer to the first graph as composed of nodes connected by links which outline patches. Corners are located inside patches and are connected by faces, which outhttps://doi.
The focus on finite-difference and finite-volume utilities, however, reflects the interests of developers rather than the potential characteristics of the package. In the second category are application-focused utilities. These utilities are typically developed for use in a particular component but have grown to have broader use within the package. For example the watershed utility submodule was developed to support the SpeciesEvolver but provides the broadly applicable ability to label watersheds. Components are Python objects with a standard interface that implement a single Earth surface process, set of equations, or analysis compatible with the component interface (e. g. , calculation of drainage density). All components require a Landlab model grid to instantiate and have a builtin function that advances the component forward in time or updates it based on the current values stored as fields.
Changes and new features added since Landlab v1. 0 Landlab v2. 0 contains many changes to the core source code that add new features. We have compiled tables describing the pre-existing, refactored, and new core capabilities of the Landlab package. These include data structures (Table  1 ), utilities (Table  2 ), new components (Table  3 ), and pre-existing or refactored components (Table  4 ). We list core package, development environment, testing, tutorial, and documentation dependencies in Table  5. This section focuses on the technical details of what has changed between Landlab v1. 0 and v2. 0. One might glean comparable information from reading the software repositories' change logs. Inclusion of the technical details here is intended to summarize key changes. In addition, where relevant, we describe why changes or improvements were made.
This explanation is intended both for current and future users of Landlab as well as for those interested in scientific software development generally. Changes that broke backward compatibility were required to incorporate some of the new features in Landlab v2. 0. This necessitated a new major version. These changes included (i) binding of the boundary condition flags to model grids (Sect. 4. 1. 3), (ii) a revision to the Component standard interface (Sect. 4. 2), (iii) deprecation and removal of certain components and utilities (Sect. 4. 3), (iv) dropping Python 2 support (following sunsetting of this version at the end of 2019 by the Python Software Foundation). Additionally, we completely revised the documentation structure (Sect. 4. 4). Landlab v2. 0 is designed to work with a number of other Python tools for numerical modeling.
Grids inherit from graphs Each Landlab model grid combines a dual graph topology with the ability to store fields at grid elements and keep track of boundary conditions. While the concept of a dual graph is not new in Landlab v2. 0, the package architecture has been revised to create a set of graph classes from which the Landlab model grids inherit (Table  1 ). The Landlab graphs describe the topology and connectivity of a dual graph of nodes-links-patches/corners-facescells and specify the x and y coordinates of the nodes and corners. The package contains support for 1-D and 2-D graphs and for graphs not yet used in Landlab grids (e. g. , DualStructuredQuadGraph). It was designed to be reusable by projects external to Landlab. While the graph capabilities do not yet support 3-D graphs, the package was designed with extension to 3-D in mind.
Building the model grids to inherit from the graph data structure results in all model grids containing a complete set of topology-derived attributes (e. g. , grid. links_at_node) and attribute naming consistency between model grids. In addition, all of the topology-derived attributes are only created when needed (just-in-time memory allocation) and are cached. This was inconsistently implemented in v1. 0 and provides an improvement for memory management and speed. The graph and model grid data structures are all built on the xarray Python package's Dataset  (Hoyer and Hamman, 2016). Using xarray. Dataset provides a number of advantages including improved input and output to the NetCDF format, use of xarray's optimized data structures, and the possibility to take advantage of xarray-compatible parallelization-related tools (e.
Updates to the component standard interface Scientific software and data are much easier to work with when they follows standards. Software tools in particular become much more accessible when they provide a standard interface: a common set of functions that look and act in a similar way across many different elements of the software. Landlab's components use a lightweight interface that is inspired by the CSDMS Basic Model Interface (BMI)  (Peckham et al. , 2013; Hutton and Piper, 2020a)  but which takes advantage of object-oriented features of the Python language, allowing it to be more compact. Landlab also includes builtin functionality that converts any Landlab component into a BMI component, for use in frameworks like the CSDMS Python Modeling Tool  (Hutton and Piper, 2020b). In addition to its interface, each Landlab component also encodes metadata in a standardized format; these metadata include, for example, information about the input and output fields of the component.
Changes to the component __init__ method The design of many numerical model programs follows the "initialize-run-finalize" pattern (e. g. ,  Peckham et al. , 2013; Hutton and Piper, 2020a ). In the BMI, the initialization step is handled by the standard initialize( ) function, and stepwise updating is handled by the update( ) function. For Landlab components, which are implemented as Python objects, the class __init__ method implements initialization, and stepwise updating is normally handled by a method called either run_one_step or update. Hobley et al. (2017, their Sect. 3. 3. 1) defined the interface for Landlab components with the function signature for instantiation (Component. __init__) and advancing forward (Component. run_one_step). The v1. 0 component instantiation interface defined with the function definition as follows: __init__(self, grid, arg1, arg2.
Changes to the component run method The v1. 0 component interface defined a run method with a function signature run_one_step(dt, * args, * * kwds), where dt represents the duration of time the model runs forward, * args represents a generic list of arguments, and * * kwds represents a generic set of keyword arguments. In practice, we found that many Landlab components were not able to follow this interface standard because it was not flexible enough. For example, some components do not require a dt and thus did not take dt. We also found the presence of * args and * * kwds in the run_one_step problematic because it complicated wrapping components with a Basic Model Interface (BMI;  Peckham et al. , 2013; Hutton and Piper, 2020a)  for use with the Python Modeling Tool (PyMT;  Hutton and Piper, 2020b).
Although it might seem odd to create functions to handle such seemingly trivial tasks, the practice has the advantage of enabling defensive programming (e. g. , a setter can check for the right data type), allowing a program to create a particular variable only when it is requested (which can save memory), and supporting built-in documentation (in the form of function documentation) for each variable. In Landlab (as in Python practice generally) getters and setters are implemented using the Python @property decorator. Those variables that use getters and setters are considered to be public, meaning that programmers using the component can easily inspect and, if desired, change their values. Other variables are considered private: used only by the component internally and not to be modified (to indicate this, the names of private variables are preceded by an underscore character).
New component metadata standard For both data and software, standardized metadata promote efficiency, interoperability, and reuse. To that end, each Landlab component includes a set of metadata in the header of the class that defines the component. Our experience with component metadata in Landlab led us to revise its design for version 2. 0. The metadata section of a Landlab component describes its input fields, output fields, field units, the type of grid element associated with each field, and a long-format description of the field. Metadata are now organized into a single Python dictionary, which has a key-value pair for each field used by the component. The new data structure makes it easier to test for completeness and consistency across components. Each key is a string indicating the field name.
"intent" A string indicating the input/output intent of the field. Valid options are "in", "out", and "inout". "optional" Boolean indicating whether the field is an optional input or output. "units" String indicating the units of the field. Some components are unit agnostic, in which case these units can be interpreted as dimensions (see description of Component. unit_agnostic in Sect. 4. 2. 4). "mapping" String indicating the type of grid element associated with the field (e. g. , node, link). "doc" String describing the field. -All public attributes must be documented properties of the Component class; that is, they have the @property standard Python decorator. This ensures that other users are able to identify what each public attribute is and prevents variable modification unless the developer explicitly permits it.
-The ModelParameterDictionary was removed because it represents an old-style input file that has been superseded by the Yaml format. -A new ChannelProfiler component replaces the previous channel-profiling submodule (landlab. plot. channel_profile). -The noclobber keyword argument for field creation was changed to clobber because the original name required double negatives and was not intuitive. noclobber=False is equivalent to clobber = True. -The ability to pass an array of flooded nodes to the run_one_step method in surface-water erosion components was removed and replaced with a keyword argument to __init__ called erode_flooded_nodes.
Reorganization of the Landlab documentation The Landlab online documentation is now consolidated onto a single Sphinx-based platform (https://landlab. readthedocs. io/, last access: 12 May 2020). Consolidating the documentation onto a single platform with a consistent interface reduces duplication of information, improves consistency, and permits comprehensive searches. The site design is similar to that of widely used scientific Python packages and was modeled after that of pandas  (McKinney, 2010). The revised documentation pages include installation instructions, a user guide (including tutorials), a guide for developers, and an API reference that contains formatted versions of inline documentation within the source code. The documentation source is written in reStructuredText format, and the source files are provided as part of the Landlab package.
Packages built to work with Landlab Landlab was designed as a generic, extensible modeling framework for Earth surface dynamics. Because Landlab exposes a BMI  (Hutton and Piper, 2020a) , it is compatible with the PyMT package  (Hutton and Piper, 2020b ) -a Python package that supports running and coupling models that expose a BMI. PyMT provides access to a suite of models written in multiple languages (e. g. , Python, Fortran, C++) and a standard interface for initializing and running. In addition, two packages have been built using Landlab to support applications in sensitivity analysis, calibration, validation, and multimodel comparison (see  Barnhart et al. , 2020a, b, c , for example applications). First, terrainbento is a Python package for multimodel analysis that provides an extensible set of 27 Landlab-built models for long-term drainage basin and landform evolution, along with general classes for handling boundary conditions through an input-file format  (Barnhart et al.
Citation of Landlab and parts of Landlab Citation of scientific software is an outstanding challenge (e. g. ,  Niemeyer et al. , 2016). Scientific software is cited less frequently than it is used (e. g. ,  Pan et al. , 2015). Indicating a recommended citation for use of Landlab is additionally challenging because, depending on the portion of Landlab used, the set of citations required may vary. We describe our recommendations for which citations to use and present a tool within Landlab to improve citation discoverability. Any time any part of Landlab is used, Hobley et al. (2017) should be cited; if the version used is > 1. 0, then this contribution should be additionally cited. These two citations acknowledge the development of the Landlab package itself. We also recommend that authors state the specific version of Landlab used (the version can be found by evaluating landlab.
__version__). Each application of Landlab may use a different set of components, each with a different citation for the software itself and general set of theory references (Tables  3  and 4 ). Additionally, some parts of Landlab may internally use others; thus a user may not easily be able to assess the entire set of elements of Landlab their application has used and what to cite for each part. This challenge is not new. For example, it is faced by the scipy package, which addresses it by providing a corepackage citation -  Virtanen et al. (2019)  -and indicating that users should look to the reference section of the documentation for additional citations. Similarly, the codes dis- tributed through the Computational Infrastructure for Geodynamics (CIG) have a citation builder that distinguishes between citations specific to the software implementation, primary citations describing the code development and numerical methods, and secondary citations that pertain to parts of the code a user may or may not have used  (Kellogg et al.
, 2018). This example from CIG highlights a further challenge: a component may have one or more citations for each of the following categories: (i) the theory behind the implemented idea, (ii) a description of the software implementation itself, (iii) any specialized algorithms developed for the implementation, and (iv) the first reported use of the software in a publication. Should one of these or all of these be the recommended and/or required citations for a given software component? We do not think it is our role to decide which citations, if any, a component author indicates as recommended or required. Additionally, it is not our place -as the software developers behind Landlab -to determine which citations best represent the theory behind an implementation. Instead we provide two places for a component author to indicate what they think the minimum required citations are as follows: a component attribute called Component.
cite_as which lists required citations for a given component and a section in the component docstring that provides the broader reference context. These two categories are reflected by the two citation columns in Tables  3  and 4. Clearly, a component developer has the authority to decide exactly what to put in either of these locations. To aid discoverability of citations, we have created the Landlab citation registry, a tool that compiles citation-related metadata for the specific set of Landlab components used in an application (Listing 1). The citation registry compiles citation information for all components currently instantiated in a Python session by automatically interrogating their cite_as properties.
Lessons on geoscientific software development In this section we highlight several lessons about software development we have learned in the processes of supporting and improving Landlab v1. 0 to its current v2. 0 state and working with the growing community of users. We reflect on these lessons because the production of research software is itself research, and there are many aspects of scientific software which are distinct from other software, notably that (i) the development lifecycle includes additional stages because the methods used to implement a piece of software may not exist at the outset of a project, (ii) requirements evolve because they are part of the research, and (iii) the state of the scientific field may be complex and evolving (e. g. ,  Carver et al. , 2016).