dataesr/scientific-paragraphs-categorization huggingface dataset:
Rainfall data Rainfall was monitored onsite using an Odyssey Logger with a tipping bucket rain gauge (NZ$ 369 ∼€ 215all prices are given as of June 2019), clear from canopy interception. The rain gauge was inspected weekly to fortnightly to verify that the hole of the rain gauge bucket was not blocked (i. e. by leaves). If the rain gauge was found to be partially blocked, the cumulative rainfall was compared to another nearby rain gauge. Blockages are obvious on such graphs. If the hole was found to be completely blocked with no water reaching the tipping pivot, data since the previous download were considered not usable. Rainfall was recorded at a 1-min timestep in the built-in logger. Total blockages were rare. The rain gauge did need calibration: the volume of rainfall was collected every month, weighed to 0.
Flow data Flows going into and out of the basin were monitored using area-velocity sensors connected to Hach Sigma 950 flowmeters (hereafter referred to as HACH). The probes recorded velocity with a Doppler ultrasonic sensor and water level with a pressure diaphragm. Monitoring of flows proved very challenging. Small weirs had to be installed in two pipes to make sure the probes remained submerged to better monitor low flows, which was very important to close the water balance. Both inlet pipes were large and likely oversized (750 and 525 mm in diameter) to accommodate high storm flows, but this proved challenging for flow monitoring as large pipes resulted in low water levels. Catchment hydrographs had long tails, with substantial and long-lasting low flows (,1 L/s). Flows (level and velocity) measured were within manufacturer-claimed accuracy ranges (-1.
Water quality (autosampling) Two autosamplers (Sigma 900 MAX portable, with 24 × 1 L samples) were installed at the inlet and at the outlet of the basin to monitor the water quality treatment performance of the basin. The autosamplers were connected to the flowmeters and samples were taken every given volume to cover a large range of Data collection in urban drainage and stormwater management systemscase studies 433 Downloaded from http://iwaponline. com/ebooks/book/chapter-pdf/919177/9781789060119_0415. pdf by TECHNISCHE UNIVERSITEIT DELFT user 11. 4. 2. 4 Water and groundwater level data Water level at the surface of the basin was monitored with four Odyssey capacitance probes, chosen primarily for their price and availability (NZ$ 248 ∼ € 140). The water level within the filter media of the basin was monitored using three of the same Odyssey capacitance probes.
The fate of infiltrated stormwater was monitored with 15 water level probes (the same Odyssey capacitance probes), installed around the basin to monitor shallow groundwater (ranging 2-5 m deep). All these probes were calibrated in the laboratory before deployment. Calibration was checked and adjusted when needed. Because of their relatively short length, it was easy enough to remove every probe from their bore during download to clean them and check the calibration with a home-built 2-m long PVC pipe filled with water. The operation took about 2 hours for all probes, so missing data during downloads were minimal. Water levels were read with an accuracy ranging from 2-5 mm, and most data were usable, except for times with flat batteries or unexplained probe issues (around 10% of the dataset).
Database, accessibility and data management Data were downloaded from the loggers, on site, every month, the operation taking about half a day to a day of work. All data were downloaded on a computer and then stored into a Dropbox folder shared by all members of the WERG. The folder was only accessible by researchers from the group. The Dropbox folder contains data of all projects of the WERG and is managed by a data manager, funded by the partnership between the WERG and Melbourne Water. The primary responsibility of the data manager is to look after data from all WERG projects. Data validation was done by or with researchers who spend time on sites. Data format and data storage were systematic and the same for all probes (flow, water level, …). Data were stored in three folders: • Raw data, containing the files obtained from the sensors.
These files were never touched, never modified. • Compiled data, containing data being processed (validation, correction…). • Final data, containing validated data (in either. csv files or. r data format), ready to be read and used in R for data analysis. Additional but crucial data validation was done manually using field notes and observations to adjust data when needed. A quality code system was set up as: 1 (good data), 2 (data might have a problem) or 3 (unusable), based on field observations (for example, if a wooden stick was observed on the probe but data still made sense and visually looked OK, the quality code would be 2). Researchers shared a spreadsheet where timings, all field observations and actions taken for the probes were recorded so that anomalies in the data could be explained later on (for example: 'on this day, probe nr x has been removed from its casing to sample groundwater', can explain a couple of timesteps containing surprising data).
Lessons and suggestions The monitoring set-up at this site required constant human inputs and labour with at least weekly site visits, indicating that a remote way to communicate with probes would save operational time. Meta-data (information about sensors, calibration, field notes) are crucial to data management and analysis, as they allow understanding of why chunks of data look suspect. The way meta-data will be recorded and used should be discussed during planning a monitoring strategy. Similarly, processing and analysing data as early as possible after collection (instead of storing long periods of data without looking at them) can avoid further trouble as many issues with probes were identified during data analysis. In this case study, weeks of data were lost before realizing that low flows were important and that therefore weirs had to be installed, and that the flowmeters needed substantial calibration effort to provide reliable data.
This was all found by trial and error. Original aims of the monitoring were achieved and stakeholders (research, council, water authority) were pleased with the monitoring results. Reliable experimental assessment of the hydrological and water quality Data collection in urban drainage and stormwater management systemscase studies 435 Downloaded from http://iwaponline. com/ebooks/book/chapter-pdf/919177/9781789060119_0415. pdf by TECHNISCHE UNIVERSITEIT DELFT user performance of the basin was obtained. To date, monitoring of this site has not yielded actual policy changes, but has enhanced confidence amongst stakeholders to implement such systems across the Melbourne area. Anglian Water Services (AWS) have adopted an integrated supply chain and holistic approach for their AMP6 (AMP6 stands for Asset Management Plan 6.
g. water levels at wet wells, pump status, treated effluent flow rate) alongside radar rainfall data were adopted for model calibration. These catchments were mainly in locations where drivers such as growth were not as highly ranked at MDM1 stage, but which were still relevant to AWS's business needs for the next AMP cycle (AMP7). The final number of catchments that had a new flow survey carried out was 143. For the selected 143 catchments, site selection for each flow monitor was made with potential hydraulic conditions, access requirements and health and safety (H&S) in mind. For example, locations on bends in the network and chambers that may have turbulent conditions were avoided. The initial selection was automated through SQLs in Infonet (asset database). Accounting for the above-mentioned considerations improved the likelihood that, when the site engineers came to install the monitors, the location would be suitable for accurate data capture.
It was also crucial during the selection of sites that traffic management and site access were accounted for, avoiding locations where costly traffic management would need to be put into place to install and remove the monitor, or where access to manholes would be difficult due to access restrictions on private land. In catchments where we had access to information on historic flow surveys, we installed monitors in the same locations where we could confirm that the data captured previously were of an acceptable standard. This increased the likelihood that it would be possible to install the monitor, and the hydraulic conditions would lend themselves to accurate data capture. Details on H&S factors considered during survey planning can be found in Section 11. 5. 4 below. Flow monitoring was accompanied by rainfall monitoring using tipping bucket rain gauges.
Rainfall monitoring Rainfall across the catchments subjected to flow monitoring was measured by means of tipping bucket rain gauges equipped with GPRS data loggers. Key rain gauge and data logger specifications are summarized in Table  11. 4. As mentioned earlier, rain gauge densities were decided upon following WaPUG guidance (Table  11. 3). The reader must be reminded that for catchments not subjected to short-term flow monitoring, available telemetry data (i. e. SCADA depth and flow records at ancillaries) and radar rainfall data were used for model re-calibration. The radar data used for this purpose were the UK Met Office data, available at 1 × 1 km 2 and 5 min resolution from the CEDA archive (http://ceda. ac/uk/, accessed 17 Dec. 2020).
Data transmission, storage and retrieval In-sewer monitors and tipping bucket rain gauges were equipped with GPRS data logging and transmission units. Data from each logger were transmitted on a daily basis via a 2G network to a central physical server. All loggers were set to transmit data from 6 a. m. every day. The logger would attempt to send data for up to 3 hours after the first attempt but would then cease in order to conserve battery life. Once in the server, data could be visualized via the RPS Flow Survey Online Viewer. The reason data were transmitted only once per day, rather than at a higher frequency, was to preserve battery life. It is worth noting that data transmission issuesrelated to poor signalwere encountered in approximately 35% of all monitoring sites. Data from the problem sites were retrieved manually on a weekly basis; this entailed renewing traffic permits at traffic-sensitive sites.
The weekly retrieval frequency allowed timely data review, and therefore timely maintenance, and avoided filling up the loggers' internal memory. The memory capacity at 2 minutes logging rate, as is the case for flow monitors, is approximately three weeks. Final datasets were available in FDV format. Data in this format can be readily imported into RPS' data processing and hydraulic modelling software packages, i. e. FlowBot data processing toolbox (described below) and InfoWorks.
Data quality assurance during and after monitoring period Flow and rainfall monitoring data were assessed on a weekly basis as well as at the end of the survey period using RPS' in-house FlowBot toolbox. FlowBot is a bespoke software solution that can be utilized by both flow survey data analysts and hydraulic modellers to visualize, manage and understand flow survey data. FlowBot allows undertaking a range of checks on the depth and flow measurements, including flow monitoring data comparison against theoretical depth vs. flow rating curves, flow monitor comparison including volumetric checks between upstream and downstream monitors, and automatic identification of dry weather and storm events based upon user-defined criteria. Many of these checks are in line with those recommended in the Guide to  Short-Term Flow Surveys (WRc, 1987).
In addition, FlowBot allows for the deployment of a machine learning algorithm (decision tree-based) to classify the performance of flow and depth monitors and ultimately identify faulty measurements. The machine learning algorithm has been trained using 30,000 days of previous human classifications, mined from previous RPS flow surveys. The outputs that are generated have a high degree of accuracy, with assessments that would usually take hours being completed in minutes. Automating this user-intensive operation reduced the time spent by engineers classifying data and allowed for more time to be spent proactively managing a flow survey, maximizing the quality of the final data. In addition to depth and flow checks, FlowBot includes features for rain gauge data quality assurance via cumulative rainfall plots.
On the whole, the use of the FlowBot toolbox led to 80%-time savings in the processing of flow survey data, in relation to conventional (more manual) methods. By undertaking weekly assessments while the survey was still on the ground, it was possible to detect and rectify any problems with given monitoring locations throughout the survey and ultimately ensure that good quality data were collected at the locations of interest. Problems encountered with monitors included power failure, transmission problems due to poor network coverage and/or location of the monitor inside sewer network, ragging, flow/depth data below monitor resolution, and rain gauge blockages, amongst others. Some of the problems could be rectified through sensor maintenance. Others required sensor re-location. Ultimately, quality assurance of flow survey data ensured that the best data possible were adopted for model calibration and verification.
Conclusions and outlook As part of Anglian Water Services' AMP6 Integrated Urban Drainage Modelling Programme, short-term (average of 12-weeks duration) flow and rainfall surveys were undertaken across the majority of the company's catchments, resulting in a total of 3,568 flow and/or depth monitors and 801 rain gauges being installed and operated between March 2017 and November 2019. The data collected through these surveys enabled standardized re-calibration of hydraulic sewer models to a level of confidence appropriate for intended use. The resulting models, alongside other company datasets (e. g. permanent monitoring data at strategic locations), will enable a range of catchment performance assessments and implementation of management strategies, which will ultimately lead to improved catchment management.
e. 12 weeks) in some of the catchments. Likewise, maintenance of and data removal from traffic sensitive sites required constant renewal of traffic permits, which was time-consuming. These problems were however tackled with proactive data review and effective work-planning and collaboration between the modelling and survey teams. The data collected as part of these short-term surveys will be stored centrally by AWS, both to provide an audit trail as well as to enable future studies such as detailed flooding and pollution investigations, further development of ML-based data assurance tools, preliminary testing of real-time analytics, amongst others. UK Water Utilities normally keep their short-term flow data in-house, and not publicly available. Depending on the type of data and the sensitivity of the catchment, access may be arranged for research on a case-by-case basis, through individual legal agreements with the researchers.
Measured variables and location of monitors A total of 75 monitoring sites have been defined throughout the project. As budget constraints did not allow for parallel monitoring at all locations, the project was designed to work with a rotation system for (part of) the sensors. This is reflected in the design of the used database. The average period for which monitors had been operated at one individual site for rotated devices was between 9 and 12 months. This period was chosen to capture seasonal variations and as a result of practicalities. The absolute minimum for monitoring at one location is deemed to be around 3 months. First sensors were installed in March 2017. As of July 2020, about 35 locations were still active without any plans for de-installation. Figure  11. 10 shows an overview on the finally selected locations for flow, water level and precipitation, but also continuous water quality monitoring (temperature, conductivity, turbidity, TSS (total suspended solids), BOD (biological oxygen demand), COD (chemical oxygen demand), NO 3 , NH 4 , all backed up by lab samples) not detailed here any further.
Flow and water level Flow is monitored at 13 locations using 10 devices based on proven technology: hydrostatic depth transducer and ultrasound Doppler velocity sensor. Such devices have been successfully used throughout many short-term campaigns over the last 20 years (using an external contractor). Usually the focus of Data collection in urban drainage and stormwater management systemscase studies 443 Downloaded from http://iwaponline. com/ebooks/book/chapter-pdf/919177/9781789060119_0415. pdf by TECHNISCHE UNIVERSITEIT DELFT user short -term campaigns, which typically run for seven weeks with preferably a minimum of three reasonable rain events being monitored  (WRc, 1987) , is on the characterization of the system's most relevant locations and ancillaries (e. g. large CSO structures, throttle locations, pumping stations and joining points of collectors) to develop a sound understanding of systems hydraulics.
In the IMPAKT! Project this classic approach was maintained for much longer (approx. 1. 5 years) to allow for the identification of seasonal variations in dry weather flows, and to allow rotation between a number of locations. The data gathered by the flow monitoring campaign are extended through the permanent installation of 20 additional water level monitoring devices (pressure gauges) at CSOs (in collectors and/or storage tanks), and 15 data loggers capturing existing level measurements in pumping stations. At these locations full flow monitoring is of less or no relevance given the poor velocity conditions, but levels monitored at pumping stations can be converted using the wet well's geometry to estimate flows as done by  Fencl et al. (2019). Desirable monitoring locations are selected based on detailed analysis of the existing sewer models or asset databases.
For all locations not requiring data roaming, virtual-private-network SIM cards have been used to allow for high data security paired with ease of use in a fixed IP address (Internet Protocol address) range (no firewall issues). Data loss due to failed communication was limited to ,1% as a result of sufficiently large data storage in the loggers, timely manual intervention and the use of high-performance antennae where required. Data collection in urban drainage and stormwater management systemscase studies 445 Downloaded from http://iwaponline. com/ebooks/book/chapter-pdf/919177/9781789060119_0415. pdf by TECHNISCHE UNIVERSITEIT DELFT user The different types of monitoring devices are reflected in different ways for data communication and had to be accounted for in the data import routines for the individual sensor types.
Data management, validation and accessibility The data management system is designed around a three-level hierarchy of monitoring site (location or mounting place), monitoring device, and data channel (a time series of any variable, e. g. flow or pump switch-on level). To accommodate this structure, the system has been designed in-house as part of the project presented here. As the structure and performance of the system have proved satisfactory, the system will be used for future projects. All data are stored in one central relational PostgreSQL™ database (http://www. postgresql. org, accessed 28 April 2021) running on a company-wide accessible dedicated virtual server. The decision for PostgreSQL™ was based on the ease of use of vectorized time series data (data with multiple measurements per timestamp) as they are frequently used for spectrometric water quality measurements.
Formulae required for these calculations are defined in the channel's meta-data, to allow for easy modification. All time series and meta-data can be consulted and manipulated by a feature-rich and flexible querying tool and free programming based on Matlab™. This tool also allows the import, visualization and analysis of frequently used data from external sources such as historians (e. g. GE Historian™), web-based data servers (e. g. waterinfo. be) and simulation results (here e. g. from Infoworks ICM™, SWMM (Storm Water Management Model). Alternatively, a read-only web interface exists for quickly creating views of time series data and a real-time overview on monitoring performance and device connectivity in a GIS-enabled, browser-based environment implemented as overlays on Openstreetmap™ (Figure  11.
12). Time series data are dynamically queried using PHP (Hyper text Pre-processor, also known as Personal Home Pages) and visualized with an existing JavaScript charting library (http://dygraphs. com/, accessed 28 April 2021). The fairly detailed and expansible data structure for meta-data is inspired by DWA-M 151 (DWA, 2014) and is aimed at rendering the database self-explanatory to users not accustomed to the project itself. For example, all monitoring locations can be visualized in Google Streetview™ through a link, allowing easy recognition of the location when on site. Each monitoring channel, device and site is assigned a state calculated in real-time which reflects the fitness of the data it represents: the data logger shown in Figure  11. 12(b) occurs in red (intervention required) as the database did not receive any data for one of its channels (Pump 5).
Daily backups of the entire database ensure it can be restored in case of irreparable data loss or damage. 11. 6. 5 Sensor operation, maintenance and budget 11. 6. 5. 1 Operation and maintenance All monitoring read-outs are visually inspected at least weekly by the personnel who will later be working with the data, e. g. for model calibration to ensure sufficient data quality and plausibility. Implausible data, high noise ratio or comparable phenomena potentially caused by poor monitoring quality will lead to device inspection. Sensor maintenance is therefore accounted for in the weekly planning of technicians to ensure short response times. All flow sensors are visually inspected on a weekly basis and calibrated under lab conditions once every year. Batteries are changed, if required. Water velocity and level are manually measured on site using a hand-held screw current meter and folding rule, respectively.
The use of meta-data should thus be enforced wherever possible. This can be easily realized by the sensor network administrator by only allowing sensors to be added into the monitoring system if a minimum amount of meta-data (e. g. proper naming, address or X-Y-coordinates of installation site) are provided so that the monitored signals are self-explanatory to colleagues and project partners. Aside from flexible data analysis tools, also low-threshold, ubiquitous access to the data should be provided to ensure project-wide adoption. The usability of the web interface (initially purely conceived for maintenance personnel) of the project presented here has led to other projects migrating their data into this database purely for its ease of data access. For any monitoring campaign that accommodates multiple sensors for more than a couple of weeks, the use of an adequate database architecture should be preferred over data storage in individual files: the amount of data currently maintained in the database (80 GB as of July 2020, after 3.