TITLE: Protocol for extracting flow hydrograph shape metrics for use in time-series flood hydrology analysis
AUTHORS: Amir Mohammad Arash, Kirstie Fryirs, Tim J. Ralph

[DESCRIPTION]
The shape characteristics of flow hydrographs hold essential information for understanding, monitoring and assessing changes in flow and flood hydrology at reach and catchment scales. However, the analysis of individual hydrographs is time consuming, making the analysis of hundreds or thousands of them unachievable. A method or protocol is needed to ensure that the datasets being generated, and the metrics produced, have been consistently derived and validated. In this study protocol, we present workflows in Python for extracting flow hydrographs with any available temporal resolution from any Open Access or publicly available gauging station records. The workflow identifies morphologically-defined flow and flood types (i.e. in-channel fresh, high flow and overbank flood) and uses them to classify hydrographs. It then calculates several at-a-station and upstream-to-downstream hydrograph shape metrics including kurtosis, skewness, peak hydrograph stage, peak arrival time, rate-of-rise, peak-to-peak travel time, flood wave celerity, flood peak attenuation, and flood wave attenuation index. Some metrics require GIS-derived data, such as catchment area and upstream-to-downstream channel distance between gauges. The output dataset provides quantified hydrograph shape metrics which can be used to track changes in flow and flood hydrographs over time, or to characterise the flow and flood hydrology of catchments and regions. The workflows are flexible enough to allow for additional hydrograph shape indicators to be added or swapped out, or to use a different hydrograph classification method that suits local conditions. The protocol could be considered a change detection tool to identify where changes in hydrology are occurring and where to target more sophisticated modelling exercises to explain the changes detected. We demonstrate the workflow using 117 Open Access gauging station records that are available for coastal rivers of New South Wales (NSW), Australia. 

[BEFORE_START]
1- Users must ensure their input Excel file contains columns labelled 'date' and 'value' for timestamp and streamflow data, respectively, to use this code effectively. Adjustments might be necessary based on specific dataset characteristics and
analysis requirements. 

2- They need to make sure that Python 3.7 is installed on their systems. The instructions of installing required libraries within Python is provided in Step 6 of this protocol.

3- Always validate the outputs (extracted hydrographs and calculated metrics) and modifications to suit your analysis needs and data format.

[GUIDELINES]
Please note that this code employs user-defined threshold-based peak identification for distinct flow types and then filters these peaks to ensure they match specific flow criteria. It organises and separates the extracted hydrographs into different folders for each flow type, preventing any mix-ups. Each hydrograph is uniquely processed, named, and saved separately to avoid duplication. Additionally, the code establishes time boundaries around each peak, ensuring that hydrographs are extracted within a defined duration, which helps in accurate assignment and avoids incorrect overlaps or misplacements among different flow types.

The analysis was conducted using Python, leveraging a suite of libraries to efficiently handle data, conduct visualisations, and perform statistical analyses. The analysis was in Python version 3.7 and the script name was ‘Hydrograph extraction code - User friendly version.’ Pandas, a versatile Python library, effortlessly handled streamflow data from CSV and Excel files. It simplified data import, transformation, and management, enabling smooth processing and analysis of the streamflow datasets. The input Excel (.xlsx) file is formatted with two columns: 'date' and 'value.' The 'date' column represents the timestamp of the streamflow measurements, while the 'value' column contains the corresponding streamflow values.

Matplotlib and Seaborn were employed for visual representation, generating graphical plots that illustrate the streamflow data over time alongside threshold lines indicating specific flow types (in-channel fresh, high flow, and overbank flood). The code used datetime and timedelta functionalities to handle the average flood hydrograph duration, allowing adjustments to durations and helping calculate durations for peak segmentation.

The core analysis of the hydrograph peaks was carried out utilising the ‘find_peaks’ function from Scipy'sSignal module. This function enabled the identification of significant peaks within the streamflow data based on user-defined thresholds. Flow hydrograph peaks were categorised into three types: in-channel freshes, high flows, and overbank floods, based on predefined threshold values provided by the user.

The subsequent steps involved the comprehensive visualisation of these identified peaks. This included generating line plots using Matplotlib and Seaborn, depicting the temporal variation of streamflow. Threshold lines corresponding to different types of flow hydrograph peaks were overlaid on these plots, providing a visual reference for overbank flood, high flow, and in-channel fresh hydrograph peaks within the streamflow data.

Following the visual representation, the analysis delved into the detailed processing of each peak. Around each identified peak, the code segmented the data to capture the dynamics before and after the peak. Specifically, a window of a specified duration (typically 5 days before and after the peak) was defined to confine the hydrograph for further analysis. Within these segments, the code performed some statistical computations, deriving key hydrograph shape metrics such as count, mean, standard deviation, minimum, maximum, kurtosis, and skewness using Pandas and Scipy's statistical functions. These metrics provided a comprehensive understanding of the characteristics of each identified peak.

In conjunction with the statistical analysis, the code also managed the storage and organisation of extracted hydrographs. It systematically created folders to categorise and store the processed hydrographs and plots these peaks based on their respective flow hydrograph types (‘Overbank Flood Peaks’, ‘High Flow Peaks’, ‘In-Channel Fresh Peaks’) within the time series data, ensuring clear and organised data management throughout the analysis.

Moreover, the code aggregated the statistical outcomes and peak data, combining them into detailed Excel files with .csv extension and image formats for each identified peak (‘In_channel_fresh_peaks_stats.csv’,’ High_flow_peaks_stats.csv’, and ‘Overbank_flood_peaks_stats.csv’). These comprehensive reports included statistical summaries and plotted representations, contributing to a holistic understanding of the identified peaks.

The required Python packages for extracting hydrograph and assigning them to one of the three flow types (i.e. in-channel fresh, high flow or overbank flood):

- Pandas: For managing data using DataFrame structures, enabling CSV and Excel ingestion.

- Matplotlib, Seaborn: Facilitating data visualisation to represent streamflow and threshold lines graphically.

- Datetime, Timedelta: Managing temporal data for date modifications and duration calculations.

- Scipy Signal: Using `find_peaks` to identify peaks within streamflow data.

- OS, Shutil: Supporting directory creation and file management.

- Numpy: Essential for numerical operations, especially array manipulations and statistical computations.
 

[STEPS]
SECTION: B. Input data preparation
3. Download the streamflow time series from BoM and WaterNSW.

The streamflow data at sub-1hour intervals is available from the Australian Bureau of Meteorology (BoM) (Fig 3) and WaterNSW 2023 (Fig 4) Open Access portals.

WaterNSW Website
 
BoM Website
 
SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
7. Introduction: 
Extract the in-channel fresh, high flow and overbank flood hydrographs by running a Python script, and calculate flow indicators (kurtosis, skewness and peak flow stage) for each flow hydrograph.

By running the Python code, the script extracts flow hydrographs and calculates kurtosis, skewness, peak value, peak date and some other statistics for each hydrograph. Before undertaking further analysis, the extracted hydrographs need to be checked to ensure that the delineated hydrographs and calculated metrics are correct and any inputs (streamflow time series, flow stages, and duration of hydrographs) adjusted where needed.
 
In Excel, RoR is calculated using peak value and peak date in Eqn 3. Total drainage area draining into each gauge and total area of each catchment is calculated using the watershed tool
in ArcHyro extension in ArcGIS Pro (Arash et al., 2023). Subcatchment area draining to each gauge is classified as small (e.g. <2000 km2), moderate (e.g. 2,000-10,000 km2) and large (e.g. >10,000 km2). Also, gauges are classified as upstream (a gauge captured 10% of the total catchment area), midstream (11%–50%), and downstream (>51%).
 
Peak-to-peak travel time of each pair of hydrographs is calculated on the peak value at the upstream and downstream gauges. The distance between upstream and downstream gauge pairs
is calculated in ArcGIS Pro by using any readily available, accurate, streamlines layer (in NSW this is the River Styles streamlines layer). The distance between gauge pairs and peak-to-peak travel time are used to calculate flood wave celerity. FWAI is calculated using the peak stage height value and contributing drainage area at the upstream and downstream gauges. 
SECTION: CODE Extraction and quantification of flow hydrographs from streamflow time series.
0.20. This is a script that checks for required Python libraries, prompts the user for installation if any are missing, and handles the installation process based on the user's choice.

 

SECTION: B. Input data preparation
4. Remove the redundant data in the excel file of streamflow timeseries.

In different places and for different gauges, stage height data may come to a user in different formats. Therefore, setting up the input data excel spreadsheet to a standard format is necessary. For example, the streamflow data download from BoM and WaterNSW portals for NSW streams needs to be adjusted and filtered before importing into the Python code. The label of the first column and row (cell A1) should be ‘date’ and the second column first row (B1) should be ‘value’. All other redundant information should be removed from the excel file. Then, this file should be saved in .xlsx format. The date data interval is 1 hour and the unit of stage height is meters. The format of the date is dd/mm/yyyy hh:mm:ss. 

 
SECTION: B. Input data preparation
5. Specify the in-channel fresh, high flow and overbank flood stages at the study gauges.

Hydrologists and river experts often classify flow stage heights into categories based on water level at each cross-section. This protocol runs on stage height hydrographs instead of discharge for several reasons including, greater data availability, ease of comparison across different streams, ease of visualisation, and to match to stakeholders’ experience in the field and emergency warnings that are issued during floods. Also, each river has a different morphology and therefore discharge or recurrence intervals for say a bankfull flow are not consistent across the landscape. Therefore, this protocol runs on three morphologically-defined flow and flood stages which can then be used to undertake like-with-like comparisons of flow and flood behaviour across catchments and regions. The three flow stages are: in-channel fresh, high flow and overbank flood. Table 1 shows definitions of different flow stages, and Fig 6 represents different flow thresholds at a conceptual cross-section.

 
  
  Type of flow stage
  
  
  Definition of the flow stage
  
  
  
  In-channel fresh
  
  
  A stage flow that is 95% higher than the low
  flow absolute threshold (7Q10) and 50% higher than the fresh flow absolute
  threshold (half bankfull stage height).
  
  
  
  High flow
  
  
  A stage
  flow that is 50% lower than the bankfull flow absolute threshold (top of the macrochannel
  bank) and 5% higher than this threshold.
  
  
  
  Overbank flood
  
  
  A stage flow that is 5% above the bankfull
  flow absolute threshold.
  
 

 
SECTION: B. Input data preparation
6. Determine the start and end time of flood hydrograph.

Preparation of other inputs include identification of the in-channel fresh, high flow and overbank flood stages at each gauge station and identification of the start time and end time of each flow or flood event of interest. The start time would typically be the onset of the precipitation event, which causes a rise in stage height. The end time normally indicates a return to base flow level. To identify the start and end times, flow record variability, local flow conditions and the purpose of study should be considered. In this study, 10 days was selected as the duration of flow and flood events. For ease, this is defined as 5 days back from the flow peak and 5 days forward from the flood peak (Fig 7).

 
SECTION: D. Expected results and technical verification
11. Verify the hydrograph extraction process by visual observation and adjust the inputs, if needed.

The comparative analysis is not available for this study. Nevertheless, the script was executed iteratively, allowing for manual observation of the generated hydrographs. This ensured that the specified inputs consistently produced hydrographs that align with real-world expectations. Additionally, thorough checks were implemented to prevent the duplication of hydrographs, and mixed outputs in the generated dataset (see Fig 9).
 
Besides, the streamflow time series sourced by BoM and WaterNSW and used for extracting hydrographs are of high quality (McMahon and Peel, 2019). The hydrograph indicators calculated in this study were successfully used in previous studies for decadal time‐series changes in flow hydrology as an indicator of flood mitigation signal and natural flood management potential (e.g., Fryirs et al., 2023; Arash et al., 2023). 

 
SECTION: D. Expected results and technical verification
12. Calculate RoR, tT, C, Ha and FWAI:

The calculated statistics in Step 11 were used to calculate rate-of-rise (RoR), peak-to-peak travel time (tT), flood wave celerity (C), flood peak attenuation (Ha) and flood wave attenuation index (FWAI) by using contributing catchment area (Ac), and upstream-to-downstream channel distance between gauges (D) derived from the NSW River Styles database (2023) in ArcGIS Pro version 3.2 (Figs 10-11; Table 3).

 
 
   
 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
8. Pre-processing step:

This is a script that checks for required Python libraries, prompts the user for installation if any are missing, and handles the installation process based on the user's choice.

 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9. The main processes:

The initial code segments primarily focused on data manipulation, threshold-based peak detection using `find_peaks`, and visual representation using Seaborn and Matplotlib libraries. These segments included functions to manage user input for thresholds, data loading, and graphical visualization, offering a comprehensive approach to visualize and identify specific types of peaks in a time series dataset.

Subsequently, the code delved deeper into peak processing, statistical analysis, and data saving. It included a function to process and save peak values, generate statistics, create Excel files and plots for each peak instance, as well as combining and storing summary statistics in separate CSV files for further analysis. This section extended the analysis to encompass detailed statistical insights and organized data into structured formats for comprehensive analysis and reporting.
SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9.1. Essential Python Libraries for Data Analysis and Visualization:

These following lines of code are used to import specific Python libraries/modules that serve various purposes (data analysis, visualization, statistical computations, file operations, and mathematical calculations):
 
 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9.2. Threshold Input and Data Loading Functions for Flood Analysis:

The following snippet is a combination of functions for user input retrieval (`get_user_input()`) and data loading (`pd.read_excel`) to work with streamflow data stored in CSV or Excel format, using Pandas DataFrames for data manipulation and analysis.

 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9.3. Peak Detection for Flood Analysis and Time Modification Functions:

The code segment demonstrates the use of the `find_peaks` function from `scipy.signal` to identify peaks in a time series dataset ('values'). It employs predefined thresholds to isolate specific types of peaks related to flood levels. It also contains a function, `modify_time`, that adjusts dates in the dataset based on certain conditions, ensuring they fall within a specified range.

 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9.4. Visualizing Hydrograph Peaks and Thresholds withSeaborn and Matplotlib:

This code segment illustrates graph plotting using Seaborn and Matplotlib libraries. It visualizes a time series dataset (`time_data`) representing stage heights, along with threshold lines based on user-defined input. Additionally, it organizes extracted hydrographs into separate folders based on peak types (overbank flood, high flow, and in-channel fresh).

 

SECTION: C. Extraction and quantification of flow hydrographs from streamflow time series.
9.5. Peak Value Processing, Statistics Generation, and File Saving:

This code section defines a function `process_peak_values()` responsible for processing, analyzing, and saving peak values obtained from different types of flood peaks identified earlier (`Overbank_flood_peaks`, `High_flow_peaks`, `In_channel_fresh_peaks`). It performs statistics generation, Excel file creation, and image plotting for each peak instance based on a specified time window (`time_delta`). Additionally, it combines and saves summary statistics for each peak type into separate CSV files.

 

SECTION: A. Overview of steps and flowchart
1. The workflow to quantify flow hydrograph metrics both at-a-station and from upstream-to-downstream
is shown in Fig. 1. This workflow contains 10 main steps:

1) Prepare input data including the hydrographs and geospatial data in Excel.
2) Process gauging station cross-sections to identify the in-channel fresh, high flow and overbank flood stages at each of the gauges in Excel.
3) Input the streamflow time series hydrographs from gauge records into Excel (in NSW these are sourced from the BoM and WaterNSW. 
4) Remove redundant data in the excel file of streamflow timeseries.
5) Specify the start and end time of the flow or flood event using the hydrographs in Excel.
6) Run the Python script to extract in-channel fresh, high flow and overbank flood hydrographs.
7) Verify the hydrograph extraction process by visual observation and adjust the inputs (remove outliers and adjust start time and end time) if needed in the Python script.
8) Calculate hydrograph metrics; kurtosis (Eqn 1), skewness (Eqn 2), peak flow stage, peak flow date and rate-of-rise (RoR) (Eqn 3) for each hydrograph in Python.
9) Calculate contributing catchment area (Ac), and upstream-to-downstream channel distance between gauges (D) in ArcGIS Pro and input them into the Python script (in NSW these metrics are calculated by using NSW River Styles streamlines layer (44) in ArcGIS Pro (36)).
10) Calculate peak-to-peak travel time (tT) (Eqn 4), flood wave celerity (C) (Eqn 5), flood peak attenuation (Ha) (Eqn 6) and flood wave attenuation index (FWAI) (Eqn 7) in Excel.

Please note that Steps 8 and 10 produce a hydrograph metrics database as an output (See Fig. 2).

 

SECTION: B. Input data preparation
2. Hierarchical structure of the hydrograph analysis protocol

Fig. 2 shows the hierarchical structure of the hydrograph analysis protocol presented in this study. It involves structured-based preparing, calculating, organising and saving data that captures different information about hydrological characteristics. This structure aims to provide a comprehensive and organised representation of metrics derived from hydrographs and geospatial data, allowing for detailed analysis and interpretation.

 

SECTION: D. Expected results and technical verification
10. Expected results:

This script creates folders and generates separate subplots for different types of identified hydrographs (Overbank Flood Peaks, High Flow Peaks, In-Channel Fresh Peaks) within the time series. These subplots showcase the peaks overlaid on the original data for visual examination.

Besides, it calculates important statistics (kurtosis, skewness, peak value, peak date and some other statistics) for each hydrograph that can be used for further analysis and reference (Table 2). Fig 8 shows an example of the application of script to extract and quantify flow hydrographs for the in-channel fresh, high flow and overbank flood flow stage classes from streamflow time series data for the Hunter River at Singleton gauging station from 01/01/1913 to 21/11/2023.