2_use_case_name: Data-driven approaches to filling missing time-series data within the San Francisco Bay-Delta
3_agency: Department of the Interior
3_abr: DOI
4_bureau: USGS
8_topic_area: Mission-Enabling
10_commercial_ai: None of the above.
11_purpose_benefits: In measurements of natural systems, incomplete time-series data are the rule, not the exception. Environmental time-series data may suffer from gaps at a variety of time scales, significantly reducing the number of observations to understand phenomena, identify change, calibrate models, and predict future behavior. Data may be missing because of sensor degradation (e.g., biofouling, mechanical issues, power failure), failure to meet basic quality assurance checks, or resampling of paired observations to a common timestamp.

Decades of water-quality time-series data (e.g., turbidity, salinity, temperature) collected throughout the San Francisco Bay-Delta (SFBD) by a variety of agencies including USGS, CA DWR, USBR, USFWS, and USACE are used as a critical indicator of estuary health. These data are no exception to the rule and contain numerous gaps. While bad data points can be relatively straightforward to identify (e.g., as anomalously high or low values), they are challenging to replace and are often flagged and left blank, which can bias long-term observational records.

We will test and develop several (~5) proposed methods to fill gaps in water-quality time-series data and quantify the uncertainty in those estimates. The methods will range in complexity, from linear regression to machine learning and deep learning approaches (Lepot and others, 2017), and will characterize uncertainty in the filled data (Cox and others, 2003).  For this proposal, we will initially restrict our scope to turbidity data, because of the relatively high biofouling rate of the optical sensors typically used to measure turbidity, as well as its ecological significance. In addition, we are already actively implementing one of the methods to fill gaps in turbidity data collected in a south S.F. Bay salt-marsh tidal creek (Figure 1) as part of a project investigating temporal variability in sediment delivery to marshes funded by the S.F. Bay Regional Monitoring Program.
12_outputs: Outputs were not documented and DOI will update the use cases once additional data is collected from submitter
16_dev_stage: Acquisition and/or Development
17_impact_type: Neither
22_dev_method: Developed in-house.
24_hisp_support: No
27_public_info: Yes
28_iqa_compliance: USGS Fundamental Science Practicies Policies
29_contains_pii: No
30_saop_review: No
31_data_catalog: Yes
34_data_docs: Documentation is complete: Documentation exists regarding the maintenance, composition, quality, and intended use of the training and evaluation data, as well as any statistical bias across model features and protected groups. 
37_custom_code: Yes
38_code_access: Yes – agency has access to source code, but it is not public.
40_has_ato: Yes
41_system_name: Science Support System (SSS)
42_dev_tools_wait: Less than 6 months
43_infra_provisioned: Yes
45_compute_request: Yes
47_timely_resources: Yes
49_existing_reuse: Use of existing data platforms: This use case is being developed on existing enterprise data and analytics platforms within the agency rather than procuring additional platforms or SaaS to operate.
50_internal_review: Documentation has been published: Complete documentation has been published to a repository or data catalog within the agency and is made accessible to other data science teams for review and feedback.
51_extension_request: No – Agency did not request an extension for this use case.