Implementation of a reporting workflow to maintain data lineage for major water resource modelling projects

In early 2007, the Australian Government commissioned the Murray-Darling Basin Sustainable Yields project to quantify current and likely future water resources throughout the Murray-Darling Basin. It was a large multi-disciplinary modelling project with more than 150 people contributing to the modelling and the production of over 80 public reports. Managing the design, authoring, revision and production of these reports produced over 10,000 files requiring 18.4 GB of storage. Contributions to the reports came from six discipline-based teams supported by coordination, data management and reporting teams. In early 2008, the Council of Australian Governments (COAG) expanded this study to include three new sustainable yields projects, namely the Tasmania Sustainable Yields project, the South-West Western Australia Sustainable Yields project and the Northern Australia Sustainable Yields project. These new projects are also producing more than 80 reports, and large volumes of reporting data. In the Murray-Darling Basin Sustainable Yields project the reporting team was responsible for managing the production of reports. This involved developing, implementing and managing a system for linking model results and text from each discipline-based team to their corresponding reporting and analysis in the reports. A reporting workflow was developed using the concept of elements (plots, tables, maps and diagrams) and text, which allowed for creation, and then continuous revision, throughout a formal three-stage review process, while maintaining data lineage. Elements and text were handled separately in the workflow. Elements were organised in Microsoft Excel workbooks, with one worksheet per element. These workbooks also contained the raw outputs from the models and therefore allowed for the post-processing of results in an organised fashion while maintaining data lineage. Text was written into a base Microsoft Word document, which contained all of the styles, headings, subheadings, and a list of elements and their associated captions. These Word documents were managed using Microsoft SharePoint web collaboration software. Using strict versioning and a check-in/check-out protocol, SharePoint was used to control access to the documents (commercial-in-confidence until government release date) and to alert the reporting team when new versions of documents were uploaded for incorporation into the workflow. Adobe Professional was used to prepare reports for publication. All of the Excel and Word documents contained a version log at their beginning. In this log, the details of the changes made to each of the versions, the date and the person responsible, were recorded. This not only traced the development of the documents, but also revealed the stage of progress through the workflow. This paper reports on the workflow described above. The workflow process and related tools allowed for the production of the reports to be successfully managed while maintaining data lineage. The use of familiar office tools (Word, Excel, Adobe Professional) ensured that the learning curve for authors and modellers was minor, and this allowed training to focus on the use of SharePoint and the need to adhere to formal protocols and conventions. This paper also identifies areas that would have benefited from more structured management. Particular reference is made to the synchronisation of the region and summary reports and to the production of maps, and how the three new sustainable yields projects are overcoming these issues. The decision to separate the modelling activities from reporting on those activities was an innovative one, and the paper reflects on some of the implications of this decision. In conclusion, the reporting workflow processes developed for the Murray-Darling Basin Sustainable Yields project have proved to be effective, adaptable and scalable as illustrated by their adoption in the three new sustainable yields projects.