Scaling computations and maintaining reproducible projects with signac, an open-source framework
Channel: csvconf
Date: 2021-06-11

Presenter: Alyssa Travitz  



Data science projects are often messy, with complex workflows and large, heterogeneous parameter spaces. Bash scripting and long file names can only get you so far, and can result in a project that is incomprehensible to collaborators or your future self. The open-source data management framework, signac, (https://signac.io) makes it easy to maintain organized and reproducible projects, and integrates with existing file-based workflows. The serverless data management and lightweight workflow model ensure that projects are easily transferable between laptops and high-performance computing environments, and the data model is well suited for high-dimensional parameter searches or hyperparameter optimization of machine learning models. In this talk, we will demonstrate the signac approach to managing data science projects, emphasizing its use in real-world scenarios. The signac framework not only increases research efficiency, but makes best practices and scalability easy and intuitive.