Project: datamanagement
Issue DM-5013: Convert Confluence DM Developer Guide to Sphinx (hack day) 
This is a hack day sprint to convert all remaining content on https://confluence.lsstcorp.org/display/LDMDG to reStructuredText content in the Sphinx project at https://github.com/lsst-sqre/dm_dev_guide and published at http://developer.lsst.io. The top priority for this sprint is to port all content into reST and have it tracked by Git. h2. Sprint ground rules # Before the sprint, clone {{https://github.com/lsst-sqre/dm_dev_guide.git}} and {{pip install -r requirements.txt}} in a Python 2.7 environment so that you can locally build the docs ({{make html}}). # Claim a page from the list below by putting your name on it. Put a checkmark on the page when you’ve merged it to the ticket branch (see below). # See http://developer.lsst.io/en/latest/docs/rst_styleguide.html for guidance on writing our style of reStructuredText. Pay attention to the [heading hierarchy|http://developer.lsst.io/en/latest/docs/rst_styleguide.html#sections] and [labelling for internal links|http://developer.lsst.io/en/latest/docs/rst_styleguide.html#internal-links-to-labels]. # If you use Pandoc to do an initial content conversion, you still need to go through the content line-by-line to standardize the reStructuredText. I personally recommend copy-and-pasting-and-formatting instead of using Pandoc. # Your Git commit messages should include the URL of the original content from Confluence. # Merge your work onto the {{tickets/DM-5013}} ticket branch. Rebase your personal work branch before merging. JSick is responsible for merging this ticket branch to {{master}}. # Put a note at the top of the confluence page with the new URL; root is {{http://developer.lsst.io/en/latest/}}. h2. Planned Developer Guide Table of Contents We’re improving the organization of DM’s Developer Guide; there isn’t a 1:1 mapping of Confluence pages to developer.lsst.io pages. Below is a proposed section organization and page structure. These sections can still be refactored based on discussion during the hack day. h3. Getting Started — /getting-started/ * ✅ *Onboarding Checklist* (Confluence: [Getting Started in DM|https://confluence.lsstcorp.org/display/LDMDG/Getting+Started+in+DM]). I’d like this to eventually be a quick checklist of things a new developer should do. It should be both a list of accounts the dev needs to have created, and a list of important developer guide pages to read next. The NCSA-specific material should be spun out. [[~jsick]] * *Communication Tools* (new + DM Confluence [Communication and Links|https://confluence.lsstcorp.org/display/DM/Communication+and+Links]). I see this as being an overview of what methods DM uses to communicate, and what method should be chosen for any circumstance. * *Finding Code on GitHub* (new). This should point out all of the GitHub organizations that a developer might come across (DM and LSST-wide), and point out important repositories within each organization. Replaces the confluence page [LSST Code Repositories|https://confluence.lsstcorp.org/display/LDMDG/LSST+Code+Repositories] h3. Processes — /processes/ * ✅ *Team Culture and Conduct Standards* (confluence) * ✅ *DM Development Workflow with Git, GitHub, JIRA and Jenkins* (new & Confluence: [git development guidelines for LSST|https://confluence.lsstcorp.org/display/LDMDG/git+development+guidelines+for+LSST] + [Git Commit Best Practices|https://confluence.lsstcorp.org/display/LDMDG/Git+Commit+Best+Practices] + [DM Branching Policy|https://confluence.lsstcorp.org/display/LDMDG/DM+Branching+Policy]) * ✅ *Discussion and Decision Making Process* (new & [confluence|https://confluence.lsstcorp.org/display/LDMDG/Discussion+and+Decision+Making+Process]) * ✅ *DM Wiki Use* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/DM+Wiki+Use]) [[~swinbank]] * ✅ *Policy on Updating Doxygen* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Policy+on+Updating+Doxygen]); needs to be addressed with TCT. Inter-link with the developer workflow page. [[~jsick]] (we’re just re-pointing the Confluence page to the workflow document) * ✅ *Transferring Code Between Packages* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Transferring+Code+Between+Packages]) [[~swinbank]] * -*Policy on Changing a Baseline Requirement*- ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Policy+on+Changing+a+Baseline+Requirement]) * ✅ *Project Planning for Software Development* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Project+Planning+for+Software+Development]) [[~swinbank]] * ✅ *JIRA Agile Usage* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/JIRA+Agile+Usage]) [[~swinbank]] * -*Technical/Control Account Manager Guide*- ([confluence|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=21397653]) (Do not port; see discussion below.) * *Licensing* (new) Need a centralized page to discuss license and copyright policies; include boilerplate statements. h3. Coding Guides — /coding/ * ✅ *Introduction* and note on stringency language (confluence: [DM Coding Style Policy|https://confluence.lsstcorp.org/display/LDMDG/DM+Coding+Style+Policy]) * ✅ *DM Python Style Guide* (confluence: [Python Coding Standard|https://confluence.lsstcorp.org/display/LDMDG/Python+Coding+Standard]) * ✅ *DM C++ Style Guide* (confluence pages: [C++ Coding Standard|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908666] + [C++ General Recommendations|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908756] + [C++ Naming Conventions|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908685] + [C++ Files|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908674] + [C++ Statements|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908706] + [C++ Layout and Comments|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=16908737] + [Policy on use of C++11/14|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283399] + [On Using ‘Using’|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283856]) * Coding Style Linters (new; draft from confluence [C++ Coding Standards Compliance|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20283861] and [Python Coding Standards Compliance|https://confluence.lsstcorp.org/display/LDMDG/Python+Coding+Standards+Compliance] * ✅ *Using C++ Templates* ([confluence|https://confluence.lsstcorp.org/pages/viewpage.action?pageId=20284190]); this page needs to severely edited or re-written, however. * ✅ *Profiling* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Profiling|]). Also add a section ‘Using Valgrind with Python' (new) [[~jsick]] * ✅ *Boost Usage* ([TRAC|https://dev.lsstcorp.org/trac/wiki/TCT/BoostUsageProposal]) [[~tjenness]] * ✅ *Software Unit Test Policy* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Software+Unit+Test+Policy]) [[~swinbank]] * ✅ *Unit Test Coverage Analysis* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Coverage+Analysis]) [[~swinbank]] * ✅ *Unit Testing Private C++ Functions* ([trac|https://dev.lsstcorp.org/trac/wiki/UnitTestingPrivateFunctions]) [[~swinbank]] h3. Writing Docs — /docs/ * *Introduction* (new): Overview of DM’s documentation needs; links resources on technical writing. * *English Style Guide* (new): Supplement the [LSST Style Manual|https://www.lsstcorp.org/docushare/dsweb/Get/Document-13016/LSSTStyleManual.pdf] and provide English style guidance specific to DM. Capitalization of different heading levels; use of Chicago Manual of Style; a ‘this, not that’ table of spelling and word choices. * ✅ *ReStructuredText Style Guide* (new) * ✅ *Documenting Stack Packages* (new) * ✅ *Documenting Python Code* (new) * ✅ *Documenting C++ Code* (confluence, adapted from [Documentation Standards|https://confluence.lsstcorp.org/display/LDMDG/Documentation+Standards]); needs improvement * ✅ *Writing Technotes* (new; port README from [lsst-technote-bootstrap|https://github.com/lsst-sqre/lsst-technote-bootstrap/blob/master/README.rst]) h3. Developer Tools — /tools/ * ✅ *Git Setup and Best Practices* (new) * ✅ *Using Git Large File Storage (LFS) for Data Repositories* (new) * ✅ *JIRA Work Management Recipes* (new) * ✅ *Emacs Configuration* ([Confluence|https://confluence.lsstcorp.org/display/LDMDG/Emacs+Support+for+LSST+Development]). See DM-5045 for issue with Emacs config repo - [~jsick] * ✅ *Vim Configuration* ([Confluence|https://confluence.lsstcorp.org/display/LDMDG/Config+for+VIM]) - [~jsick] h3. Developer Services — /services/ * ✅ *NCSA Nebula OpenStack Guide* (Confluence: [User Guide|https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+User+Guide] + [Starting an Instance|https://confluence.lsstcorp.org/display/LDMDG/Introduction+to+Starting+a+Nebula+Instance] + [Using Snapshots|https://confluence.lsstcorp.org/display/LDMDG/Start+an+Instance+using+a+base+snapshot+with+the+LSST+Stack]. Add the [Vagrant instructions too from SQR-002|http://sqr-002.lsst.io]? [[~jsick]] * ✅ *Using lsst-dev* (Confluence: [notes Getting Started|https://confluence.lsstcorp.org/display/LDMDG/Getting+Started+in+DM] + [Developer Tools at NCSA|https://confluence.lsstcorp.org/display/LDMDG/Developer+Tools+at+NCSA] * ✅ *Using the Bulk Transfer Server at NCSA* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Using+the+Bulk+Transfer+Server+at+NCSA]) [[~jsick]] h3. Build, Test, Release — /build-ci/ * *Eups for LSST Developers* (new) [[~swinbank]] * ✅ *The LSST Software Build Tool* → ‘Using lsstsw and lsst-build' ([confluence|https://confluence.lsstcorp.org/display/LDMDG/The+LSST+Software+Build+Tool]); lsstsw and lsst-build documentation. [[~swinbank]] * *Using DM’s Jenkins for Continuous Integration* (new) [~frossie] * ✅ *Adding a New Package to the Build*([confluence|https://confluence.lsstcorp.org/display/LDMDG/Adding+a+new+package+to+the+build]) [[~swinbank]] * ✅ *Distributing Third-Party Packages with Eups* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Distributing+third-party+packages+with+EUPS]) [[~swinbank]] * ✅ *Triggering a Buildbot Build* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Triggering+a+Buildbot+Build]) [~frossie] * ✅ *Buildbot Errors FAQ* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Buildbot+FAQ+on+Errors]) [~frossie] * * Buildbot configuration ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Buildbot+Configuration+and+Setup] [~frossie] * *Creating a new DM Stack Release* ([confluence|https://confluence.lsstcorp.org/display/LDMDG/Creating+a+new+DM+Stack+Release]); though this page or a modern equivalent should probably belong with the software docs? [~frossie] _A lot of work should go into this section._ Have something about Scons? Or maybe that belongs in the doc of each relevant software product. h2. Leftover Confluence pages h3. The following pages should be moved to a separate Confluence space run by NCSA: * [NCSA Nebula OpenStack Issues|https://confluence.lsstcorp.org/display/LDMDG/NCSA+Nebula+OpenStack+Issues] * [DM System Announcements|https://confluence.lsstcorp.org/display/LDMDG/DM+System+Announcements] * [NCSA Development Servers|https://confluence.lsstcorp.org/display/LDMDG/DM+Development+Servers] h3. The following pages are either not relevant, generally misplaced, or need to be updated/recalibrated: * [Git Crash Course|https://confluence.lsstcorp.org/display/LDMDG/Git+Crash+Course] * [Basic Git Operations|https://confluence.lsstcorp.org/display/LDMDG/Basic+Git+Operations] * [Handling Git Push Problems|https://confluence.lsstcorp.org/display/LDMDG/Handling+Git+Push+Problems] * [LSST Code Repositories|https://confluence.lsstcorp.org/display/LDMDG/LSST+Code+Repositories]; see the proposed “Finding Code on GitHub” page for a replacement. * [Standards and Policies|https://confluence.lsstcorp.org/display/LDMDG/Standards+and+Policies]: this is a good TOC for the Confluence docs; but not longer needed for the new docs. * [Documentation Guidelines|https://confluence.lsstcorp.org/display/LDMDG/Documentation+Guidelines]. Some of this could be re-purposed into an intro to the ‘Writing Documentation’ section; some of this should go in a ‘Processes' page. * [DM Acknowledgements of Use|https://confluence.lsstcorp.org/display/LDMDG/DM+Acknowledgements+of+Use]: this probably belongs in documentation for the software projects that actually used this work.
Story points: 5