Package: dsc
Theproject wikiis the
main source of documentation for both developers and users of the DSC
project. If you are new to the concept of DSC, it may worth reading thisblog
postto understand the motivation behind this project.This work is supported by the the Gordon and Betty Moore Foundation via
an Investigator Award to Matthew Stephens,Grant
GBMF4559, as part of theData-Driven Discovery
program.
If you have any questions or want to share some information with the
developer / user community, please open agithub
issue.Developer notesUpgrading DSC to latest development versionFor most users, we recommend installing themost recent stable
release.
If you would like to upgrade your existing installation of DSC to the
most recent (unstable) development version, follow these steps.DSC is closely developed in parallel withSoS. Therefore, the development
version of DSC (maintained in themasterbranch of the GitHub
repository) typically requires the development version of SoS.To update,pipinstallgit+https://github.com/vatlab/sos-Upipinstallgit+https://github.com/stephenslab/dsc-UInstall dscrutils from sourceAssuming the working directory in your R environment is thedscrepository, run the following code in R to install the latest
development version of the dscrutils R package:getwd()# Should be ... /dscinstall.packages("dscrutils",repos=NULL,type="source")Project maintenanceAlthough relatively stable and usable in practice, DSC is still actively
being developed. Occasionally upgrades to the most recent version will
lead to changes of file signatures that triggers rerun of existing
benchmark even if they have not been changed. When this happens we will
indicate in bold in our release note below that “a file signature clean
up is recommended” (see release note 0.2.7.7 for example). That means
after such DSC upgrades you should rerun your benchmark with-sexisting(or,--touch) option to update file signatures. If
possible, it is recommended that you rerun your benchmark from scratch
(if resources can afford) with-snoneinstead of-sexistingto
skip all existing files. We apologize for the inconveniences it incurs.Change Log0.4.3Issue #94 revisited.[minor] Fix syntax incompatibility with earlier versions ofmsgpack.[minor] Fix a bug of not updating meta-database when-sexistingis used.A file signature clean up is required after this upgrade.0.4.2Issue #214 for container support.SoS bumped to version 0.21.5 for container support.0.4.0Issue #194Issue #202[minor] #209, #196Improve scripts command options.A file signature clean up is required after this upgrade.0.3.x0.3.10Issue #154Add a remote job optionnodes_per_joband provide such mechanism
to run on multiple nodes.SoS bumped to version 0.20.2.0.3.9Issue #179Issue #182Issue #183SoS bumped to version 0.19.11 for additional improvements in cluster
job submission.0.3.8Issue #171 is partially improved.0.3.7dscquerynow properly handlesNULLmodule output variables.Some performance improvements on I/O database.SoS bumped to version 0.19.7 for various bug fixes on cluster job
submitter.0.3.6Now SQLite keywords can be used as module names: no error message
will be triggered.Introduce an explicit syntax for base module: a module without
executable is considered base module.Bumprpy2version requirment version 3.0.0+ for benchmark with R
and Python modules.SoS bumped to version 0.19.5 for improved memory usage on job
submitter and improved Python module version check.0.3.5Major improvements todscrqueryPerformance improvement #168Output format improvement #1740.3.4Reimplementdsc-queryanddscrqueryfor improved handling of
missing value #145.Reimplementdscrquery’sconditionstatement to make it more
R-user friendly.Add unknown command argument #162.[minor] IntroduceDSC::run::default, for the behavior of running
the script without targets.[minor] #161SoS bumped to version 0.19.1 for improved sockets management and
improved R library auto-installation.0.3.3[minor] #160SoS bumped to version 0.18.4 for performance optimizations.0.3.2Various improvements for remote job submission and execution (mostly
on SoS).SoS bumped to version 0.18.1 to support these changes.0.3.1Improved database I/O performance to cope with file system latency.Improved Python to R data flow.Paralleled data extraction indscquery.SoS bumped to version 0.17.4 to support a new implementation of job
queues.0.3.0SoS bumped to version 0.16.9 to support a new implementation of
signatures.[minor] Bug fix #147.0.2.x0.2.9.1Stop moving library imports to the front of scripts, due to various
side effect.0.2.9.0SoS bumped to version 0.9.16.0 for optimized remote task file
management.[minor] Bug fixes.A file signature clean up is required after this upgrade.0.2.8.6Change in query behavior #1450.2.8.5Bug fixes for cluster execution #142, #143, #144.Add-doption to output DAG #141.Removed-poption because it triggers rerun and cannot be easily
implemented otherwise.SoS bumped to version 0.9.14.10 for many of the fixes above.A file signature clean up is recommended after this upgrade.0.2.8.4Fix running Python 3 based modules on Mac computer withhomebrewinstalled Python #140.0.2.8.3dsc-iocan now convert CSV to HTML with pop-up figures.Addgroupsandload.pkloptions todscrutils::dscquery.[minor] Bug fixes.0.2.8.2Add-poption to print stdout and stderr to screen.SoS bumped to version 0.9.14.1 forImproved parallel slot management.Improved messaging on executed steps (use-v3to display in
DSC).0.2.8.1Minor file check performance optimization.Force overwrite convertedpkltordsindscutils::dscquery, as a save default.0.2.8Input string parameter behavior has changed since this version. Now
un-quoted strings will be treated input script code; string parameters
will have to be quoted. A new DSC configuration parser has been
implemented to overcomepyYAMLrestrictions. Please submit a bug
report if the new parser misbehaves.A file signature clean up is recommended after this upgrade.0.2.7.11[minor] More stringent check on improper module names ending with_{digits}.0.2.7.10Stop adding script hash to default seed #136.[minor] SoS bumped to version 0.9.13.8 a bug fix release.A file signature clean up is recommended after this upgrade.0.2.7.9Minor touches on 0.2.7.8 – just a celebration of the 1,000-th commit to
the DSC repo on github, after 2 years and 3 months into this project.0.2.7.8Implement a preliminary%includefeature to provide alternative
code organization style.Allow for!operator inList()andDict().SoS bumped to version 0.9.13.7 for improved remote job support.[minor] Various bug fixes.0.2.7.7Improvements for module with shell executables and command options.Improvements for remote execution #131.Improved logging.Bug fixes #126, #127.SoS bumped to version 0.9.13.4 for #128 and related.A file signature clean up is recommended after this upgrade.0.2.7.6Add new featuredscrutils::shiny_plotto display simple benchmark
results.[minor] Display unused modules with-hoption.0.2.7.5Add R / Python packages and version display with-hoption.Add.gitignorefor cache folder when a git environment is
detected.SoS bumped to 0.9.13.3 that now bundles thepbsmodule.0.2.7.4Improved R’s sessionInfo format.Bug fixes #119, #121, #122[minor] Error message improvements.0.2.7.3More stringent R library and command executable check.[minor] Fix a regression bug on path due to 0.2.7.2.0.2.7.2Improved Windows path support.[minor] Fix a bug with nested tuple withraw().0.2.7.1Dump individual data object with scripts usingdsc-query*.pklanddsc-query*.rds.[minor] Improve behavior for length 1 vector in R’s list withR()operator.[minor] Various bug fixes.0.2.7#92paired
parameter input convention.#90and#93useRmdfiles as module executables.#94and#95addedDSC::replicateand command option--replicate.EnhanceR()operator due to use ofdscrutilspackage. This packages is now required to parse DSC file whenRmodules are involved.Add, by default, a variableDSC_DEBUGto output files that saves
various runtime info.SoS bumped to 0.9.13.2Support R github package force install when version mismatches.Force usepipto install local development version.#97Improved
error logging and reporting behavior.[minor] Revert fromruamel.yamltoyamlfor better
performance.[minor]#96[minor]#98[minor] Various bug fixes.0.2.6.5Bring back partial mixed languages support.Piplines with mixed R
and Python code can communicate data of limited types (recursively
support array, matrix, dataframe), via ``rpy2`` as in versions prior
to 0.2.5.x. Support for additional languages will be implemented on
need basis withHDF5format#86.0.2.6.4Add adsc-iocommand to convert between pythonpickleand RRDSfiles – an internal command for data conversion and a test
forrpy2configuration.0.2.6.3Inline module executable via language interpreters (eg.R(),Python()).0.2.6.2[minor] Ignore leading.infile():file(.txt)andfile(txt)are equivalent.[minor] Disallow derivation of modules from ensemble.[minor] Various bug fixes.0.2.6.1Internally replaceRDSformat withHDF5format for Python
routines.Pipeline with mixed languages is now officially broken at
this point until the next major release that supports ``HDF5`` in
R.SoS required version bumped to 0.9.12.7 for relevant upstream bug
fixes for remote host computing.[minor] Various bug fixes.0.2.6Bring back--hostoption; add a companion option--to-hostto
facilicate sending resources to remote computer.Add--truncateswitch.SoS required version bumped to 0.9.12.3 for relevant upstream bug
fixes.[minor] Improved command interface.0.2.5.2SoS required version bumped to 0.9.12.2 for relevant upstream bug
fixes.0.2.5.1Change inseedbehavior: since this releaseseedwill no
longer be a DSC keyword. Users are responsible to set seeds on their
own.[minor] Allow for both lower case and capitalized operator namesFile/file, List/list, Dict/dict.0.2.5New syntax release, compatible with SoS 0.9.12.1.Removed--hostoption due to upstream changes.0.1.x0.1.0First release, compatible with SoS 0.6.4.