Package: lms-waltz
waltzA software system for synchronizing curricular materials between a
Learning Management Systems (LMS) and your local filesystem.Major features include:Pull and push data between an LMS and the file systemLightweight, friendly Markdown-based format for local resourcesDiffing of local and remote contentList and search for remote resourcesWe aim to support the following LMSes:Canvas (pages, assignments, quizzes)BlockPy (assignment, groups) <- In-progressInstallationYou can install Waltz from PyPi (the package name islms-waltzeven though the module and command line script iswaltz):foo@bar:~$pipinstalllms-waltzYou can also install our dev version from GitHub:foo@bar:~$pipinstallgit+https://github.com/acbart/waltz.gitSetup WaltzWaltz can synchronize content between a local directory and a remote server.
You'll need to initialize the local directory, whether it is empty or already has your learning materials.foo@bar:~$waltzinitThis will create two new files which should both be included in your.gitignore:.waltzis a plain-text YAML file with settings for the current course repository..waltz.dbis a SQLite database used in file uploading/downloading.Setup a ServiceBefore you can start interacting with an LMS, you'll need to configure an
instance of a service. For example, to configure a new Canvas service, you'll need the following:A short, easy-to-type name for the service instance (e.g.,ud_canvasorcs1014_canvas)API tokenThe URL base for your Canvas site (e.g.,https://canvas.instructure.com/)The Course ID that you want to connect to (usually a large number)Then, you can use the command below:foo@bar:~$waltzconfigurecanvas<name>--base<url>--course<id>--token<token>A more concrete example:foo@bar:~$waltzconfigurecanvasud_canvas--basehttps://udel.instructure.com/--course4234343432343--token1081~zJIOJSDFBBSDFIJAIJ==>...If things went well, you can list the available services:foo@bar:~$waltzlistThe following services are available:local:./canvas:ud_canvasblockpy: (none configured)By default, there's a Local service for the current directory (representing the connection to the filesystem).
You can have more than one instance of a service, which can allow you to
access multiple data sources from one course (e.g., to transfer resources between semesters).
For convenience, you can refer to the first instance of a service by the service's name.
In our case, anywhere that we useud_canvaswe could usecanvasinstead.Identifying ResourcesYou can list available resources for a service:foo@bar:~$waltzlist<service><category>So the following code checks the Pages for the Canvas course. Note that
you could use the specific instance name instead ofcanvas, and either
the singular or plural form ofpages.foo@bar:~$waltzlistcanvaspagesRemote    Local    Title                      Path--------  -------  -------------------------  --------------------Yes       Yes      Abstraction                pages\Abstraction.mdYes       Yes      NetLogo-2                  NetLogo-2.mdYes       No       Overview of FunctionsYes       No       SyllabusYes       No       ThemesYes       Yes      Turtles                    pages\Turtles.mdThelocalservice allows you to omit the category (but you can filter by category if you want):foo@bar:~$waltzlistlocalResource         Title                        Path---------------  ---------------------------  ---------------------------------------------------------[unknown]        NetLogo-2                    NetLogo-2.md[unknown]                                     pages\Abstraction.md[Page]           Turtles                      pages\Turtles.md[quiz]           Backup Quiz                  quizzes\Backup Quiz.md[unknown]                                     quizzes\Mangled Resource.md[quiz]           New Quiz                     quizzes\New Quiz.md[assignment]     Video Upload Test            quizzes\Video Upload Test.md[quiz question]  Complex Matching Question    quizzes\New Quiz Questions\Complex Matching Question.md[quiz question]  Essay Question               quizzes\New Quiz Questions\Essay Question.mdThe local service only knows the resource category and title if that information
is available in the front-matter of the Markdown file. Otherwise, it can
only report the path of potential resource files.Similar to Git, you can usewaltzcommands from child folders and the
system will search up for the configuration and use it appropriately.Managing ResourcesThe two basic commands for moving resources arepullandpush, moving resources between
the local service and a remote one. If the local version does not already exist,
it will be created in the current directory with a filename based on the title of
the resource from the LMS.foo@bar:~$waltzpull<service><category><title>foo@bar:~$waltzpush<service><category><title>Generally, you want to fully specify the remote service, resource category,
and title of the resource you want to use. This avoids ambiguity.foo@bar:~$ waltz pull canvas assignment "Project 3"
foo@bar:~$ ls
Project 3.md
foo@bar:~$ waltz push canvas assignment "Project 3"However, the commands attempt to work intelligently to find the appropriate file given minimal amounts
of information, particularly for pushing. If the given title matches the title of a file
in the current directory (either based on the filename or the front-matter of the file),
and that file has a resource specified, then the appropriate service can automatically be inferred.foo@bar:~$waltzpush"Turtles"If the desired files are not in the present folder, it will search subfolders.TOOD: We are working on ways to further disambiguate resources that have
the same name. This will be accomplished using their IDs or some other attribute.
In general, however, you can avoid ambiguity by organizing the file system
and your LMS resources appropriately.Diffing ResourcesPushing and pulling resources can get quite complex, so it can be
helpful to find out how the local version of a resource is different
from the remote. Thediffcommand has similar syntax to the other
actions, but does not modify the filesystem when it is run.foo@bar:~$waltzdiffcanvaspage"Turtles"By default,diffing will create a new HTML file adjacent to the
resource and automatically open that file in the browser. There is also
simplified console output available.Download, Decode, Encode, and UploadPulling is actually a two-step process: first youdownloada resource, and then youdecodeit.
Similarly,Pushing requires you toencodea resource and thenuploadit.Theuploadanddownloadcommands (which are available in Waltz) move data between Canvas and the
local Waltz SQLite database. Generally, we download raw data from the remote service (e.g., a JSON blob)
and upload using whatever API is available for the service.Theencodeanddecodecommands (also available) move data between the local filesystem and the
local Waltz SQLite database. Generally, this means we are decoding the raw data into a Markdown file
or potentially even multiple files.TemplatesTODO: This feature is still in progressThe HTML body of a Resource can be built using a Template. We use the
Jinja2 format to embed data.Previewing ResourcesTODO: This feature is still in progressIf you want to generate a web-friendly local version of a resource, you
can use thepreviewcommand.Building Public VersionsTODO: This feature is still in progressFrequently, you want to create public versions of assignments that are
viewable by folks who are not authorized to see the answers or other
components of a resource. We distinguish betweenpublic(viewable by
anyone),private(viewable by other teachers), andsecret(viewable
only by local course staff).You can use the--hide_answersflag to remove answers and other
private information from a resource, generating a new file in a parallel
repository.Secret information usually includes data covered by FERPA, such as
students' grades or assignment submissions. Secret information is harder
to secure in a VCS, but can be done through encryption. In general,
however, data that is meant to be secret should be kept in a different
repository. If you want it alongside the data, though, perhaps we could
encrypt/decrypt the file, suspending git commands until all relevant
files are re-encrypted?TerminologyResource: A curricular material to be transferred around. Resources can be represented in different styles. Some resources can be composed of other resources.Category: Resources of the same type. The "Pages" category is distinct from the "Assignment" category. Categories do not belong to a particular Service, meaning their names must be meaningfully distinct if they are different kinds of things.Title: A name that, idealistically, is globally unique within a resource.Markdown: A plain-text format that we use to store files locally. Meant to be easily edited. Can have Front-matter.Front-matter: At the top of Markdown files, you can put "---" above and below YAML-encoded data to store extra data. We particularly look for awaltzattribute with additional Waltz information.Service Specific InformationThis section details information specific to individual services.LocalWaltz is not opinionated about file system organization. You are free to
arrange your data into directories as you see fit. By default, new files
are stored in the current working directory. However, existing files
will be matched as best they can.Largely, Waltz encourages resources to decode themselves as Markdown
files. In particular, Markdown files with front-matter can be parsed and
have special Waltz-specific information. The only requirement is to have
an attribute namedwaltz. We also strongly encourage atitleandresourceattribute within.---
waltz:
  title: The Resource's Full Proper Nameresource: Page---The body of the fileAdditional attributes can be stored in the Markdown, and services
attempt to leave them unmangled when downloading updated versions.CanvasInstructure's Canvas has a rich API that allows us to control resources
relatively well.Code in Markdown files will be syntax-highlighted.Uploading data into Canvas is somewhat limited. To get around the
limitations, we store additional front-matter as HTML content hidden
inside of an invisible DIV tag at the start of the resource's body. If
that information is deleted on the server, however, there isn't much to
be done to reclaim it. Be careful when deleting large swathes of data if
you are relying on this feature! Also note that we do not currently
encode any extrawaltzfront-matter, so you should be sure not to rely
on extra data stored in that attribute.You can use the--allswitch to bulk download resources.You can use the--destination <path>parameter to set the location
where files will be decoded to.Canvas PagesA Canvas Page is decoded into a Markdown file with very minimal
front-matter.Canvas AssignmentsA Canvas Page is decoded into a Markdown file with very minimal
front-matter.Canvas QuizzesBy default, quizzes are pulled as multiple files. The core file will
hold the quiz itself, with the questions stored as a list of their
names. Those questions' content will be then be stored as separate files
in an adjacent folder named"<Quiz Title> Questions/". All question
types are supported.Groups are also supported, and will be represented inside of the core
Markdown file alongside other questions. However, due to a limitation of
the Canvas API, we are not able to keep the order of grouped questions -
further, we cannot guarantee that question groups will be properly
disposed of. When using quizzes, keep an eye on the Canvas side to make
sure that the content is appropriately transferred.A Canvas Quiz can be pulled as a combined single file.foo@bar:~$waltzpullcanvasquiz"My Quiz"--combineYou can specify a question bank to reuse questions between quizzes.Canvas FilesTODO: This feature is still in progressYou can have either a folder dedicated to being a file system, or
identify an individual file with an adjacent Markdown file.Linking ResourcesTODO: This feature is still in progressWe attempt to decode URLs to images, files, and other resources as
intelligently as possible so that they make sense locally and remotely.Full Command Listfoo@bar:~$waltzresetfoo@bar:~$waltzinitfoo@bar:~$waltzconfigure<service><name>foo@bar:~$waltzlist<service><category>foo@bar:~$waltzpull<service><category><title>foo@bar:~$waltzpush<service><category><title>foo@bar:~$waltzdownload<service><category><title>foo@bar:~$waltzupload<service><category><title>foo@bar:~$waltzencode<service><category><title>foo@bar:~$waltzdecode<service><category><title>foo@bar:~$waltzdiff<service><category><title>