Package: tutor-contrib-aspects
Aspects Learner Analytics combines several free, open source, tools to add analytics and reporting capabilities to the Open edX platform. This plugin offers easy installation, configuration, and deployment of these tools usingTutor. The tools Aspects uses are:ClickHouse, a fast, scalable analytics database that can be run anywhereApache Superset, a data visualization platform and data APIOpenFUN Ralph, a Learning Record store (and more) that can validate and store xAPI statements in ClickHouseVector, a log forwarding tool that can be used to forward tracking log and xAPI data to ClickHouseevent-routing-backends, an Open edX plugin that transforms tracking logs into xAPI and optionally forwards them to one or more Learning Record Stores in near real timeevent-sink-clickhouse, an Open edX plugin that exports course structure and high level data to ClickHouse at publish timedbt, a tool to build data pipelines from SQL queries. The dbt project used by this plugin isaspects-dbt.Seehttps://github.com/openedx/openedx-aspectsfor more details about the Aspects architecture and high level documentation.Aspects is a community developed effort combining the Cairn project by Overhang.io and the OARS project by EduNEXT, OpenCraft, and Axim Collaborative.Note: Aspects is beta and not yet production ready! Please feel free to experiment with the system and offer feedback about what you’d like to see by adding Issues in this repository. Current details on the beta progress can be found here:https://openedx.atlassian.net/wiki/spaces/COMM/pages/3861512203/Aspects+BetaCompatibilityThis plugin is compatible with Tutor 15.0.0 and later and is expected to be compatible with Open edX releases from Nutmeg forward.InstallationAspects is implemented as a Tutor plugin. Documentation will be coming soon to cover how to install Aspects in non-Tutor environments, but by far the easiest way to try and install it is via Tutor. These instructions assume you are running atutor localinstall, which is the fastest and easiest way to get started.Install Tutor:https://docs.tutor.overhang.io/install.html#installCreate an admin user on the LMS:https://docs.tutor.overhang.io/whatnext.html#logging-in-as-administratorInstall the Aspects plugin (in your Tutor Python environment):pip install tutor-contrib-aspectsEnable the plugins:tutor plugins enable aspectsSave the changes to the environment:tutor config saveBecause we’re installing new applications in LMS (event-routing-backends, event-sink-clickhouse) you will need to rebuild your openedx image:tutor images build openedx --no-cacheRun the initialization scripts:tutor local do initAt this point you should have a working Tutor / Aspects environment, but with no way to create data! There are a few options for how to proceed.If you would just like to see some data populated in the charts without loading a real course in the LMS you can create test data in the database (use--helpfor usage):tutor local do load-xapi-test-dataOR Load the test course and generate real data from the LMS:https://docs.tutor.overhang.io/whatnext.html#importing-a-demo-courseLog into the LMS with your admin user and enroll / proceed through the demo courseOR If you are adding Aspects to an existing LMS that already has dataSink course data from the LMS to clickhouse (seehttps://github.com/openedx/openedx-event-sink-clickhousefor more information):tutor local do dump-data-to-clickhouse --options "--object course_overviews"Sink Historical event data to ClickHouse:tutor [dev|local] do transform_tracking_logs \
  --source_provider LOCAL --source_config '{"key": "/openedx/data", "container":
     "logs", "prefix": "tracking.log"}' \
  --transformer_type xapi

# Note that this will work only for default tutor installation. If you store your tracking logs any other way, you need to change the source_config option accordingly.
# See https://event-routing-backends.readthedocs.io/en/latest/howto/how_to_bulk_transform.html#sources-and-destinations for details on how to change the source_config option.If your assets have changed since the last time you ran init, you will need to rebuild the aspects-superset image and re-import the assets:tutor images build aspects-superset --no-cache
tutor local do import-assetsYou should now have data to look at in Superset! Log in tohttps://superset.local.overhang.io/with your admin account and you should see charts with your data.Superset AssetsAspects maintains the Superset assets in this repository, specifically the dashboards,
charts, datasets, and databases. That means that any updates made here will be reflected
on your Superset instance when you update your deployment.But it also means that any local changes you make to these assets will be overwritten
when you update your deployment. To prevent your local changes from being overwritten,
please create new assets and make your changes there instead. You can copy an existing
asset by editing the asset in Superset and selecting “Save As” to save it to a new name.# Note: If you are using custom assets you will need to rebuild your aspects-superset
# image on your local machine withtutor images build aspects-superset –no-cache.Sharing Charts and DashboardsTo share your charts with others in the community, use Superset’s “Export” button to
save a zip file of your charts and related datasets.WarningThe exported datasets will contain hard-coded references to your particular
databases, including your database hostname, port, and username, in some cases
it may also contain database passwords. It is vital that you review the
database and dataset files before sharing them.To import charts or dashboards shared by someone in the community:Expand the zip file and look for any files added underdatabases.
Update thesqlalchemy_urito match your database’s connection details.Compress the files back into a.zipfile.On the Charts or Dashboards page, use the “Import” button to upload your.zipfile.Contributing Charts and Dashboards to AspectsThe Superset assets provided by Aspects can be found in the templatedtutoraspects/templates/aspects/build/aspects-superset/openedx-assets/assets/directory. For the most part,
these files are what Superset exports, but with some crucial differences
which make these assets usable across all Tutor deployments.To contribute assets to Aspects:Fork this repository and have a locally running Tutor set up with this plugin
installed.Export the assets you want to contribute as described inSharing Charts and DashboardsRun the command:tutor aspects import_superset_zip ~/Downloads/your_file.zipThis command will copy the files from your zip to the assets directory and
attempt to warn you if there are hard coded connection settings where it expects
template variables. These are usually in database and dataset assets, and those are
often assets that already exist. The warnings look like:WARN: fact_enrollments.yaml has schema set to reporting instead of a setting.Check the diff of files and update any database connection strings or table names
to use Tutor configuration template variables instead of hard-coded strings, e.g.
replaceclickhousewith{{CLICKHOUSE_HOST}}. Passwords can be left as{{CLICKHOUSE_PASSWORD}}, though be aware that if you are adding new
databases, you’ll need to updateSUPERSET_DB_PASSWORDSin the init scripts.
Here is the default connection string for reference:``clickhousedb+connect://{{CLICKHOUSE_REPORT_URL}}``You will likely also run into issues where our SQL templates have been expanded into
their actual SQL. If you haven’t changed the SQL of these queries (stored intutoraspects/templates/openedx-assets/queriesyou can just revert that change back
to theirincludevalues such as:sql: “{% include ‘openedx-assets/queries/fact_enrollments_by_day.sql’ %}”The script will also warn about missing_rolesin dashboards. Superset does not export
these, so you will need to manually add this key with the roles that are necessary to
view the dashboard. See the existing dashboards for how this is done.Re-build youraspects-supersetimage withtutor images build aspects-superset –no-cacheRun the commandtutor aspects check_superset_assetsto confirm there are no
duplicate assets, which can happen when you rename an asset, and will cause import
to fail. The command will automatically delete the older file if it finds a duplicate.Check that everything imports correctly by runningtutor local do import-assetsand confirming there are no errors.Double check that your database password did not get exported before committing!Commit and submit a PR with screenshots of your new chart or dashboards, along with an
explanation of what data question they answer.Virtual datasets in SupersetSuperset supports creating virtual datasets, which are datasets defined using a SQL query instead of mapping directly to an underlying database object. Aspects leverages virtual datasets, along withSQL templating, to make better use of table indexes.To make it easier for developers to manage virtual datasets, there is an extra step that can be done on the output oftutor aspects serialize. Thesqlsection of the dataset yaml can be moved to its own file in thequeriesdirectory and included in the yaml like so:sql:"{%include'openedx-assets/queries/query.sql'%}"However, please keep in mind that the assets declaration is itself a jinja template. That means that any jinja used in the dataset definition should be escaped. There are examples of how to handle this in the existing queries, such asdim_courses.sql.LicenseThis software is licensed under the terms of the AGPLv3.