Package: redash-stmo
Redashextensions forsql.telemetry.mozilla.org.Or as it should have been called:St. Moredash;)OverviewInherits Redash’s Docker setupredash-stmo is using Redash’s own Docker image for development to implement a
close development/production parity and extends it in various ways, e.g.
an own docker-compose configuration, an own docker-entrypoint script.Specifically it uses Mozilla’s “rc” tagged version of the Redash Docker
image, which includes (at the time of writing this, 2019-06-13) many
customizations from Mozilla’s pseudo-temporary and regularly updated Redash
fork. The “rc” tagged Docker image is updated every time a “rebase” from
upstream Redash happens and is put to testing in the “release” Redash
environment on Mozilla’s server.Please review theRedash Docker installation guidelinesbefore continuing. It’s important to know
those basics since many decisions for redash-stmo were derived from it.
Thank you.Is mounted under /extensionThe current working directory (the directory with thisREADME.rst) is
mounted under the path/extensionby docker-compose inside the Docker
container.Runs with Redash in /appSince it reuses the Redash Docker image, you can find all the Redash setup
under the/appdirectory inside the Docker container.Uses Redash’s “entrypoints” for discoveryThe way Redash finds new extensions is by using the so called “entrypoints”
of Python packages, metadata that is specified and distributed in
Python packages, that is read out by Redash at runtime to find the filesystem
locations for Redash extensions.That’s true for three kinds of entrypoints:redash.extensionsPython callables to be used to extend the Redash Flask app, e.g.redash_stmo.data_sources.health:extension.redash.bundlesPython packages that contain additional front-end files for the
webpack build process, e.g.redash_stmo.data_sources.link.redash.scheduled_jobsPython callables that return parameters for scheduled RQ jobs,
e.g.redash_stmo.data_sources.health:scheduled_job.Hooks into WebpackSince Redash extensions like redash-stmo can also provide additional Webpack
bundles, the development setup runs Redash’sbundle-extension scriptperiodically
to copy the files from redash-stmo to the right place for webpack to pick
them up (/app/client/app/extensions).See the section about the webpack development server below for more
information.Development workflowWe provide some convenience Make tasks to be run from your host machine
(not inside the Docker container) to ease this non-trivial application setup:Build and update the local Docker imageMany of the helpers below will implicitely run docker-compose to start the
Redash containers and in effect automatically build the local Docker image
as well if it doesn’t exist.If you’d like to build the local Docker image separately or if you’d like to
fetch the latest version of the base Redash Docker image (or its child
image of the Mozilla Redash fork), e.g. in the event of a new rebase by
Mozilla staff, please run the following:make buildBehind the scenesThis will rundocker-composebuild--pullwhich will pull updates to
the Docker images used by the docker-compose setup, including the Redash,
Redis and Postgres images.Create the databaseOn you command line run thisONCEto create the database for
Redash/redash-stmo setup:make databaseThis uses Redash’s own ability and redash-stmo is just set up to reuse it.Behind the scenesThis will run docker-compose to create the server container that is
running the Redash Python server and in effect the Redis and Postgres
containers, too.It will then initialize the Postgres tables needed for Redash.Install npm modulesThen we’ll install the Redash npm modules inside the server container:make node_modulesBehind the scenesThis will runnpm installinside the server in the/appdirectory,
which is the directory with Redash’s code from the Redash Docker base
image.NOTE, the redash-stmo development setup mounts the/app/node_modulesdirectory as a separate Docker volume, that will be maintained by Docker
and won’t show up in or transfer to the host machine where Docker is
running.Start the containersTo start the whole set of Docker containers for a working environment
(Redash server, RQ workers, Redis, Postgres) all you need to run is this:make upBehind the scenesThis is pretty simply runningdocker-composeup, to launch all
containers of the redash-stmo Docker setup.NOTE: Thisrequiresfirst installing npm modules inside the container
above and creating the database as well!Run webpack devserverIf you’re developing a Redash extension that includes an additional webpack
bundle (which will need to be included in Redash’s webpack build process
to be shipped in the client application bundle) you’ll want to use the webpack
development server.It automatically compiles the Redash client application bundle on files
changes and proxies requests for the Redash server via a proxy running
on port 8080 (instead of the usual Redash port of 5000).After starting the containers using the description in the above step,
open a second terminal andadditionally run:make devserverBehind the scenesThis will run the webpack devserver in another instance of the server
container (not the same as when runningmake up) and runs a script
that listens for files changes to.jsand.jsxfiles in the/extensiondirectory.When changes are detected, it’ll automatically run Redash’sbundle-extensionsscript that does the heavy lifting of copying
the changed extension files into the/app/client/app/extensionsdirectory, which triggers the webpack devserver to recompile the
client application bundle.NOTE: Thisrequiresopening the Redash instance viahttp://localhost:8080/instead ofhttp://localhost:5000/to go through
the webpack devserver.Start shellIn case you need to do any debugging or file system checks inside the
server container, you can create a bash shell by running:make bashBehind the scenesAny changes you make here outside the/extensiondirectory
(which is mounted as a Docker volume with the current working directory on
the Docker host machine) and the following directores are not persisted.List of directories inside the container that are mounted as Docker volumes:/extensionMaps the current working directory (where this README.rst is located)
on the host machine for developing the extension./home/redash/.cacheUsed by pip and other scripts,/app/client/distDirectory to retain webpack build results, so webpack builds don’t take
as long on consecutive runs./home/redash/.localDirectory for “user-installed” Python packages. If you’d like you can
easily install additonal Python packages with the Docker container user
Redash usingpip install--user<package>. Installed scripts from
those packages will be found under/home/redash/.local/binbut
are also automatically added toPATH./app/node_modulesDirectory for npm modules, that are installed when runningnpm installinside of/appin the container. Retained to make use of native npm
caching between consecutive runs.Run testsRunning the Python based tests requires first creating a separate database
(implemented by thetest_databaseMake task) and then running the test
runner inside the container. The test database is not the same as the
databse in use for regular development (e.g. to not overwrite development
data).Frontend or integration tests are currently not supported.To run the tests (from the host machine) run:make testThis will automatically run thetest_databaseMake task before running
the tests.Behind the scenesWhen launching the tests runner it’ll the regular server container,
but also set theREDASH_DATABASE_URLenvironment variable to the
test database to prevent overwriting any data that you added to the
database the regular Redash interface (e.g. data sources, queries etc).By default it usespytestto run
the Python tests in/extension, with a number of parameters as
defined in thepytest.ini.If you’d like to add additional parameters to pytest simply appened the
command line arguments inpytest.ini.Alternatively, e.g. if you’d like to usepdbto debug a test, do this:create the test database from the host machinemake test_databasestart a Bash shell in the containermake bashset theREDASH_DATABASE_URLenv var in the containerexportREDASH_DATABASE_URL="postgresql://postgres@postgres/tests"change direcotry to extensio codecd /extensionrun the tests with whatever parameterpytest-vvv--pdbIssues & questionsSee theissue tracker on GitHubto open tickets if you have issues or questions about Redash-STMO.