Package: commcare-export
CommCare Exporthttps://github.com/dimagi/commcare-exportA command-line tool (and Python library) to generate customized exports from theCommCare HQREST API.User documentationChangelogInstallation & Quick StartFollowing commands are to be run on a terminal or a command line.Once on a terminal window or command line, for simplicity, run commands from the home directory.PythonCheck for python installed.
This tool istested with Python 3.6, 3.7, and 3.8.$python--version
$python3--versionIf python is installed, all of its available versions would be listed.If python isn't installed, Install Python 3.8 fromthis link.Virtualenv (Optional)It is recommended to set up a virtual environment for CommCare Export
to avoid conflicts with other python applications.More about virtualenvs onhttps://docs.python.org/3/tutorial/venv.htmlSetup a virtual environment using:$python3.8-mvenv.commcare-export-venv# update python version for the one installedActivate virtual environment by running:source.commcare-export-venv/bin/activateNote: virtualenv needs to be activated each time you start a new terminal session or command line prompt.For convenience, to avoid doing that, you can create an alias to activate virtual environments in
".commcare-export-venv" directory by adding the following to your.bashrcor.zshrcfile:aliascommcare-export-venv='if [[ -d .commcare-export-venv ]] ; then source .commcare-export-venv/bin/activate ; fi'Then you can activate virtual environments with simply typing$commcare-export-venvInstall CommCare ExportInstall CommCare Export viapip$ python3 -m pip install wheel
$ python3 -m pip install commcare-exportCommCareHQSign up forCommCare HQif you have not already.Create a project space and application.Visit the Release Manager, make a build, click the star to release it.Use Web Apps and fill out some forms.Modify one of example queries in theexamples/directory, modifying the "Filter Value" column
to match your form XMLNS / case type.
Seethis pageto
determine the XMLNS for your form.Now you can run the following examples:$ commcare-export \
     --query examples/demo-registration.xlsx \
     --project YOUR_PROJECT \
     --output-format markdown

$ commcare-export \
     --query examples/demo-registration.json \
     --project YOUR_PROJECT \
     --output-format markdown

$ commcare-export \
     --query examples/demo-deliveries.xlsx \
     --project YOUR_PROJECT \
     --output-format markdown

$ commcare-export \
     --query examples/demo-deliveries.json \
     --project YOUR_PROJECT \
     --output-format markdownYou'll see the tables printed out. Change to--output-format sql --output URL_TO_YOUR_DB --since DATEto
sync all forms submitted since that date.Example query files are provided in both Excel and JSON format.  It is recommended
to use the Excel format as the JSON format may change upon future library releases.Command-line UsageThe basic usage of the command-line tool is with a saved Excel or JSON query (see how to write these, below)$ commcare-export --commcare-hq <URL or alias like "local" or "prod"> \
                  --username <username> \
                  --project <project> \
                  --api-version <api version, defaults to latest known> \
                  --version <print current version> \
                  --query <excel file, json file, or raw json> \
                  --output-format <csv, xls, xlsx, json, markdown, sql> \
                  --output <file name or SQL database URL> \
                  --users <export data about project's mobile workers> \
                  --locations <export data about project's location hierarchy> \
                  --with-organization <export users, locations and joinable form or case tables>Seecommcare-export --helpfor the full list of options.There are example query files for the CommCare Demo App (available on the CommCare HQ Exchange) in theexamples/directory.--outputCommCare Export uses SQLAlachemy'screate_engineto establish a database connection. This is based off of theRFC-1738protocol. Some common examples:# Postgres
postgresql+psycopg2://scott:tiger@localhost/mydatabase

# MySQL
mysql+pymysql://scott:tiger@localhost/mydatabase

# MSSQL
mssql+pyodbc://scott:tiger@localhost/mydatabases?driver=ODBC+Driver+17+for+SQL+ServerExcel QueriesAn Excel query is any.xlsxworkbook. Each sheet in the workbook represents one table you wish
to create. There are two grouping of columns to configure the table:Data Source: Set this toformto export form data, orcasefor case data.Filter Name/Filter Value: These columns are paired up to filter the input cases or forms.Field: The destination in your SQL database for the value.Source Field: The particular field from the form you wish to extract. This can be any JSON path.JSON QueriesJSON queries are a described in the table below. You build a JSON object that represents the query you have in mind.
A good way to get started is to work from the examples, or you could make an Excel query and run the tool
with--dump-queryto see the resulting JSON query.User and Location DataThe --users and --locations options export data from a CommCare project that
can be joined with form and case data. The --with-organization option does all
of that and adds a field to Excel query specifications to be joined on.Specifying the --users option or --with-organization option will export an
additional table named 'commcare_users' containing the following columns:ColumnTypeNoteidTextPrimary keydefault_phone_numberTextemailTextfirst_nameTextgroupsTextlast_nameTextphone_numbersTextresource_uriTextcommcare_location_idTextForeign key into the commcare_locations tablecommcare_location_idsTextcommcare_primary_case_sharing_idTextcommcare_projectTextusernameTextThe data in the 'commcare_users' table comes from theList Mobile Workers
API endpoint.Specifying the --locations option or --with-organization options will export
an additional table named 'commcare_locations' containing the following columns:ColumnTypeNoteidTextcreated_atDatedomainTextexternal_idTextlast_modifiedDatelatitudeTextlocation_dataTextlocation_idTextPrimary keylocation_typeTextlongitudeTextnameTextparentTextResource URI of parent locationresource_uriTextsite_codeTextlocation_type_administrativeTextlocation_type_codeTextlocation_type_nameTextlocation_type_parentTextlocation level codeTextColumn name depends on project's organizationlocation level codeTextColumn name depends on project's organizationThe data in the 'commcare_locations' table comes from the Location API
endpoint along with some additional columns from the Location Type API
endpoint. The last columns in the table exist if you have set up
organization levels for your projects. One column is created for each
organization level. The column name is derived from the Location Type
that you specified. The column value is the location_id of the containing
location at that level of your organization. Consider the example organization
from theCommCare help page.
A piece of the 'commcare_locations' table could look like this:location_idlocation_type_namechwsupervisorclinicdistrict939fa8DistrictNULLNULLNULL939fa8c4cbefClinicNULLNULLc4cbef939fa8a9ca40SupervisorNULLa9ca40c4cbef939fa84545b9CHW4545b9a9ca40c4cbef939fa8In order to join form or case data to 'commcare_users' and 'commcare_locations'
the exported forms and cases need to contain a field identifying which user
submitted them. The --with-organization option automatically adds a field
called 'commcare_userid' to each query in an Excel specification for this
purpose. Using that field, you can use a SQL query with a join to report
data about any level of you organization. For example, to count the number
of forms submitted by all workers in each clinic:SELECTl.clinic,COUNT(*)FROMform_tabletLEFTJOIN(commcare_usersuLEFTJOINcommcare_locationslONu.commcare_location_id=l.location_id)ONt.commcare_userid=u.idGROUPBYl.clinic;Note that the table names 'commcare_users' and 'commcare_locations' are
treated as reserved names and the export tool will produce an error if
given a query specification that writes to either of them.The export tool will write all users to 'commcare_users' and all locations to
'commcare_locations', overwriting existing rows with current data and adding
rows for new users and locations. If you want to remove obsolete users or
locations from your tables, drop them and the next export will leave only
the current ones. If you modify your organization to add or delete levels,
you will change the columns of the 'commcare_locations' table and it is
very likely you will want to drop the table before exporting with the new
organization.Python Library UsageAs a library, the variouscommcare_exportmodules make it easy toInteract with the CommCare HQ REST APIExecute "Minilinq" queries against the API (a very simple query language, described below)Load and save JSON representations of Minilinq queriesCompile Excel configurations to Minilinq queriesTo directly access the CommCare HQ REST API:fromcommcare_export.checkpointimportCheckpointManagerWithDetailsfromcommcare_export.commcare_hq_clientimportCommCareHqClient,AUTH_MODE_APIKEYfromcommcare_export.commcare_minilinqimportget_paginator,PaginationModeusername='some@username.com'domain='your-awesome-domain'hq_host='https://commcarehq.org'API_KEY='your_secret_api_key'api_client=CommCareHqClient(hq_host,domain,username,API_KEY,AUTH_MODE_APIKEY)case_paginator=get_paginator(resource='case',pagination_mode=PaginationMode.date_modified)case_paginator.init()checkpoint_manager=CheckpointManagerWithDetails(None,None,PaginationMode.date_modified)cases=api_client.iterate('case',case_paginator,checkpoint_manager=checkpoint_manager)forcaseincases:print(case['case_id'])To issue aminilinqquery against it, and then print out that query in a JSON serialization:importgetpassimportjsonfromcommcare_export.minilinqimport*fromcommcare_export.commcare_hq_clientimportCommCareHqClientfromcommcare_export.commcare_minilinqimportCommCareHqEnvfromcommcare_export.envimportBuiltInEnvapi_client=CommCareHqClient(url="http://www.commcarehq.org",project='your_project',username='your_username',password='password',version='0.5')source=Map(source=Apply(Reference("api_data"),Literal("form"),Literal({"filter":{"term":{"app_id":"whatever"}}})),body=List([Reference("received_on"),Reference("form.gender"),]))query=Emit('demo-table',[Literal('Received On'),Literal('Gender')],source)printjson.dumps(query.to_jvalue(),indent=2)results=query.eval(BuiltInEnv()|CommCareHqEnv(api_client)|JsonPathEnv())iflen(list(env.emitted_tables()))>0:# with writers.Excel2007TableWriter("excel-output.xlsx") as writer:withwriters.StreamingMarkdownTableWriter(sys.stdout)aswriter:fortableinenv.emitted_tables():writer.write_table(table)Which will output JSON equivalent to this:{"Emit":{"headings":[{"Lit":"Received On"},{"Lit":"Gender"}],"source":{"Map":{"body":{"List":[{"Ref":"received_on"},{"Ref":"form.gender"}]},"name":None,"source":{"Apply":{"args":[{"Lit":"form"},{"Lit":{"filter":{"term":{"app_id":"whatever"}}}}],"fn":{"Ref":"api_data"}}}}},"table":"demo-table"}}MiniLinq ReferenceThe abstract syntax can be directly inspected in thecommcare_export.minilinqmodule. Note that the choice between functions and primitives is deliberately chosen
to expose the structure of the MiniLinq for possible optimization, and to restrict the overall language.Here is a description of the astract syntax and semanticsPythonJSONWhich is evaluates toLiteral(v){"Lit": v}JustvReference(x){"Ref": x}Whateverxresolves to in the environmentList([a, b, c, ...]){"List": [a, b, c, ...}The list of whata,b,cevaluate toMap(source, name, body){"Map": {"source": ..., "name": ..., "body": ...}Evalsbodyfor each elem insource. Ifnameis provided, the elem will be bound to it, otherwise it will replace the whole env.FlatMap(source, name, body){"FlatMap": {"source" ... etc}}Flattens after mapping, like nested list comprehensionsFilter(source, name, body)etcBind(value, name, body)etcBinds the result ofvaluetonamewhen evaluatingbodyEmit(table, headings, rows)etcEmitstablewithheadingsandrows. Note thattableis a string,headingsis a list of expressions, androwsis a list of lists of expressions. See explanation below for emitted output.Apply(fn, args)etcEvaluatesfnto a function, and all ofargs, then applies the function to the args.Built in functions likeapi_dataand basic arithmetic and comparison are provided via the environment,
referred to be name usingRef, and utilized viaApply.List of builtin functions:FunctionDescriptionExample Usage+, -, *, //, /, >, <, >=, <=Standard MathlenLengthboolBoolstr2boolConvert string to boolean. True values are 'true', 't', '1' (case insensitive)str2dateConvert string to datebool2intConvert boolean to integer (0, 1)str2numParse string as a numberformat-uuidParse a hex UUID, and format it into hyphen-separated groupssubstrReturns substring indexed by [first arg, second arg), zero-indexed.substr(2, 5) of 'abcdef' = 'cde'selected-atReturns the Nth word in a string. N is zero-indexed.selected-at(3) - return 4th wordselectedReturns True if the given word is in the value.selected(fever)count-selectedCount the number of wordsjson2strConvert a JSON object to a stringtemplateRender a string template (not robust)template({} on {}, state, date)attachment_urlConvert an attachment name into it's download URLform_urlOutput the URL to the form view on CommCare HQcase_urlOutput the URL to the case view on CommCare HQuniqueOuptut only unique values in a listOutput FormatsYour MiniLinq may define multiple tables with headings in addition to their body rows by usingEmitexpressions, or may simply return the results of a single query.If your MiniLinq does not contain anyEmitexpressions, then the results of the expression will be
printed to standard output as pretty-printed JSON.If your MiniLinqdoescontainEmitexpressions, then there are many formats available, selected
via the--output-format <format>option, and it can be directed to a file with the--output <file>command-line option.csv: Each table will be a CSV file within a Zip archive.xls: Each table will be a sheet in an old-format Excel spreadsheet.xlsx: Each table will be a sheet in a new-format Excel spreadsheet.json: The tables will each be a member of a JSON dictionary, printed to standard outputmarkdown: The tables will be streamed to standard output in Markdown format (very handy for debugging your queries)sql: All data will be idempotently "upserted" into the SQL database you specify, including creating the needed tables and columns.DependenciesRequired dependencies will be automatically installed via pip. But since
you may not care about all export formats, the various dependencies there
are optional. Here is how you might install them:# To export "xlsx"
$ pip install "commcare-export[xlsx]"

# To export "xls"
$ pip install "commcare-export[xls]"

# To sync with a Postgres database
$ pip install "commcare-export[postgres]"

# To sync with a mysql database
$ pip install "commcare-export[mysql]"

# To sync with a database which uses odbc (e.g. mssql)
$ pip install "commcare-export[odbc]"

# To sync with another SQL database supported by SQLAlchemy
$ pip install "commcare-export[base_sql]"
# Then install the python package for your databaseContributing0. Sign up for GitHub, if you have not already, athttps://github.com.1. Fork the repository athttps://github.com/dimagi/commcare-export.2. Clone your fork, install into avirtualenv, and start a feature branch$ mkvirtualenv commcare-export
$ git clone git@github.com:dimagi/commcare-export.git
$ cd commcare-export
$ pip install -e ".[test]"
$ git checkout -b my-super-duper-feature3. Make your edits.4. Make sure the tests pass. The best way to test for all versions is to sign up forhttps://travis-ci.organd turn on automatic continuous testing for your fork.$ py.test
=============== test session starts ===============
platform darwin -- Python 2.7.3 -- pytest-2.3.4
collected 17 items

tests/test_commcare_minilinq.py .
tests/test_excel_query.py ....
tests/test_minilinq.py ........
tests/test_repeatable_iterator.py .
tests/test_writers.py ...

============ 17 passed in 2.09 seconds ============5. Type hints are used in theenvandminilinqmodules. Check that any changes in those modules adhere to those types:$ mypy --install-types @mypy_typed_modules.txt6. Push the feature branch up$ git push -u origin my-super-duper-feature7. Visithttps://github.com/dimagi/commcare-exportand submit a pull request.8. Accept our gratitude for contributing: Thanks!Release process1. Create a tag for the release$ git tag -a "X.YY.0" -m "Release X.YY.0"
$ git push --tags2. Create the source distribution$ python setup.py sdistEnsure that the archive (dist/commcare-export-X.YY.0.tar.gz) has the correct version number (matching the tag name).3. Upload to pypi$ pip install twine
$ twine upload -u dimagi dist/commcare-export-X.YY.0.tar.gz4. Verify uploadhttps://pypi.python.org/pypi/commcare-export5. Create a release on githubhttps://github.com/dimagi/commcare-export/releasesTesting and Test DatabasesThe following command will run the entire test suite (requires DB environment variables to be set as per below):$ py.testTo run an individual test class or method you can run, e.g.:$ py.test -k "TestExcelQuery"
$ py.test -k "test_get_queries_from_excel"To exclude the database tests you can run:$ py.test -m "not dbtest"When running database tests, supported databases are PostgreSQL, MySQL, MSSQL.To run tests against selected databases can be done using test marks as follows:py.test -m [postgres,mysql,mssql]Database URLs can be overridden via environment variables:POSTGRES_URL=postgresql://user:password@host/
MYSQL_URL=mysql+pymysql://user:password@host/
MSSQL_URL=mssql+pyodbc://user:password@host/Postgresql$ docker pull postgres:9.6
$ docker run --name ccexport-postgres -p 5432:5432 -e POSTGRES_PASSWORD=postgres -d postgres:9.6
$ export POSTGRES_URL=postgresql://postgres:postgres@localhost/Docker postgres image docsMySQL$ docker pull mysql
$ docker run --name ccexport-mysql -p 3306:3306 -e MYSQL_ROOT_PASSWORD=pw -e MYSQL_USER=travis -e MYSQL_PASSWORD='' -d mysql

# create travis user
$ docker run -it --link ccexport-mysql:mysql --rm mysql sh -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
mysql> CREATE USER 'travis'@'%';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'travis'@'%';MSSQL$ docker pull mcr.microsoft.com/mssql/server:2017-latest 
$ docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Password@123" -p 1433:1433 --name mssql1 -d microsoft/mssql-server-linux:2017-latest

# install driver
$ curl https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add -
$ echo "deb [arch=amd64] https://packages.microsoft.com/ubuntu/$(lsb_release -rs)/prod $(lsb_release -rs) main" | sudo tee /etc/apt/sources.list.d/mssql-release.list

$ sudo apt-get update
$ sudo ACCEPT_EULA=Y apt-get install msodbcsql17
$ odbcinst -q -dMSSQL for Mac OS$ docker pull mcr.microsoft.com/mssql/server:2017-latest 
$ docker run -e "ACCEPT_EULA=Y" -e "MSSQL_SA_PASSWORD=Password@123" -p 1433:1433 --name mssql1 -d microsoft/mssql-server-linux:2017-latest

# Install driver
$ brew install unixodbc freetds

# Add the following 5 lines to /usr/local/etc/odbcinst.ini
[ODBC Driver 17 for SQL Server]
Description=FreeTDS Driver for Linux & MSSQL
Driver=/usr/local/lib/libtdsodbc.so
Setup=/usr/local/lib/libtdsodbc.so
UsageCount=1

# Create a soft link from /etc/odbcinst.ini to actual file
sudo ln -s /usr/local/etc/odbcinst.ini /etc/odbcinst.iniIntegration TestsRunning the integration tests requires API credentials from CommCare HQ
that have access to thecorporadomain. This user should only have
access to the corpora domain.These need to be set as environment variables as follows:export HQ_USERNAME=<username>
export HQ_API_KEY=<apikey>For Travis builds these are included as encrypted vars in the travis
config.