Package: earthmover
earthmovertransforms collections of tabular source data (flat files, FTP files, database tables/queries) into text-based (JSONL, XML) data via YAML configuration.Table of ContentsRequirementsInstallationSetupUsageFeaturesTestsDesignPerformanceLimitationsBest PracticesChangelogContributingLicenseBundlesRequirementsPython 3andpip.Installationpip install earthmoverSetupRunning the tool requiressource data, such as CSV or TSV files or a relational database tableJinjatemplatesdefining the desired output format (JSON, XML, etc.)aYAML configurationfile specifying the source data, doing any necessary transformations (joins, value mappings, etc.), and destinations (output files) to write toItem 1 is usually your own data. Items 2 & 3 together may be shared as a reusable "bundle" (zip file); seeavailable bundlesfor more information and a list of published bundles.If you  already have a bundle, continue to theusagesection below.If you develop a bundle for a particular source data system or format, please consider contributing it to the community by publishing it online and emailing the link totreitz@edanalytics.orgto get it listedhere.Source dataThis tool is designed to operate on tabular data in the form of multiple CSV or TSV files, such as those created by an export from some software system, or from a set of database tables.There are few limitations on the source data besides its format (CSV or TSV). Generally it is better to avoid using spaces in column names, however this can be managed by renaming columns as described in thesourcesYAML configuration section below.TemplatesAfter transforming the source data, this tool converts it to a text-based file like JSON(L) or XML based on a template using the Jinja templating language.Briefly, Jinja interpolates variables in double curly braces{{...}}to actual values, as well as providing other convenience functionality, such as string manipulation functions, logic blocks like if-tests, looping functionality, and much more. See the examples inexample_projects/, or check outthe official Jinja documentation.Note that templates mayincludeother templates, specified relative to the path from whichearthmoveris run - seeexample_projects/06_subtemplates/earthmover.yamlandexample_projects/06_subtemplates/mood.jsontfor an example.YAML configurationWhen updating to 0.2.xA breaking change was introduced in version 0.2 of Earthmover.
Before this update, each operation under a transformation required asourcebe defined.
This allowed inconsistent behavior where the results of an upstream operation could be discarded if misdefined.Thesourcekey has been moved into transformations as a required field.
In unary operations, the source is the output of the previous operation (or the transformationsourceif the first defined).
In operations with more than one source (i.e.,joinandunion), the output of the previous operation is treated as the first source;
any additional sources are defined using thesourcesfield.For example:# Before                          # AftertransA:                           transA:source:$sources.Aoperations:                       operations:-operation: add_columns          - operation:add_columnssource:$sources.Acolumns:                          columns:-A:"a"- A:"a"-B:"b"- B:"b"-operation: union                - operation:unionsources:                          sources:-$transformations.transA-$sources.B                        - $sources.B-$sources.C                        - $sources.CTo ensure the user has updated their templates accordingly, the key and valueversion: 2is mandatory at the beginning of Earthmover templates going forward.All the instructions for this tool — where to find the source data, what transformations to apply to it, and how and where to save the output — are specified in a single YAML configuration file. Example YAML configuration files and projects can be found inexample_projects/.The YAML configuration may alsocontain Jinjaandenvironment variable references.The general structure of the YAML involves the following sections:version, with required value2(Earthmover 0.2.x and later)config, which specifies options like the logging level and parameter defaultsdefinitionsis anoptionalway to specify reusable values and blockssources, where each source file is listed with details like the number of header rowstransformations, where source data can be transformed in various waysdestinations, where transformed data can be mapped to JSON templates and Ed-Fi endpoints and sent to an Ed-Fi APISection 1 has general options. Sections 2, 3, and 4 define aDAGwhich enables efficient data processing. Below, we document each section in detail:configTheconfigsection of theYAML configurationspecifies various options for the operation of this tool.A sampleconfigsection is shown here; the options are explained below.config:output_dir:./state_file:~/.earthmover.csvlog_level:INFOtmp_dir:/tmpshow_stacktrace:Trueshow_graph:Truemacros:>{% macro example_macro(value) -%}prefixed-int-{{value|int}}{%- endmacro %}parameter_defaults:SOURCE_DIR:./sources/show_progress:True(optional)output_dirdetermines where generated JSONL is stored. The default is./.(optional)state_filedetermines the file which maintainstool state. The default is~/.earthmover.csvon *nix systems,C:/Users/USER/.earthmover.csvon Windows systems.(optional) Specify alog_levelfor output. Possible values areERROR: only output errors like missing required sources, invalid references, invalidYAML configuration, etc.WARNING: output errors and warnings like when the run log is getting longINFO: all errors and warnings plus basic information about whatearthmoveris doing: start and stop, how many rows were removed by adistinct_rowsorfilter_rowsoperation, etc. (This is the defaultlog_level.)DEBUG: all output above, plus verbose details about each transformation step, timing, memory usage, and more. (Thislog_levelis recommended fordebuggingtransformations.)(optional) Specify thetmp_dirpath to use when dask must spill data to disk. The default is/tmp.(optional) Specify whether to show a stacktrace for runtime errors. The default isFalse.(optional) Specify whether or notshow_graph(default isFalse), which requiresPyGraphVizto be installed and createsgraph.pngandgraph.svgwhich are visual depictions of the dependency graph.(optional) Specify Jinjamacroswhich will be available within any Jinja template content throughout the project. (This can slow performance.)(optional) Specifyparameter_defaultswhich will be used if the user fails to specify a particularparameterorenvironment variable.(optional) Specify whether toshow_progresswhile processing, via a Dask progress bar.definitionsThedefinitionssection of theYAML configurationis an optional section you can use to define configurations which are reused throughout the rest of the configuration.earthmoverdoes nothing special with this section, it's just interpreted by the YAML parser. However, this can be a very useful way to keep your YAML configurationDRY– rather than redefine the same values, Jinja phrases, etc. throughout your config, define them once in this section and refer to them later usingYAML anchors, aliases, and overrides.An exampledefinitionssection is shown below:definitions:operations:-&student_join_opoperation:joinjoin_type:leftleft_key:student_idright_key:student_id...date_to_year_jinja:&date_to_year"{%raw%}{{val[-4:]}}{%endraw%}"...transformations:roster:operations:-<<:*student_join_opsources:-$sources.roster-$sources.studentsenrollment:operations:-<<:*student_join_opsources:-$sources.enrollment-$sources.students...academic_terms:operations:-operation:duplicate_columnssource:$sources.academic_termscolumns:start_date:school_year-operation:modify_columnscolumns:school_year:*date_to_yearsourcesThesourcessection of theYAML configurationspecifies source data the tool will work with.A samplesourcessection is shown here; the options are explained below.sources:districts:connection:"ftp://user:pass@host:port/path/to/districts.csv"tx_schools:connection:"postgresql://user:pass@host/database"query:>select school_id, school_name, school_websitefrom schema.schoolswhere school_address_state='TX'courses:file:./data/Courses.csvheader_rows:1columns:-school_id-school_year-course_code-course_title-subject_idmore_schools:file:./data/Schools.csvheader_rows:1columns:-school_id-school_name-address-phone_numberexpect:-low_grade != ''-high_grade != ''-low_grade|int <= high_grade|intEach source must have a name (which is how it is referenced by transformations and destinations) such asdistricts,courses,tx_schools, ormore_schoolsin this example. Three types ofsourcesare currently supported:File sources must specify the relative or absolute path to the sourcefile. Supported file types areRow-based formats:.csv: Specify the number ofheader_rows, and (ifheader_rows> 0, optionally) overwrite thecolumnnames. Optionally specify anencodingto use when reading the file (the default is UTF8)..tsv: Specify the number ofheader_rows, and (ifheader_rows> 0, optionally) overwrite thecolumnnames. Optionally specify anencodingto use when reading the file (the default is UTF8)..txt: a fixed-width text file; column widths are inferred from the first 100 lines.Column-based formats:.parquet,.feather,.orc— these require thepyarrowlibrary, which can be installed withpip install pyarrowor similarStructured formats:.json: Optionally specify aobject_type(frameorseries) andorientation(seethese docs) to interpret different JSON structures..jsonlor.ndjson: reads files with a flat JSON structure per line..xml: Optionally specify anxpathtoselect a set of nodesdeeper in the XML..html: Optionally specify a regex tomatchforselecting one of many tablesin the HTML. This can be used to extract tables from a live web page.Excel formats:.xls,.xlsx,.xlsm,.xlsb,.odf,.odsand.odt— optionally specify thesheetname (as a string) or index (as an integer) to load.Other formats:.pklor.pickle: apickledPython object (typically a Pandas dataframe).sas7bdat: aSAS data file.sav: aSPSS data file.dta: aStata data fileFile type is inferred from the file extension, however you may manually specifytype:(csv,tsv,fixedwidth,parquet,feather,orc,json,jsonl,xml,html,excel,pickle,sas,spss, orstata) to forceearthmoverto treat a file with an arbitrary extension as a certain type. Remote file paths (https://somesite.com/path/to/file.csv) generally work.Database sources are supported viaSQLAlchemy. They must specify a databaseconnectionstring and SQLqueryto run.FTP file sources are supported viaftplib. They must specify an FTPconnectionstring.For any source, optionally specify conditions youexpectdata to meet which, if not true for any row, will cause the run to fail with an error. (This can be useful for detecing and rejecting NULL or missing values before processing the data.) The format must be a Jinja expression that returns a boolean value. This is enables casting values (which are all treated as strings) to numeric formats like int and float for numeric comparisons.The examples above showuser:passin theconnectionstring, but if you are version-controlling your YAML you must avoid publishing such credentials. Typically this is done viaenvironment variablesorcommand line parameters, which are both supported by this tool. Such environment variable references may be used throughout your YAML (not just in thesourcessection), and are parsed at load time.transformationsThetransformationssection of theYAML configurationspecifies how source data is manipulated by the tool.A sampletransformationssection is shown here; the options are explained below.transformations:courses:source:$sources.coursesoperations:-operation:map_valuescolumn:subject_idmapping:01:1 (Mathematics)02:2 (Literature)03:3 (History)04:4 (Language)05:5 (Computer and Information Systems)-operation:joinsources:-$sources.schoolsjoin_type:innerleft_key:school_idright_key:school_id-operation:drop_columnscolumns:-address-phone_numberThe above example shows a transformation of thecoursessource, which consists of an ordered list of operations. A transformation defines a source to which a series of operations are applied. This source may be an original$sourceor another$transformation. Transformation operations each require further specification depending on their type; the operations are listed and documented below.Frame operationsunionConcatenates the transformation source with one or more sources sources of the same shape.-operation:unionsources:-$sources.courses_list_1-$sources.courses_list_2-$sources.courses_list_3joinJoins the transformation source with one or more sources.-operation:joinsources:-$sources.schoolsjoin_type:inner | left | rightleft_key:school_idright_key:school_id# or:left_keys:-school_id-school_yearright_keys:-school_id-school_year# optionally specify columns to (only) keep from the left and/or right sources:left_keep_columns:-left_col_1-left_col_2right_keep_columns:-right_col_1-right_col_2# or columns to discard from the left and/or right sources:left_drop_columns:-left_col_1-left_col_2right_drop_columns:-right_col_1-right_col_2# (if neither ..._keep nor ..._drop are specified, all columns are retained)Joining can lead to a wide result; the..._keep_columnsand..._drop_columnsoptions enable narrowing it.Besides the join column(s), if a columnmy_columnwith the same name exists in both tables and is not dropped, it will be renamedmy_column_xandmy_column_y, from the left and right respectively, in the result.Column operationsadd_columnsAdds columns with specified values.-operation:add_columnscolumns:-new_column_1:value_1-new_column_2:"{%raw%}{%ifTrue%}Jinjaworkshere{%endif%}{%endraw%}"-new_column_3:"{%raw%}Referencevaluesfrom{{AnotherColumn}}inthisnewcolumn{%endraw%}"-new_column_4:"{%raw%}{%ifcol1>col2%}{{col1|float+col2|float}}{%else%}{{col1|float-col2|float}}{%endif%}{%endraw%}"Use Jinja:{{value}}refers to this column's value;{{AnotherColumn}}refers to another column's value. AnyJinja filtersandmath operationsshould work. Reference the current row number with{{__row_number__}}or a dictionary containing the row data with{{__row_data__['column_name']}}.You must wrap Jinja expressionsin{%raw%}...{%endraw%}to avoid them being parsed at YAML load time.rename_columnsRenames columns.-operation:rename_columnscolumns:old_column_1:new_column_1old_column_2:new_column_2old_column_3:new_column_3duplicate_columnsDuplicates columns (and all their values).-operation:duplicate_columnscolumns:existing_column1:new_copy_of_column1existing_column2:new_copy_of_column2drop_columnsRemoves the specified columns.-operation:drop_columnscolumns:-column_to_drop_1-column_to_drop_2keep_columnsKeeps only the specified columns, discards the rest.-operation:keep_columnscolumns:-column_to_keep_1-column_to_keep_2combine_columnsCombines the values of the specified columns, delimited by a separator, into a new column.-operation:combine_columnscolumns:-column_1-column_2new_column:new_column_nameseparator:"_"Defaultseparatoris none - values are smashed together.modify_columnsModify the values in the specified columns.-operation:modify_columnscolumns:state_abbr:"{%raw%}XXX{{value|reverse}}XXX{%endraw%}"school_year:"{%raw%}20{{value[-2:]}}{%endraw%}"zipcode:"{%raw%}{{value|int**2}}{%endraw%}"Use Jinja:{{value}}refers to this column's value;{{AnotherColumn}}refers to another column's value. AnyJinja filtersandmath operationsshould work. Reference the current row number with{{__row_number__}}or a dictionary containing the row data with{{__row_data__['column_name']}}.You must wrap Jinja expressionsin{%raw%}...{%endraw%}to avoid them being parsed at YAML load time.map_valuesMap the values of a column.-operation:map_valuescolumn:column_name# or, to map multiple columns simultaneouslycolumns:-col_1-col_2mapping:old_value_1:new_value_1old_value_2:new_value_2# or a CSV/TSV with two columns (from, to) and header row:map_file:path/to/mapping.csvdate_formatChange the format of a date column.-operation:date_formatcolumn:date_of_birth# orcolumns:-date_column_1-date_column_2-date_column_3from_format:"%b%d%Y%H:%M%p"to_format:"%Y-%m-%d"Thefrom_formatandto_formatmust followPython's strftime() and strptime() formats.snake_case_columnsForce the names of all columns tosnake_case.-operation:snake_case_columnsRow operationsdistinct_rowsRemoves duplicate rows.-operation:distinct_rowscolumns:-distinctness_column_1-distinctness_column_2Optionally specify thecolumnsto use for uniqueness, otherwise all columns are used. If duplicate rows are found, only the first is kept.filter_rowsFilter (include or exclude) rows matching a query.-operation:filter_rowsquery:school_year < 2020behavior:exclude | includeThe query format is anything supported byPandas.DataFrame.query. Specifyingbehaviorasexcludewraps the Pandasquery()withnot().Group operationsgroup_byReduce the number of rows by grouping, and add columns with values calculated over each group.-operation:group_bygroup_by_columns:-student_idcreate_columns:num_scores:count()min_score:min(item_score)max_score:max(item_score)avg_score:mean(item_score)item_scores:agg(item_score,;)Valid aggregation functions arecount()orsize()- the number of rows in each groupmin(column)- the minumum (numeric) value incolumnfor each groupstr_min(column)- the minumum (string) value incolumnfor each groupmax(column)- the maximum (numeric) value incolumnfor each groupstr_max(column)- the maximum (string) value incolumnfor each groupsum(column)- the sum of (numeric) values incolumnfor each groupmean(column)oravg(column)- the mean of (numeric) values incolumnfor each groupstd(column)- the standard deviation of (numeric) values incolumnfor each groupvar(column)- the variance of (numeric) values incolumnfor each groupagg(column,separator)- the values ofcolumnin each group are concatenated, delimited byseparator(defaultseparatoris none)Numeric aggregation functions will fail with errors if used on non-numeric column values.Note the difference betweenmin()/max()andstr_min()/str_max(): given a list like10, 11, 98, 99, 100, 101, return values arefunctionreturnmin()10str_min()10max()101str_max()99destinationsThedestinationssection of theYAML configurationspecifies how transformed data is materialized to files.A sampledestinationssection is shown here; the options are explained below.destinations:schools:source:$transformations.school_listtemplate:./json_templates/school.jsontextension:jsonllinearize:Truecourses:source:$transformations.course_listtemplate:./json_templates/course.jsontextension:jsonllinearize:Truecourse_report:source:$transformations.course_listtemplate:./json_templates/course.htmltextension:htmllinearize:Falseheader:<html><body><h1>Course List:</h1>footer:</body></html>For each file you want materialized, provide thesourceand thetemplatefile — a text file (JSON, XML, HTML, etc.) containing Jinja with references to the columns ofsource. The materialized file will containtemplaterendered for each row ofsource, with an optionalheaderprefix andfooterpostfix. Files are materialized using your specifiedextension(which is required).IflinearizeisTrue, all line breaks are removed from the template, resulting in one output line per row. (This is useful for creating JSONL and other linear output formats.) If omitted,linearizeisTrue.Global optionsAny source, transformation, or destination may also specifydebug: Truewhich will output the dataframe shape and columns after the node completes processing. This can be very useful while building and debugging.Additionally, theshow_progressboolean flag can be specified on any source, transformation, or destination to display a progress bar while processing.Finally,repartitioncan be passed to any node to repartition the node in memory before continuing to the next node.
Set either the number of bytes, or a text representation (e.g., "100MB") to shuffle data into new partitions of that size.
(Note: this configuration is advanced, and its use may drastically affect performance.)UsageOnce you have the requiredsetupand your source data, run the transformations withearthmoverrun-cpath/to/config.yamlIf you omit the optional-cflag,earthmoverwill look for anearthmover.yamlin the current directory.See a help message withearthmover-h
earthmover--helpSee the tool version withearthmover-v
earthmover--versionFeaturesThis tool includes several special features:Jinja in YAML configurationYou may use Jinja in the YAML configuration, which will be parsed at load time. Only theconfigsection may not contain Jinja, except formacroswhich are made available both at parse-time for Jinja in the YAML configuration and at run-time for Jinja inadd_columnsormodify_columnstransformation operations.The following exampleloads 9 source filesadds a column indicating the source file each row came fromunions the sources togetherif an environment variable or parameterDO_FILTERING=Trueis passed, filters out certain rowsconfig:show_graph: Trueparameter_defaults:DO_FILTERING: "False"sources:{%foriinrange(1,10)%}source{{i}}:file: ./sources/source{{i}}.csvheader_rows: 1{%endfor%}transformations:{%foriinrange(1,10)%}source{{i}}:source: $sources.source{{i}}operations:- operation: add_columnscolumns:- source_file:{{i}}{%endfor%}stacked:source: $transformations.source1operations:- operation: unionsources:{%foriinrange(2,10)%}- $transformations.source{{i}}{%endfor%}{%if"${DO_FILTERING}"=="True"%}- operations: filter_rowsquery: school_year < 2020behavior: exclude{%endif%}destinations:final:source: $transformations.stackedtemplate: ./json_templates/final.jsontextension: jsonllinearize: TrueSelectorsRun only portions of theDAGby using a selector:earthmoverrun-cpath/to/config.yaml-speople,people_*This processes all DAG paths (from sources to destinations) through any matched nodes.Optional SourcesIf you specify thecolumnslist andoptional: Trueon a filesourcebut leave thefileblank,earthmoverwill create an empty dataframe with the specified columns and pass it through the rest of theDAG. This, combined with the use ofenvironment variable referencesand/orcommand-line parametersto specify asource'sfile, provides flexibility to include data when it's available but still run when it is missing.Environment variable referencesIn yourYAML configuration, you may reference environment variables with${ENV_VAR}. This can be useful for making references to source file locations dynamic, such assources:people:file:${BASE_DIR}/people.csv...Note:becauseos.path.expandvars()does not expand variables within single quotesin Python under Windows,Windows users should avoid placing environment variable references (orCLI parameterreferences) inside single quote stringsin their YAML.Command-line parametersSimilarly, you can specify parameters via the command line withearthmoverrun-cpath/to/config.yaml-p'{"BASE_DIR":"path/to/my/base/dir"}'earthmoverrun-cpath/to/config.yaml--params'{"BASE_DIR":"path/to/my/base/dir"}'Command-line parameters override any environment variables of the same name.StateThis toolmaintains state about past runs.Subsequent runs only re-process if something has changed – theYAML configurationitself, data files ofsources,value_mappingCSVs oftransformations, template files ofdestinations, or CLI parameters. (Changes are tracked by hashing files; hashes and run timestamps are stored in the file specified byconfig/state_file.) You may choose to override this behavior and force reprocessing of the whole DAG, regardless of whether files have changed or not, using the-for--forcecommand-line flag:earthmoverrun-cpath/to/config.yaml-f
earthmoverrun-cpath/to/config.yaml--force-regenerateTo further avoid computing input hashes and not log a run to thestate_file, use the-kor--skip-hashingflag:earthmoverrun-cpath/to/config.yaml-k
earthmoverrun-cpath/to/config.yaml--skip-hashing(This makes a one-time run on large input files faster.) If earthmover skips running because nothing has changed, it returns bash exit code99(this was chosen becauseit signals a "skipped" task in Airflow).Structured output of run resultsTo produce a JSON file with metadata about the run, invoke earthmover withearthmoverrun-cpath/to/config.yaml--results-file./results.jsonFor example, forexample_projects/09_edfi/, a sample results file would be:{"started_at":"2023-06-08T10:21:42.445835","working_dir":"/home/someuser/code/repos/earthmover/example_projects/09_edfi","config_file":"./earthmover.yaml","output_dir":"./output/","row_counts":{"$sources.schools":6,"$sources.students_anytown":1199,"$sources.students_someville":1199,"$destinations.schools":6,"$transformations.all_students":2398,"$destinations.students":2398,"$destinations.studentEducationOrganizationAssociations":2398,"$destinations.studentSchoolAssociations":2398},"completed_at":"2023-06-08T10:21:43.118854","runtime_sec":0.673019}TestsThis tool ships with a test suite covering all transformation operations. It can be run withearthmover -t, which simply runs the tool on theconfig.yamland toy data in theearthmover/tests/folder. (The DAG is pictured below.) Renderedearthmover/tests/output/are then compared against theearthmover/tests/expected/output; the test passes only if all files match exactly.Run tests withearthmover-tDesignSome details of the design of this tool are discussed below.YAML parsingearthmoverallows Jinja templating expressions in its YAML configuration files. (This is similar to howAnsible Playbookswork.)earthmoverparses the YAML in several steps:Extract only theconfigsection(if any), in order to make available anymacroswhen parsing the rest of the Jinja + YAML. Theconfigsectiononlymay not contain any Jinja(besidesmacros).Load the entire Jinja + YAML as a string and hydrate allenvironment variableorparameterreferences.Parse the hydrated Jinja + YAML string with anymacrosto plain YAML.Load the plain YAML string as a nested dictionary and begin building and processing theDAG.Note that due to step (3) above,runtimeJinja expressions (such as column definitions foradd_columnsormodify_columnsoperations) should be wrapped with{%raw%}...{%endraw%}to avoid being parsed when the YAML is being loaded.DAGThe mapping of sources through transformations to destinations is modeled as a directed acyclic graph (DAG). Eachcomponentof the DAG is run separately.Each component is materialized intopological order. This minimizes memory usage, as only the data from the current and previous layer must be retained in memory.DataframesAll data processing is done usingPandas DataframesandDask, with values stored as strings (orCategoricals, for memory efficiency in columns with few unique values). This choice of datatypes prevents issues arising from Pandas' datatype inference (like inferring numbers as dates), but does require casting string-representations of numeric values using Jinja when doing comparisons or computations.Performance & LimitationsTool performance depends on a variety of factors including source file size and/or database performance, the system's storage performance (HDD vs. SSD), memory, and transformation complexity. But some effort has been made to engineer this tool for high throughput and to work in memory- and compute-constrained environments.Smaller source data (which all fits into memory) processes very quickly. Larger chunked sources are necessarily slower. We have tested with sources files of 3.3GB, 100M rows (synthetic attendance data): creating 100M lines of JSONL (30GB) takes around 50 minutes on a modern laptop.Thestate featureadds some overhead, as hashes of input data and JSON payloads must be computed and stored, but this can be disabled if desired.Best PracticesIn this section we outline some suggestions for best practices to follow when usingearthmover, based on our experience with the tool. Many of these are based on best practices for usingdbt, to whichearthmoveris similar, althoughearthmoveroperates on dataframes rather than database tables.Project Structure PracticesA typicalearthmoverproject might have a structure like this:project/
├── README.md
├── sources/
│   └── source_file_1.csv
│   └── source_file_2.csv
│   └── source_file_3.csv
├── earthmover.yaml
├── output/
│   └── output_file_1.jsonl
│   └── output_file_2.xml
├── seeds/
│   └── crosswalk_1.csv
│   └── crosswalk_2.csv
├── templates/
│   └── json_template_1.jsont
│   └── json_template_2.jsont
│   └── xml_template_1.xmlt
│   └── xml_template_2.xmltGenerally you should separate the mappings, transformations, and structure of your data – which are probablynotsensitive – from the actual input and output – which may be large and/or sensitive, and therefore should not be committed to a version control system. This can be accomplished in two ways:include a.gitignoreor similar file in your project which excludes thesources/andoutput/directories from being committed the repositoryremove thesources/andoutput/directories from your project and updateearthmover.yaml'ssourcesanddestinationsto reference another location outside theproject/directoryWhen dealing with sensitive source data, you may have to comply with security protocols, such as referencing sensitive data from a network storage location rather than copying it to your own computer. In this situation, option 2 above is a good choice.To facilitateoperationalization, we recommended usingenvironment vairablesorcommand-line parametersto pass input and output directories and filenames toearthmover, rather than hard-coding them intoearthmover.yaml. For example, rather thanconfig:output_dir:path/to/outputs/...sources:source_1:file:path/to/inputs/source_file_1.csvheader_rows:1source_2:file:path/to/inputs/source_file_2.csvheader_rows:1...destinations:output_1:source:$transformations.transformed_1...output_2:source:$transformations.transformed_2...instead consider usingconfig:output_dir:${OUTPUT_DIR}...sources:source_1:file:${INPUT_DIR}${INPUT_FILE_1}header_rows:1source_2:file:${INPUT_DIR}${INPUT_FILE_2}header_rows:1...destinations:output_1:source:$transformations.transformed_1...output_2:source:$transformations.transformed_2...and then run withearthmoverearthmover.yaml-p'{ "OUTPUT_DIR": "path/to/outputs/", "INPUT_DIR": "path/to/inputs/", "INPUT_FILE_1": "source_file_1.csv", "INPUT_FILE_2": "source_file_2.csv" }'Note that with this pattern you can also useoptional sourcesto only create one of the outputs if needed, for exampleearthmoverearthmover.yaml-p'{ "OUTPUT_DIR": "path/to/outputs/", "INPUT_DIR": "path/to/inputs/", "INPUT_FILE_1": "source_file_1.csv" }'would only createoutput_1ifsource_1hadrequired: False(sinceINPUT_FILE_2is missing).Development practicesWhile YAML is a data format, it is best to treat theearthmoverYAML configurationas code, meaning you shouldversionit!avoid putting credentials and other sensitive information in the configuration; rather specify such values asenvironment variablesorcommand-line parameterskeep your YAMLDRYby usingJinja macrosandYAML anchors and aliasesRemember thatcode is poetry: it should be beautiful! To that endCarefully choose concise, good names for yoursources,transformations, anddestinations.Good names forsourcescould be based on their source file/table (e.g.studentsforstudents.csv)Good names fortransformationsindicate what they do (e.g.students_with_mailing_addresses)Good names fordestinationscould be based on the destination file (e.g.student_mail_merge.xml)Add good, descriptive comments throughout your YAML explaining any assumptions or non-intuitive operations (including complex Jinja).Likewise put Jinja comments in your templates, explaining any complex logic and structures.Keep YAML concise by composingtransformationoperations where possible. Many operations likeadd_columns,map_values, and others can operate on multiplecolumnsin a dataframe.At the same time, avoid doing too much at once in a singletransformation; splitting multiplejoinoperations into separate transformations can makedebuggingeasier.Debugging practicesWhen developing your transformations, it can be helpful tospecifyconfig»log_level: DEBUGandtransformation»operation»debug: Trueto verify the columns and shape of your data after eachoperationturn onconfig»show_stacktrace: Trueto get more detailed error messagesavoid name-sharing for asource, atransformation, and/or adestination- this is allowed but can make debugging confusinginstall pygraphvizand turn onconfig»show_graph: True, then visually inspect your transformations ingraph.pngfor structural errorsuse a linter/validator to validate the formatting of your generated dataYou can remove these settings once yourearthmoverproject is ready for operationalization.Operationalization practicesTypicallyearthmoveris used when the same (or simlar) data transformations must be done repeatedly. (A one-time data transformation task may be more easily done withSQLiteor a similar tool.) When deploying/operationalizingearthmover, whether with a simple scheduler likecronor an orchestration tool likeAirfloworDagster, considerspecifying conditions youexpectyoursourcesto meet, soearthmoverwill fail on source data errorsspecifyingconfig»log_level: INFOand monitoring logs for phrases likedistinct_rowsoperation removed NN duplicate rowsfilter_rowsoperation removed NN rowsChangelogSeeCHANGELOG.ContributingBugfixes and new features (such as additional transformation operations) are gratefully accepted via pull requests here on GitHub.ContributionsCover image created withDALL • E miniLicenseSeeLicense.BundlesBundles are pre-built data mappings for converting various data formats to Ed-Fi format using this tool. They consist of a folder with CSV seed data, JSON template files, and aconfig.yamlwith sources, transformations, and destinations.Here we maintain a list of bundles for various domain-specific uses:Bundles for transformingassessment data from various vendors to the Ed-Fi data standardIf you develop bundles, please consider contributing them to the community by publishing them online and emailing the link totreitz@edanalytics.orgto get them listed above.