Package: lightbeam
lightbeamtransmits payloads from JSONL files into anEd-Fi API.Table of ContentsRequirementsInstallationSetupUsageFeaturesDesignPerformanceLimitationsChangelogContributingLicenseRequirementsPython 3,pip, and connectivity to an Ed-Fi API.Installationpip install lightbeamSetupRunning the tool requiresa folder of JSONL files, one for each Ed-Fi Resource and Descriptor to populatea YAML configuration file
An example YAML configuration is below, followed by documentation of each option.state_dir:~/.lightbeam/data_dir:./namespace:ed-fiedfi_api:base_url:https://api.schooldistrict.org/v5.3/apiversion:3mode:year_specificyear:2021client_id:yourIDclient_secret:yourSecretconnection:pool_size:8timeout:60num_retries:10backoff_factor:1.5retry_statuses:[429,500,502,503,504]verify_ssl:Truecount:separator:,fetch:page_size:100force_delete:Truelog_level:INFOshow_stacktrace:True(optional)state_diris wherestateis stored. The default is~/.lightbeam/on *nix systems,C:/Users/USER/.lightbeam/on Windows systems.(optional) Specify thedata_dirwhich contains JSONL files to send to Ed-Fi. The default is./. The tool will look for files like{Resource}.jsonlor{Descriptor}.jsonlin this location, as well as directory-based files like{Resource}/*.jsonlor{Descriptor}/*.jsonl. Files with.ndjsonor simply.jsonextensions will also be processed. (More info at thendjsonstandard page.)(optional) Specify thenamespaceto use when accessing the Ed-Fi API. The default ised-fibut others includetpdmor custom values. To send data to multiple namespaces, you must use a YAML configuration file andlightbeam sendfor each.Specify the details of theedfi_apito which to connect including(optional) Thebase_urlwhich serves a JSON object specifying the paths to data endpoints, Swagger, and dependencies. The default ishttps://localhost/api(the address of an Ed-Fi APIrunning locally in Docker), but the location varies depending on how Ed-Fi is deployed.Theversionas one of3or2(2is currently unsupported).(optional) Themodeas one ofshared_instance,sandbox,district_specific,year_specific, orinstance_year_specific.(required ifmodeisyear_specificorinstance_year_specific) Theyearused to build the resource URL. The default is the current year.(required ifmodeisinstance_year_specific) Theinstance_codeused to build the resource URL. The default is none.(required) Specify theclient_idto use when connecting to the Ed-Fi API.(required) Specify theclient_secretto use when connecting to the Ed-Fi API.Specify theconnectionparameters to use when making requests to the API including(optional) Thepool_size. The default is 8. The optimal setting depends on the Ed-Fi API's capabilities.(optional) Thetimeout(in seconds) to wait for each connection attempt. The default is60seconds.(optional) Thenum_retriesto do in case of request failures. The default is10.(optional) Thebackoff_factorto use for the exponential backoff. The default is1.5.(optional) Theretry_statuses, that is, the HTTPS response codes to consider as failures to retry. The default is[429, 500, 501, 503, 504].(optional) Whether toverify_ssl. The default isTrue. Set toFalsewhen working withlocalhostAPIs or to live dangerously.(optional) forlightbeam count, optionally change theseparatorbetweenRecordsandEndpoint. The default is a "tab" character.(optional) forlightbeam fetch, optionally specify the number of records (page_size) to GET at a time. The default is 100, but if you're trying to extract lots of data from an API increase this to the largest allowed (which depends on the API, but is often 500 or even 5000).(optional) Skip the interactive confirmation prompt (for programmatic use) when using thedeletecommand. The default isFalse(prompt).(optional) Specify alog_levelfor output. Possible values areERROR: only output errors like missing required sources, invalid references, invalidYAML configuration, etc.WARNING: output errors and warnings like when the run log is getting longINFO: all errors and warnings plus basic information about whatearthmoveris doing: start and stop, how many rows were removed by adistinct_rowsorfilter_rowsoperation, etc. (This is the defaultlog_level.)DEBUG: all output above, plus verbose details about each transformation step, timing, memory usage, and more. (Thislog_levelis recommended fordebuggingtransformations.)(optional) Specify whether to show a stacktrace for runtime errors. The default isFalse.Usagelightbeamrecognizes several commands:countlightbeamcount-cpath/to/config.yamlPrints to the console (or to your--results-file, if specified) a record count for each endpoint in your Ed-Fi API.By default, resourcesand descriptors(all endpoints) are counted. You can change this by usingselectors, such as-e *Descriptors.Endpoint counts printed to the console (if you don't specify a--results-file) include only endpoints with more than zero records. Endpoint counts saved in a--results-fileinclude all available endpoints, even those with zero records.Whether printed to the console or a--results-file, output will include columnsRecordsandEndpointseparated by a separator specified ascount.separatorin yourYAML configuration(default is a "tab" character).fetchlightbeamfetch-cpath/to/config.yamlFetches the payloads of selected endpoints from your Ed-Fi API and saves them, each on their own line, to JSONL files in yourdata_dir.Optionally specify--query '{"studentUniqueId": 12345}'or-q '{"key": "value"}'to add query parameters to every GET request. This can be useful if you want tofetchdata for just a specific record (and related data). For example:lightbeamfetch-sstudent*-e*Descriptors-q'{"studentUniqueId":12345}'-did,_etag,_lastModifiedDateOptionally specify--keep-keys idor-k idto keep only specific keys from every payload. This can be useful to reduce the amount of data stored if you only need certain fields. It is used internally bytruncateto onlyfetchtheids or payloads to thendeletebyid.Optionally specify--drop-keys id,_etag,_lastModifiedor-d idto remove specific keys from every payload. This can be useful if you want tofetchdata from one Ed-Fi API and then turn around andsendit to another.validatelightbeamvalidate-cpath/to/config.yamlYou mayvalidateyour JSONL before transmitting it. This checks that the payloadsare valid JSONconform to the structure described in the Swagger documents forresourcesanddescriptorsfetched from your APIcontain valid descriptor values (fetched from your API and/or from descriptor values in your JSONL files)contain unique values for any natural keyThis command will not find invalid reference errors, but is helpful for finding payloads that are invalid JSON, are missing required fields, or have other structural issues.sendlightbeamsend-cpath/to/config.yamlSends your JSONL payloads to your Ed-Fi API.validate+sendlightbeamvalidate+send-cpath/to/config.yamlThis is a shorthand for sequentially runningvalidateand thensend. It can be useful to catching errors in automated pipelines earlier in thevalidatestep before you actuallysendproblematic data to your Ed-Fi API.deletelightbeamdelete-cpath/to/config.yamlDelete payloads bydeterming the natural key (set of required fields) for each endpointiterating through your JSONL payloads and looking up each one via aGETrequest to the API filtering for the natural key valuesif exactly one result is returned,DELETEing it byidPayload hashes are also deleted fromsaved state. Endpoints are processed in reverse-dependency order to prevent delete failures due to data dependencies.Note that the default profile for most Ed-Fi API credentials prevents deletion of certain core resources (student,school, etc.), even if your credentials were used to create the records. If you get API errors trying to delete records, you may need "no further auth" API credentials.Running thedeletecommand will prompt you to type "yes" to confirm. This confirmation prompt can be disabled (for programmatic use) by specifyingforce_delete: Truein your YAML.truncatelightbeamtruncate-cpath/to/config.yamlTruncates (empties) your Ed-Fi API for selected endpoints, in dependency-order.USE WITH CAUTION!truncateworks by fetching theidof every record for a given endpoint and then deleting all records by ID.Truncateing a resource will also clear out thesaved statefor it.Note that the default profile for most Ed-Fi API credentials prevents deletion of certain core resources (student,school, etc.), even if your credentials were used to create the records. If you get API errors trying to delete records, you may need "no further auth" API credentials.Running thetruncatecommand will prompt you to type "yes" to confirm. This confirmation prompt can be disabled (for programmatic use) by specifyingforce_delete: Truein your YAML.truncateis a convenience command which should be used sparingly, as it can generate large numbers ofdeletesrecords and cause performance issues when pulling fromdeletesendpoints. If you want to wipe an entire Ed-Fi ODS, a better approach may be to drop and recreate the database (and re-send Descriptors and other default resources as needed).Other optionsSee a help message withlightbeam-h
lightbeam--helpSee the tool version withlightbeam-v
lightbeam--versionFeaturesThis tool includes several special features:SelectorsSend only a subset of resources or descriptors in yourdata_dirusing-sor--selector:lightbeamsend-cpath/to/config.yaml-sschools,students,studentSchoolAssociationsor, similarly, exclude some resources or descriptors using-eor--exclude:lightbeamsend-cpath/to/config.yaml-e*DescriptorsSelection and exclusion may be a single or comma-separated list of strings or a wildcards (beginning or ending with*). For example:lightbeamsend-cpath/to/config.yaml-sstudent*,parent*-e*Associations,*Descriptorswould process resources likestudentSchoolAttendanceEventsandparents, but notstudentSchoolAssociations,studentParentAssociations, or any Descriptors.Environment variable referencesIn yourYAML configuration, you may reference environment variables with${ENV_VAR}. This can be useful for passing sensitive data like credentials tolightbeam, such as...edfi_api:client_id:${EDFI_API_CLIENT_ID}client_secret:${EDFI_API_CLIENT_SECRET}...Command-line parametersSimilarly, you can specify parameters via the command line withlightbeamsend-cpath/to/config.yaml-p'{"CLIENT_ID":"populated", "CLIENT_SECRET":"populatedSecret"}'lightbeamsend-cpath/to/config.yaml--params'{"CLIENT_ID":"populated", "CLIENT_SECRET":"populatedSecret"}'Command-line parameters override any environment variables of the same name.StateThis toolmaintains state about payloads previously dispatched to the Ed-Fi APIto avoid repeatedly resending the same payloads. This is done by maintaining apickledPython dictionary of payload hashes for each Ed-Fi resource and descriptor, together with a timestamp and HTTP status code of the last response. The files are located in theconfigfile'sstate_dirand have names like{resource}.dator{descriptor}.dat.By default, only new, never-before-seen payloads aresentordeleted.You may choose to resend payloads last sent beforetimestampusing the-tor--older-thancommand-line flag:lightbeamsend-cpath/to/lightbeam.yaml-t2020-12-25T00:00:00
lightbeamsend-cpath/to/lightbeam.yaml--older-than2020-12-25T00:00:00Or you may choose to resend payloads last sent aftertimestampusing the-nor--newer-thancommand-line flag:lightbeamsend-cpath/to/lightbeam.yaml-n2020-12-25T00:00:00
lightbeamsend-cpath/to/lightbeam.yaml--newer-than2020-12-25T00:00:00Or you may choose to resend payloads that returned a certain HTTP status code(s) on the last send using the-ror--retry-status-codescommand-line flag:lightbeamsend-cpath/to/lightbeam.yaml-r200,201
lightbeamsend-cpath/to/lightbeam.yaml--retry-status-codes200,201These three options may be composed;lightbeamwill resend payloads that match any conditions (logical OR).Finally, you can ignore prior state and resend all payloads using the-for--forceflag:lightbeamsend-cpath/to/lightbeam.yaml-f
lightbeamsend-cpath/to/lightbeam.yaml--forceCacheTo reduce runtime,lightbeamcaches the resource and descriptor Swagger docs it fetches from your Ed-Fi API as well as the descriptor values for up to a month. This way, the data does not have to be re-loaded from your API on every run. The cached files are stored in thecachedirectory within yourstate_dir. You may runlightbeamwith the-wor--wipeflag to clear this cached data and force re-fetching the API metadata:lightbeamsend-cpath/to/config.yaml-w
lightbeamsend-cpath/to/config.yaml--wipeStructured output of run resultsTo produce a JSON file with metadata about the run, invoke lightbeam withlightbeamsend-cpath/to/config.yaml--results-file./results.jsonA sample results file could be:{"started_at":"2023-06-08T17:18:25.053207","working_dir":"/home/someuser/code/sandbox/testing_lightbeam","config_file":"lightbeam.yml","data_dir":"./","api_url":"https://some-ed-fi-api.edu/api","namespace":"ed-fi","resources":{"studentSchoolAssociations":{"failed_statuses":{"400":{"400: { \"message\": \"The request is invalid.\", \"modelState\": { \"request.schoolReference.schoolId\": [ \"JSON integer 1234567899999 is too large or small for an Int32. Path 'schoolReference.schoolId', line 1, position 328.\" ] } }":{"files":{"./studentSchoolAssociations.jsonl":{"line_numbers":"6,4,5,7,8","count":5}}},"400: { \"message\": \"Validation of 'StudentSchoolAssociation' failed.\\n\\tStudent reference could not be resolved.\\n\" }":{"files":{"./studentSchoolAssociations.jsonl":{"line_numbers":"1,3,2","count":3}}},"count":8},"409":{"409: { \"message\": \"The value supplied for the related 'studentschoolassociation' resource does not exist.\" }":{"files":{"./studentSchoolAssociations.jsonl":{"line_numbers":"9,10,12,14,16,13,11,15,17,18,19,21,22,20","count":14}}},"count":14}},"records_processed":22,"records_skipped":0,"records_failed":22}},"completed_at":"2023-06-08T17:18:26.724699","runtime_sec":1.671492,"total_records_processed":22,"total_records_skipped":0,"total_records_failed":22}DesignSome details of the design of this tool are discussed below.Resource-dependency orderingJSONL files are sent to the Ed-Fi API in resource-dependency order, which avoids "missing reference" API errors when populating multiple endpoints.Asynchronous requestslightbeamachieves exceptional performance by makingasynchronousrequests to the Ed-Fi API - up toconnection.pool_size(in yourYAML configuration) at a time.Performance & LimitationsTool performance depends on primarily on the performance of the Ed-Fi API, which in turn depends on the compute resources which back it. Typically the bottleneck is write performance to the database backend (SQL server or Postgres). If you uselightbeamto ingest a large amount of data into an Ed-Fi API (not a recommended use-case), consider temporarily scaling up your database backend.For reference, we have achieved throughput rates in excess of 100 requests/second against an Ed-Fi ODS & API running in Docker on a laptop.ChangelogSeeCHANGELOG.ContributingBugfixes and new features (such as additional transformation operations) are gratefully accepted via pull requests here on GitHub.ContributionsCover image created withDALL • E miniLicenseSeeLicense.