Package: edfi-api-client
Ed-Fi API Client Python PackageQuick Guidefromedfi_api_clientimportEdFiClient# Client connection with Ed-Fi3 ODSapi=EdFiClient(BASE_URL,CLIENT_KEY,CLIENT_SECRET,api_version=3)# Get the total row-count for the 'students' resource in the ODSstudents=api.resource('students')students.total_count()# Pull all rows for the 'staffs' resource deletes endpoint (setting a custom page-size)staffs=api.resource('staffs',get_deletes=True)forrowinstaffs.get_rows(page_size=500):pass# Pull all rows for the 'studentStaffAssociations' resource as pages (retrying when given authentication-timeout errors)ssa=api.resource('studentStaffAssociations')# OR 'student_staff_associations'forpageinssa.get_pages(retry_on_failure=True):pass# Pull all rows for the enrollment students composite, filtering by section IDenrollment_students=api.composite('students',filter_type='sections',filter_id='12345')forrowinenrollment_students.get_rows():passEdFiClientEdFiClientserves as the interface with the ODS.
If credentials are provided, a session with the ODS is automatically authenticated.
Some methods do not require credentials to be called.Arguments:ArgumentDescriptionbase_url[Required] The root url of the API server, without any trailing components likedata/v3orapi/v2.0client_keyThe keyclient_secretThe secretapi_versionEither 2 or 3, depending on the suite number of the API (Default 3)api_modeThe API mode of the ODS (e.g.,shared_instance,year_specific, etc.). If empty, the mode will automatically be inferred from the ODS' Swagger spec (Ed-Fi 3 only).api_yearThe year of data to connect to if accessing ayear_specificorinstance_year_specificODS.instance_codeThe instance code if accessing aninstance_year_specificODS.If eitherclient_keyorclient_secretare empty, a session with the ODS will not be established.All code examples in this document use verbose-logging to more-explicitly show interactions with the API.
It is recommended to setverbose=Truewhile working interactively with the API.>>>fromedfi_api_clientimportEdFiClient>>>api=EdFiClient(BASE_URL,verbose=True)Clientkeyandsecretnotprovided.ConnectionwithODSwillnotbeattempted.# OR>>>fromedfi_api_clientimportEdFiClient>>>api=EdFiClient(BASE_URL,CLIENT_KEY,CLIENT_SECRET,verbose=True)ConnectiontoODSsuccessful!AttributesAuthentication with the ODS is not required:resourcesresourcesThis method is unavailable in Ed-Fi2.Retrieve a list of namespaced-resources from theresourcesSwagger payload.>>>api.resources[('ed-fi','academicWeeks'),('ed-fi','accounts'),('ed-fi','accountCodes'),...]descriptorsdescriptorsThis method is unavailable in Ed-Fi2.Retrieve a list of namespaced-descriptors from thedescriptorsSwagger payload.>>>api.descriptors[('ed-fi','absenceEventCategoryDescriptors'),('ed-fi','academicHonorCategoryDescriptors'),...]MethodsAuthentication with the ODS is not required:get_infoget_infoThis method is unavailable in Ed-Fi2.Ed-Fi3 provides an informative payload at the ODS base URL.
This contains versioning by suite and build, API mode, and URLs for authentication and data management.>>>api.get_info(){'apiMode':'Shared Instance','build':'2022.6.1.2034','dataModels':[{'name':'Ed-Fi','version':'3.3.0-a'}],'informationalVersion':'5.2','suite':'3','urls':{'dataManagementApi':'{BASE_URL}/data/v3/','dependencies':'{BASE_URL}/metadata/data/v3/dependencies','oauth':'{BASE_URL}/oauth/token','openApiMetadata':'{BASE_URL}/metadata/','xsdMetadata':'{BASE_URL}/metadata/xsd'},'version':'5.2'}get_api_modeget_api_modeThis method is unavailable in Ed-Fi2.Each Ed-Fi3 ODS has a declared API mode that alters how users interact with the ODS. This is a shortcut-method for finding the API mode of the Ed-Fi ODS via the payload retrieved usingEdFiClient.get_info(), formatted in snake_case.>>>api.get_api_mode()'shared_instance'This method is called automatically whenapi_modeis left undefined by the user.get_ods_versionget_ods_versionThis method is unavailable in Ed-Fi2.This is a shortcut-method for finding the version of the Ed-Fi ODS via the payload retrieved usingEdFiClient.get_info().>>>api.get_ods_version()'5.2'get_data_model_versionget_data_model_versionThis method is unavailable in Ed-Fi2.This is a shortcut-method for finding the data model version of the Ed-Fi ODS' 'ed-fi' namespace via the payload retrieved usingEdFiClient.get_info().>>>api.get_data_model_version()'3.3.0-a'get_swaggerget_swaggerThis method is unavailable in Ed-Fi2.The entire Ed-Fi API is outlined in an OpenAPI Specification (i.e., Swagger Specification).
There is a separate Swagger defined for each component type (e.g., resources, descriptors, etc.).Ifcomponentis unspecified,resourceswill be collected.>>>api.get_swagger(component='resources')# Default{'swagger':...,'basePath':...,'consumes':...,'definitions':...,...}Returns anEdFiSwaggerclass containing the complete JSON payload, as well as extracted metadata from the Swagger.is_edfi2is_edfi2This boolean filter returns whether the client-connection to the ODS is via Ed-Fi2.
Ed-Fi3 introduces many new features that are utilized heavily in this package.>>>api.is_edfi2()FalseAuthentication with the ODS is required:get_newest_change_versionget_newest_change_versionThis method is unavailable in Ed-Fi2.This method requires a connection to the ODS.Starting in Ed-Fi3, each row in the ODS is linked to an ODS-wide "change version" parameter, which allows for narrow time-windows of data to be filtered for delta-ingestions, instead of only full-ingestions.
This method returns the newest change version defined in the ODS.>>>api.get_newest_change_version()59084739resourceresourceThis method requires a connection to the ODS.Use this method to initialize an EdFiResource (i.e. EdFiEndpoint).
This object contains methods to pull rows and resource metadata from the API.>>>api.resource(name='students',# Name of resourcenamespace='ed-fi',# Default ; custom resources use a different namespaceget_deletes=False,# Default ; set to `True` to access the /deletes endpointparams={},# Optional; used to pass parameters to API calls**kwargs# Optional; alternative way to pass parameters to API calls)<Resource[edFi/students]>name,params, andkwargscan be formatted insnake_caseorcamelCase.descriptordescriptorThis method requires a connection to the ODS.Use this method to initialize an EdFiResource (i.e. EdFiEndpoint).
This object contains methods to pull rows and descriptor metadata from the API.Note that although descriptors and resources are saved at the same endpoint in the ODS, descriptors do not use their /deletes endpoint.>>>api.descriptor(name='sexDescriptors',# Name of descriptornamespace='ed-fi',# Default ; custom resources use a different namespaceparams={},# Optional; used to pass parameters to API calls**kwargs# Optional; alternative way to pass parameters to API calls)<Resource[edFi/sexDescriptors]>name,params, andkwargscan be formatted insnake_caseorcamelCase.compositecompositeThis method requires a connection to the ODS.Use this method to initialize an EdFiComposite (i.e. EdFiEndpoint).
This object contains methods to pull rows and composite metadata from the API.Note: The only composite currently defined in the API isenrollment.>>>api.composite(name='students',# Name of composite resourcenamespace='ed-fi',# Default ; custom resources use a different namespacecomposite='enrollment',# Default ; name of compositefilter_type=None,# Optional; used to filter composites by ID and typefilter_id=None,# Optional; used to filter composites by ID and typeparams={},# Optional; used to pass parameters to API calls**kwargs# Optional; alternative way to pass parameters to API calls)<EnrollmentComposite[edFi/students]>name,params, andkwargscan be formatted insnake_caseorcamelCase.EdFiEndpointEdFiEndpointis an abstract base class for interfacing with API endpoints.
All methods that returnEdFiEndpointand child classes require a session with the API.>>>students=api.resource('students',min_change_version=52028375,max_change_version=53295015)>>>students<Resourcewith2parameters[edFi/students]># AND/OR>>>students_composite=api.composite('students')>>>students_composite<EnrollmentComposite[edFi/students]>AttributesdescriptiondescriptionThis attribute retrieves the Ed-Fi endpoint's description if present in its respective Swagger payload.>>>api.resource('bellSchedules').description'This entity represents the schedule of class period meeting times.'has_deleteshas_deletesThis attribute returns whether a deletes path is present the Ed-Fi endpoint's respective Swagger payload.>>>api.resource('bellSchedules').has_deletesTrueMethodspingpingThis method pings the endpoint and returns a Response object with scrubbed JSON data.
This offers a shortcut for verifying claim-set permissions without needing to pull data from the ODS.>>>res=students.ping()>>>res<Response[200]>>>>res.json(){'message':'Ping was successful! ODS data has been intentionally scrubbed from this response.'}getgetThis method retrieves one GET-request of JSON rows from the specified endpoint.
This can be used to verify the structure of the data or to collect a small sample for testing.An optional limit can be provided.
If unspecified, the default limit will be retrieved.
(This value must be less than the hard-coded limit of the ODS, or the request will fail.)>>>students.get(limit=1)[GetResource]Endpoint:{BASE_URL}/data/v3/ed-fi/students[GetResource]Parameters:{}[{'id':'abc123','studentUniqueId':'987654','birthDate':'1970-01-01',...}]Because this GET does not use pagination, the return is a list, not a generator.get_rows / get_pagesget_rows / get_pagesThese are the primary methods for retrieving all JSON rows from the specified endpoint and parameters.
The only difference in function is whether the rows are returned individually or in batches (i.e., pages).
Iteration continues until no rows are returned.Both methods use identical arguments.
Under the hood,get_rows()implementsget_pages(), but unnests the rows before returning.>>>student_rows=students.get_rows(page_size=500,# The limit to pass to the parameters. Overwrites parameter if already defined.retry_on_failure=False,# Reconnect session if request fails and reattempt (e.g., if authentication expires).max_retries=5,# If `retry_on_failure is True`, how many attempts before giving up.max_wait=500,# If `retry_on_failure is True`, max wait time for exponential backoff before giving up.step_change_version=False,# Only available for resources/descriptors. See [Change Version Stepping] below.change_version_step_size=50000,# Only available for resources/descriptors. See [Change Version Stepping] below.)<generatorobjectEdFiEndpoint.get_rowsat0x7f7472650f90>>>>list(student_rows)[PagedGetResource]Endpoint:{BASE_URL}/data/v3/ed-fi/students[PagedGetResource]Parameters:{'minChangeVersion':52028375,'maxChangeVersion':53295015,'limit':500,'offset':0}[PagedGetResource]@Retrieved500rows.Pagingoffset...# ...[PagedGetResource]Parameters:{'minChangeVersion':52028375,'maxChangeVersion':53295015,'limit':500,'offset':4000}[PagedGetResource]@Retrieved135rows.Pagingoffset...[PagedGetResource]@Retrievedzerorows.Endingpagination.[{'id':'abc123','studentUniqueId':'987654','birthDate':'1970-01-01',...},...]To circumvent memory constraints, these methods return generators instead of lists.total_counttotal_countThis method returns the total count of rows for the given endpoint, as declared by the API.
This action is completed by sending a limit 0 GET request to the API with theTotal-Countheader set toTrue.>>>students.total_count()4135total_count()is currently only implemented for resources, not composites.Change Version SteppingThe Ed-Fi API already has pagination built-in via thelimitandoffsetparameters passed in GET-requests.Here is an example of what calls to the API look like using pagination (page size 500), charted across time by change versions.This client provides a second type of pagination that uses change versions to improve performance when pulling from the API, referred to here aschange version stepping.A change version window of a specified length is defined, and calls to the API pass the min and max change versions of this window.
Ordinary pagination still occurs within each window until zero rows are returned, after which the change version window steps and the process is repeated.Here is an example of what calls to the API look like using change version stepping (step-window size 2000 and page size 500).Note that change versions are currently accessible only forresources, not for composites.Why is change version stepping recommended when pulling from the API?We can imagine requests sent to the Ed-Fi API as SQL select statements against the underlying ODS.
For example, the code below makes repeated calls to the API, paging by 500 until all rows are retrieved.>>>students=api.resource('students',schoolYear='2022')>>>students.get_rows(page_size=500)This code is semantically identical to the following SQL statements:SELECT*FROMstudentsWHEREschoolYear='2022'LIMIT500OFFSET0;SELECT*FROMstudentsWHEREschoolYear='2022'LIMIT500OFFSET500;SELECT*FROMstudentsWHEREschoolYear='2022'LIMIT500OFFSET1000;-- etc.This works fine for small-volume resources.
However, asoffsetincreases, the computational-runtime of the query increases with it.
For large-volume resources (e.g.studentSectionAttendanceEvents), this could translate to the following:SELECT*FROMstudentSectionAttendanceEventsLIMIT500OFFSET100000000;This is the equivalent of calculating the first 100,000,500 rows of data, but only collecting the final 500.
In practice, the connection to the ODS will time-out and need to re-authenticated before this query returns.Luckily, the Ed-Fi3 "change versions" feature provides a helpful workaround for this.
By specifying a min- and max-change-version in the query, a filtered select is applied that never reaches high offset.>>>students=api.resource('students',min_change_version=0,max_change_version=50000)>>>students.get_rows(page_size=500)By definition, a change-version window will never contain more rows than the size of that window.
Therefore, because the change version window defined above is only 50000 (i.e.,max_change_version - min_change_version),
the final API-call will be equivalent to the following:SELECT*FROMstudentsWHEREchangeVersionBETWEEN0AND50000LIMIT500OFFSET50000Settingstep_change_version = Trueinget_rows()orget_pages()turns on change version stepping.
Usechange_version_step_sizeto set the width of each stepping window (default 50000).>>>students=api.resource('students',min_change_version=52028375,max_change_version=53295015)>>>student_rows=students.get_rows(page_size=500,step_change_version=True,change_version_step_size=50000# Default value. This is NOT optimized. Raise it to reduce API calls.)>>>list(student_rows)[PagedGetResource]Endpoint:{BASE_URL}/data/v3/ed-fi/students[PagedGetResource]Parameters:{'minChangeVersion':52028375,'maxChangeVersion':52078375,'limit':500,'offset':0}[PagedGetResource]@Retrieved101rows.Pagingoffset...[PagedGetResource]Parameters:{'minChangeVersion':52028375,'maxChangeVersion':52078375,'limit':500,'offset':500}[PagedGetResource]@Retrievedzerorows.Steppingchangeversion...[PagedGetResource]Parameters:{'minChangeVersion':52078376,'maxChangeVersion':52128375,'limit':500,'offset':0}[PagedGetResource]@Retrieved500rows.Pagingoffset...# ...[PagedGetResource]Parameters:{'minChangeVersion':53278376,'maxChangeVersion':53295015,'limit':500,'offset':0}[PagedGetResource]@Retrievedzerorows.Steppingchangeversion...[PagedGetResource]@Changeversionexceededmax.Endingpagination.To ingest all rows for a resource, find the ODS' newest change version and apply this tomax_change_version, as below:>>>max_change_version=api.get_newest_change_version()>>>students=api.resource('students',min_change_version=0,max_change_version=max_change_version)>>>students.get_rows(page_size=500,step_change_version=True)Things to note when using change version stepping:Change version stepping usually results in more requests made to the API; however, they are far less likely to overwhelm it as with high offsets.Using change version stepping requires bothmin_change_versionandmax_change_versionbe defined within either the resource'sparamsorkwargs.
If either are undefined, an error is raised.The defaultchange_version_step_sizeis set to50000.
This value is not optimized. Try raising it to send fewer requests to the API.Important Caveat: API De-synchronization when Change-Version SteppingThere is a known problem that can occur when pulling from the API using change-version limits and without snapshotting.
If any rows within the change-version window are updated mid-pull, their change-version is updated and they escape the window.
When this occurs, all other rows in the window shift to fill the place of the missing row, resulting in rows entering previously-pulled offset-windows and being missed in subsequent calls to the API.
This leads to a gradual de-synchronization between the API and datalakes built from the API.We have added a new offset-pagination method to counteract this bug, known as "reverse paging."
By default, whenstep_change_version=Truein resource pulls, requests are made to the API starting at the greatest offset and iterating backwards until offset zero.
If a row is updated and a shift occurs mid-pull, one or more rows in the change version may be ingested multiple times, but no rows will be lost altogether.For example:Say there are 15 rows in thestudentsresource with change versions between 0 and 20.
We pull these rows using a page-size of 4.Say that before our fourth (and final) API call, record number 6 is updated and leaves the change-version window.
Records 7 through 15 will shift to fill its place.
When this occurs, record number 13 will shift from page 4 into a page that has already been ingested.
Therefore, it will be missed from the final output.Using reverse-paging, page 4 will be ingested first. When record number 6 is updated and the rows shift, record 13 will move into page 3 and will be ingested a second time.
However, this row will not be lost.