Package: edfi-lms-ds-loader
lms-ds-loaderThe Ed-Fi LMS Data Store Loader is a utility for loading CSV files in the
Learning Management System Unified Data Model (LMS-UDM) into a Learning
Management System Data Store (LMS-DS) database.The application processes each file in the input file system by date order, as
indicated in the file name. If a record is in a file one day, and missing on the
next day, then the system "soft deletes" that record by setting the current
timestamp into thedeletedatcolumn. Similarly, if a previously soft deleted record
reappears later, the record is "un-soft deleted" and updated with any new values.This functionality requires that a root level directory only contains files for
one LMS provider. Thus if an education organization uses multiple LMS providers,
then each LMS Extractor needs to write files to a separate, dedicated directory,
and the LMS DS Loader must be run once for each extractor's output directory.What's NewVersion 1.1:The SQL script names have been modified so that it is easier to add
new scripts in the future. If you have previously run version 1.0.0,
then you will need to run the following script once manually before
running the 1.1.x version of this tool:SQL Update Scriptupdate lms.MigrationJournal set script = '0001_initialize_lms_database' where script = 'initialize_lms_database';
update lms.MigrationJournal set script = '0002_create_processed_files_table' where script = 'create_processed_files_table';
update lms.MigrationJournal set script = '0003_create_user_tables' where script = 'create_user_tables';
update lms.MigrationJournal set script = '0004_create_section_tables' where script = 'create_section_tables';
update lms.MigrationJournal set script = '0005_create_assignment_tables' where script = 'create_assignment_tables';
update lms.MigrationJournal set script = '0006_create_section_association_tables' where script = 'create_section_association_tables';
update lms.MigrationJournal set script = '0007_create_assignment_submission_tables' where script = 'create_assignment_submission_tables';
update lms.MigrationJournal set script = '0008_create_section_activity_tables' where script = 'create_section_activity_tables';
update lms.MigrationJournal set script = '0009_create_system_activity_tables' where script = 'create_system_activity_tables';
update lms.MigrationJournal set script = '0010_create_attendance_tables' where script = 'create_attendance_tables';
update lms.MigrationJournal set script = '0011_remove_startdate_enddate_from_sectionassociation' where script = 'remove_startdate_enddate_from_sectionassociation';
update lms.MigrationJournal set script = '0012_add_mapping_columns_for_edfi_student_and_section' where script = 'add_mapping_columns_for_edfi_student_and_section';Limitations as of November 2021PostgreSQL support currently ony covers the Users file. Other files will soon be supported.Getting StartedSQL Server support requires the Microsoft ODBC 17 driver, which is newer than
the ones that come with most operating systems.Windows:choco install sqlserver-odbcdriverLinux
instructions(has not been tested yet)Requires Python 3.9+ and Poetry.Install required Python packages:poetryinstallThe database account used when running the tool needs to be a member of the
following roles in the destination database:db_datareaderdb_datawriterdb_ddladminNote that the tool automatically manages deployment of the LMS-DS tables into
the destination database, under thelmsschema.ConfigurationSupported parameters:DescriptionRequiredCommand Line ArgumentEnvironment VariableCSV file pathyes-cor--csvpathCSV_PATHDatabase Engine ("mssql" or "postgresql")no (default: mssql)-eor--engineDB_ENGINEDB Serveryes-sor--serverDB_SERVERDB Portno (default: 1433)--portDB_PORTDB Nameyes-dor--dbnameDB_NAMEDB Username **no (no default)-uor--usernameDB_USERNAMEDB Password **no (no default)-por--passwordDB_PASSWORDUse integrated security **no (default: false)-ior--useintegratedsecurityUSE_INTEGRATED_SECURITYLog level*no (default: INFO)-lor--log-levelLOG_LEVELEncrypt db connectionno (default: False)-nor--encryptENCRYPT_SQL_CONNECTIONTrust db server certificateno (default: False)-tor--trust-certificateTRUST_SERVER_CERTIFICATE* Valid values for the optionallog level:DEBUGINFO(default)WARNINGERRORCRITICAL** If using integrated security, DB Username and password won't be required,
otherwise they are required.Running the ToolFor detailed help, executepoetry run python edfi_lms_ds_loader -h.Sample call using full integrated security, loading from the sample files
directory:poetryrunpythonedfi_lms_ds_loader\--serverlocalhost\--dbnamelms_toolkit\--useintegratedsecurity\--csvpath../../docs/sample-outDeveloper NotesDatabase-Enabled Integration TestsThis package contains SQL Server-integrated tests, in addition to the standard
unit tests. The specialized tests are in thetests_integration_mssqldirectory
and they will run automatically if you callpytestwith no arguments.Before running the tests, create an empty SQL Server database named
"test_integration_lms_toolkit":sqlcmd -Q "create database test_integration_lms_toolkit". This is required to run the SQL Server
integration pytests.To run only theunit tests:poetry run pytest tests. To run only the
integration tests,poetry run pytest tests_integration_mssql.DockerIf you do not have local installations of Microsoft SQL Server (MSSQL) or
PostgreSQL, then you can try running them in Docker. See theeng/dockerfolder for more information. Integration tests
currently only support MSSQL.To create the test database database after starting the MSSQL Docker container:dockerexeclms_toolkit_mssql/opt/mssql-tools/bin/sqlcmd-Slocalhost-USA-P'<a good password>'-Q'create database test_integration_lms_toolkit'Writing New TestsPlease note and follow the existing pattern for database-enabled integration
tests, as this pattern was carefully tuned to support use of database
transactions for rolling back changes and thereby leaving behind a clean
database.Dev OperationsStyle check:poetry run flake8Static typing check:poetry run mypy .Run unit tests:poetry run pytestRun unit tests with code coverage:poetry run coverage run -m pytestView code coverage:poetry run coverage reportAlso seebuild.pyfor
use of the build script.Adding New MigrationsCreate SQL Server and PostgreSQL SQL scripts underedfi_lms_ds_loader/scripts/<engine name>, using the same file name for
both. Start from the artifact created by MetaEd, and then update the script
with these modifications:Remove theIdcolumn and the default constraint for that column.Add aDeletedAtcolumn as a nullableDatetime2.Duplicate the table definition and prefix the duplicate with "stg_".In the staging table, change thexyzIdentifierprimary key column name toStagingId, and leave out theDeletedAtcolumn.Use;(semi-colon) terminators at the end of each SQL statement for both
languages. Do not useGOin the SQL Server files, as the application is not
coded to parse it.Add the new script name to theMIGRATION_SCRIPTSconstant at the top ofedfi_lms_ds_loader/migrator.py.Adding New Files UploadsCreate the required table and staging table in a new migration.Ensure that thefile-utilsshared library correctly maps the data types for
the new file.Update theedfi_lms_ds_loader/loader_facade.pyto pull in the additional
file type and upload it.Legal InformationCopyright (c) 2022 Ed-Fi Alliance, LLC and contributors.Licensed under theApache License, Version 2.0(the "License").Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.SeeNOTICESfor
additional copyright and license notifications.