Package: lokii
lokiiis a powerful package that enables the generation of relational datasets, specifically tailored to
facilitate the creation of robust development environments. Withlokii, you can effortlessly generate diverse
datasets that mimic real-world scenarios, allowing for comprehensive end-to-end testing of your applications.Project structurelokiileverages the hierarchical structure of the file system to discover groups and nodes. Each dataset
consists of nodes, which are defined using.node.pyfiles. For instance, in the context of a database, each
node represents a table. Furthermore, you can even group nodes under database schemas within the database. Groups
defines how generated node data will be exported. You can recognize group files by their.group.pyfile extension.# example project directory structureproj_dir├──group_1│├──group_1.group.py│├──node_2.node.py│└──node_2.node.py├──group_2│├──node_3.node.py│└──node_4.node.py├──group_3.group.py├──node_5.node.py└──node_6.node.pyNode DefinitionNode file defines how each item will be generated. There are special variables and functions in node
definition files.name: Name of the node, filename will be used if not providedsource: Source query for retrieve dependent parameters for each itemitem: Generation function that will return each item in node# offices.node.pyfromfakerimportFaker# use your favorite tools to generate data# you can even use database connection, filesystem or AIfake=Faker()# if you want you can override the node name if not provided filename will be used# can be used in source queries if you want to retrieve rows that depends on another node# name = "business.offices"# define a query that returns one or more rowssource="SELECT * FROM range(10)"# item function will be called for each row in `source` query resultdefitem(args):address=fake.address().split("\n")return{"officeCode":args["id"],"city":fake.city(),"phone":fake.phone_number(),"addressLine1":address[0],"addressLine2":address[1],"state":fake.city(),"country":fake.country(),"postalCode":fake.postcode(),"territory":fake.administrative_unit(),}Group DefinitionGroup file defines how each node data will be exported. There are special functions in group definition files.before: Called once before export operationexport: Called for every node in the groupafter: Called once after export operation# filesystem.group.pyimportosimportshutilfromcsvimportDictWriterout_path="out_data"defbefore(args):"""Executed before export function.:param args: contains node names that belongs to this group:type args: {"nodes": list[str]}"""ifos.path.exists(out_path):# always clear your storage before starting a new exportshutil.rmtree(out_path)os.makedirs(out_path)defexport(args):"""Executed for all nodes that belongs to this group:param args: contains node name, node columns and a batch iterator:type args: {"name": str, "cols": list[str], "batches": list[dict]}"""node_name=args["name"]node_cols=args["cols"]batches=args["batches"]# out_data/offices.csvout_file_path=os.path.join(out_path,node_name+".csv")withopen(out_file_path,'w+',newline='',encoding='utf-8')asoutfile:writer=DictWriter(outfile,fieldnames=node_cols)writer.writeheader()forbatchinbatches:writer.writerows(batch)defafter(args):"""Executed after export function.:param args: contains node names that belongs to this group:type args: {"nodes": list[str]}"""passUpload to PyPIYou can create the source distribution of the package by running the command given below:python3setup.pysdistInstall twine and upload pypi forfinnetdevlabusername.pip3installtwine
twineuploaddist/*RequirementsPackage requirements are handled using pip. To install them dopip install -r requirements.txt
pip install -r requirements.dev.txtTestsTesting is set up usingpytestand coverage is handled
with the pytest-cov plugin.Run your tests withpy.testin the root directory.Coverage is run by default and is set in thepytest.inifile.