Package: makeit
MAKE IT!GNU Make for applications.Extensible execution model languageExpressive and concise task description languageTask are not deferred to remoteBuild-in async supportRationaleRethinking of Doit package.Similar libraries -- what's differentSee Luigi, Airflow, doit.Main conceptsTwo main concepts of the library are Task and Artifact.
A task depends on artifacts and produces artifacts.The use of artifacts is twofold. First, artifacts are used to build a graph (DAG) of the execution.
On the other side, artifacts' fingerprints (i.e., md5) are used to define weather task should be run or not.Task definitionThe simplest way to define a task is to derive a dataclass class from makeit.DataclassTask
and implement methodexecute, as in the example below.The class makeit.DataclassTask closely interweaves with the standard dataclasses library,
allowing users to benefit from a rich dataclasses support from IDEs.One implicit dependency is the source code if a task (obtained by inspect.getsource(task.class)),
if variable DataclassTask.depends_on_source_code_of_self is set (it is set by default).
However, this is not available in the environments like Jupyter.importdataclassesfrommakeitimportFile,Dependency,Target,DataclassTaskfrompathlibimportPath@dataclasses.dataclassclassProcess(DataclassTask):# derive from DataclassTask -- this adds required and additional functions like dependencies, ...input_1:Path|Dependency# this is a file dependencyinput_2:Path|Dependency# many dependencies are possiblemany_files:list[Path]|Dependency# and even list of dependencies allowedparameter:float# this is a parameter -- not a dependency or a target# however, it is used in label() method (via dataclasses method __str__),# therefore it helps to parametrize tasks.# See also method DataclassTask.md5 -- it is helpful for creating unique target names.target:File|Target# many targets are possible as welltarget_2:File|Target=None# this can be initialized in the dataclasses default method __post_init__class_variable=12# this is a class variable -- makeit disregards it, as dataclasses does.def__post_init__(self):# our task has parameters (parameter: float)# -- so it is better to save results into a unique target (file in this example)# DataclassTask.md5 is roughly equivalent to md5hash(str(self)).self.target_2=Path('datafolder')/self.md5(".csv")defexecute(self):# Implement task's logictext=self.input_1.read_text()+self.input_2.read_text()self.target.path.write_text(text)Tasks executionThe simplest way to execute tasks is to use the functionexecute_on_change:frommakeitimportexecute_on_changetasks=[...]execute_on_change(tasks,backend="makeit.json")Filebackendserves as a storage for execution. Creates one, if it does not exist.Reporteris used to log DAG execution, default implementation prints out colorized output to stderr.
None can be used to make it silent.Setdag_nameif you have several dags in one project, and you want to avoid names clashing.If parametercapture_onlyis set, tasks are not executed, only fingerprints are captured
and stored in the backend. Can not be set simultaneously with execute_all.If parameterexecute_allis set, all fingerprints are disregarded, everything will be executed.
The advantage overexecute_alwaysis that all fingerprints are stored in the backend.Parametertest_targets_fingerprintscontrols checking of targets fingerprints. If it is set and
a target has been modified externally, the task will be executed.Parameterstrictcontrols tasks execution. If target is not created or dependency does not exist,
exception is raised.Execution reportingThere is a build-in logging execution reporter.Example withTqdmReporterfrommakeit.coreimportReporter,TaskEventclassTqdmReporter(Reporter):defreport_task_event(self,event:TaskEvent,task_name:str,reason:str|None):passdefon_dag_start(self,dag_name:str,total_number_of_tasks:int):passdefon_dag_end(self):passUsing graphviz to plot DAGfrommakeit.contrib.graphvizimportrender_online,create_dottasks=[...]dot=create_dot(tasks)# graphviz.Digraph can be rendered to pdf, png, ...render_online(dot,'https://...')# ... or simply drawn onlineSome remarks:dotted lines connect task with its parametersfile artifacts are marked by a folder-shaped boxgreen or red color denote weather file exists or notlabels are shortened as much as possible in order to get readable pictureAdvanced tasks executionUsemakeit.mp_execute_on_changeto run tasks in multiprocessing. This function has almoust the
same signature asexecute_on_change, however it has additional parametern_jobscontrolling number of
worker processes.Usemakeit.aio_execute_on_changeto
run async tasks (in this case methodasync def a_execute(self)should be implemented instead).
This function has also own specific parameters controlling queue size and number of async tasks.Advanced task definitionRegistering a type conversion.Subclass directly frommakeit.core.Taskand implement all needed abstract methods.Extra dependencies/targets. Task dependencies.Typical workflowsWorkflowDescriptionBackend roleMakeExecute task if targets are older than dependencies-"Luigi"Execute task if and only if target does not exist-Make 2.0Execute task if dependency has changedFingerprint is stored in backend, i.e. MD5 or modification timeLuigi 2.0Pickup new files from folder, parse and processBackend remembers which files have been already seenAlwaysExecute all tasks (in correct order)-MakeCompare files' timestamps, if dependencies are newer than targets then recalculate the targets."Luigi"Run task if and only if targets are absent.