Package: acnestis
Acnestis - collect, aggregate and convertA very simple tool that allows you collect data from different sources, change it ia different waysInstall$pipinstallacnestisYou can do in declarative wayconvert data in the current folderaggregate data from different sourcesinheritence - when you collect data from different source - you can replace any file and those files will be used in the child repoOk, this is confusing. Show me some examplefirst of all, you can find some examples in thetests/datafolder(source folders, or from-folders, may contain both yaml and py files, that doesn't mean both are required. We just make alternatives for testing and illustration)Acnestis declorationin order to declarate folder as acnestis folder you should do one of the followingcreate.acnestis.yamlfile in that foldercreate.acnestis.pyfile in that folder. Py-file has an identical functionality, but some more power (of course)create both.asnestis.yamland.acnestis.py- in that case yaml-file will be used as the main one, but can use declared global variables from the py-filedeclarate some of the sub-folders (even non-existed), in the current acnestis-declorationIn order to process acnestis folders - you should do$acnestingprocessfrom/folderto/folderif no acnestis decloration in thefrom/folderit will just copy files toto/folder. iffrom/folderhas at least one acnestis decloration - it will be used for processing that folder, the rest will be copiedSimple convertionFrom:tests/data/002_concat_poemto:tests/data/002_concat_poem_resultthis is a very simple example, where all files from folder poem.txt will be connected into a single filewe declaratepoem.txtfolder as acnestis folderPy version:fromacnestis.processingimportProcessorfromacnestis.stepsimportconcat_filesPROCESSOR=Processor([concat_files("poem.txt")],as_file="poem.txt")YAML version:steps:-name:concat_filesinto_file:poem.txtas_file:poem.txtIt has one single stepconcat_fileswith attributeinto_file: poem.txt- so it simply concat all the files into oneAlso processing has attributeas_filemeans the folder will be replaced with one filepoem.txtDocker for more complex convertion*Fromtests/data/006_docker_svgoto:tests/data/006_docker_svgo_resultyou can build own convertion tools using dockerPy version:fromacnestis.processingimportProcessorfromacnestis.stepsimportdockerPROCESSOR=Processor([docker("acnestis_svgo",skip_pull=True)],)YAML version:steps:-name:dockerimage:acnestis_svgoskip_pull:trueone single step of using docker imageacnestis_svgofor processing acnestis folder.Dockerfile of acnestis_svgois very simple:# Base imageFROMnode:20-alpine# Install SVGORUNnpminstall-gsvgo# Execute the SVGO command when the container startsENTRYPOINT["svgo","-f","/data_input","-o","/data_output"]you can use simple python-code for convertionFromtests/data/004_codetotests/data/004_code_resultNow let's play with aggregation.we want not only convert data, but first collect it from different sourcesFor the testing and examples we will useoduvan/acnestis-test-repogithub repository and its branches.simple use of git-repoFrom:data/007_gittodata/007_git_resultas any folders in the given repo are declarated as acnestis - we will see a simple copy files from the repoPy version:fromacnestis.processingimportProcessorfromacnestis.stepsimportgitPROCESSOR=Processor([git("https://github.com/oduvan/acnestis-test-repo.git",branch="main")],)YAML version:steps:-name:giturl:https://github.com/oduvan/acnestis-test-repo.gitbranch:maininjection in the childFrom:tests/data/008_git_processingto:tests/data/008_git_processing_resultYAML version:steps:-name:giturl:https://github.com/oduvan/acnestis-test-repo.gitis very simple, but the master branch of the repo contains an acnestis-folder, which means it will be processed after checkout into the current one.During the checkoutpoem.txtfolder will be created and all of the files in the folder will be connectedbut in the current repo we created a folder poem.txt with file3.txtso that file become a part of processing processMore complex examples:for more example checktests/datafolder009_git_subfolder- we don't need all of the files from the git repo, but only specific folder010_subprocess_git- we declarate a specific acnestis folder for processing011_sub_git_copy- using of copy and rm steps013_simple_aggregateand014_aggregate_two_poems- more complex aggregation examples