Package: scrippy-core
scrippy_corescrippy_coreis the mainPython3module of theScrippyframework for the normalization ofPythonscripts writing.This module provides all the basic functionalities such as configuration file management, log and execution history management, script access control, concurrent execution management, etc.PrerequisitesSystemDebian and derivativespython3python3-pippython-devbuild-essentialPython modulesList of required modulesThe modules listed below will be installed automatically.prettytablecoloredlogsargcompletefilelockPyYAMLInstallationCompleteScrippyframework installation and configurationTheScrippyframework and all its packages can be installed with theScrippyinstaller and configuration helper.pipinstallscrippy
scrippyinstallThe commands listed above will install theScrippyscripting framework with default values in the user home directory.Withpipsudopipinstallscrippy-coreManual installation from sourcegitclonehttps://codeberg.org/scrippy/scrippy-core.gitcdscrippy-core
python-mpipinstall-rrequirements.txt
makeinstallEnvironment configuration detailsAt the start of each script,Scrippylooks for its global configuration file at the following locations in the following order:/etc/scrippy/scrippy.yml, allows package mainteners to provide a defaultScrippyconfiguration (i.e:vendor config)~/.config/scrippy/scrippy.yml, allows users to configureScrippywithout requiring administrator permissions/usr/local/etc/scrippy/scrippy.yml, allows the system administrator to configureScrippy.When multiple configuration files exist within the same system, the configuration files are merged so that a user can override a default configuration without being able to override a configuration set up by the system administrator.TheScrippyconfiguration file must define a number of directories that will be useful to scripts based onScrippy.KeyPurposeenv.logdirDirectory of execution logs of scripts based onScrippyenv.histdirDirectory of execution history filesenv.reportdirDirectory of report filesenv.tmpdirTemporary directory for scripts based onScrippyenv.templatedirdirDirectory of template filesenv.confdirDirectory of configuration files for scripts based onScrippyenv.datadirDirectory of data used by scripts based onScrippyModel ofScrippyexecution environment configuration file:env:logdir:/var/log/scrippyhistdir:/var/scrippy/histreportdir:/var/scrippy/reportstmpdir:/var/tmp/scrippydatadir:/var/scrippy/datatemplatedir:/var/scrippy/templatesconfdir:/usr/local/etc/scrippy/confCreation of the directories defined in the configuration file/usr/local/etc/scrippy/scrippy.ymlPython script to create the necessary directories:importosimportyamlconf_file="/usr/local/etc/scrippy/scrippy.yml"withopen(conf_file,"r")asconf:scrippy_conf=yaml.load(conf,Loader=yaml.FullLoader)forrepinscrippy_conf["env"]:os.makedirs(scrippy_conf["env"][rep],0o0775)Enabling auto-completion (optional)If your shell has thewhencecommand (i.ezsh), the argument parser (argparse) can be used to supply auto-completion. Seeargcomplete's documentation.To activate it, launch the following command (installed with theargcompletePython module):sudoactivate-global-python-argcompleteRefresh yourBashenvironment.source/etc/profileFormalismScripts using thescrippy_coremodule must follow a certain formalism in order to ensure standardization of their format while facilitating the implementation of certain advanced functionalities such as the control of the validity of the configuration or the management of optional parameters.Thus each script must include in itsdoc stringa declarative header and a set of predefinedsections.Basic modelThe basic model below can be used as a code snippet:#!/bin/env python3"""--------------------------------------------------------------------------------@author         : <Author>@date           : <Date of current script version>@version        : <X.Y.Z>@description    : <Brief description (one line) of the utility of the script>--------------------------------------------------------------------------------Update:<X.Y.Z>  <Date> - <Author> - <Reason>: <Description of update>--------------------------------------------------------------------------------List of authorized users or group:@user:<username>@group:<group name>--------------------------------------------------------------------------------Concurrent executions:@max_instance: <INT>@timeout: <INT>@exit_on_wait: <BOOL>@exit_on_timeout: <BOOL>--------------------------------------------------------------------------------List of mandatory configuration parameters:@conf:<section>|<key>|<type>|<secret>--------------------------------------------------------------------------------List of execution options and argument parameters:@args:<name>|<type>|<help>|<number of arguments>|<default value>|<conflicts>|<implies>|<required>|<secret>--------------------------------------------------------------------------------Functioning:---------------<Detailed description of the script>..."""#-------------------------------------------------------------------------------#  Initialization of the environment#-------------------------------------------------------------------------------importscrippy_corefromscrippy_coreimportlogger#-------------------------------------------------------------------------------#  Definition of functions and classes#-------------------------------------------------------------------------------class<Class>(<object>):def__init__(self,[<param>,...]):[...]def<function>([<param>,...]):[...]#-------------------------------------------------------------------------------#  Main processing#-------------------------------------------------------------------------------defmain():withscrippy_core.ScriptContext()as_context:# Retrieve config if necessaryconfig=_context.configargs=_context.args[...]#-------------------------------------------------------------------------------#  Entry point#-------------------------------------------------------------------------------if__name__=='__main__':main()Amain()functionmust:Be defined in theDefinition of functions and classessectionBe called in theEntry pointsectionDirectly enclose its contents bywith scrippy_core.ScriptContext() as _context:which manages and activates all the functionalities of scripts written from thescrippy_coremodule.defmain():withscrippy_core.ScriptContext()as_context:# Script environement is accessed through `_context.env`# Script arguments are accessed through `_context.args`# Script configuration are accessed through `_context.config`if__name__=='__main__':main()Header elementsTheauthor,date,version,descriptionelements aremandatoryand will be automatically displayed by the--helpoption.VersionA script's version number is in the format X.Y.Z with:X, the major version identifierYis the minor version identifierZ, the correction version identifierMajor version X: It is worth "0" during development, the script is considered invalid and should not be called by other scripts or used in production.Once the script is validated, the version should be 1.0.0 (first stable version).Xshould be incremented if changes in the code no longer guarantee backward compatibility.The minor version identifiersYand the correction version identifiersZmust be reset to zero when the major version identifierXis incremented.Minor version Y: Must be incremented when adding new functionalities or improving the code that has no impact on backward compatibility.The correction version identifierZmust be reset to zero when the minor version identifier is incremented.Correction version Z: Must be incremented if only retro-compatible corrections are introduced.A correction is defined as an internal change that corrects incorrect behavior (Bug).Zcan be incremented for typographical or grammatical correction.Update:In addition to the version number, modification date, and modification author, each line of the script history must indicate the reason for the modification.<Reason>can take one of the following values:cre: Script creationevo: Script evolution (adding functionality, improving code, etc.)bugfix: Correction of unexpected behavior (bug)typo: Typo correction, adding comments, and any action that does not modify the code.Authorized Users and GroupsThescrippy_coremodule adds a verification layer to the script execution to ensure that a script is executed by a specific user or a user belonging to a specific group.Placed in the header, a line such as@user:harry.finkwill prevent execution by any user other thanharry.fink.It is possible to define several authorized users by multiplying the declarations:@user:harry.fink
@user:luiggi.vercottiIt is also possible to authorize groups of users with a line such as@group:montywhich ensures that only a user from themontygroup executes the script.In the same way as for users, it is possible to multiply group declarations:@group:monty
@group:pythonIn the absence of@userand@groupdeclarative lines, file permissions take precedence, and in all cases, file permissions have priority.Attention:If a group and a user are declared,both conditions must be metfor the user to be authorized to execute the script.Management of Concurrent ExecutionsThe optional declarations@max_instance,@timeout,@exit_on_wait, and@exit_on_timeoutmake it possible to define the number of concurrent executions of the same script as well as the behavior of the script if necessary:DeclarationTypeUtilityDefault value@max_instanceIntegerMaximum number of parallel executions0 (infinite)@timeoutIntegerMaximum waiting time expressed in seconds if@exit_on_timeoutis set to true0 (infinite)@exit_on_timeoutBoolean (true,1,on)Exits the script with an error when the waiting time is reachedFalse@exit_on_waitBoolean (true,1,on)Immediately exits the script with an error in case of waiting, even if the waiting time is not reachedFalseThe waiting occurrences are executed sequentially in the order in which they are recorded in the execution queue.In the following example, two occurrences of the script are allowed. A third execution will be put on hold for 10 seconds, after which the script will exit with an error if it cannot obtain a execution slot.@max_instance: 2
  @timeout: 10
  @exit_on_wait: false
  @exit_on_timeout: trueManagement and Verification of Mandatory Configuration ParametersAny script may have an optional configuration file.This configuration file will be automatically loaded at script start if:located in the directory defined by theenv::confdirvariable set by theScrippymain configuration filehas the same name as the script that must load it, stripped of its extension and suffixed with the.confextension.This way the scriptexp_test_logs.pywill automatically load the configuration fileexp_test_logs.conf.A script configuration file is a simpleinifile divided into as many sections as necessary:[log]level=ERROR[database]host=srv.flying.circusport=5432user=arthur.pewteybase=ministry_of_silly_walkspassword=parrot# The section name may hashave spaces[my section]my variable=my valueIn such a file:Indentation is possible but optionalA line beginning with#or;is considered a comment, even if it is indented.All values are strings:It is up to the developer to convert the variable value to the desired type during processing (seeRetrieving a Value of a Particular Type).Spaces are accepted in the section name, key name, or value.Validating the configuration fileIn order to validate the configuration file, the script must contain a set of lines starting with@confand describing the configuration file in itsdocstring.The declaration lines of the configuration format must respect the following format:@conf:<section>|<key>|<value_type>[|<secret>]<value_type>must be one of the following recognized types:str(string of characters)int(integer)float(floating point number)bool(boolean)secretisoptionaland if defined, must have the valuetrueorfalse.Ifsecretis defined and has the valuetrue, the configuration parameter value will be considered asecretand will be redacted in the logging files.Example:From the following declaration:@conf:log|level|str
@conf:database|port|int
@conf:sql|verbose|boolean
@conf:sql|database|str|false
@conf:sql|password|str|trueThe following configuration file will be checked:[log]
  level = error
[database]
  port = 5432
[sql]
  verbose = True
  database = monty
  password = d34dp4rr0tAll occurrences of the passwordd34dp4rr0twill be replaced by*******in the logging files and on the standard output.No control over the values of the parameters is performed, so there is no need to indicate them. Only the structure of the configuration and the type of the keys are checked.During the configuration check and if the logging level is set todebug, the loaded configuration will be displayed on the standard output and in the log (beware of the presence of passwords in the configuration when using thedebuglogging level)."""@conf:database|port|int@conf:database|base|str@conf:database|host|str@conf:database|password|str@conf:database|user|str@conf:local|dir|str@conf:src|port|int@conf:src|host|str@conf:src|dir|str@conf:src|user|str@conf:dst|port|int@conf:dst|host|str@conf:dst|dir|str@conf:dst|user|str"""importscrippy_corefromscrippy_coreimportloggerwithscrippy_core.ScriptContext()as_context:# retrieving the configurationconfig=_context.configIf one of the sections or keys described by@confis absent from the configuration file or if the type described for a key does not match the type found in the configuration file for that key, a critical error is raised and the script immediately exits with an error.Extra sections or keys found in the configuration file that are not declared will simply be ignored during validation but will remain available to the script.Thus in the above example, the[mail]section not being defined in@confneither its presence nor its validity will be checked.Retrieving a configuration parameter valueRetrieving a parameter value from the configuration file is done through the_context.config.get()method."""@conf:database|port|int@conf:database|base|str@conf:database|host|str@conf:database|password|str@conf:database|user|str"""importloggingimportscrippy_corefromscrippy_coreimportloggerwithscrippy_core.ScriptContext()as_context:config=_context.configlogger.info(config.get('database','host'))In the above example, the value of thehostkey in thedatabasesection will be displayed on the screen.If the section or key does not exist, an error is raised andthe script will immediately raise a critical error.Retrieving a value of a particular typeUnless the 'param_type' parameter is set to one of the authorized values (str(default),int,floatorbool), the returned type is always a string.Calling theConfig.get()method with the wrong type will raise an error andthe script will immediately raise a critical error."""@conf:log|level|str@conf:database|port|int@conf:database|base|str@conf:database|host|str@conf:database|password|str@conf:database|user|str"""importscrippy_corefromscrippy_coreimportloggerwithscrippy_core.ScriptContext()as_context:config=_context.configlogger.info(config.get('database','port','int'))Reserved Sections and Keys:Some sections and keys are automatically read and interpreted during the import of thescrippy_coremodule.These configuration keys are optional, just like the configuration file.Log level, read and applied automatically:[log]level=<str>Activation of logging and history (True by default):[log]file=<bool>More details in theRuntime Logging Managementsection.ExamplesAll examples in this documentation are based on all or part of the following configuration file:[log]level=infofile=true[database]host=srv.flying.circusport=5432user=arthur.pewteybase=ministry_of_silly_walkspassword=dead parrot[local]dir=/tmp/transfert[src]host=srv.source.circusport=22user=harry.finkdir=/home/harry.fink/data[dst]host=srv.destination.circusport=22user=luigi.vercottidir=/home/luigi.vercotti/received[mail]host=srv.mail.circusport=25from=Luiggi Vercottifrom_addr=luiggi.vercotti@circus.comto=Harry Finkto_addr=harry.fink@circus.comsubject=Execution ReportExecution options managementScript options can be managed through declarations in thedocstring.The declaration of an option is composed of 8 fields, some of which are required:@args:<name>|<type>|<help>|<number of arguments>|<default value>|<conflicts>|<implies>|<required>|<secret>with:name: The name of the option (required)type: One of the following values:str,int,float,choice,bool(default: str). If the type ischoice, the list of possible choices must be entered in thedefault valuefield as a list of words separated by commas.help: A string summarising the purpose of this option (required).number of arguments: The number of arguments taken by the option. This field is mandatory for all types exceptbool, where the declared number of arguments is ignored. Its value is usually an integer but can take the value+when the number of arguments is greater than 1 but is not known in advance, or?when the number of arguments can be equal to zero.default value: The default value of the option (optional).trueis the default value forbooloptions.conflictx: The list of options that conflict with the current option (optional, list of options separated by commas).implies: The list of options induced by the current option (optional, list of options separated by commas).required: A Boolean (trueorfalse) indicating whether the option is mandatory or not.secret: A Boolean (trueorfalse) indicating whether the option value should be kept secret.The value of an option whosesecretattribute is set totruewill be automatically masked in log files as well as in standard output.The declaration of the options will automatically generate the help of the script accessible with the--helpoption.The following options will also be automatically generated:--version: Displays the script version number from the information contained in the header.--source-code: Displays the source code of the script.--hist [NB_EXECUTION (default:10)]: Displays the execution history of the script.--log [SESSION]: Displays the content of the log file corresponding to the session whose identifier is passed as an argument. By default, the latest session is displayed.--debug: Forces the logging level to DEBUG (Changes in log level during execution are then ignored).--no-stdout: Disable log output to standard output except forerrorandcritialmessages.--no-log-file: Prevents the creation of log and history files.The--help,--version,--hist,--log,--debug,--no-stdoutand--no-log-fileoptions should not be declared.ExamplesThe following script can be called with the following options and arguments:--env: Mandatory. Accepts one of the following values: qualif, preprod or prod--appname: Optional. A free string with a default value of "aviva"--now: Boolean option. If used, its value will betrue(false by default) and the--dateoption cannot be used.--date: A sequence of 3 integers from which a date will be created (i.e. 24 02 2019). This option conflicts with the--nowoption.--email: A free string. This option is mandatory if the--nowoption is used.#!/bin/env python3"""--------------------------------------------------------------------------------@author         : Florent Chevalier@date           : 2019-07-27@version        : 1.0.0@description    : test arguments--------------------------------------------------------------------------------Update :1.0.0  2019-07-27   - Florent Chevalier   - Cre : Production--------------------------------------------------------------------------------@args:appname|str|Application name|1|aviva|||false@args:date|int|Planning date(day, month, year)|3||now|email|false@args:email|str|Notification email address|1||||false@args:now|bool|Apply immediately||false|date||false--------------------------------------------------------------------------------"""importdatetimeimportscrippy_corefromscrippy_coreimportloggerdefprint_contact(email):logger.info(" - Contact:{}".format(email))defprint_appname(appname):logger.info(" - Application:{}".format(appname))defprint_date(date):logger.info(" - Date:{}".format(date))defmain():withscrippy_core.ScriptContext()as_context:args=_context.argsifargs.date:date="{}/{}/{}".format(args.date[0],args.date[1],args.date[2])ifargs.now:date=datetime.datetime.now().strftime('%d/%m/%Y')logger.info("[+] Report:")print_date(date)print_appname(args.appname)print_contact(args.email)if__name__=='__main__':main()Execution log managementExecution logging is done using thescrippy_core.loggerobject.Two types of logs are simultaneously available:Standard output: Colored display towardssys.stdoutLog file: A file located in log directory defined by theenv::logdirvariable set by theScrippymain configuration file.The name of the log file is extrapolated from the name of the script as follows:<script_name>_<timestamp>_<pid>.log.If a configuration file exists for the script and it contains a[log]section indicating a log level with thelevelkey, then the indicated log level is automatically applied.If the configuration file contains a[log]section with afilekey whose value isfalse, then no log file will be created and only standard output will receive the logs.Defining the log level through the configuration file:[log]
  level = warningThe value of the log level in the configuration file is case-insensitive.The available log levels, from least verbose to most verbose, arethe logging levels of the standard logging modulecriticalerrorwarninginfodebugNote that thedebuglog level displays the entire configuration file as well as other details that could prove to be a source of information leakage. It is not recommended to use it in production.All scripts written using thescrippy_coremodule have the following logging options:--no-log-file: When this option is used, no execution log is recorded on disk. This option does not prevent output on the screen.--debug: When this option is used, the log level is forced toDEBUG. In this case, the script does not take into account a possible configuration parameter indicating the opposite.These two options can be combined and will override their equivalent defined in the configuration file.Error managementTheScrippyframework is intended to be used in production environments where debug trace or even stack traces should be avoided.For the above reason, if an exception is not handled and propagates up to thescript context, asys.exit(1)will be triggered with a brief error messagewith no stack trace.To display the stack trace, the log level must be set todebug.Please, debug your scripts in testing environment !Execution historyThe execution history file is located in the history directory defined by theenv::histdirvariable set by theScrippymain configuration fileThe execution history file will be created and named<script_name>.dbat the first script execution.This file is ansqlite3database that lists all executions of a script.For each execution the following information are stored:session identifierStart date of the executionEnd date of the executionDuration of the executionUser who initiated the executionUser who actually ran the script (in case ofsudo)Return code of the execution (0 if OK, other value if KO)Set of parameters passed as arguments to the scriptName of the error if the execution did not end correctly (0 otherwise)If the history file already exists at execution time, it will be updated with the parameters of the new execution.History is automatically enabled by enclosing the call to themainfunction with the declarationwith scrippy_core.ScriptContext() as _context:withscrippy_core.ScriptContext()as_context:main()Each script execution is assigned a session allowing each execution to be uniquely identified.This session is composed of:atimestamprepresenting the execution timethe process identifier (PID)Exemple:1568975414.6954327_10580This session identifier is reported in theSessioncolumn of the history and allows the corresponding log to be found (See the--logoption inExecution options management).RetentionThe number of executions kept in the history file is90by default.It is possible to override this value by specifying the desired retention number using the declarationwith scrippy_core.ScriptContext(retention=100) as _context:withscrippy_core.ScriptContext(retention=100)as_context:config=_context.configargs=_context.args<...>Displaying execution historyAll scripts based on thescrippy_coremodule have an--histoption that allows you to display the last executions.exp_test_script.py--histThe number of executions to display can be specified by passing an integer parameter to the--histoption.exp_test_script.py--hist2For each script execution, the execution history records the following information:The execution dateThe original userThe effective user (sudo)The unique session identifierThe exit code (0 by default)The list of options and arguments passed to the scriptTemporary workspaceThe declarationwith scrippy_core.ScriptContext() as _context:automatically creates a temporary workspace that willautomatically be destroyed with its contents at the end of the script.The workspace is located in the temporary directory defined by theenv::tmpdirvariable set by the [Scrippymain configuration file].The exact path to this workspace can be obtained using theworkspace_pathattribute of the execution context_context.withscrippy_core.ScriptContext()as_context:workspace_path=_context.workspace_path...In the previous example, theworkspace_pathvariable will contain the path to the temporary working directory, whose name will be constructed as follows:scrippy_core.SCRIPPY_TMPDIR/<SCRIPT NAME>_<SESSION ID>Exemple:/var/tmp/scrippy/exp_transfert_ftp_1574391960.6696503_102257This temporary workspace, which will beautomatically destroyed with its contents at the end of the script, is a directory that can be used to create temporary files used by the script.#!/bin/env python3importosimportscrippy_corefromscrippy_coreimportloggerdefcreate_file(workspace_path):tmp_file="fichier.tmp"logger.info("[+] Creating temporary file:{}".format(os.path.join(workspace_path,tmp_file)))withopen(os.path.join(workspace_path,tmp_file),'a')astmpfile:logger.info("[+] Writing in temporary file")tmpfile.write("Nobody expects the Spanish inquisition !")defmain():withscrippy_core.ScriptContext()as_context:config=_context.configcreate_file(_context.workspace_path)if__name__=='__main__':main()Accessing the environmentThe environement is accessible through the_context.envdictionnary.It contains all the initial environement variables that may be useful to scripts.#!/bin/env python3importosimportscrippy_corefromscrippy_coreimportloggerdefcreate_report(reportdir,filename):report_filename=os.join(reportdir,filename)logger.info(f"[+] Creating report file:{report_filename}")withopen(report_filename),'a')asreport:logger.info("[+] Writing report file")report.write("Nobody expects the Spanish inquisition !")defmain():withscrippy_core.ScriptContext()as_context:config=_context.configcreate_report(_context.env.get("reportdir"),"inquisition.txt")if__name__=='__main__':main()Tips and guidelinesA log isnever too verbosePriority should be given to readabilityand maintainability rather than compactness and technicalityBreak down the code intosmall functionsEach function should log its entry pointand where possible, the parameters it receivesMaximize variable namingand transfer as many variables as possible to the configuration file or run-time options (no hard coded values)Simplify the main algorithmof the program to its simplest expressionHandle errors as finely as possibleand as close as possibleAvoid global variables at all costMinimize information leaksin log files by using thesecretattributes of options and arguments whose values are sensitive information (login, password, connection server, etc.)Create amain()functioncontaining the script's main algorithmCreate the environment (objects and variables used at the global level) in theEntry pointsection.Additional modulesTheScrippyframework, of whichscrippy-coreis the core, has modules that facilitate the writing of advancedPythonscripts in accordance with Scrippy's basic principles.ModuleUtilityscrippy-templateTemplate file management (based onJinja2)scrippy-remoteImplementation ofSSH/SFTPandFTPprotocolsscrippy-mailImplementation ofSMTP,POPandSpamassassinprotocolsscrippy-gitGit repository managementscrippy-dbDatabase management (PostgreSQLandOracle)scrippy-apiUsingREST API(based onresquets)All available modules are available athttps://codeberg.org/scrippy