Package: keboola.component
This library provides a Python wrapper over theKeboola Common Interface. It simplifies all tasks related to
the communication of theDocker componentwith the Keboola
Connection that is defined by the Common Interface. Such tasks are config manipulation, validation, component state, I/O
handling, I/O metadata and manifest files, logging, etc.It is being developed by the Keboola Data Services team and officially supported by Keboola. It aims to simplify the
Keboola Component creation process, by removing the necessity of writing boilerplate code to manipulate with the Common
Interface.Another useful use-case is within the KeboolaPython Transformationsto simplify the I/O handling.LinksAPI Documentation:API docsSource code:https://github.com/keboola/python-componentPYPI project
code:https://test.pypi.org/project/keboola.component-kds/Documentation:https://developers.keboola.com/extend/component/python-component-libraryPython Component Cookiecutter template
project:https://bitbucket.org/kds_consulting_team/cookiecutter-python-componentQuick startInstallationThe package may be installed via PIP:pip install keboola.componentCore structure & functionalityThe package contains two core modules:keboola.component.interface- Core methods and class to initialize and handle
theKeboola Common Interfacetaskskeboola.component.dao- Data classes and containers for objects defined by the Common Interface such as manifest
files, metadata, environment variables, etc.keboola.component.base- Base classes to build the Keboola Component applications from.CommonInterfaceCore class that serves to initialize the docker environment. It handles the following tasks:Environment initialisationLoading
allenvironment variablesLoading theconfiguration fileand
initialization of thedata folderState fileprocessing.LoggingData foldermanipulationManifest fileprocessingConfig validationMetadata manipulationOAuthconfiguration handling.InitializationThe core class iskeboola.component.interface.CommonInterface, upon it's initialization the environment is created.
e.g.data folder initialized (either from the Environment Variable or manually)config.json is loadedAll Environment variables are loadedThe optional parameterdata_folder_pathof the constructor is the path to the data directory. If not
provided,KBC_DATADIRenvironment variablewill be
used.The class can be either extended or just instantiated and manipulated like object. TheCommonInterfaceclass is
exposed in thekeboola.componentnamespace:fromkeboola.componentimportCommonInterface# init the interface# A ValueError error is raised if the KBC_DATADIR does not exist or contains non-existent path.ci=CommonInterface()To specify the data folder path manually use this code:fromkeboola.componentimportCommonInterface# init the interface# A ValueError error is raised if the data folder path does not exist.ci=CommonInterface(data_folder_path='/data')Loading configuration parameters:The below example loads initializes the common interface class and automatically loading config.json from thedata folderwhich is defined by an environment
variableKBC_DATADIR, if the variable is not present, and error is raised. To override the data folder location
provide thedata_folder_pathparameter into constructor.NOTE:Theconfigurationobject is initialized upon access and a ValueError is thrown if theconfig.jsondoes not
exist in the data folder. e.g.cfg = ci.configurationmay throw a ValueError even though the data folder exists and
ci (CommonInterface)
is properly initialized.fromkeboola.componentimportCommonInterface# Logger is automatically set up based on the component setup (GELF or STDOUT)importloggingSOME_PARAMETER='some_user_parameter'REQUIRED_PARAMETERS=[SOME_PARAMETER]# init the interface# A ValueError error is raised if the KBC_DATADIR does not exist or contains non-existent path.ci=CommonInterface()# A ValueError error is raised if the config.json file does not exists in the data dir.# Checks for required parameters and throws ValueError if any is missing.ci.validate_configuration(REQUIRED_PARAMETERS)# print KBC Project ID from the environment variable if present:logging.info(ci.environment_variables.project_id)# load particular configuration parameterlogging.info(ci.configuration.parameters[SOME_PARAMETER])Processing input tables - Manifest vs I/O mappingInput and output tables specified by user are listed in theconfiguration file.
Apart from that, all input tables provided by user also include manifest file with additional metadata.Tables and their manifest files are represented by thekeboola.component.dao.TableDefinitionobject and may be loaded
using the convenience methodget_input_tables_definitions(). The result object contains all metadata about the table,
such as manifest file representations, system path and name.Manifest & input folder contentfromkeboola.componentimportCommonInterfaceimportlogging# init the interfaceci=CommonInterface()input_tables=ci.get_input_tables_definitions()# print path of the first table (random order)first_table=input_tables[0]logging.info(f'The first table named: "{first_table.name}" is at path:{first_table.full_path}')# get information from table manifestlogging.info(f'The first table has following columns defined in the manifest{first_table.columns}')Using I/O mappingimportcsvfromkeboola.componentimportCommonInterface# initialize the libraryci=CommonInterface()# get list of input tables from the input mapping ()tables=ci.configuration.tables_input_mappingj=0fortableintables:# get csv file nameinName=table.destination# read input table manifest and get it's physical representationtable_def=ci.get_input_table_definition_by_name(table.destination)# get csv file name with full path from output mappingoutName=ci.configuration.tables_output_mapping[j].full_path# get file name from output mappingoutDestination=ci.configuration.tables_output_mapping[j]['destination']I/O table manifests and processing resultsThe component may define
outputmanifest filesthat define options on storing the results back to the Keboola Connection Storage. This library provides methods that
simplifies the manifest file creation and allows defining the export options and metadata of the result table using
helper objectsTableDefinitionandTableMetadata.TableDefinitionobject serves as a result container containing all the information needed to store the Table into the
Storage. It contains the manifest file representation and initializes all attributes available in the manifest.This object represents both Input and Output manifests. All output manifest attributes are exposed in the class.There are convenience methods for result processing and manifest creationCommonInterface.write_manifest. Also it is
possible to create the container for the output table using theCommonInterface.create_out_table_definition().Example:fromkeboola.componentimportCommonInterfacefromkeboola.componentimportdao# init the interfaceci=CommonInterface()# create container for the resultresult_table=ci.create_out_table_definition('my_new_result_table',primary_key=['id'],incremental=True)# write some contentwithopen(result_table.full_path,'w')asresult:result.write('line')# add some metadataresult_table.table_metadata.add_table_description('My new table description')# add column datatyperesult_table.table_metadata.add_column_data_type('id',dao.SupportedDataTypes.STRING,source_data_type='VARCHAR(100)',nullable=True,length=100)# write manifestci.write_manifest(result_table)Get input table by namefromkeboola.componentimportCommonInterface# init the interfaceci=CommonInterface()table_def=ci.get_input_table_definition_by_name('input.csv')Initializing TableDefinition object from the manifest filefromkeboola.componentimportdaotable_def=dao.TableDefinition.build_from_manifest('data/in/tables/table.csv.manifest')# print table.csv full-path if present:print(table_def.full_path)# rows countprint(table_def.rows_count)Retrieve raw manifest file definition (CommonInterface compatible)To retrieve the manifest file representation that is compliant with Keboola Connection Common Interface use
thetable_def.get_manifest_dictionary()method.fromkeboola.componentimportdaotable_def=dao.TableDefinition.build_from_manifest('data/in/tables/table.csv.manifest')# get the  manifest file representationmanifest_dict=table_def.get_manifest_dictionary()Processing input filesSimilarly as tables, files and their manifest files are represented by thekeboola.component.dao.FileDefinitionobject
and may be loaded using the convenience methodget_input_files_definitions(). The result object contains all metadata
about the file, such as manifest file representations, system path and name.Theget_input_files_definitions()supports filter parameters to filter only files with a specific tag or retrieve only
the latest file of each. This is especially useful because the KBC input mapping will by default include all versions of
files matching specific tag. By default, the method returns only the latest file of each.fromkeboola.componentimportCommonInterfaceimportlogging# init the interfaceci=CommonInterface()input_files=ci.get_input_files_definitions(tags=['my_tag'],only_latest_files=True)# print path of the first file (random order) matching the criteriafirst_file=input_files[0]logging.info(f'The first file named: "{input_files.name}" is at path:{input_files.full_path}')When working with files it may be useful to retrieve them in a dictionary structure grouped either by name or a tag
group. For this there are convenience methodsget_input_file_definitions_grouped_by_tag_group()andget_input_file_definitions_grouped_by_name()fromkeboola.componentimportCommonInterfaceimportlogging# init the interfaceci=CommonInterface()# group by taginput_files_by_tag=ci.get_input_file_definitions_grouped_by_tag_group(only_latest_files=True)# print list of files matching specific taglogging.info(input_files_by_tag['my_tag'])# group by nameinput_files_by_name=ci.get_input_file_definitions_grouped_by_name(only_latest_files=True)# print list of files matching specific namelogging.info(input_files_by_name['image.jpg'])Processing state filesState filescan be easily written and
loaded using theget_state_file()andwrite_state_file()methods:fromkeboola.componentimportCommonInterfacefromdatetimeimportdatetimeimportlogging# init the interfaceci=CommonInterface()last_state=ci.get_state_file()# print last_updated if existslogging.info(f'Previous job stored following last_updated value:{last_state.get("last_updated","")})')# store new state fileci.write_state_file({"last_updated":datetime.now().isoformat()})LoggingThe library automatically initializes STDOUT or GELF logger based on the presence of theKBC_LOGGER_PORT/HOSTenvironment variable upon theCommonInterfaceinitialization. To use the GELF logger just enable the logger for your
appplication in the Developer Portal. More info in
thededicated article.Once it is enabled, you may just log your messages using the logging library:fromkeboola.componentimportCommonInterfacefromdatetimeimportdatetimeimportlogging# init the interfaceci=CommonInterface()logging.info("Info message")TIP:When the logger verbosity is set toverboseyou may leverageextrafields to log the detailed message in
the detail of the log event by adding extra fields to you messages:logging.error(f'{error}. See log detail for full query. ',extra={"failed_query":json.dumps(query)})You may also choose to override the settings by enabling the GELF or STDOUT explicitly and specifying the host/port
parameters:fromkeboola.componentimportCommonInterfaceimportosimportlogging# init the interfaceci=CommonInterface()os.environ['KBC_LOGGER_ADDR']='localhost'os.environ['KBC_LOGGER_PORT']=12201ci.set_gelf_logger(log_level=logging.INFO,transport_layer='UDP')logging.info("Info message")ComponentBaseBase classfor general Python components. Base your components on this class for simpler debugging.It performs following tasks by default:Initializes the CommonInterface.For easier debugging the data folder is picked up by default from../datapath, relative to working directory when
theKBC_DATADIRenv variable is not specified.Ifdebugparameter is present in theconfig.json, the default logger is set to verbose DEBUG mode.Executes sync actions ->runby default. See the sync actions section.Constructor arguments:data_path_override: optional path to data folder that overrides the default behaviour
(KBC_DATADIRenvironment variable). May be also specified by-dor--datacommandline argumentRaises:UserException- on config validation errors.Example usage:importcsvimportloggingfromdatetimeimportdatetimefromkeboola.component.baseimportComponentBase,sync_actionfromkeboola.componentimportUserException# configuration variablesKEY_PRINT_HELLO='print_hello'# list of mandatory parameters => if some is missing,# component will fail with readable message on initialization.REQUIRED_PARAMETERS=[KEY_PRINT_HELLO]REQUIRED_IMAGE_PARS=[]classComponent(ComponentBase):defrun(self):'''Main execution code'''# ####### EXAMPLE TO REMOVE# check for missing configuration parametersself.validate_configuration_parameters(REQUIRED_PARAMETERS)self.validate_image_parameters(REQUIRED_IMAGE_PARS)params=self.configuration.parameters# Access parameters in data/config.jsonifparams.get(KEY_PRINT_HELLO):logging.info("Hello World")# get last state data/in/state.json from previous runprevious_state=self.get_state_file()logging.info(previous_state.get('some_state_parameter'))# Create output table (Tabledefinition - just metadata)table=self.create_out_table_definition('output.csv',incremental=True,primary_key=['timestamp'])# get file path of the table (data/out/tables/Features.csv)out_table_path=table.full_pathlogging.info(out_table_path)# DO whatever and save into out_table_pathwithopen(table.full_path,mode='wt',encoding='utf-8',newline='')asout_file:writer=csv.DictWriter(out_file,fieldnames=['timestamp'])writer.writeheader()writer.writerow({"timestamp":datetime.now().isoformat()})# Save table manifest (output.csv.manifest) from the tabledefinitionself.write_manifest(table)# Write new state - will be available next runself.write_state_file({"some_state_parameter":"value"})# ####### EXAMPLE TO REMOVE END# sync action that is executed when configuration.json "action":"testConnection" parameter is present.@sync_action('testConnection')deftest_connection(self):connection=self.configuration.parameters.get('test_connection')ifconnection=="fail":raiseUserException("failed")elifconnection=="succeed":# this is ignored when run as sync action.logging.info("succeed")"""Main entrypoint"""if__name__=="__main__":try:comp=Component()# this triggers the run method by default and is controlled by the configuration.action paramtercomp.execute_action()exceptUserExceptionasexc:logging.exception(exc)exit(1)exceptExceptionasexc:logging.exception(exc)exit(2)Table Schemas in ComponentBaseIn cases of a static schemas of output/input tables, the schemas can be defined using a JSON Table Schema. For output
mapping these json schemas can be automatically turned into out table definitions.JSON Table Schema example file{"name":"product","description":"this table holds data on products","parent_tables":[],"primary_keys":["id"],"fields":[{"name":"id","base_type":"string","description":"ID of the product","length":"100","nullable":false},{"name":"name","base_type":"string","description":"Plain-text name of the product","length":"1000","default":"Default Name"}]}Out table definition from schema exampleThe example below shows how a table definition can be created from a json schema using the ComponentBase. The schema is
located in the 'src/schemas' directory.importcsvfromkeboola.component.baseimportComponentBaseDUMMY_PRODUCT_DATA=[{"id":"P0001","name":"juice"},{"id":"P0002","name":"chocolate bar"},{"id":"P0003","name":"Stylish Pants"},]classComponent(ComponentBase):def__init__(self):super().__init__()defrun(self):product_schema=self.get_table_schema_by_name('product')product_table=self.create_out_table_definition_from_schema(product_schema)withopen(product_table.full_path,'w')asoutfile:writer=csv.DictWriter(outfile,fieldnames=product_table.columns)writer.writerows(DUMMY_PRODUCT_DATA)self.write_manifest(product_table)Sync ActionsFrom the documentationSync actions:Action provide a way to execute very quick tasks in a single Component, using a single code base. The default
component’s action (run) executes as a background, asynchronous job. It is queued, has plenty of execution time, and
there are cases when you might not want to wait for it. Apart from the defaultrun, there can be synchronous actions
with limited execution time and you must wait for them. When we refer toactions, we mean synchronous actions. Using
actions is fully optional.Use CaseFor example, in our database extractor, the main task (runaction) is the data extraction itself. But we also want to
be able to test the database credentials and list tables available in the database. These tasks would be very helpful in
the UI. It is not possible to do these things directly in the browser. Setting up a separate component would bring an
overhead of maintaining both the extractor’s Docker image and the new component.Sync Action limitationsData is exchanged viastdoutorstderr.Sync actions need to be registered in the Developer Portal first.Following are handled by the decorator automaticallyAll success responses have to output valid JSON string. Meaning nothing can log into the stdout during the action
executionFor success action the output needs to be always{"status":"success"}in stdout.Framework SupportDecoratorsync_actionwas added. It takes one parameteraction_namethat will create mapping between the actual
method and the sync action name registered in the Developer Portal.Decorated methods can also be called from within the program and return values.They can log normally -> when run as sync action all logging within the method is muted.When a return value is produced, it is expected to bedictorlistobject. These will be printed to stdout at the
end.Exceptions can be thrown normally and the message will be propagated to the platform.Action output & examplesEach action has to have specific output based on type of the UI element that the action is triggered with. It can either
have no return value (success / fail type of actions) or UI element specific output.For convenience each output variant is represented by classes specified inkeboola.component.sync_actionsmodule.ValidationResultResult expected by validation button element.fromkeboola.component.baseimportComponentBase,sync_actionfromkeboola.component.sync_actionsimportValidationResult,MessageTypeclassComponent(ComponentBase):defrun(self):pass@sync_action('validate_example')defvalidate_message(self)->ValidationResult:returnValidationResult('Some warning **markdown** message',MessageType.WARNING)SelectElementElement of a dynamic (multi)select UI element. The UI objects expects list of such elements.fromkeboola.component.baseimportComponentBase,sync_actionfromkeboola.component.sync_actionsimportValidationResult,MessageType,SelectElementfromtypingimportListclassComponent(ComponentBase):defrun(self):pass@sync_action('validate_example')defvalidate_message(self)->List[SelectElement]:return[SelectElement(value="value1",label="Value 1 label"),SelectElement(value="value2",label="Value 2 label")]No outputSome actions like test connection button expect only success / failure type of result with no return value.fromkeboola.component.baseimportComponentBase,sync_actionfromkeboola.componentimportUserExceptionimportloggingclassComponent(ComponentBase):def__init__(self):super().__init__()@sync_action('testConnection')deftest_connection(self):# this is ignored when run as sync action.logging.info("Testing Connection")print("test print")params=self.configuration.parametersconnection=params.get('test_connection')ifconnection=="fail":raiseUserException("failed")elifconnection=="succeed":# this is ignored when run as sync action.logging.info("succeed")LicenseMIT licensed, seeLICENSEfile.