Package: pdpcli
PdpCLIQuick LinksIntroductionInstallationTutorialBasic UsageData Reader / WriterPluginsIntroductionPdpCLI is a pandas DataFrame processing CLI tool which enables you to build a pandas pipeline powered bypdpipefrom a configuration file. You can also extend pipeline stages and data readers / writers by using your own python scripts.FeaturesProcess pandas DataFrame from CLI without wrting Python scriptsSupport multiple configuration file formats: YAML, JSON, JsonnetRead / write data files in the following formats: CSV, TSV, JSON, JSONL, pickled DataFrameImport / export data with multiple protocols: S3 / Databse (MySQL, Postgres, SQLite, ...) / HTTP(S)Extensible pipeline and data readers / writersInstallationInstalling the library is simple using pip.$ pip install "pdpcli[all]"TutorialBasic UsageWrite a pipeline config fileconfig.ymllike below. Thetypefields underpipelinecorrespond to the snake-cased class names of thePdpipelineStages. Other fields such asstageandcolumnsare the parameters of the__init__methods of the corresponging classes. Internally, this configuration file is converted to Python objects bycolt.pipeline:type:pipelinestages:drop_columns:type:col_dropcolumns:-name-jobencode:type:one_hot_encodecolumns:sextokenize:type:tokenize_textcolumns:contentvectorize:type:tfidf_vectorize_token_listscolumn:contentmax_features:10Build a pipeline by training ontrain.csv. The following command generages a pickled pipeline filepipeline.pklafter training. If you specify a URL of  file path, it will be automatically downloaded and cached.$ pdp build config.yml pipeline.pkl --input-file https://github.com/altescy/pdpcli/raw/main/tests/fixture/data/train.csvApply the fitted pipeline totest.csvand get output of a processed fileprocessed_test.jsonlby the following command. PdpCLI automatically detects the output file format based on the file name. In this example, the processed DataFrame will be exported as the JSON-Lines format.$ pdp apply pipeline.pkl https://github.com/altescy/pdpcli/raw/main/tests/fixture/data/test.csv --output-file processed_test.jsonlYou can also directly run the pipeline from a config file without fitting pipeline.$ pdp apply config.yml test.csv --output-file processed_test.jsonlIt is possible to override or add parameters by adding command line arguments:pdp apply config.yml test.csv pipeline.stages.drop_columns.column=nameData Reader / WriterPdpCLI automatically detects a suitable data reader / writer based on a given file name.
If you need to use the other data reader / writer, add areaderorwriterconfig toconfig.yml.
The following config is an exmaple to use SQL data reader.
SQL reader fetches records from the specified database and converts them into a pandas DataFrame.reader:type:sqldsn:postgres://${env:POSTGRES_USER}:${env:POSTGRES_PASSWORD}@your.posgres.server/your_databaseConfig files are interpreted byOmegaConf, so${env:...}is interpolated by environment variables.Prepare yuor SQL filequery.sqlto fetch data from the database:select*fromyour_tablelimit1000You can execute the pipeline with SQL data reader via:$ POSTGRES_USER=user POSTGRES_PASSWORD=password pdp apply config.yml query.sqlPluginsBy using plugins, you can extend PdpCLI. This plugin feature enables you to use your own pipeline stages, data readers / writers and commands.Add a new stageWrite your plugin scriptmypdp.pylike below.Stage.register("<stage-name>")registers your pipeline stages, and you can specify these stages by writingtype: <stage-name>in your config file.importpdpcli@pdpcli.Stage.register("print")classPrintStage(pdpcli.Stage):def_prec(self,df):returnTruedef_transform(self,df,verbose):print(df.to_string(index=False))returndfUpdateconfig.ymlto use your plugin.pipeline:
    type: pipeline
    stages:
        drop_columns:
        ...

        print:
            type: print

        encode:
        ...Execute command with--module mypdpand you can see the processed DataFrame after runningdrop_columns.$ pdp apply config.yml test.csv --module mypdpAdd a new commandYou can also add new commands not only stages.Add the following script tomypdp.py. Thisgreetcommand prints out a greeting message with your name.@pdpcli.Subcommand.register(name="greet",description="say hello",help="say hello",)classGreetCommand(pdpcli.Subcommand):requires_plugins=Falsedefset_arguments(self):self.parser.add_argument("--name",default="world")defrun(self,args):print(f"Hello,{args.name}!")To register this command, you need to create the.pdpcli_pluginsfile in which module names are listed for each line. Due to module importing order, the--moduleoption is unavailable for command registration.$ echo "mypdp" > .pdpcli_pluginsRun the following command and get a message like below. By using the.pdpcli_pluginsfile, it is is not needed to add the--moduleoption to a command line for each execution.$ pdp greet --name altescy
Hello, altescy!