Package: pipelinewise-target-s3-csv
pipelinewise-target-s3-csvSingertarget that uploads loads data to S3 in CSV format
following theSinger spec.This is aPipelineWisecompatible target connector.How to use itThe recommended method of running this target is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atTarget S3 CSVIf you want to run thisSinger Targetindependently please read further.InstallFirst, make sure Python >=3.7 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-target-s3-csvormakevenvTo runLike any other target that's following the singer specification:some-singer-tap | target-s3-csv --config [config.json]It's reading incoming messages from STDIN and using the properties inconfig.jsonto upload data into Postgres.Note: To avoid version conflicts runtapandtargetsin separate virtual environments.Configuration settingsRunning the target connector requires aconfig.jsonfile. An example with the minimal settings:{"s3_bucket":"my_bucket"}Profile based authenticationProfile based authentication used by default using thedefaultprofile. To use another profile setaws_profileparameter inconfig.jsonor set theAWS_PROFILEenvironment variable.Non-Profile based authenticationFor non-profile based authentication setaws_access_key_id,aws_secret_access_keyand optionally theaws_session_tokenparameter in theconfig.json. Alternatively you can define them out ofconfig.jsonby settingAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENenvironment variables.Full list of options inconfig.json:PropertyTypeRequired?Descriptionaws_access_key_idStringNoS3 Access Key Id. If not provided,AWS_ACCESS_KEY_IDenvironment variable will be used.aws_secret_access_keyStringNoS3 Secret Access Key. If not provided,AWS_SECRET_ACCESS_KEYenvironment variable will be used.aws_session_tokenStringNoAWS Session token. If not provided,AWS_SESSION_TOKENenvironment variable will be used.aws_endpoint_urlStringNoAWS endpoint URL.aws_profileStringNoAWS profile name for profile based authentication. If not provided,AWS_PROFILEenvironment variable will be used.s3_bucketStringYesS3 Bucket names3_key_prefixString(Default: None) A static prefix before the generated S3 key names. Using prefixes you candelimiterString(Default: ',') A one-character string used to separate fields.quotecharString(Default: '"') A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.add_metadata_columnsBoolean(Default: False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix_SDC_. The column names are following the stitch naming conventions documented athttps://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the_SDC_DELETED_ATmetadata column. Without theadd_metadata_columnsoption the deleted rows from singer taps will not be recongisable in Snowflake.encryption_typeStringNo(Default: 'none') The type of encryption to use. Current supported options are: 'none' and 'KMS'.encryption_keyStringNoA reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. '1234abcd-1234-1234-1234-1234abcd1234'). This field is ignored if 'encryption_type' is none or blank.compressionStringNoThe type of compression to apply before uploading. Supported options arenone(default) andgzip. For gzipped files, the file extension will automatically be changed to.csv.gzfor all files.naming_conventionStringNo(Default: None) Custom naming convention of the s3 key. Replaces tokensdate,stream, andtimestampwith the appropriate values.Supports "folders" in s3 keys e.g.folder/folder2/{stream}/export_date={date}/{timestamp}.csv.Honors thes3_key_prefix,  if set, by prepending the "filename". E.g. naming_convention =folder1/my_file.csvand s3_key_prefix =prefix_results infolder1/prefix_my_file.csvtemp_dirString(Default: platform-dependent) Directory of temporary CSV files with RECORD messages.To run tests:Define environment variables that requires running the testsexport TARGET_S3_CSV_ACCESS_KEY_ID=<s3-access-key-id>
  export TARGET_S3_CSV_SECRET_ACCESS_KEY=<s3-secret-access-key>
  export TARGET_S3_CSV_BUCKET=<s3-bucket>
  export TARGET_S3_CSV_KEY_PREFIX=<s3-key-prefix>Install python test dependencies in a virtual env and run unit and integration testsmakevenvTo run unit tests:makeunit_testTo run integration tests:makeintegration_testTo run pylint:Install python dependencies and run python lintermakevenvpylintLicenseApache License Version 2.0SeeLICENSEto see the full text.