Package: target-s3-jsonl
target-s3-jsonlSingertarget that uploads loads data to S3 in JSONL format
following theSinger spec.target-s3-jsonlis aSingerTarget which intend to work with regularSingerTap. It take the output of the tap and export it as aJSON Linesfiles into an AWS S3 bucket.This package is built using thetarget-corelibrary.InstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.DefaultsNote: To avoid version conflicts runtapandtargetsin separate virtual environments.python-mvenv~/.virtualenvs/target-s3-jsonl
~/.virtualenvs/target-s3-jsonl/bin/pipinstalltarget-s3-jsonlHeadpython-mvenv~/.virtualenvs/target-s3-jsonl
~/.virtualenvs/target-s3-jsonl/bin/pipinstall--upgradegit+https://github.com/ome9ax/target-s3-jsonl.git@mainAlternativepython-mvenv~/.virtualenvs/target-s3-jsonlsource~/.virtualenvs/target-s3-jsonl/bin/activate
pipinstalltarget-s3-jsonl
deactivateUsageLike any other target that's following the singer specificiation:some-singer-tap | target-s3-jsonl --config [config.json]It's reading incoming messages from STDIN and using the properites inconfig.jsonto upload data into AWS S3.Configuration settingsRunning the the target connector requires aconfig.jsonfile. An example with the minimal settings:{"s3_bucket":"my_bucket"}Profile based authenticationProfile based authentication used by default using thedefaultprofile. To use another profile setaws_profileparameter inconfig.jsonor set theAWS_PROFILEenvironment variable.Non-Profile based authenticationFor non-profile based authentication setaws_access_key_id,aws_secret_access_keyand optionally theaws_session_tokenparameter in theconfig.json. Alternatively you can define them out ofconfig.jsonby settingAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENenvironment variables.Full list of options inconfig.json:Inherited fromtarget-corePropertyTypeMandatory?Descriptionpath_templateString(Default: None) Custom naming convention of the s3 key. Replaces tokensdate,stream, andtimestampwith the appropriate values.Supports datetime and other python advanced string formatting e.g.{stream:_>8}_{timestamp:%Y%m%d_%H%M%S}.jsonor{stream}/{timestamp:%Y}/{timestamp:%m}/{timestamp:%d}/{timestamp:%Y%m%d_%H%M%S_%f}.json.Supports "folders" in s3 keys e.g.folder/folder2/{stream}/export_date={date}/{timestamp}.json.Honors thes3_key_prefix,  if set, by prepending the "filename". E.g. path_template =folder1/my_file.jsonand s3_key_prefix =prefix_results infolder1/prefix_my_file.jsonmemory_bufferIntegerMemory buffer's size used for non partitioned files before storing the data into the temporary file. 64Mb used by default if unspecified.file_sizeIntegerFile partitinoning bysize_limit. File parts will be created. Thepath_templatemust contain a part section for the part number. Example"path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json".compressionStringThe type of compression to apply before uploading. Supported options arenone(default),gzip, andlzma. For gzipped files, the file extension will automatically be changed to.json.gzfor all files. Forlzmacompression, the file extension will automatically be changed to.json.xzfor all files.timezone_offsetIntegerOffset value in hour. Use offset0hours is you want thepath_templateto useutctime zone. Thenullvalues is used by default.work_dirString(Default: platform-dependent) Directory of temporary JSONL files with RECORD messages.Specific Fortarget-s3-jsonlPropertyTypeMandatory?Descriptions3_bucketStringYesS3 Bucket nameaws_profileStringAWS profile name for profile based authentication. If not provided,AWS_PROFILEenvironment variable will be used.aws_endpoint_urlStringAWS endpoint URL.aws_access_key_idStringS3 Access Key Id. If not provided,AWS_ACCESS_KEY_IDenvironment variable will be used.aws_secret_access_keyStringS3 Secret Access Key. If not provided,AWS_SECRET_ACCESS_KEYenvironment variable will be used.aws_session_tokenStringAWS Session token. If not provided,AWS_SESSION_TOKENenvironment variable will be used.encryption_typeString(Default: 'none') The type of encryption to use. Current supported options are: 'none' and 'KMS'.encryption_keyStringA reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. '1234abcd-1234-1234-1234-1234abcd1234'). This field is ignored if 'encryption_type' is none or blank.role_arnStringThe ARN of the role to assumeTestInstall the toolspipinstalltoxRun pytesttox-epyLint & Static typing validationtox-elint,staticReleaseUpdate the version number at the beginning oftarget-s3-jsonl/target_s3_json/__init__.pyMerge the changes PR intomainRelease the new version in githubLicenseApache License Version 2.0