Package: target-core
Target CoreSingertarget-coreprovide safe tools to easily build newtargetsfollowing theSinger specconventionandprotocol.How to use ittarget-coreis aSingerTarget which intend to work with regularSingerTap.The Goal is to use this package as a foundation to build other taps focusing on the core tools, reducing the energy spent on maintaining the common parts.Packages extending thetarget-coretarget-s3-jsonlInstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use avenv:Defaultspython-mvenvvenv
.venv/bin/activate
pipinstall--upgradetarget-coreHeadpython-mvenvvenv
.venv/bin/activate
pipinstall--upgradehttps+git@gitlab.com:omegax/target-core.gitIsolated virtual environmentpython-mvenv~/.virtualenvs/target-core
~/.virtualenvs/target-core/bin/pipinstalltarget-coreAlternativepython-mvenv~/.virtualenvs/target-coresource~/.virtualenvs/target-core/bin/activate
pipinstalltarget-core
deactivateTo runLike any other target that's following the singer specificiation:some-singer-tap | target-core --config [config.json]It's reading incoming messages from STDIN and using the properites inconfig.jsonto upload the data.Note: To avoid version conflicts runtapandtargetsin separate virtual environments.Configuration settingsRunning the the target connector requires aconfig.jsonfile. An example with the minimal settings:{"path_template":"{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json"}Profile based authenticationProfile based authentication used by default using thedefaultprofile. To use another profile setaws_profileparameter inconfig.jsonor set theAWS_PROFILEenvironment variable.Non-Profile based authenticationFor non-profile based authentication setaws_access_key_id,aws_secret_access_keyand optionally theaws_session_tokenparameter in theconfig.json. Alternatively you can define them out ofconfig.jsonby settingAWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENenvironment variables.Full list of options inconfig.json:PropertyTypeMandatory?Descriptionpath_templateString(Default: None) Custom naming convention of the s3 key. Replaces tokensstream, anddate_timewith the appropriate values.Supports datetime and other python advanced string formatting e.g.{stream}_{date_time:%FT%T.%f}.jsonlor{stream:_>8}/{date_time:%Y}/{date_time:%m}/{date_time:%d}/{date_time:%Y%m%d_%H%M%S_%f}.json.Supports "folders" in s3 keys e.g.my_folder/my_sub_folder/{stream}/export_date={date}/{date_time}.json.timezone_offsetIntegerOffset value in hour. Use offset0hours is you want thepath_templateto useutctime zone. Thenullvalues is used by default.memory_bufferIntegerMemory buffer's size used before storing the data into the temporary file. 64Mb used by default if unspecified. This value is used to partition the files by size.file_sizeIntegerFile size limit. File parts will be created. Thepath_templatemust contain a part section for the part number. Example"path_template": "{stream}_{date_time:%Y%m%d_%H%M%S}_part_{part:0>3}.json".work_dirString(Default: platform-dependent) Directory of temporary JSONL files with RECORD messages.compressionStringThe type of compression to apply before uploading. Supported options arenone(default),gzip, andlzma. For gzipped files, the file extension will automatically be changed to.json.gzfor all files. Forlzmacompression, the file extension will automatically be changed to.json.xzfor all files.TestInstall the toolspipinstall.[test,lint]Run pytestpytest-pno:cacheproviderLintflake8--show-source--statistics--count--extend-exclude.virtualenvsReleaseUpdate the version number at the beginning oftarget-core/target_core/__init__.pyMerge the changes PR intomainCreate a taggit tag -a 1.0.0 -m 'Release version 1.0.0'Release the new version in githubLicenseApache License Version 2.0