Package: target-jsonl-webhdfs
target-jsonl-webhdfsASingertarget that writes data to HDFS cluster in the JSONL (JSON Lines) format.
This is fork of theTarget-jsonlrepo.How to use ittarget-jsonlworks together with any otherSinger Tapto move data from sources likeBraintree,FreshdeskandHubspotto JSONL formatted files.InstallWe will usetap-exchangeratesapito pull currency exchange rate data from a public data set as an example.First, make sure Python 3 is installed on your system or follow these installation instructions forMacorUbuntu.It is recommended to install each Tap and Target in a separate Python virtual environment to avoid conflicting dependencies between any Taps and Targets.# Install tap-exchangeratesapi in its own virtualenvpython3-mvenv~/.virtualenvs/tap-exchangeratesapisource~/.virtualenvs/tap-exchangeratesapi/bin/activate
pipinstalltap-exchangeratesapi
deactivate# Install target-jsonl in its own virtualenvpython3-mvenv~/.virtualenvs/target-jsonlsource~/.virtualenvs/target-jsonl/bin/activate
pipinstalltarget-jsonl
deactivateRunWe can now runtap-exchangeratesapiand pipe the output totarget-jsonl.~/.virtualenvs/tap-exchangeratesapi/bin/tap-exchangeratesapi|~/.virtualenvs/target-jsonl/bin/target-jsonlThe data by default will be written to a file calledexchange_rate-{timestamp}.jsonlin your working directory.›catexchange_rate-{timestamp}.jsonl{"CAD":1.3954067515,"HKD":7.7503228187,"ISK":147.1130787678,"PHP":50.5100534957,"DKK":6.8779745434,"HUF":327.9376498801,"CZK":25.018446781,"GBP":0.8059214167,"RON":4.4673491976,"SEK":9.9002029146,"IDR":15321.0016602103,"INR":75.6516325401,"BRL":5.4711307877,"RUB":73.6220254566,"HRK":6.9765725881,"JPY":106.548607268,"THB":32.420217672,"CHF":0.9750046117,"EUR":0.9223390518,"MYR":4.3475373547,"BGN":1.8039107176,"TRY":6.988286294,"CNY":7.0764619074,"NOK":10.3973436635,"NZD":1.6446227633,"ZAR":18.4316546763,"USD":1.0,"MXN":24.1217487548,"SGD":1.4152370411,"AUD":1.5361556908,"ILS":3.5102379635,"KRW":1218.9540675152,"PLN":4.1912931194,"date":"2020-04-29T00:00:00Z"}Optional Configurationtarget-jsonltakes an optional configuration file that can be used to set formatting parameters like the delimiter - seeconfig.sample.jsonfor examples. To runtarget-jsonlwith the configuration file, use this command:~/.virtualenvs/tap-exchangeratesapi/bin/tap-exchangeratesapi|~/.virtualenvs/target-jsonl/bin/target-jsonl-cmy-config.jsonHere is a brief description of the optional config keysdestination_path- Specifies where to write the resulting.jsonlfile to. By default, the file gets written in your working directory.custom_name- Specifies a custom name for the filename, instead of the stream name (i.e.{custom_name}-{timestamp}.jsonl, asummingdo_timestamp_fileistrue). By default, the stream name will be used.do_timestamp_file- Specifies if the file should get timestamped. By default, the resulting file will have a timestamp in the file name (i.e.exchange_rate-{timestamp}.jsonlas described above in theRunsection). If this option gets set tofalse, the resulting file will not have a timestamp associated with it (i.e.exchange_rate.jsonlin our example).webhdfs- Boolean variable to enable webhdfs writing.webhdfs_url- Specifies url for connection to the webhdfs service (i.e.http://hostname:port).webhdfs_user- Specifies user that will be use for connect to the webhdfs service.Copyright © 2022 Andy Huynh, Stanislav Lysikov