Package: datateer-upload-agent
Datateer upload-agentThis is a command-line tool for uploading data into your Datateer data lake.The upload agent pushes files into an AWS S3 bucket, where the files are picked up for ingestion and further processingQuick startEnsure you have python and pip installed, then follow these steps:Install withpip install datateer-upload-agentDo one-time agent configuration withdatateer config upload-agentDo one-time feed configuration withdatateer config feedUpload data withdatateer upload <feed_key> <path>ConceptsAll data in the data lake has the following metadata:Aprovideris an organization that is providing data. This could be your organization if you are pushing data from an internal database or applicationAsourceis the system or application that is providing data. A provider can provide data from one or more systemsAfeedis an independent data feed. A source can provide one or more feeds. For example, if the source is a database, each feed could represent a single table or view. If the source is an API, each feed could represent a single entity.Afileis a data file like a CSV file. It is a point-in-time extraction of a feed, and it is what you upload using the agent.CommandsUploadingUpload a filedatateer upload orders_feed ./my_exported_data/orders.csvwill upload the file at./my_exported_data/orders.csvusing the feed keyorders_feedConfiguringConfigure the upload agentdatateer config upload-agentwill ask you a series of questions to configure your agentDatateer client code:Raw bucket name:Access key:Access secret:If you need to reconfigure the agent, just rerundatateer config upload-agentConfigure a new feeddatateer config feedwill ask a series of questions to configure a new feedProvider:xyzData Source:internal_app1Feed:ordersFeed key [orders]:orders_feedReconfigure an existing feeddatateer config feed --update orders_feedwill rerun the configuration questions for the feed with the keyorders_feedShow configdatateer config upload-agent --showwill show you your existing configurationclient-code:xyzraw-bucket:xyz-pipeline-raw-202012331213123432341213access-key:ABC***access-secret:123***feeds:31) Feed "customers" will upload to xyz/internal_app1/customers/2) Feed "orders_feed" will upload to xyz/internal_app1/orders/3) Feed "leads" will upload to salesforce/salesforce/leadsFeed "abc" will upload to provider/source/feedData File RequirementsThe data lake supports CSV, TSV, and JSONL filesThe first row of the data file must contain header namesAdding new data fields or removing data fields are both supportedYou should strive to be consistent with your header names over time. The data lake can handle changes, but it will likely confuse anyone using the feedsConfiguration - detailed infoConfiguration can be handled completely through thedatateer configcommands. If you need more details, this section provides more details on how configuration works and where it is stored.LocationHere is where the Datateer upload agent will look for configuration information, in order of preference:In a relative directory named.datateer, in a file namedconfig.yml.In the future, we may add global configuration in the user's home directory or in environment variablesSchemaAn example configuration file will look like this:client-code:xyzupload-agent:raw-bucket:xyz-pipeline-raw-202012331213123432341213access-key:ABC***access-secret:123***feeds:customers:provider:xyzsource:internal_app1feed:customersorders_feed:provider:xyzsource:internal_app1feed:ordersleads:provider:salesforcesource:salesforcefeed:leadsDevelopmentTo develop in this repo:Install poetry and activate shell withpoetry shellRunpoetry installTo test runpytestorptwTo run locally, install withpip install -e .