Package: pipelinewise-tap-twilio
pipelinewise-tap-twilioSingertap that extracts data from aTwilio APIand produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.This tap:Extracts the following resources:accountsaddressesdependent_phone_numbersapplicationsavailable_phone_number_countriesavailable_phone_numbers_localavailable_phone_numbers_mobileavailable_phone_numbers_toll_freeincoming_phone_numberskeyscallsconferencesconference_participantsoutgoing_caller_idsrecordingsusage_recordsusage_triggerstranscriptionsqueuesmessage_mediaalertsExtracts TaskRouter resources:workspacesactivitieseventstaskstask_channelstask_queuescumulative_statisticsworkersworker_channelsworkflowsExtracts Programmable Chat resources (membersandchat_messagesareFULL_TABLEsynced, so take care syncing them, they result in a lot of request/data):servicesroleschat_channelsmemberschat_messagesusersOutputs the schema for each resourceIncrementally pulls data based on the input stateStreamsStandard Endpoints:accountsEndpoint:https://api.twilio.com/2010-04-01/Accounts.jsonPrimary key fields: sidReplication strategy: FULL_TABLETransformations: subresources_to_arrayaddressesEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses.jsonParent: accountPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arraydependent_phone_numbersEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Addresses/{ParentId}/DependentPhoneNumbers.jsonParent: addressesPrimary key fields: sidReplication strategy: FULL_TABLETransformations: subresources_to_arrayapplicationsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Applications.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayavailable_phone_number_countriesEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers.jsonParent: accountsPrimary key fields: country_codeReplication strategy: FULL_TABLETransformations: subresources_to_arrayavailable_phone_numbers_localEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Local.jsonParent: available_phone_number_countriesPrimary key fields: iso_country, phone_numberReplication strategy: FULL_TABLETransformations: subresources_to_arrayavailable_phone_numbers_mobileEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/Mobile.jsonParent: available_phone_number_countriesPrimary key fields: iso_country, phone_numberReplication strategy: FULL_TABLETransformations: subresources_to_arrayavailable_phone_numbers_toll_freeEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/AvailablePhoneNumbers/{ParentId}/TollFree.jsonParent: available_phone_number_countriesPrimary key fields: iso_country, phone_numberReplication strategy: FULL_TABLETransformations: noneincoming_phone_numbersEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/IncomingPhoneNumbers.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arraykeysEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Keys.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arraycallsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Calls.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayconferencesEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayconference_participantsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Conferences/{ParentId}/Participants.jsonParent: conferencesPrimary key fields: uriReplication strategy: FULL_TABLETransformations: subresources_to_arrayoutgoing_caller_idsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/OutgoingCallerIds.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayrecordingsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Recordings.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arraytranscriptionsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Transcriptions.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayqueuesEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Queues.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arraymessage_mediaEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Messages/{ParentId}/Media.jsonParent: messagesPrimary key fields: sidReplication strategy: FULL_TABLETransformations: subresources_to_arrayusage_recordsEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Records.jsonParent: accountsPrimary key fields: account_sid, category, start_dateReplication strategy: INCREMENTALTransformations: subresources_to_arrayusage_triggersEndpoint:https://api.twilio.com/2010-04-01/Accounts/{AccountSid}/Usage/Triggers.jsonParent: accountsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: subresources_to_arrayalertsEndpoint:https://monitor.twilio.com/v1/AlertsPrimary key fields: sidReplication strategy: INCREMENTALTransformations: noneTaskRouter Endpoints:workspacesEndpoint:https://taskrouter.twilio.com/v1/WorkspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: noneactivitiesEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/ActivitiesParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: noneeventsEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/EventsParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonetasksEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TasksParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonetask_channelsEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskChannelsParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonetask_queuesEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueuesParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonecumulative_statisticsEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/TaskQueues/{ParentId}/CumulativeStatisticsParent: task_queuesReplication strategy: FULL_TABLETransformations: noneworkersEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/WorkersParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonechannelsEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/Workers/{ParentId}/ChannelsParent: workersPrimary key fields: sidReplication strategy: FULL_TABLETransformations: noneworkflowsEndpoint:https://taskrouter.twilio.com/v1/Workspaces/{ParentId}/WorkflowsParent: workspacesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: noneProgrammable Chat Endpoints:servicesEndpoint:https://chat.twilio.com/v2/ServicesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonerolesEndpoint:https://chat.twilio.com/v2/Services/{ParentId}/RolesParent: servicesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonechat_channelsEndpoint:https://chat.twilio.com/v2/Services/{ParentId}/ChannelsParent: servicesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: nonemembersEndpoint:https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/MembersParent: workspacesPrimary key fields: sidReplication strategy: FULL_TABLETransformations: nonechat_messagesEndpoint:https://chat.twilio.com/v2/Services/{ParentId}/Channels/{ParentId}/MessagesParent: workspacesPrimary key fields: sidReplication strategy: FULL_TABLETransformations: noneusersEndpoint:https://chat.twilio.com/v2/Services/{ParentId}/UsersParent: servicesPrimary key fields: sidReplication strategy: INCREMENTALTransformations: noneAuthenticationThis tap authenticates to the Twilio API using Basic Auth.To set up authentication simply include your Twilioaccount_sidandauth_tokenin the tap config.Quick StartInstallpython3-mvenvvenv
.venv/bin/activate
pipinstall--upgradepip
pipinstall.Create your tap'sconfig.jsonfile. Theapi_keyis available in the twilio Console UI (seeAuthenticationabove). Thedate_window_daysis the integer number of days (between the from and to dates) for date-windowing through the date-filtered endpoints (default = 30). Thestart_dateis the absolute beginning date from which incremental loading on the initial load will start.{"account_sid":"YOUR_ACCOUNT_SID","auth_token":"YOUR_AUTH_TOKEN","start_date":"2019-01-01T00:00:00Z","user_agent":"tap-twilio <api_user_email@your_company.com>",}Optionally, also create astate.jsonfile.currently_syncingis an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.{"currently_syncing":"registers","bookmarks":{"acounts":"2020-03-23T10:31:14.000000Z","...":"2020-03-23T00:00:00.000000Z"}}Run the Tap in Discovery Mode
This creates a catalog.json for selecting objects/fields to integrate:tap-twilio--configconfig.json--discover>catalog.jsonSee the Singer docs on discovery modehere.Run the Tap in Sync Mode (with catalog)For Sync mode:>tap-twilio--configtap_config.json--catalogcatalog.jsonMessages are written to standard output following the Singer specification.
The resultant stream of JSON data can be consumed by a Singer target.To run testsInstall python test dependencies in a virtual env and run nose unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install -e .[test]To run unit tests:pytest tests/unit