Package: pipelinewise-tap-slack
pipelinewise-tap-slackSingertap that extracts data from aSlackworkspace and produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atTap SlackIf you want to run thisSinger Tapindependently please read further.InstallationIt is highly recommended installingtap-slackin it's own isolated virtual environment. For example:python3-mvenv~/.venvs/tap-slacksource~/.venvs/tap-slack/bin/activate
pip3installpipelinewise-tap-slack
deactivateSetupThe tap requires aSlack API tokento interact with your Slack workspace. You can obtain a token for a single workspace by creating a newSlack Appin your workspace and assigning it the relevantscopes. As of right now, the minimum required scopes for this App are:channels:historychannels:joinchannels:readfiles:readgroups:readlinks:readreactions:readremote_files:readremote_files:writeteam:readusergroups:readusers.profile:readusers:readusers:read.emailThis scope is only required if you want to extract the user emails as well.Create a config file containing the API token and a start date, e.g.:{"token":"xxxx","start_date":"2020-05-01T00:00:00"}Private channelsOptionally, you can also specify whether you want to sync private channels or not by adding the following to the config:"private_channels":"false"By default, private channels will be synced.Joining Public ChannelsBy adding the following to your config file you can have the tap auto-join all public channels in your ogranziation."join_public_channels":"true"If you do not elect to have the tap join all public channels you must invite the bot to all channels you wish to sync.Specify channels to syncBy default, the tap will sync all channels it has been invited to. However, you can limit the tap to sync only the channels you specify by adding their IDs to the config file, e.g.:"channels":["abc123","def345"]Note this needs to be channel ID, not the name, asrecommended by the Slack API. To get the ID for a channel, either use the Slack API orfind it in the URL.Archived ChannelsYou can control whether or not the tap will sync archived channels by including the following in the tap config:"exclude_archived":"false"It's important to note that a botCANNOTjoin an archived channel, so unless the bot was added to the channel prior to it being archived it will not be able to sync the data from that channel.Date WindowingDue to the potentially high volume of data when syncing certain streams (messages, files, threads)
this tap implements date windowing based on a configuration parameter.including"date_window_size":"5"Will cause the tap to sync 5 days of data per request, for applicable streams. The default value if
one is not defined is to window requests for 7 days at a time.UsageIt is recommended to follow Singerbest practiceswhen running taps eitheron their ownorwith a Singer target.In practice, it will look something like the following:~/.venvs/tap-slack/bin/tap-slack--configslack.config.json--catalogcatalog.json|~/.venvs/target-stitch/bin/target-stitch--configstitch.config.jsonReplicationThe Slack Conversations API does not natively store last updated timestamp information about a Conversation. In addition, Conversation records are mutable. Thus,tap-slackrequires aFULL_TABLEreplication strategy to ensure the most up-to-date data in replicated when replicating the following Streams:Channels(Conversations)Channel Members(Conversation Members)TheUsersstreamdoesstore information about when a User record was last updated, sotap-slackuses that timestamp as a bookmark value and prefers using anINCREMENTALreplication strategy.Table SchemasChannels (Conversations)Table Name:channelsDescription:Primary Key Column:idReplication Strategy:FULL_TABLEAPI Documentation:LinkChannel Members (Conversation Members)Table Name:channel_membersDescription:Primary Key Columns:channel_id,user_idReplication Strategy:FULL_TABLEAPI Documentation:LinkMessages (Conversation History)Table Name:messagesDescription:Primary Key Columns:channel_id,tsReplication Strategy:INCREMENTALAPI Documentation:LinkUsersTable Name:usersDescription:Primary Key Column:idReplication Strategy:INCREMENTALAPI Documentation:LinkThreads (Conversation Replies)Table Name:threadsDescription:Primary Key Columns:channel_id,ts,thread_tsReplication Strategy:FULL_TABLEfor each parentmessageAPI Documentation:LinkUser GroupsTable Name:user_groupsDescription:Primary Key Column:idReplication Strategy:FULL_TABLEAPI Documentation:LinkFilesTable Name:filesDescription:Primary Key Column:idReplication Strategy:INCREMENTALquery filtered using date windows and lookback windowAPI Documentation:LinkRemote FilesTable Name:remote_filesDescription:Primary Key Column:idReplication Strategy:INCREMENTALquery filtered using date windows and lookback windowAPI Documentation:LinkTesting the TapInstall test dependenciesmakevenvTo run tests:makeunit_testLintingInstall test dependenciesmakevenvTo run linter:makepylint