Package: pipelinewise-tap-mixpanel
pipelinewise-tap-mixpanelSingertap that extracts data from aMixpanel APIand produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible tap connector.This tap:Pulls raw data from theMixpanel Event Export APIand theMixpanel Query API.Extracts the following resources:Export (Events)Engage (People/Users)FunnelsAnnotationsCohortsCohort MembersRevenueOutputs the schema for each resourceIncrementally pulls data based on the input stateUses date-windowing to chunk/loop throughexport,revenue,funnels.Incorporates attribution window for latency look-back to accommodate delays in data reconciliation.StreamsexportEndpoint:https://data.mixpanel.com/api/2.0/exportPrimary key fields:event,time,distinct_idReplication strategy: INCREMENTAL (query filtered)Bookmark:timeBookmark query field:from_date,to_dateTransformations: De-nestpropertiesto root-level, re-name properties with leading$...tomp_reserved_..., convert datetimes from project timezone to UTC.Optional parametersexport_eventsto export only certain eventsengageEndpoint:https://mixpanel.com/api/2.0/engagePrimary key fields:distinct_idReplication strategy: FULL_TABLE (all records, every load)Transformations: De-nest$propertiesto root-level, re-name properties with leading$...tomp_reserved_....funnelsEndpoint 1 (name, id):https://data.mixpanel.com/api/2.0/exportEndpoint 2 (date, measures):https://mixpanel.com/api/2.0/funnelsPrimary key fields:funnel_id,dateParameters:funnel_id: {funnel_id} (from Endpoint 1)unit: dayReplication strategy: INCREMENTAL (query filtered)Bookmark:dateBookmark query field:from_date,to_dateTransformations: Combine Endpoint 1 & 2 results, convertdatekeys to list toresultslist-array.revenueEndpoint:https://mixpanel.com/api/2.0/engage/revenuePrimary key fields:dateParameters:unit: dayReplication strategy: INCREMENTAL (query filtered)Bookmark:dateBookmark query field:from_date,to_dateTransformations: Convertdatekeys to list toresultslist-array.annotationsEndpoint:https://mixpanel.com/api/2.0/annotationsPrimary key fields:dateReplication strategy: FULL_TABLETransformations: None.cohortsEndpoint:https://mixpanel.com/api/2.0/cohorts/listPrimary key fields:idReplication strategy: FULL_TABLETransformations: None.cohort_members (engage)Endpoint:https://mixpanel.com/api/2.0/cohorts/listPrimary key fields:distinct_id,cohort_idParameters:filter_by_cohort: {cohort_id} (fromcohortsendpoint)Replication strategy: FULL_TABLETransformations: For eachcohort_idincohortsendpoint, queryengageendpoint withfilter_by_cohortparameter to create list ofdistinct_idfor eachcohort_id.AuthenticationThe Mixpanel API uses Basic Authorization with theapi_secretfrom the tap config in base-64 encoded format. It is slightly different than normal Basic Authorization with username/password. All requests should include this header with theapi_secretas the username, with no password:Authorization:Basic <base-64 encoded api_secret>More details may be found in theMixpanel API Authenticationinstructions.Quick StartInstallpython3-mvenvvenv
.venv/bin/activate
pipinstall--upgradepip
pipinstall.Create your tap'sconfig.jsonfile.  The tap config file for this tap should include these entries:start_date- the default value to use if no bookmark exists for an endpoint (rfc3339 date string)user_agent(string, optional): Process and email for API logging purposes. Example:tap-mixpanel <api_user_email@your_company.com>api_secret(string,ABCdef123): an API secret for each project in Mixpanel. This can be found in the Mixpanel Console, upper-right Settings (gear icon), Organization Settings > Projects and in the Access Keys section. For this tap, only the api_secret is needed (the api_key is legacy and the token is used only for uploading data). Each Mixpanel project has a different api_secret; therefore each Singer tap pipeline instance is for a single project.date_window_size(integer,30): Number of days for date window looping through transactional endpoints with from_date and to_date. Default date_window_size is 30 days. Clients with large volumes of events may want to decrease this to 14, 7, or even down to 1-2 days.attribution_window(integer,5): Latency minimum number of days to look-back to account for delays in attributing accurate results.Default attribution window is 5 days.project_timezone(string likeUS/Pacific): Time zone in which integer date times are stored. The project timezone may be found in the project settings in the Mixpanel console.More info about timezones.select_properties_by_default(trueorfalse): Mixpanel properties are not fixed and depend on the date being uploaded. During Discovery mode and catalog.json setup, all current/existing properties will be captured. Setting this config parameter to true ensures that new properties on events and engage records are captured. Otherwise new properties will be ignored.denest_properties(trueorfalse): To denest large and nested JSON Mixpanel responses in theextractandengagestreams. To avoid very wide schema you can disable the denesting feature and the original JSON response will be sent in the RECORD message as plain object. Defaultdenest_propertiesistrue.{"api_secret":"YOUR_API_SECRET","date_window_size":"30","attribution_window":"5","project_timezone":"US/Pacific","select_properties_by_default":"true","denest_properties":"true","start_date":"2019-01-01T00:00:00Z","user_agent":"tap-mixpanel <api_user_email@your_company.com>"}If you want to export only certain events from theRaw export APIthen addexport_eventsoption to theconfig.jsonand list the required event names:"export_events":["event_one","event_two"]Optionally, also create astate.jsonfile.currently_syncingis an optional attribute used for identifying the last object to be synced in case the job is interrupted mid-stream. The next run would begin where the last job left off.{"currently_syncing":"engage","bookmarks":{"export":"2019-09-27T22:34:39.000000Z","funnels":"2019-09-28T15:30:26.000000Z","revenue":"2019-09-28T18:23:53Z"}}Run the Tap in Discovery Mode
This creates a catalog.json for selecting objects/fields to integrate:tap-mixpanel--configconfig.json--discover>catalog.jsonSee the Singer docs on discovery modehere.Run the Tap in Sync Mode (with catalog) andwrite out to state fileFor Sync mode:>tap-mixpanel--configtap_config.json--catalogcatalog.jsonMessages are written to standard output following the Singer specification.
The resultant stream of JSON data can be consumed by a Singer target.
To load to json files to verify outputs:>tap-mixpanel--configtap_config.json--catalogcatalog.json|target-json>state.json
>tail-1state.json>state.json.tmp&&mvstate.json.tmpstate.jsonTo pseudo-load toStitch Import APIwith dry run:>tap-mixpanel--configtap_config.json--catalogcatalog.json|target-stitch--configtarget_config.json--dry-run>state.json
>tail-1state.json>state.json.tmp&&mvstate.json.tmpstate.jsonTestInstall python test dependencies in a virtual env and run nose unit and integration testspython3-mvenvvenv
.venv/bin/activate
pipinstall--upgradepip
pipinstall.[test]Run unit testspytest tests/unittests