Package: pipelinewise-tap-google-analytics
pipelinewise-tap-google-analyticsThis is aSingertap that reads data from Google Analytics and produces JSON-formatted data following theSinger spec.This is aPipelineWisecompatible target connector.How to use itThe recommended method of running this tap is to use it fromPipelineWise. When running it from PipelineWise you don't need to configure this tap with JSON files and most of things are automated. Please check the related documentation atGoogle AnalyticsIf you want to run thisSinger Tapindependently please read further.UsageThis tap is a fork ofMeltano's Google Analytics tapthat:Pulls raw data from theGoogle Analytics Reporting API.Supports generating as many reports as the user of the tap wants (following Google's limit of 7 dimensions and 10 metrics per report).Generates a valid Catalog that follows theSinger spec.As the Google Analytics Reports are defined dynamically and there are practically infinite combinations of dimensions and metrics a user can ask for, the entities and their schema (i.e. the Catalog for this tap) are not static. So, this tap behaves more or less similarly to a tap extracting data from a Data Source (e.g. a Postgres Database).The difference oftap-google-analyticsto a database tap is that the Catalog (available entities/streams and their schema) is dynamic but not available to be discovered at run time by connecting to the Data Source. It must be dynamically generated based on the reports the user wants to generate by connecting to the Google Analytics Reporting API.To that end, this tap uses an additional JSON file for the definition of the reports that the user wants to be generated. You can check, as an example, the JSON file used as a default intap-google-analytics/defaults/default_report_definition.json. Those report definitions could be part of theconfig.json, but we prefer to keepconfig.jsonsmall and clean and provide the definitions by using an additional file.Based on the report(s) definition, it generates a valid Catalog that follows theSinger spec.It then behaves as any Singer compatible tap and uses that Catalog (or any Catalog generated by atap-google-analytics) to generate the requested reports. The additional JSON file for defining the reports is only required for generating an initial Catalog.When no report definitions are provided by the user,tap-google-analyticsgenerates a default Catalog with some common reports provided:website_overview: Most common metrics (users, new users, sessions, avg session duration, page views, bounce rate, etc) per daytraffic_sources: Most common metrics per day, source, medium and social networkpages: Most common page metrics (page views, unique page views, avg time on page, etc) per day, host name and page pathlocations: Most common metrics per day and various location dimensions (continent, country, region, city, etc)monthly_active_users: Monthly active users (past 30 days) per dayweekly_active_users: Weekly active users (past 7 days) per daydaily_active_users: Daily active users (past 1 day) per daydevices: Most common metrics per day, device category, operating system and browserInstallFirst, make sure Python 3 is installed on your system or follow these
installation instructions forMacorUbuntu.It's recommended to use a virtualenv:python3-mvenvvenvpipinstallpipelinewise-tap-google-analyticsorpython3-mvenvvenv.venv/bin/activatepipinstall--upgradepippipinstall.Authorization Methodstap-google-analyticssupports two different ways of authorization:Service account based authorization, where an administrator manually creates a service account with the appropriate permissions to view the account, property, and view you wish to fetch data fromOAuthaccess_tokenbased authorization, where this tap gets called with a validaccess_tokenandrefresh_tokenproduced by an OAuth flow conducted in a different system.If you're setting uptap-google-analyticsfor your own organization and only plan to extract from a handful of different views in the same limited set of properties, Service Account based authorization is the simplest. When you create a service account Google gives you a json file with that service account's credentials called theclient_secrets.json, and that's all you need to pass to this tap, and you only have to do it once, so this is the recommended way of configuringtap-google-analytics.If you're building something where a wide variety of users need to be able to give access to their Google Analytics,tap-google-analyticscan use anaccess_tokengranted by those users to authorize it's requests to Google. Thisaccess_tokenis produced by a normal Google OAuth flow, but this flow is outside the scope oftap-google-analytics. This is useful if you're integratingtap-google-analyticswith another system, like Stitch Data might do to allow users to configure their extracts themselves without manual config setup. This tap expects anaccess_token,refresh_token,client_idandclient_secretto be passed to it in order to authenticate as the user who granted the token and then access their data.Required Analytics Reporting APIs & OAuth ScopesIn order fortap-google-analyticsto access your Google Analytics Account, it needs the Analytics Reporting APIandthe Analytics API (which are two different things) enabled. If using a service account to authorize, these need to be enabled for a project inside the same organization as your Google Analytics account (see below), or if using an OAuth credential set, they need to be enabled for the project the OAuth client ID and secret come from.If using the OAuth authorization method, the OAuth flow conducted elsewhere must request at minimum theanalytics.readonlyOAuth scope to get anaccess_tokenauthorized to hit these APIsCreating service account credentialsIf you have already have a validclient_secrets.jsonfor a service account, or if you are using OAuth based authorization, you can skip the rest of this section.As a first step, you need to create or use an existing project in the Google Developers Console:Sign in to the Google Account you are using for managing Google Analytics (you must have Manage Users permission at the account, property, or view level).Open theService accounts page. If prompted, select a project or create a new one to use for accessing Google Analytics.Click Create service account.In the Create service account window, type a name for the service account, and select Furnish a new private key. Then click Save and store it locally asclient_secrets.json.If you already have a service account, you can generate a key by selecting 'Edit' for the account and then selecting the option to generate a key.Your new public/private key pair is generated and downloaded to your machine; it serves as the only copy of this key. You are responsible for storing it securely.Add service account to the Google Analytics accountThe newly created service account will have an email address that looks similar to:quickstart@PROJECT-ID.iam.gserviceaccount.comUse this email address toadd a userto the Google analytics view you want to access via the API. For usingtap-google-analyticsonlyRead & Analyze permissionsare needed.Enable the APIsVisit theGoogle Analytics Reporting APIdashboard and make sure that the project you used in theCreate credentialsstep is selected.From this dashboard, you can enable/disable the API for your account, set Quotas and check usage stats for the service account you are using withtap-google-analytics.Visit theGoogle Analytics APIdashboard, make sure that the project you used in theCreate credentialsstep is selected and enable the API for your account.Configuration SettingsA sample config fortap-google-analyticsmight look like this:sample_config.json{"key_file_location":"client_secrets.json",// can also use `oauth_credentials`, see below"view_id":"123456789","reports":"reports.json","start_date":"2019-05-01T00:00:00Z","end_date":"2019-06-01T00:00:00Z"}Required configuration parameters:view_id: The ID for the Google Analytics View you want to fetch data from.You can easily find it by usingGoogle Analytics Account Explorer.start_date: The earliest date you want to fetch data forOptional parameters:end_date: The last date for your report (not included). (Default: Today)We choose to follow an open [start_date, end_date) definition to easily chunk tap executions by month or week (using airflow or other tools).By using that approach, fetching data for May 2019 can be achieved by setting:"start_date": "2019-05-01T00:00:00Z",
"end_date": "2019-06-01T00:00:00Z"key_file_location: If using Service Account based authorization, path to theclient_secrets.jsonfile you generated and downloaded during the "Create credentials" step.oauth_credentials: If using OAuth based authorization, a nested JSON object with the whole config looking like this:{"oauth_credentials":{"access_token":"<ya29.GlxtB_access_token_gobbledegook>","refresh_token":"<ya29.GlxtB_refresh_tokeN_gobbledegook>","client_id":"<something.apps.googleusercontent.com>","client_secret":"<some client secret string>"},"view_id":...}quota_user: an arbitrary string to use as an identifier for the user this library is being invoked on behalf of. Since the Google Analytics APIs have restrictive call quotas, they have a parameter for the API that allows server side applications liketap-google-analyticsto make invocations under different users that each have a quota. Seehttps://developers.google.com/analytics/devguides/reporting/core/v4/limits-quotasandhttps://developers.google.com/analytics/devguides/reporting/core/v4/parametersfor more information. This parameter is only necessary if you are running into 429 Too Many Request errors from the GA API when running this tap.reports: path to a file with the definition of the reports to be generated.If not provided and the tap runs without a--catalogalso provided, usetap-google-analytics/defaults/default_report_definition.jsonas the default definition.Thereports.jsonfile structure expected by thereportsconfig key is really simple:reports.json[
  { "name" : "name of stream to be used",
    "dimensions" :
    [
      "Google Analytics Dimension",
      "Another Google Analytics Dimension",
      ... up to 7 dimensions per stream ...
    ],
    "metrics" :
    [
      "Google Analytics Metric",
      "Another Google Analytics Metric",
      ... up to 10 metrics per stream ...
    ]
  },
  {
  	... another stream definition ...
  },
  ... as many streams / reports as the user wants ...
]For example, if you want to extract user stats per day in a users_per_day stream and session stats per day and country in a sessions_per_country_day stream:[
  { "name" : "users_per_day",
    "dimensions" :
    [
      "ga:date"
    ],
    "metrics" :
    [
      "ga:users",
      "ga:newUsers"
    ]
  },
  { "name" : "sessions_per_country_day",
    "dimensions" :
    [
      "ga:date",
      "ga:country"
    ],
    "metrics" :
    [
      "ga:sessions",
      "ga:sessionsPerUser",
      "ga:avgSessionDuration"
    ]
  }
]You can checktap-google-analytics/defaults/default_report_definition.jsonfor a more lengthy, detailed example.Runtap-google-analytics-cconfig.jsonImplementation detailsThis tap makes some explicit decisions:All Google Analytics Metrics and Dimensions used are preserved with their name changed toga_XXXfromga:XXXe.g.ga:date-->ga_dateAll dimensions are part of the stream's key definition (table-key-properties).The {start_date, end_date} parameters for the report query are also added to the schema as {report_start_date, report_end_date}.This is important for defining the date range the records are for, especially when 'ga:date' is not part of the requested Dimensions.If 'ga:date' has not been added as one of the Dimensions, then the {report_start_date, report_end_date} attributes are also added as keys.For example, if a user requests to see user stats by device or by source, the {start_date, end_date} can be used as part of the key uniquelly identifying the generated stats.That way we can properly identify rows with no date dimension included and also update those rows over overlapping runs of the tap.All streams and attributes are set to"inclusion": "automatic"There is a custom metadata keyword defined for all schema attributes:ga_type: dimension | metricThis keyword is required for processing the catalog afterwards and generating the query to be send to GA API, as dimensions and metrics are not treated equally (they are sent as separate lists of attributes)The tap dynamically fetches the available dimensions and metrics and their data type. We have to use the Google Analytics API V3 for fetching those lists, as they are not provided in the Analytics Reporting API V4.We use those lists to check for invalid dimension or metric names requested by the user and inform her about those errors before the tap runs and starts making requests to the Google Analytics API.We also use the dynamically fetched dimensions and metrics lists to set the data type for those attributes and cast the values accordingly (in case of integer or numeric values)Tap shortcomings (contributions are more than welcome):This tap does not currently use any STATE information for incrementally extracting data. This is currently mitigated by allowing for chunked runs using [start_date, end_date), but we should definitely add support for using STATE messages.The difficulty on that front is on dynamically deciding which attributes to use for capturing state for ad-hoc reports that do not include thega:datedimension or other combinations of Time Dimensions.We may be checking for valid Google Analytics Metrics and Dimensions and report back to the user with errors, but we are not checking for valid combinations. Not all dimensions and metrics can be queried together and we should add a black list of metrics and dimensions that can not be used with specific dimensions (e.g. ga:impressions with ga:browser) or with other metrics (e.g. ga:XXdayUsers with other ga:XXdayUsers or ga:users).TheAvailable Metrics/Dimensions and combos pageprovides a way to discover those restrictions.