Package: pipelinewise-tap-github
pipelinewise-tap-githubSingertap that produces JSON-formatted data from the GitHub API following theSinger spec.This is aPipelineWisecompatible tap connector.This tap:Pulls raw data from theGitHub REST APIExtracts the following resources from GitHub for a single repository:AssigneesCollaboratorsCommitsIssuesPull RequestsCommentsReviewsReview CommentsStargazersOutputs the schema for each resourceIncrementally pulls data based on the input stateQuick startInstallWe recommend using a virtualenv:python3-mvenvvenv
.venv/bin/activate
pipinstall--upgradepip
pipinstall.Create a GitHub access tokenLogin to your GitHub account, go to thePersonal Access Tokenssettings
page, and generate a new token with at least thereposcope. Save this
access token, you'll need it for the next step.Create the config fileCreate a JSON file containing the required fields and/or the optional ones.
You can decide between allow-list or deny-list strategy combining organization with repos_include and repos_exclude using wildcards.ConfigRequired?Descriptionaccess_tokenyesThe access token to access github apistart_dateyesThe date inclusive to start extracting the dataorganizationnoThe organization you want to extract the data fromrepos_includenoAllow list strategy to extract selected repos data from organization. Supports wildcard matchingrepos_excludenoDeny list to extract all repos from organization except the ones listed. Supports wildcard matchinginclude_archivednotrue/false to include archived repos. Default falseinclude_disablednotrue/false to include disabled repos. Default falserepositoryno(DEPRECATED) Allow list strategy to extract selected repos data from organization(has priority over repos_exclude)max_rate_limit_wait_secondsnoMax time to wait if you hit the github api limit. DEFAULT to 600sExample:{"access_token":"ghp_16C7e42F292c6912E7710c838347Ae178B4a","organization":"singer-io","repos_exclude":"*tests* api-docs","repos_include":"tap* getting-started pipelinewise-github","start_date":"2021-01-01T00:00:00Z","include_archived":false,"include_disabled":false,"max_rate_limit_wait_seconds":800}You can also passsinger-io/tap-github another-org/tap-octopusonrepos_include.For retro compatibility you can passrepository: "singer-io/tap-github singer-io/getting-started":warning:If you have very small repos with total size less than 0.5KB: These will currently be excluded, as the Github repositories API returnssize: 0for these, andtap_github/__init__.pycurrently usessize <= 0as a way to filter out repos with no commits.Run the tap in discovery mode to get properties.json filetap-github--configconfig.json--discover>properties.jsonIn the properties.json file, select the streams to syncEach stream in the properties.json file has a "schema" entry.  To select a stream to sync, add"selected": trueto that stream's "schema" entry.  For example, to sync the pull_requests stream:...
"tap_stream_id": "pull_requests",
"schema": {
  "selected": true,
  "properties": {
    "updated_at": {
      "format": "date-time",
      "type": [
        "null",
        "string"
      ]
    }
...Run the applicationtap-githubcan be run with:tap-github--configconfig.json--propertiesproperties.jsonTo run testsInstall python test dependencies in a virtual env and run nose unit and integration testspython3 -m venv venv
  . venv/bin/activate
  pip install --upgrade pip
  pip install -e .[test]To run unit tests:pytest tests/unittests