Package: tracker-exporter
Yandex.Tracker ETLExport issue metadata & agile metrics, transform and load to OLAP data storage. Metrics based on issue changelog.⚠️ImportantVersions 1.x.x incompatible with 0.1.x. New versions works only on Python >= 3.10You can fork this repository and refine the tool the way you want. Or use it as it is - this will allow you to build basic analytics on the tasks from Yandex.Tracker.Require:Python>=3.10.*Clickhouse + specifictables(how to runmigration)What does this tool do?ETL– Export, transform, load.It's simple. It doesn't do anything supernatural, it doesn't have Rocket Science in it.This is a simple ant with some mathematical abilities that takes data from one place, sorts/transforms/adapts/calculate them and puts them in another place.Sometimes he has to go to a lot of endpoints to collect what needs to be taken to the storage (that's the way Yandex.Tracker API).Collects:Issue metadata (i.e. title, author, assignee, components, tags, status, etc)Issue changelog (i.e the history of all the events that occurred with the task)Calculated issue metrics by status (i.e. the time spent in a particular status)Tech statsMetrics based on 100,000+ constantly changing production issuesCPU usage: from2%to10%Memory usage (RSS):from48MBto256MBAverage processing time per issue (metrics + issue metadata): 1.5 secondsAverage processing time per issue (with full changelog export):7 secondsWhy is it taking so long?This is how the tracker API and the library I use work. To get additional information about the task, you need to make a subquery in the API. For example, get the status name, employee name, and so on. When collecting data about a single task, more than several dozen HTTP requests can be executed.This is also the answer to the question why the tool is not asynchronous. Limits in the API would not allow effective use of concurrency.The processing speed of one issue depends on how many changes there are in the issue in its history. More changes means longer processing.Extend exported issue data by your custom fieldsJust declare yourmain.pymodule in which extended theTrackerIssuemodel using multiple inheritance likefromtracker_exporter.models.issueimportTrackerIssuefromtracker_exporter.utils.helpersimportvalidate_resourcefromtracker_exporterimportrun_etlclassCustomIssueFields:def__init__(self,issue:Issues)->None:self.foo_custom_field=validate_resource(issue,"fooCustomField")self.bar_custom_field=validate_resource(issue,"barCustomField")classExtendedTrackerIssue(TrackerIssue,CustomIssueFields):def__init__(self,issue:Issues)->None:super().__init__(issue)CustomIssueFields.__init__(self,issue)run_etl(issue_model=ExtendedTrackerIssue)See full examplehereUsageNativeInstall from source# prepare virtual environmentpython3-mvenvvenvsourcevenv/bin/activate
makeinstall# configure environment variablesexportEXPORTER_TRACKER__TOKEN=your_tokenexportEXPORTER_TRACKER__CLOUD_ORG_ID=your_org_idexportEXPORTER_CLICKHOUSE__HOST=localhostexportEXPORTER_CLICKHOUSE__PORT=8123exportEXPORTER_STATEFUL="true"exportEXPORTER_STATE__STORAGE=jsonfileexportEXPORTER_STATE__JSONFILE_STRATEGY=localexportEXPORTER_STATE__JSONFILE_PATH=./state.json# runtracker-exporterInstall from PyPIpip3installtracker-exporter
tracker-exporterConfigure via .env fileRead about the settingsheretracker-exporter--env-file/home/akimrx/tracker/.settingsDockercdyandex-tracker-exporter/docker
docker-composeup-d
dockerlogstracker-exporter-fOn-premise arch exampleOn-premise ClickhouseSo, you can install Clickhouse with Proxy viaAnsible role inside project (previous versions).Edit the inventory fileansible/inventory/hosts.ymland just run ansible-playbook.Attention:For the role to work correctly, docker must be installed on the target server.Example Clickhouse installation:gitclonehttps://github.com/akimrx/yandex-tracker-exporter.gitcdyandex-tracker-exporter
gitcheckoutv0.1.19
python3-mvenvvenv&&sourcevenv/bin/activate
pip3install-rrequirements-dev.txtcdansible
ansible-playbook-iinventory/hosts.ymlplaybooks/clickhouse.yml--limitagileAlso, you can usethis extended Clickhouse roleYandex.Cloud – Cloud FunctionsCreate a Managed Clickhouse clusterHow to:https://cloud.yandex.com/en/docs/managed-clickhouse/operations/cluster-createSet user for exporter, example:agileSet a database name, example:agileEnableServerless accessflagFor testing enable host public accessEnableAccess from the management consoleflagRun migration or manual create tables (see migration blockhere, seesql)Create Cloud FunctionHow to:https://cloud.yandex.com/en/docs/functions/quickstart/create-function/python-function-quickstartUse Python >= 3.10Copy/paste example content fromexamples/serverless(code)Set entrypoint:main.handler(for code from examples)Set function timeout to600, because the launch can be long if there are a lot of updated issues during the collection periodSet memory to512MBor moreAdd environment variables (see variables blockhere)EXPORTER_TRACKER__TOKEN=XXXXXXXXXXXXXXXXEXPORTER_TRACKER__CLOUD_ORG_ID=123456EXPORTER_TRACKER__SEARCH__RANGE=2hEXPORTER_ENABLE__UPLOAD="true"EXPORTER_CLICKHOUSE__PROTO=httpsEXPORTER_CLICKHOUSE__CACERT_PATH=/etc/ssl/certs/ca-certificates.crtEXPORTER_CLICKHOUSE__PORT=8443EXPORTER_CLICKHOUSE__HOST=rc1b-xxxxxx.mdb.yandexcloud.netEXPORTER_CLICKHOUSE__USER=agileEXPORTER_CLICKHOUSE__PASSWORD=xxxxEXPORTER_CHANGELOG_EXPORT_ENABLED="false"Release functionRun testSee logsServerless database connection without public accessIf you don't want to enable clickhouse public access, use service account with such permissions -serverless.mdbProxies.userand set environment variables below:EXPORTER_CLICKHOUSE__HOST=akfd3bhqk3xxxxxxxxxxx.clickhouse-proxy.serverless.yandexcloud.netEXPORTER_CLICKHOUSE__SERVERLESS_PROXY_ID=akfd3bhqk3xxxxxxxxxxxxxHow to create database connection:https://cloud.yandex.com/en/docs/functions/operations/database-connectionAlso, theEXPORTER_CLICKHOUSE__PASSWORDvariable with service account must be replaced by IAM-token. Keep this in mind.
Probably, you should get it in the function code, because the IAM-token works for a limited period of time.Create TriggerHow to:https://cloud.yandex.com/en/docs/functions/quickstart/create-trigger/timer-quickstartCreate new triggerChoose typeTimerSet interval every hour:0 * ? * * *Select your functionCreate serverless service account or use an existing oneSave triggerYandex.Cloud – Serverless ContainersTODOVisualizationYou can use any BI/observability tool for visualization, for example:Yandex DataLens (btw, this isopensource)Apache SupersetPowerBIGrafanaMigrationBased ongo-migratetool.Download and install go-migrate toolmacOSwgethttps://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.darwin-amd64.tar.gz-Omigrate.tar.gz

tarxvfmigrate.tar.gz
mvmigrate~/binLinuxwgethttps://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.linux-amd64.tar.gz-Omigrate.tar.gz

tar-xvfmigrate.tar.gz
mvmigrate/usr/local/binRun migrationExample bash script below.See full example scripthere#!/usr/bin/env bashset-EeuopipefailCLICKHOUSE_HOST="localhost"CLICKHOUSE_TCP_PORT=9000CLICKHOUSE_HTTP_PORT=8123CLICKHOUSE_USER="default"CLICKHOUSE_PASSWORD="strongpassword"MIGRATION_SOURCE_PATH="file://${PWD}/../migrations/clickhouse"MIGRATION_HISTORY_TABLE="ci_gomigrate_migrations"MIGRATION_DATABASE="agile"MIGRATION_CLICKHOUSE_DSN="clickhouse://${CLICKHOUSE_HOST}:${CLICKHOUSE_TCP_PORT}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${MIGRATION_DATABASE}&x-multi-statement=true&x-migrations-table=${MIGRATION_HISTORY_TABLE}"prepare_migration(){echo"CREATE DATABASE IF NOT EXISTS${MIGRATION_DATABASE}"|\curl"http://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?user=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}"--data-binary@-}run_migration(){migrate-verbose\-source$MIGRATION_SOURCE_PATH\-database$MIGRATION_CLICKHOUSE_DSN\up}prepare_migration
run_migrationConfiguration via environment variablesSee config declarationhereGeneral settingsvariabledescriptionEXPORTER_STATEFULEnable stateful mode. RequiredEXPORTER_STATE__*params. Default isFalseEXPORTER_STATEFUL_INITIAL_RANGEInitial search range when unknown last state. Default:1wEXPORTER_CHANGELOG_EXPORT_ENABLEDEnable export all issues changelog to Clickhouse.Can greatly slow down exports(x5 - x10). Default isFalseEXPORTER_LOGLEVELETL log level. Default:infoEXPORTER_LOG_ETL_STATSEnable logging transform stats every N iteration. Default isTrueEXPORTER_LOG_ETL_STATS_EACH_N_ITERHow many iterations must pass to log stats. Default is100EXPORTER_WORKDAYSWorkdays for calculate business time. 0 - mon, 6 - sun. Default:[0,1,2,3,4]EXPORTER_BUSINESS_HOURS_STARTBusiness hours start for calculate business time. Default:09:00:00EXPORTER_BUSINESS_HOURS_ENDBusiness hours end for calculate business time. Default:22:00:00EXPORTER_DATETIME_RESPONSE_FORMATYandex.Tracker datetime format in responses. Default:%Y-%m-%dT%H:%M:%S.%f%zEXPORTER_DATETIME_QUERY_FORMATDatetime format for search queries. Default:%Y-%m-%d %H:%M:%SEXPORTER_DATETIME_CLICKHOUSE_FORMATDatetime format for Clickhouse. Default:%Y-%m-%dT%H:%M:%S.%fEXPORTER_ETL_INTERVAL_MINUTESInterval between run ETL. Default:30(minutes)EXPORTER_CLOSED_ISSUE_STATUSESStatuses for mark issue as closed. Default:closed,rejected,resolved,cancelled,releasedEXPORTER_NOT_NULLABLE_FIELDSFields that should never be null (e.g. dates). Default: all datetime fieldsTracker settingsvariabledescriptionEXPORTER_TRACKER__LOGLEVELLog level for Yandex.Tracker SDK. Default:warningEXPORTER_TRACKER__TOKENOAuth2 token. Required ifEXPORTER_TRACKER__IAM_TOKENis not passedEXPORTER_TRACKER__ORG_IDYandex360 organization ID. Required ifEXPORTER_TRACKER__CLOUD_ORG_IDis not passedEXPORTER_TRACKER__IAM_TOKENYandex.Cloud IAM token. Required ifEXPORTER_TRACKER__TOKENis not passedEXPORTER_TRACKER__CLOUD_ORG_IDYandex.Cloud organization ID. Required ifEXPORTER_TRACKER__ORG_IDis not passedEXPORTER_TRACKER__TIMEOUTYandex.Tracker HTTP requests timeout. Default:10(sec)EXPORTER_TRACKER__MAX_RETRIESYandex.Tracker HTTP requests max retries. Default:10EXPORTER_TRACKER__LANGUAGEYandex.Tracker language. Default:enEXPORTER_TRACKER__TIMEZONEYandex.Tracker timezone. Default:Europe/MoscowEXPORTER_TRACKER__SEARCH__QUERYCustom query for search issues. This variable has the highest priority and overrides other search parameters. Default is emptyEXPORTER_TRACKER__SEARCH__RANGESearch issues window. Has no effect in stateful mode. Default:2hEXPORTER_TRACKER__SEARCH__QUEUESInclude or exclude queues in search. Example:DEV,SRE,!TEST,!TRASHDefault is emptyEXPORTER_TRACKER__SEARCH__PER_PAGE_LIMITSearch results per page. Default:100Clickhouse settingsvariabledescriptionEXPORTER_CLICKHOUSE__ENABLE_UPLOADEnable upload data to Clickhouse. Default isTrueEXPORTER_CLICKHOUSE__HOSTClickhouse host. Default:localhostEXPORTER_CLICKHOUSE__PROTOClickhouse protocol: http or https. Default:httpEXPORTER_CLICKHOUSE__PORTClickhouse HTTP(S) port. Default:8123EXPORTER_CLICKHOUSE__CACERT_PATHPath to CA cert. Only for HTTPS proto. Default is emptyEXPORTER_CLICKHOUSE__SERVERLESS_PROXY_IDYandex Cloud Functions proxy ID. Default is emptyEXPORTER_CLICKHOUSE__USERNAMEClickhouse username. Default:defaultEXPORTER_CLICKHOUSE__PASSWORDClickhouse password. Can be empty. Default is emptyEXPORTER_CLICKHOUSE__DATABASEClickhouse database. Default:agileEXPORTER_CLICKHOUSE__ISSUES_TABLEClickhouse table for issues metadata. Default:issuesEXPORTER_CLICKHOUSE__ISSUE_METRICS_TABLEClickhouse table for issue metrics. Default:issue_metricsEXPORTER_CLICKHOUSE__ISSUES_CHANGELOG_TABLEClickhouse table for issues changelog. Default:issues_changelogEXPORTER_CLICKHOUSE__AUTO_DEDUPLICATEExecuteOPTIMIZEafter eachINSERT. Default isTrueEXPORTER_CLICKHOUSE__BACKOFF_BASE_DELAYBase delay for backoff strategy. Default:0.5(sec)EXPORTER_CLICKHOUSE__BACKOFF_EXPO_FACTORExponential factor for multiply every try. Default:2.5(sec)EXPORTER_CLICKHOUSE__BACKOFF_MAX_TRIESMax tries for backoff strategy. Default:3EXPORTER_CLICKHOUSE__BACKOFF_JITTEREnable jitter (randomize delay) for retries. Default:TrueState settingsvariabledescriptionEXPORTER_STATE__STORAGEStorage type for StateKeeper. Can be:jsonfile,redis,custom. Default:jsonfileEXPORTER_STATE__REDIS_DSNConnection string for Redis state storage when storage type isredis. Default is empty.EXPORTER_STATE__JSONFILE_STRATEGYFile store strategy forjsonfilestorage type. Can bes3orlocal. Default:localEXPORTER_STATE__JSONFILE_PATHPath to JSON state file. Default:./state.jsonEXPORTER_STATE__JSONFILE_S3_BUCKETBucket fors3strategy. Default is emptyEXPORTER_STATE__JSONFILE_S3_REGIONRegion fors3strategy. Default iseu-east-1EXPORTER_STATE__JSONFILE_S3_ENDPOINTEndpoint URL fors3strategy. Default is emptyEXPORTER_STATE__JSONFILE_S3_ACCESS_KEYAWS access key id fors3strategy. Default is emptyEXPORTER_STATE__JSONFILE_S3_SECRET_KEYAWS secret key fors3strategy. Default is emptyEXPORTER_STATE__CUSTOM_STORAGE_PARAMSSettings for custom storage params asdict. Default:{}Observability settingsvariabledescriptionEXPORTER_MONITORING__METRICS_ENABLEDEnable send statsd tagged metrics. Default isFalseEXPORTER_MONITORING__METRICS_HOSTDogStatsD / statsd host. Default:localhostEXPORTER_MONITORING__METRICS_PORTDogStatsD / statsd port. Default:8125EXPORTER_MONITORING__METRICS_BASE_PREFIXPrefix for metrics name. Default:tracker_exporterEXPORTER_MONITORING__METRICS_BASE_LABELSList of tags for metrics. Example:["project:internal",]. Default is emptyEXPORTER_MONITORING__SENTRY_ENABLEDEnable send exception stacktrace to Sentry. Default isFalseEXPORTER_MONITORING__SENTRY_DSNSentry DSN. Default is emptyMonitoringBased on DogStatsD tagged format. VictoriaMetrics compatible.Metric nameMetric typeLabelsDescriptiontracker_exporter_issue_transform_time_secondstime-Duration of transform per task (data packing to the model)tracker_exporter_issues_total_processed_countcount-Total issues processedtracker_exporter_issues_search_time_secondstime-Yandex.Tracker search duration time in secondstracker_exporter_issues_without_metricscount-Issues with empty metrics (no changelog)tracker_exporter_issue_prefetch_secondstime-Pre-transform data duration in secondstracker_exporter_comments_fetch_secondstime-Comments fetch duration in secondstracker_exporter_etl_duration_secondstime-ETL full pipeline duration in secondstracker_exporter_etl_upload_statusgauge-Last upload status, 1 - success, 2 - failtracker_exporter_export_and_transform_time_secondstime-Overall export and transform duration in secondstracker_exporter_upload_to_storage_time_secondstime-Overall insert duration time in secondstracker_exporter_last_update_timestampgauge-Last data update timestamptracker_exporter_clickhouse_insert_time_secondstimedatabase, tableInsert per table duration time in secondstracker_exporter_clickhouse_inserted_rowscountdatabase, tableInserted rows per tabletracker_exporter_clickhouse_deduplicate_time_secondstimedatabase, tableOptimize execute time duration in seconds