Package: indexdigest
index-digestAnalyses your database queries and schema and suggests indices improvements. You can useindex-digestasyour database linter. The goal is toprovide the user with actionable reportsinstead of just a list of statistics and schema details. Inspired byPercona'spt-index-usage.NEWYou can alsouseindex-digestas GitHub's Action.What this tool doesindex-digestdoes the following:it checks the schema of all tables in a given database and suggests improvements (e.g. removal of redundant indices, adding a primary key to ease replication, dropping tables with just a single column or no rows)if provided with SQL queries log (via--sql-logoption) it:checks if all tables, columns and indices are used by these queriesreports text columns with character set different thanutfreports queries that do not use indicesreports queries that use filesort, temporary file or full table scanreports queries that are not quite kosher (e.g.LIKE "%foo%",INSERT IGNORE,SELECT *,HAVINGclause, highOFFSETin pagination queries)if run with--analyze-dataswitch it:reports tables with old data (by querying forMIN()value of time column) where data retency can be reviewedreports tables with not up-to-date data (by querying forMAX()value of time column)if run with--check-empty-databasesswitch it:report empty databases on the current MySQL serverThis toolsupports MySQL 5.7, 8.0, 8.1,Percona Server8.0 and MariaDB 10.1, 10.2, 10.5, 10.6and runs underPython 3.8+.Results can be reported in a human-readable form, as YAML or sent to syslog and later aggregated & processed using ELK stack.Requirements & installFrompypipip install indexdigestFrom gitgit clone git@github.com:macbre/index-digest.git && cd index-digest
sudo apt-get install libmysqlclient-dev python3-dev virtualenv

virtualenv -ppython3 env
source env/bin/activate
make installWhen using MacOS, you should followthismysql_configinstallation steps.Running testsWe assume that the test database is running locally on port 53306. You can use the following to test your changes locally before pushing them (this one uses MySQL 8.0.20):docker run --rm -p 53306:3306 --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3 -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -e "MYSQL_DATABASE=index_digest" --name=index_digest_mysql mysql:8.0.22 "--default-authentication-plugin=mysql_native_password"Wait until the server is up and running.mysql --protocol=tcp --port=53306 -u root --password="" -v < setup.sql
./sql/populate.sh
mysql --protocol=tcp --port=53306 -uindex_digest -pqwerty index_digest -v -e '\s; SHOW TABLES; SHOW DATABASES;'

make testUsing DockerSeehttps://hub.docker.com/r/macbre/index-digest/$ docker run --network=host -t macbre/index-digest:latest mysql://index_digest:qwerty@debian/index_digest  | head -n 20
------------------------------------------------------------
Found 61 issue(s) to report for "index_digest" database
------------------------------------------------------------
MySQL v5.7.22 at debian
index-digest v1.2.0
------------------------------------------------------------
redundant_indices → table affected: 0004_id_foo

✗ "idx" index can be removed as redundant (covered by "PRIMARY")

  - redundant: UNIQUE KEY idx (item_id, foo)
  - covered_by: PRIMARY KEY (item_id, foo)
  - schema: CREATE TABLE `0004_id_foo` (
      `item_id` int(9) NOT NULL AUTO_INCREMENT,
      `foo` varbinary(16) NOT NULL DEFAULT '',
      PRIMARY KEY (`item_id`,`foo`),
      UNIQUE KEY `idx` (`item_id`,`foo`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1
  - table_data_size_mb: 0.015625
  - table_index_size_mb: 0.015625
...How to run it?$ index_digest -h
index_digest

Analyses your database queries and schema and suggests indices improvements.

Usage:
  index_digest DSN [--sql-log=<file>] [--format=<formatter>] [--analyze-data] [--checks=<checks> | --skip-checks=<skip-checks>] [--tables=<tables> | --skip-tables=<skip-tables>]
  index_digest (-h | --help)
  index_digest --version

Options:
  DSN               Data Source Name of database to check
  --sql-log=<file>  Text file with SQL queries to check against the database
  --format=<formatter>  Use a given results formatter (plain, syslog, yaml)
  --analyze-data    Run additional checks that will query table data (can be slow!)
  --checks=<list>   Comma-separated lists of checks to report
  --skip-checks=<list> Comma-separated lists of checks to skip from report
  --tables=<list>   Comma-separated lists of tables to report
  --skip-tables=<list> Comma-separated lists of tables to skip from report
  -h --help         Show this screen.
  --version         Show version.

Examples:
  index_digest mysql://username:password@localhost/dbname
  index_digest mysql://index_digest:qwerty@localhost/index_digest --sql-log=sql.log
  index_digest mysql://index_digest:qwerty@localhost/index_digest --skip-checks=non_utf_columns
  index_digest mysql://index_digest:qwerty@localhost/index_digest --analyze-data --checks=data_too_old,data_not_updated_recently
  index_digest mysql://index_digest:qwerty@localhost/index_digest --analyze-data --skip-tables=DATABASECHANGELOG,DATABASECHANGELOGLOCK

Visit <https://github.com/macbre/index-digest>SQL query logIt's a text file with a single SQL query in each line (no line breaks are allowed). Lines that do start with--(SQL comment) are ignored. The file can begenerated usingquery-digestwhen--sql-logoutput mode is selected.An example:-- A commentselect*from0002_not_used_indicesorderbyidselect*from0002_not_used_indiceswherefoo='foo'andid=2selectcount(*)from0002_not_used_indiceswherefoo='foo'/* foo bar */select*from0002_not_used_indiceswherebar='foo'INSERTIGNOREINTO`0070_insert_ignore`VALUES('123',9,'2017-01-01');FromMySQL slow query logMySQL's slow query log needs to be pre-processed first (to remove comments and timestamps):cat mysql-slow.log | egrep -v '^(SET timestamp|#|throttle: )' > queries.logThen you can runindex_digest --sql-log=queries.log ....Formattersindex-digestcan return results in various formats (use--formatto choose one).plainEmits human-readable report to a console. You can disable colored and bold text by setting env variableANSI_COLORS_DISABLED=1.syslogPushes JSON-formatted messages via syslog, so they can be aggregated using ELK stack.
UseSYSLOG_IDENTenv variable to customize syslog'sidentmessages are sent with (defaults toindex-digest).Dec 28 15:59:58 debian index-digest[17485]: {"meta": {"version": "index-digest v0.1.0", "database_name": "index_digest", "database_host": "debian", "database_version": "MySQL v5.7.20"}, "report": {"type": "redundant_indices", "table": "0004_id_foo", "message": "\"idx\" index can be removed as redundant (covered by \"PRIMARY\")", "context": {"redundant": "UNIQUE KEY idx (id, foo)", "covered_by": "PRIMARY KEY (id, foo)", "schema": "CREATE TABLE `0004_id_foo` (\n  `id` int(9) NOT NULL AUTO_INCREMENT,\n  `foo` varbinary(16) NOT NULL DEFAULT '',\n  PRIMARY KEY (`id`,`foo`),\n  UNIQUE KEY `idx` (`id`,`foo`)\n) ENGINE=InnoDB DEFAULT CHARSET=latin1", "table_data_size_mb": 0.015625, "table_index_size_mb": 0.015625}}}yamlOutputs YML file with results and metadata.ChecksYou can select which checks should be reported by the tool by using--checkscommand line option. Certain checks can also be skipped via--skip-checksoption. Refer toindex_digest --helpfor examples.Number of checks: 24redundant_indices: reports indices that are redundant and covered by othernon_utf_columns: reports text columns that have characters encoding set tolatin1(utf is the way to go)missing_primary_index: reports tables with no primary or unique key (seeMySQL bug #76252andWikia/app#9863).Primary keys can be enforced on MySQL config levelsince 8.0.13 (viasql_require_primary_keyvariable).test_tables: reports tables that seem to be test leftovers (e.g.some_guy_test_table)single_column: reports tables with just a single columnempty_tables: reports tables with no rowsgeneric_primary_key: reports tables witha primary key onidcolumn(a more meaningful name should be used)use_innodb: reports table using storage engines different thanInnoDB(a default for MySQL 5.5+ and MariaDB 10.2+)low_cardinality_index: reportsindices with low cardinalityAdditional checks performed on SQL logYou need to provide SQL log file via--sql-logoptionnot_used_columns: checks which columns were not used by SELECT queriesnot_used_indices: checks which indices are not used by SELECT queriesnot_used_tables: checks which tables are not used by SELECT queriesqueries_not_using_index: reports SELECT queries that do not use any indexqueries_using_filesort: reports SELECT queries that require filesort (a sort can’t be performed from an index and quicksort is used)queries_using_temporary: reports SELECT queries that require a temporary table to hold the resultqueries_using_full_table_scan: reports SELECT queries that require afull table scanselects_with_like: reports SELECT queries that useLIKE '%foo'conditions (they can not use an index)insert_ignore: reportsqueries usingINSERT IGNOREselect_star: reportsqueries usingSELECT *having_clause: reportsqueries usingHAVINGclausehigh_offset_selects: reportSELECT queries using high OFFSETAdditional checks performed on tables dataYou need to use--analyze-datacommand line switch. Please note that these checks will query your tables.These checks can take a while if queried columns are not indexed.data_too_old: reports tables that have really old data, maybe it's worth checking if such long data retention is actually needed (defaults to three months threshold, can be customized viaINDEX_DIGEST_DATA_TOO_OLD_THRESHOLD_DAYSenv variable)data_not_updated_recently: reports tables that were not updated recently, check if it should be up-to-date (defaults a month threshold, can be customized viaINDEX_DIGEST_DATA_NOT_UPDATED_RECENTLY_THRESHOLD_DAYSenv variable)Additional checks performed across database on the current MySQL serverYou need to use--check-empty-databasescommand line switch.empty_database: reports databases that have noBASE TABLEtables (as provided byinformation_schema.TABLES)An example report$index_digestmysql://index_digest:qwerty@localhost/index_digest--sql-log sql/0002-not-used-indices-log------------------------------------------------------------Found85issue(s)toreportfor"index_digest"database------------------------------------------------------------MySQLv5.7.21atdebianindex-digestv1.0.0------------------------------------------------------------redundant_indices→tableaffected:0004_id_foo✗"idx"indexcanberemovedasredundant(coveredby"PRIMARY")-redundant:UNIQUEKEYidx(id,foo)-covered_by:PRIMARYKEY(id,foo)-schema:CREATETABLE`0004_id_foo`(`id`int(9)NOTNULLAUTO_INCREMENT,`foo`varbinary(16)NOTNULLDEFAULT'',PRIMARYKEY(`id`,`foo`),UNIQUEKEY`idx`(`id`,`foo`))ENGINE=InnoDBDEFAULTCHARSET=latin1-table_data_size_mb:0.015625-table_index_size_mb:0.015625------------------------------------------------------------redundant_indices→tableaffected:0004_id_foo_bar✗"idx_foo"indexcanberemovedasredundant(coveredby"idx_foo_bar")-redundant:KEYidx_foo(foo)-covered_by:KEYidx_foo_bar(foo,bar)-schema:CREATETABLE`0004_id_foo_bar`(`id`int(9)NOTNULLAUTO_INCREMENT,`foo`varbinary(16)NOTNULLDEFAULT'',`bar`varbinary(16)NOTNULLDEFAULT'',PRIMARYKEY(`id`),KEY`idx_foo`(`foo`),KEY`idx_foo_bar`(`foo`,`bar`),KEY`idx_id_foo`(`id`,`foo`))ENGINE=InnoDBDEFAULTCHARSET=latin1-table_data_size_mb:0.015625-table_index_size_mb:0.046875------------------------------------------------------------missing_primary_index→tableaffected:0034_querycache✗"0034_querycache"tabledoesnothaveanyprimaryoruniqueindex-schema:CREATETABLE`0034_querycache`(`qc_type`varbinary(32)NOTNULL,`qc_value`int(10)unsignedNOTNULLDEFAULT'0',`qc_namespace`int(11)NOTNULLDEFAULT'0',`qc_title`varchar(255)CHARACTERSETlatin1COLLATElatin1_binNOTNULLDEFAULT'',KEY`qc_type`(`qc_type`,`qc_value`))ENGINE=InnoDBDEFAULTCHARSET=utf8------------------------------------------------------------test_tables→tableaffected:0075_some_guy_test_table✗"0075_some_guy_test_table"seemstobeatesttable-schema:CREATETABLE`0075_some_guy_test_table`(`id`int(9)NOTNULLAUTO_INCREMENT,`name`varchar(255)NOTNULL,PRIMARYKEY(`id`))ENGINE=InnoDBDEFAULTCHARSET=utf8------------------------------------------------------------single_column→tableaffected:0074_bag_of_ints✗"0074_bag_of_ints"hasjustasinglecolumn-schema:CREATETABLE`0074_bag_of_ints`(`id`int(9)NOTNULLAUTO_INCREMENT,PRIMARYKEY(`id`))ENGINE=InnoDBDEFAULTCHARSET=utf8------------------------------------------------------------empty_tables→tableaffected:0089_empty_table✗"0089_empty_table"tablehasnorows,isitreallyneeded?-schema:CREATETABLE`0089_empty_table`(`id`int(9)NOTNULLAUTO_INCREMENT,PRIMARYKEY(`id`))ENGINE=InnoDBDEFAULTCHARSET=latin1------------------------------------------------------------generic_primary_key→tableaffected:0094_generic_primary_key✗"0094_generic_primary_key"hasaprimarykeycalledid,useamoremeaningfulname-schema:CREATETABLE`0094_generic_primary_key`(`id`int(9)NOTNULLAUTO_INCREMENT,`foo`varchar(16)NOTNULLDEFAULT'',PRIMARYKEY(`id`))ENGINE=InnoDBDEFAULTCHARSET=latin1------------------------------------------------------------use_innodb→tableaffected:0036_use_innodb_myisam✗"0036_use_innodb_myisam"usesMyISAMstorageengine-schema:CREATETABLE`0036_use_innodb_myisam`(`item_id`int(9)NOTNULLAUTO_INCREMENT,`foo`int(8)DEFAULTNULL,PRIMARYKEY(`item_id`))ENGINE=MyISAMDEFAULTCHARSET=latin1-engine:MyISAM------------------------------------------------------------not_used_indices→tableaffected:0002_not_used_indices✗"test_id_idx"indexwasnotusedbyprovidedqueries-not_used_index:KEYtest_id_idx(test,id)------------------------------------------------------------not_used_tables→tableaffected:0020_big_table✗"0020_big_table"tablewasnotusedbyprovidedqueries-schema:CREATETABLE`0020_big_table`(`id`int(9)NOTNULLAUTO_INCREMENT,`val`int(9)NOTNULL,`text`char(5)NOTNULL,PRIMARYKEY(`id`),KEY`text_idx`(`text`))ENGINE=InnoDBAUTO_INCREMENT=100001DEFAULTCHARSET=utf8-table_size_mb:5.03125-rows_estimated:100405------------------------------------------------------------insert_ignore→tableaffected:0070_insert_ignore✗"INSERT IGNORE INTO `0070_insert_ignore` VALUES (9,..."queryusesariskyINSERTIGNORE-query:INSERTIGNOREINTO`0070_insert_ignore`VALUES(9,'123','2017-01-01');-schema:CREATETABLE`0070_insert_ignore`(`id`int(9)NOTNULL,`text`char(5)NOTNULL,`time`datetimeDEFAULTNULL,UNIQUEKEY`id`(`id`))ENGINE=InnoDBDEFAULTCHARSET=utf8------------------------------------------------------------non_utf_columns→tableaffected:0032_latin1_table✗"name"textcolumnhas"latin1"charactersetdefined-column:name-column_character_set:latin1-column_collation:latin1_swedish_ci-schema:CREATETABLE`0032_latin1_table`(`item_id`int(9)NOTNULLAUTO_INCREMENT,`name`varchar(255)DEFAULTNULL,`utf8_column`varchar(255)CHARACTERSETutf8COLLATEutf8_polish_ciNOTNULL,`ucs2_column`varchar(255)CHARACTERSETucs2DEFAULTNULL,`utf8mb4_column`varchar(255)CHARACTERSETutf8mb4DEFAULTNULL,`utf16_column`varchar(255)CHARACTERSETutf16DEFAULTNULL,`utf32_column`varchar(255)CHARACTERSETutf32DEFAULTNULL,`binary_column`varbinary(255)DEFAULTNULL,`latin_blob`blob,PRIMARYKEY(`item_id`))ENGINE=InnoDBDEFAULTCHARSET=latin1------------------------------------------------------------(...)------------------------------------------------------------queries_using_filesort→tableaffected:0020_big_table✗"SELECT val, count(*) FROM 0020_big_table WHERE id ..."queryusedfilesort-query:SELECTval,count(*)FROM0020_big_tableWHEREidBETWEEN10AND20GROUPBYval-explain_extra:Usingwhere;Usingtemporary;Usingfilesort-explain_rows:11-explain_filtered:None-explain_key:PRIMARY------------------------------------------------------------queries_using_temporary→tableaffected:0020_big_table✗"SELECT val, count(*) FROM 0020_big_table WHERE id ..."queryusedtemporary-query:SELECTval,count(*)FROM0020_big_tableWHEREidBETWEEN10AND20GROUPBYval-explain_extra:Usingwhere;Usingtemporary;Usingfilesort-explain_rows:11-explain_filtered:None-explain_key:PRIMARY------------------------------------------------------------queries_using_full_table_scan→tableaffected:0020_big_table✗"SELECT * FROM 0020_big_table"querytriggeredfulltablescan-query:SELECT*FROM0020_big_table-explain_rows:9041------------------------------------------------------------selects_with_like→tableaffected:0020_big_table✗"SELECT * FROM 0020_big_table WHERE text LIKE '%00'"queryusesLIKEwithleft-mostwildcard-query:SELECT*FROM0020_big_tableWHEREtextLIKE'%00'-explain_extra:Usingwhere-explain_rows:100623------------------------------------------------------------select_star→tableaffected:bar✗"SELECT t.* FROM bar AS t"queryusesSELECT*-query:SELECTt.*FROMbarASt;------------------------------------------------------------having_clause→tableaffected:sales✗"SELECT s.cust_id,count(s.cust_id) FROM SH.sales s ..."queryusesHAVINGclause-query:SELECTs.cust_id,count(s.cust_id)FROMSH.salessGROUPBYs.cust_idHAVINGs.cust_id!='1660'ANDs.cust_id!='2'(...)------------------------------------------------------------low_cardinality_index→tableaffected:0020_big_table✗"num_idx"indexon"num"columnhaslowcardinality,checkifitisneeded-column_name:num-index_name:num_idx-index_cardinality:2-schema:CREATETABLE`0020_big_table`(`item_id`int(9)NOTNULLAUTO_INCREMENT,`val`int(9)NOTNULL,`text`char(5)NOTNULL,`num`int(3)NOTNULL,PRIMARYKEY(`item_id`),KEY`text_idx`(`text`),KEY`num_idx`(`num`))ENGINE=InnoDBAUTO_INCREMENT=100001DEFAULTCHARSET=utf8-value_usage:33.24788541334185(...)------------------------------------------------------------data_too_old→tableaffected:0028_data_too_old✗"0028_data_too_old"hasrowsadded184daysago,considerchangingretentionpolicy-diff_days:184-data_since:2017-08-1712:03:44-data_until:2018-02-1712:03:44-date_column_name:timestamp-schema:CREATETABLE`0028_data_too_old`(`item_id`int(8)unsignedNOTNULLAUTO_INCREMENT,`cnt`int(8)unsignedNOTNULL,`timestamp`timestampNOTNULLDEFAULTCURRENT_TIMESTAMPONUPDATECURRENT_TIMESTAMP,PRIMARYKEY(`item_id`))ENGINE=InnoDBAUTO_INCREMENT=5DEFAULTCHARSET=latin1-rows:4-table_size_mb:0.015625------------------------------------------------------------data_not_updated_recently→tableaffected:0028_data_not_updated_recently✗"0028_data_not_updated_recently"hasthelatestrowadded40daysago,considercheckingifitshouldbeup-to-date-diff_days:40-data_since:2017-12-2912:03:44-data_until:2018-01-0812:03:44-date_column_name:timestamp-schema:CREATETABLE`0028_data_not_updated_recently`(`item_id`int(8)unsignedNOTNULLAUTO_INCREMENT,`cnt`int(8)unsignedNOTNULL,`timestamp`timestampNOTNULLDEFAULTCURRENT_TIMESTAMPONUPDATECURRENT_TIMESTAMP,PRIMARYKEY(`item_id`))ENGINE=InnoDBAUTO_INCREMENT=4DEFAULTCHARSET=latin1-rows:3-table_size_mb:0.015625------------------------------------------------------------high_offset_selects→tableaffected:page✗"SELECT /* CategoryPaginationViewer::processSection..."queryusestoohighoffsetimpactingtheperformance-query:SELECT/* CategoryPaginationViewer::processSection */page_namespace,page_title,page_len,page_is_redirect,cl_sortkey_prefixFROM`page`INNERJOIN`categorylinks`FORCEINDEX(cl_sortkey)ON((cl_from=page_id))WHEREcl_type='page'ANDcl_to='Spotify/Song'ORDERBYcl_sortkeyLIMIT927600,200-limit:200-offset:927600------------------------------------------------------------empty_database→tableaffected:index_digest_empty✗"index_digest_empty"databasehasnotables------------------------------------------------------------Queriesperformed:100Success storiesWant to add your entry here? Submit a pull requestBy runningindex-digestatWikiaon shared database clusters (including tables storing ~450 mm of rows with 300+ GiB of data) we were able toreclaim around 1.25 TiB of MySQL storage space across all replicas.Read morePercona Database Performance BlogHigh Performance MySQL, 3rd Edition by Vadim Tkachenko, Peter Zaitsev, Baron SchwartzPercona | Indexing 101: Optimizing MySQL queries on a single tablePercona |pt-index-usage/find unused indexesSlidesPercona | MySQL Indexing: Best Practices