Project: TiDB (App Definition and Development / Database)
File: 2020-09-30-index-usage-information.md

Author(s): Last updated: Sep. 30, 2020 Discussion at: N/A This document describes the design of recording index usage information. There may be unused indexes in the database. In addition, modifying database tables, indexes, and query statements may cause some indexes to not be used in the future. Users may want to clear these long-unused indexes to relieve storage and performance pressure. Related issues: https://github.com/pingcap/tidb/issues/14998 https://github.com/pingcap/tidb/issues/14607 https://github.com/pingcap/tidb/issues/17508 https://asktug.com/t/topic/34614/12 Design system tables to record index usage information. The system table is designed as follows: The system table is located in `mysql` database and is named `SCHEMAINDEXUSAGE`. Columns of `SCHEMAINDEXUSAGE`: | Column name | Data type | Description | |--|--|-| | TABLE_ID | bigint | ID of the table or view on which the index is defined.| | INDEX_ID | bigint | ID of the index.| | QUERY_COUNT | longlong | Number of the SQL using this index.| | ROWSSELECTED| longlong | Number of rows read from the index. We can check the average fetched rows count of each query of the index through `ROWSREAD` / `QUERY_COUNT`.| | LASTUSEDAT | timestamp | The last time of the SQL using this index.| Typical usage: `select * from mysql.SCHEMAINDEXUSAGE`. ```sql create table SCHEMAINDEXUSAGE ( TABLE_ID bigint(21), INDEX_ID bigint(21), QUERY_COUNT BIGINT, ROUWS_SELECTED BIGINT, LASTUSEDAT timestamp, PRIMARY KEY (SCHEMANAME, TABLENAME, INDEX_NAME) ); ``` According to the , `INDEXID` should be `bigint(21)`. Similarly, tells us that `TABLEID` should be `bigint(21)`. ```sql insert into mysql.SCHEMAINDEXUSAGE values (schemaA, tableA, indexA, count, row, used_at) on duplicate key update querycount=querycount+count, rowsselected=rowsselected+row, lastusedat=greatest(lastusedat, used_at) ``` Statistics update is divided into statistics and persistence of index usage information. Index usage information is counted by the exec-info collector. And persistence is periodically writing data to system tables. We add a global variable `index-usage-sync-lease` to control the persistence cycle. It is set to 1 minute by default. In addition, add a global SQL variable to control whether to turn on this feature. Due to MySQL compatibility, add the system table `SCHEMAUNUSEDINDEXES`. Create a view `SCHEMAUNUSEDINDEXES` on table `SCHEMAINDEXUSAGE`. Columns of it: | Column name | Data type | Description | | -- | | | | object_schema | varchar | The schema name. | | object_name | varchar | The table name. | | index_name | varchar | The unused index name.| ```sql create view informationschema.schemaunused_indexes as select i.tableschema as tableschema, i.tablename as tablename, i.indexname as indexname from mysql.tidbindexes as i left join mysql.schemaindex_usage as u on i.tableschema=u.tableschema and i.tablename=u.tablename and i.indexname=u.indexname where u.querycount=0 or u.querycount is null; ``` We use `TABLEID` and `INDEXID` as ID `mysql.SCHEMAINDEXUSAGE`. Because of `TABLEID` and `INDEXID` is not user-friendly, we need a more user-friendly view. Columns of it: | Column name | Data type | Description | |--|--|-| | TABLE_SCHEMA | varchar | Name of the database on which the table or view is

| TABLE_NAME | varchar | Name of the table or view on which the index is defined.| | INDEX_NAME | varchar | Name of the index.| | QUERY_COUNT | longlong | Number of the SQL using this index.| | ROWSSELECTED| longlong | Number of rows read from the index. We can check the average fetched rows count of each query of the index through `ROWSREAD` / `QUERY_COUNT`.| | LASTUSEDAT | timestamp | The last time of the SQL using this index.| ```sql create view informationschema.schemaindex_usage as select idx.tableschema as tableschema, idx.tablename as tablename, idx.keyname as indexname, stats.querycount as querycount, stats.rowsselected as rowsselected from mysql.schemaindexusage as stats, informationschema.tidbindexes as idx, information_schema.tables as tables where tables.tablename = idx.tableschema AND tables.tablename = idx.tablename AND tables.tidbtableid = stats.table_id AND idx.indexid = stats.indexid ``` Users may have just completed a deployment which changes query patterns such that they expect there will be unused indexes. They can potentially look at the `LASTUSEDAT` column, but sometimes flushing is more desired. Similar usage: `FLUSH INDEXSTATISTICS` from https://www.percona.com/doc/percona-server/LATEST/diagnostics/userstats.html. SQL Syntax: `FLUSH SCHEMAINDEXUSAGE` Users can use this to initialize SCHEMAINDEXUSAGE as ```sql delete from mysql.schemaindexusage; ``` And it needs a check. Read privilege: Reading SCHEMAINDEXUSAGE and SCHEMAUNUSEDINDEXES need permission. A user can see only the rows in the tables that correspond to tables for which the user has the proper access privileges, such as information_schema.tables. Write privilege: These tables are read-only. You can use FLUSH SCHEMAINDEXUSAGE command to reinitialize tables, which requires the RELOAD privilege. Doc: The view shows indexes for which there are no events, which indicates that they are not being used. By default, rows are sorted by schema and table. User interface SQL: `select * from schemaunusedindexes;` Columns of schemaunusedindexes object_schema (The schema name) object_name (The table name) index_name (The unused index name) The data for this view comes from the . The table contains the following columns objecttypje, objectschema, objectname, indexname In addition, there are columns related to statistical information with different granularities, such as: statistical information of all read operations, statistical information of write operations, or statistical information of all operations. Doc: returns counts of different types of index operations and the time each type of operation was last performed. User interface SQL: `select * from sys.dmdbindexusagestats;` Columns of sys.dmdbindexusagestats database_id object_id index_id The number and final execution time of different types of index operations, including seek, scan, lookup, update. Each operation distinguishes user operations and system operations Whenever the index is used, the information in the table will be updated. Doc: displays statistics about index usage gathered from the database. User interface SQL: `select * from v$object_usage;` Columns of v$object_usage index_name table_name monitoring used start_monitoring end_monitoring Oracle can set whether to monitor an index. MySQL supports `SCHEMAUNUSEDINDEXES`. We are considering compatibility and also support this view. My implementation plan is in The test method is similar to general statistics.