Package: mindexer
mindexerAn experimental tool to recommend indexes for MongoDB based on a query workload and sample of the data.Descriptionmindexeris a command line tool written in Python to recommend indexes for MongoDB. It uses queries logged to thesystem.profilecollection (find out more aboutprofiling in MongoDB) and a small random sample of the original collection to determine the indexes best suited for a workload and dataset.DisclaimerPlease note: This tool is not officially supported or endorsed by MongoDB Inc. The code is released for use "AS IS" without any warranties of any kind, including, but not limited to its installation, use, or performance. Do not run this tool against critical production systems. It is recommended to usemindexerin a test / QA environment that closely resembles your production system.InstallationThis tool requires python 3.x and pip on your system. To installmindexer, run the following command:pipinstallmindexerUsagemindexerproposes indexes for a collection by evaluating index candidates based on the query workload and the data distribution.Step 1. Collect QueriesIn order to provide queries tomindexer, you need to record a typical workload to thesystem.profilecollection using theMongoDB Profiler.Ideally, you should first turn off the profiler, delete thesystem.profilecollection for the database in which the collection resides, then re-enable the profiler and run the query workload against the collection.Assuming you want to determine indexes for themydatabase.mycollectioncollection, you can run the following commands in themongosh shell:// switch to the correct databaseusemydatabase// disable the profilerdb.setProfilingLevel(0)// delete the system.profile collectiondb.system.profile.drop()// re-enable the profiler at level 2, logging all operationsdb.setProfilingLevel(2)For more fine-grained control over which operations are logged to the profiler, you can specify filters and sample rate, as discussed in theMongoDB Profilerdocumentation. For example, you could limit profiling to onlyquerieson the target collection like so:// re-enable the profiler to log only queries on the mydatabase.mycollection namespacedb.setProfilingLevel(2,{filter:{op:"query",ns:"mydatabase.mycollection"},});Once the workload has been collected, you can disable the profiler again as shown above.Step 2. Execute mindexerWith the workload recorded insystem.profileyou can runmindexer, providing the MongoDB URI connection string, database and collection name for which to recommend indexes.Examplemindexer --uri mongodb://my.mongodb.url:27017 --db mydatabase --collection mycollectionmindexerqueries thesystem.profilecollection to find all queries related to the provided collection. Currently, only a subset of the query language is supported, see below for limitations. Unsupported queries will be skipped.mindexerthen extracts a sample of the original collection and stores it in a temporary collection in themindexer_samplesdatabase. By default,mindexerextracts 0.1% of the collection size (sample ratio = 0.001) using an aggregation with a$samplestage, but this is configurable via the--sample-ratiocommand line argument.Based on the found queries,mindexerproduces a list of potential index candidates, which are then evaluated by running a number of queries against the sample collection extracted in the previous step. These queries help determine cardinalities of sub-queries to score the index candidates.The final output is a list of indexes thatmindexerdetermined to be beneficial. The indexes are sorted in order of their scores, with the top indexes having the highest scores (and thus most benefit given the workload).Limitationsmindexeris an early prototype and does not support the full MongoDB query language yet.mindexersupportsfindqueries over one or more predicates (fields) with the following query operators: equality (via simple key/value pairs, e.g.{foo: "bar"}), ranges ($gt,$gte,$lt,$lte),$in,$exists,$regex,$sizeand negations$neand$nin. Also supported are projections, sorts and limits.The following features are not yet supported:Disjunctions in queries ($or,$nor) or multiply nested$andconjunctions (top-level$andis supported)Aggregation pipelines via theaggregatecommandMore advanced query operators like$textCompound index mixed sort order is currently not considered, all fields are sorted in ascending (1) orderAnything not explicitly mentioned as supported aboveTestsTo execute the unit tests, run from the top-level directory:python -m unittest discover ./tests