Package: django-query-inspector
A collection of tools to:count and trace db queries for debugging purposes or to optimize themrender a Queryset (or a list of dictionaries) in various formatsexport a Queryset to a spreadsheetinspect the SQL activity happening under the hood of a Django projectand more …Contents1Quick start2Does it work?3Query counting4Execute SQL statements5App settings6@query_debugger7Tracing queries in real-time8Inspecting queries in a unit test9Tracing10Inspect a queryset with qsdump11Queryset rendering12Custom rendering13Transposing resulting table14Download the queryset as CSV or Excel file (xlsx)15Generic helpers16Exporters17Helper management commands17.1History18v1.2.619v1.2.520v1.2.421v1.2.322v1.2.223v1.2.124v1.2.025v1.1.1226v1.1.1127v1.1.1028v1.1.929v1.1.830v1.1.731v1.1.632v1.1.533v1.1.434v1.1.335v1.1.236v1.1.137v1.1.038v1.0.939v1.0.840v1.0.741v1.0.642v1.0.543v1.0.444v1.0.345v1.0.246v1.0.147v1.0.048v0.0.649v0.0.550v0.0.451v0.0.352v0.0.253v0.0.11Quick startInstallation:pip install django-query-inspectorAdd “query_inspector” to your INSTALLED_APPS setting like this:INSTALLED_APPS = [
    ...
    'query_inspector',
]Add “QueryCountMiddleware” to your MIDDLEWARE setting like this:MIDDLEWARE = [
    ...
    'query_inspector.middleware.QueryCountMiddleware',
]Optionally, include styles in your base template:<link href="{% static 'query_inspector.css' %}" rel="stylesheet" />Optional dependencies:sqlparsetermcolorpygmentstabulatexlsxwriter2Does it work?Running the unit tests from your project:python manage.py test -v 2 query_inspector --settings=query_inspector.tests.test_settingsRunning the unit tests from your local fork:cd django-query-inspector
./runtests.pyor:coverage run --source='.' runtests.py
coverage report3Query countingA middleware that prints DB query counts in Django’s runserver console output (only in DEBUG mode).Adapted from:Django Querycountby Brad MontgomerySettingMeaningIGNORE_ALL_REQUESTSDisables query countIGNORE_REQUEST_PATTERNSA list of regexp patterns to bypass matching requestsIGNORE_SQL_PATTERNSA list of regexp patterns to bypass matching queriesTHRESHOLDSHow many queries are interpreted as high or medium (and the color-coded output)DISPLAY_ALLTrace all queries (even when not duplicated)DISPLAY_PRETTIFIEDUse pygments and sqlparse for queries tracingCOLOR_FORMATTER_STYLEColor formatter style for PygmentsRESPONSE_HEADERCustom response header that contains the total number of queries executed (None = disabled)DISPLAY_DUPLICATESControls how the most common duplicate queries are displayed (None = displayed)Default settings (to be overridden in projects’ settings):QUERYCOUNT = {
    'IGNORE_ALL_REQUESTS': True,
    'IGNORE_REQUEST_PATTERNS': [],
    'IGNORE_SQL_PATTERNS': [],
    'THRESHOLDS': {
        'MEDIUM': 50,
        'HIGH': 200,
        'MIN_TIME_TO_LOG': 0,
        'MIN_QUERY_COUNT_TO_LOG': 0
    },
    'DISPLAY_ALL': True,
    'DISPLAY_PRETTIFIED': True,
    'COLOR_FORMATTER_STYLE': 'monokai',
    'RESPONSE_HEADER': 'X-DjangoQueryCount-Count',
    'DISPLAY_DUPLICATES': 0,
}4Execute SQL statementsIt is possible to execute a SQL statements against the default db connection using the following helper:query_inspector.sql.perform_query(sql,params,log=False,validate=True)The resulting recordset will be returned as a list of dictionaries.Or, you can save it in the Django admin (model query_inspector.Query),
then click the “Preview” button.If the query contains named parameters (such as%(name)s), a form will be displayed to collect the
actual values before execution.Inspired by:django-sql-dashboarddjango-sql-explorerquery_preview5App settingsQUERY_INSPECTOR_QUERY_SUPERUSER_ONLY = True
QUERY_INSPECTOR_QUERY_DEFAULT_LIMIT = 0
QUERY_INSPECTOR_QUERY_STOCK_QUERIES = []
QUERY_INSPECTOR_QUERY_STOCK_VIEWS = None
DEFAULT_CSV_FIELD_DELIMITER = ';'
QUERY_INSPECTOR_SQL_BLACKLIST = (
    'ALTER',
    'RENAME ',
    ...
QUERY_INSPECTOR_SQL_WHITELIST = (
    'CREATED',
    'UPDATED',
    ...keyexampleSITECOPY_REMOTE_HOSTproject.somewhere.com”SITECOPY_REMOTE_PROJECT_INSTANCEproject”SITECOPY_REMOTE_MEDIA_FOLDER/home/project/public/media/”SITESYNC_WEBSERVER_PROCESS_NAMEproject_gunicorn’SITESYNC_SUPERVISOR_URLhttp://admin:PASSWORD@localhost:9090/RPC2’DUMP_LOCAL_DATA_TARGET_FOLDERBASE_DIR/dumps/localhost’6@query_debuggerDecorator to check how many queries are executed when rendering a specific view.Adapted from:Django select_related and prefetch_related: Checking how many queries reduce using these methods with an exampleby Goutom RoyExamples:from query_inspector import query_debugger

@query_debugger
def tracks_list_view(request):
    ...

class TrackAjaxDatatableView(AjaxDatatableView):

    ...

    @query_debugger
    def dispatch(self, request, *args, **kwargs):
        ...Result:query_debugger7Tracing queries in real-timeOn rare occasions, you might want to trace queries immediately as they happen
while stepping through the code.For that aim, configure the ‘django.db.backends’ logger in your settings;
to print formatted and colored queries, provided pygments and sqlparse have been
installed, use thequery_inspector.log.QueryLogHandlerhandler:LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'db_console': {
            'level': 'DEBUG',
            #'class': 'logging.StreamHandler',
            'class': 'query_inspector.log.QueryLogHandler',
        },
    },
    'loggers': {
        'django.db.backends': {
            'handlers': ['db_console', ],
            'level': 'DEBUG',
        },
    },
}8Inspecting queries in a unit testThis is not obvious, since unit tests are run with DEBUG disabled.Django provides a convenientCaptureQueriesContextfor this:importpprintfromdjangoimportdbfromdjango.test.utilsimportCaptureQueriesContextfromquery_inspectorimportprettyprint_querydeftext_whatever(self):db.reset_queries()withCaptureQueriesContext(db.connection)ascontext:...doyourstuff...num_queries=context.final_queries-context.initial_queriesprint('num_queries:%d'%num_queries)#pprint.pprint(context.captured_queries)forrowincontext.captured_queries:prettyprint_query(row['sql'])print('time: '+row['time'])More examples are available here:Python django.test.utils.CaptureQueriesContext() Examples9TracingSome helper functions are available to print formatted and colored text in the console.Optional requirements:sqlparsetermcolorpygmentstabulateFunctions:def trace(message, color=’yellow’, on_color=None, attrs=None, prompt=’’, prettify=False)Display ‘message’, optionally preceed by ‘prompt’;
If ‘prettify’ is True, format message with pprintColor support provided by:https://pypi.org/project/termcolor/def prettyprint_query(query, params=None, colorize=True, prettify=True, reindent=True)Display the specified SQL statementdef prettyprint_queryset(qs, colorize=True, prettify=True, reindent=True)Display the SQL statement implied by the given querysetdef trace_func(fn):Decorator to detect: function call, input parameters and return valuedef qsdump(* fields, queryset, max_rows=None, render_with_tabulate=True, title=””)See belowdef qsdump2(queryset, include, exclude, max_rows=None, render_with_tabulate=True, title=””)Calls qsdump() building the field list from either “include” or “exclude” parameterResults:prettyprint_querysettrace_func10Inspect a queryset with qsdumpWith qsdump you can:display the formatted SQL statementdisplay the content of the querysetParameters:fields:one or more field names; ‘*’ means ‘all’queryset:the queryset to be inspectedmax_rows:optionally limit the numer of rowsrender_with_tabulate=Trueuse “tabulate” when availabletitle=””optional titleExample:qsdump('*', queryset=tracks, max_rows=10)qsdump11Queryset renderingA few templatetags are available to render either a queryset or a list of dictionaries:def render_queryset_as_table(* fields, queryset, options={})
def render_queryset_as_csv(* fields, queryset, options={})
def render_queryset_as_text(* fields, queryset, options={})Sample usage:{% load static query_inspector_tags %}

<link href="{% static 'query_inspector.css' %}" rel="stylesheet" />

<table class="simpletable smarttable">
    {% render_queryset_as_table "id" "last_name|Cognome" "first_name|Nome" ... queryset=operatori %}
</table>Parameters:queryset: a queryset of a list of dictionaries with data to renderedoptions:max_rows: max n. of rows to be rendered (None=all)format_date:  date formatting string; see:https://docs.djangoproject.com/en/dev/ref/settings/#date-formathttps://docs.djangoproject.com/en/dev/ref/templates/builtins/#dateadd_totals: computes column totals and append results as bottom rowtranspose: flag to transpose the resulting tablefields: a list of field specifiers, espressed as:“fieldname”, or“fieldname|title”, or“fieldname|title|extra_classes”Field “extra classes” with special styles:“percentage”: render column as %“enhanced”“debug-only”render_querysetMore templatetags:def pdb(element)
def ipdb(element)
def format_datetime(dt, include_time=True, include_seconds=False, exclude_date=False)
def format_date(dt)
def format_datetime_with_seconds(dt)
def format_time(t, include_seconds=False)
def format_time_with_seconds(t)
def format_timedelta(td_object, include_seconds=True)
def format_timediff(t1, t2, include_seconds=True)
def timeformat_seconds(seconds)
def timeformat(seconds)
# def format_number(value, decimals, grouping )
def queryset_as_json(qs)
def object_as_dict(instance, fields=None, exclude=None)
def object_as_json(instance, fields=None, exclude=None, indent=0)12Custom renderingFor greated control of the final rendering, you can retrieve headers and data rows separately (as lists)
using:def render_queryset_as_table(* fields, queryset, options={})For example, the equivalent of:print(render_queryset_as_text(*fields,queryset=queryset,options=options))can be reproduced as follows:headers,rows=render_queryset_as_data(*fields,queryset=queryset,options=options)print('|'.join(headers))forrowinrows:print('|'.join(row))print("")13Transposing resulting tableOccasionally, you might need to switch columns and rows in the resulting table;
this can be obtained by adding a‘transpose’: Trueto theoptions.Currently available forrender_queryset_as_data()andrender_queryset_as_table().Alternatively, you can transpose a queryset withdjango-pandasas follows:importpandasaspdfromdjango_pandas.ioimportread_framedf=read_frame(queryset)table_html=df.transpose().to_html()print(table_html)14Download the queryset as CSV or Excel file (xlsx)For historical reasons, we provide two different approaches to export the queryset as a spreadsheet:with the class SpreadsheetQuerysetExporter (seeExportersbelow)parsing the queryset with the aid ofrender_queryset_as_tableThe first requires a proper Queryset, while the second should work with either a Queryset
or a list of dictionares.In both cases, two helper view functions are available to build the HTTP response
required for attachment download:export_any_queryset(request, queryset, filename, excluded_fields=[], included_fields=[], csv_field_delimiter = ";")

export_any_dataset(request, *fields, queryset, filename, csv_field_delimiter = ";")The helper function normalized_export_filename(title, extension) might be used
to build filenames consistently.Sample usage:fromdjango.utilsimporttimezonefromquery_inspector.viewsimportnormalized_export_filenamefromquery_inspector.viewsimportexport_any_datasetdefexport_tracks_queryset(request,file_format='csv'):queryset=Track.objects.select_related('album','album__artist',)filename=normalized_export_filename('tracks',file_format)returnexport_any_queryset(request,queryset,filename,excluded_fields=[],included_fields=[],csv_field_delimiter=";")defexport_tracks_dataset(request,file_format='csv'):queryset=Track.objects.select_related('album','album__artist',)filename='%s_%s.%s'%(timezone.localtime().strftime('%Y-%m-%d_%H-%M-%S'),"tracks",file_format,)fields=["id","name|Track","album|Album",]returnexport_any_dataset(request,*fields,queryset=queryset,filename=filename)then in your template:<divstyle="text-align: right;"><divclass="toolbar"><label>Export Tracks queryset:</label><ahref="/tracks/download_queryset/xlsx/"class="button">Download (Excel)</a><ahref="/tracks/download_queryset/csv/"class="button">Download (CSV)</a></div><br/><divclass="toolbar"><label>Export Tracks dataset:</label><ahref="/tracks/download_dataset/xlsx/"class="button">Download (Excel)</a><ahref="/tracks/download_dataset/csv/"class="button">Download (CSV)</a></div></div>where:urlpatterns=[...path('tracks/download_queryset/csv/',views.export_tracks_queryset,{'file_format':'csv',}),path('tracks/download_queryset/xlsx/',views.export_tracks_queryset,{'file_format':'xlsx',}),path('tracks/download_dataset/csv/',views.export_tracks_dataset,{'file_format':'csv',}),path('tracks/download_dataset/xlsx/',views.export_tracks_dataset,{'file_format':'xlsx',}),...]15Generic helpersdef get_object_by_uuid_or_404(model, uuid_pk)Calls get_object_or_404(model, pk=uuid_pk)
but also prevents “badly formed hexadecimal UUID string” unhandled exceptiondef prettify_json(data)Given a JSON string, returns it as a safe formatted HTML
Sample usage in Model:def summary_prettified(self):
    return prettify_json(self.summary)then add it to the list of readonly_fields in the ModelAdmindef cleanup_queryset(queryset)Remove multiple joins on the same table, if anyWARNING: can alter the origin queryset order16Exportersclass XslxFile(object)XSLX writerRequires: xlsxwriterdef open_xlsx_file(filepath, mode=”rb”)Utility to open an archive supporting the “with” statementSample usage:with open_xlsx_file(filepath) as writer:
    self.export_queryset(writer, fields, queryset)
assert writer.is_closed()class SpreadsheetQuerysetExporter(object)Helper class to export a queryset to a spreadsheet.Sample usage:writer = csv.writer(output, delimiter=field_delimiter, quoting=csv.QUOTE_MINIMAL)
exporter = SpreadsheetQuerysetExporter(writer, file_format='csv')
exporter.export_queryset(
    queryset,
    included_fields=[
        'id',
        'description',
        'category__id',
        'created_by__id',
    ]
)See also:Download the queryset as CSV or Excel file (xlsx)17Helper management commandsA few management commands are provided to:quickly download database and/or media file from a remote project’s instancesave/restore a backup copy of database and/or media files to/from a local backup folderDatabase actions require Postrgresql; downloading from remote instance requires
read access via SSH.You’re advised to double-check implied actions by dry-running these commands
before proceeding.sitecopy: Syncs database and media files from remote project “project” running on remote server “project.somewhere.com”Usage:usage: manage.py sitecopy [-h] [--dry-run] [--quiet] [--host HOST] [-v {0,1,2,3}] [--settings SETTINGS]

Syncs database and media files for project "gallery" from remote server "gallery.brainstorm.it"

optional arguments:
  -h, --help            show this help message and exit
  --dry-run, -d         Dry run (simulate actions)
  --quiet, -q           do not require user confirmation before executing commands
  --host HOST           Default: "gallery.brainstorm.it"
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be used.dump_local_data: Dump local db and media for backup purposes (and optionally remove old backup files)Settings:DUMP_LOCAL_DATA_TARGET_FOLDER = getattr(settings, 'DUMP_LOCAL_DATA_TARGET_FOLDER', os.path.join(settings.BASE_DIR, '..', 'dumps', 'localhost'))Usage:usage: manage.py dump_local_data [-h] [--target target] [--dry-run] [--max-age MAX_AGE] [--no-gzip] [--legacy]
                                 [-v {0,1,2,3}] [--settings SETTINGS]

Dump local db and media for backup purposes (and optionally remove old backup files)

optional arguments:
  -h, --help            show this help message and exit
  --target target, -t target
                        choices: db, media, all; default: db
  --dry-run, -d         Dry run (simulation)
  --max-age MAX_AGE, -m MAX_AGE
                        If > 0, remove backup files old "MAX_AGE days" or more
  --no-gzip             Do not compress result
  --legacy              use legacy Postgresql command syntax
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be used.restore_from_local_data: Restore db and media from local backupsSettings:DUMP_LOCAL_DATA_TARGET_FOLDER = getattr(settings, 'DUMP_LOCAL_DATA_TARGET_FOLDER', os.path.join(settings.BASE_DIR, '..', 'dumps', 'localhost'))Usage:usage: manage.py restore_from_local_data [-h] [--target target] [--dry-run] [--no-gzip] [--source-subfolder SOURCE_SUBFOLDER]
                                         [-v {0,1,2,3}] [--settings SETTINGS]
                                         prefix

Restore db and media from local backups; source folder is "/Volumes/VMS3/django_storage/gallery/dumps/localhost"

positional arguments:
  prefix                Initial substring to match the filename to restore from; provide enough characters to match a single file

optional arguments:
  -h, --help            show this help message and exit
  --target target, -t target
                        choices: db, media, all; default: db
  --dry-run, -d         Dry run (simulation)
  --no-gzip             Do not compress result
  --source-subfolder SOURCE_SUBFOLDER, -s SOURCE_SUBFOLDER
                        replaces "localhost" in DUMP_LOCAL_DATA_TARGET_FOLDER
  -v {0,1,2,3}, --verbosity {0,1,2,3}
                        Verbosity level; 0=minimal output, 1=normal output, 2=verbose output, 3=very verbose output
  --settings SETTINGS   The Python path to a settings module, e.g. "myproject.settings.main". If this isn't provided, the
                        DJANGO_SETTINGS_MODULE environment variable will be used.load_stock_queries: Load stock (readonly) queries from settings.QUERY_INSPECTOR_QUERY_STOCK_QUERIES listApplication should provide a list of stock queries as follows:SQL_QUERIES = [{
    'slug': '...',
    'title': '...',
    'sql': """
        select
        ...
        """,
    'notes': "...",
}, {
    ...
}]Ideally, you should run this command at deployment time, to make sure that stock queries
are always updated with sources.During development, a “Reload stock queries” button is also available in the changelist.QUERY_INSPECTOR_QUERY_STOCK_QUERIES can be either a list, or a callable which returns a list.Additionally, you can optionally specify insettings.QUERY_INSPECTOR_QUERY_STOCK_VIEWSa callable to list the sql views Models to be included in Stock queries17.1History18v1.2.6preverve existing (non-stock) duplicates when reloading stock queries19v1.2.5“enabled” boolean field added“slug” is no longer unique, and a Query.objects.get_query_for_slug(query_name)
is available for retriving the active query associated with a certain slug
and detect any duplication20v1.2.4QUERY_INSPECTOR_QUERY_STOCK_QUERIES can be either a list, or a callable which returns a listIdentify stock queries created from views and materialized views21v1.2.3Add sql_views in stock queries22v1.2.2export preview as CSV, JSONL or XLSX23v1.2.1comment out “signal” from sitecopy management command24v1.2.0sitecopy refactored to support sitesync25v1.1.12Fix normalized_export_filename()26v1.1.11Specify default_auto_field to avoid unexpected migrations27v1.1.10Examples for “sitecopy” settings28v1.1.9Prepare for Django 4.0Support for Sqlite3 params29v1.1.8Added missing migration30v1.1.7Stock queries support31v1.1.6Fix: remove duplicates from extract_named_parameters()32v1.1.5Fix readme33v1.1.4qsdump2() helper addedQuery: persist parameters and duplicate34v1.1.3fix preview styles again35v1.1.2fix styles36v1.1.1Query preview: add horizontal scrolling to results table37v1.1.0SQL query preview from adminfix format_datetime() for naive datetimes38v1.0.9added “params” and “reindend” parameters to prettyprint_query()added “reindend” to prettyprint_queryset()39v1.0.8[fix] remove division by zero when computing average for and empty list of rows40v1.0.7QueryCountMiddleware can be used as standalone41v1.0.6optionally Transpose rendered tablesslugify “field-…” class in rendered HTML tablessupport “field1__field2” syntax to span relationships42v1.0.5“dump_local_data” management command now supports sqlite and Windows platform43v1.0.4fix syntax error due to wrong indentation44v1.0.3render_value_as_text() optionally preserves numeric formats45v1.0.2use apply_autofit() in export_any_queryset()46v1.0.1fix unprettified duplicate_queries dump47v1.0.0fix format_datetime48v0.0.6normalized_export_filename() helperimproved documentation49v0.0.5Tracing queries in real-timeInspecting queries in a unit testHelper management commands50v0.0.4render_queryset_as_data added for greated control of the final renderingqsdump supports tabulatedownload the queryset as a spreadsheet51v0.0.3querycounter middlewarequery_debugger decoratortracing helperstempletags helpersexport a Queryset to a spreadsheet52v0.0.2unit tests reorganized53v0.0.1Initial setup