Repository: adamchainz/django-mysql

Issue:
ignore_index (and similar) aren't rewriting the query
Hi, I'm trying to use ignore_index() from django-mysql to stop mysql from incorrectly using an inefficient index (and use a composite index instead etc...)

I've got:

```
pip:
Django==2.2.6
django-mysql==3.7.0
mysqlclient==1.4.4

MySQL 5.6.47 via Brew
macOS High Sierra
```
```
# in settings:

INSTALLED_APPS.append("django_mysql")
DJANGO_MYSQL_REWRITE_QUERIES = True
```

```
# in the view:
from django_mysql.models import add_QuerySetMixin
results = FooBar.objects.all()
results = results.filter(foobar=somevariable)
results = add_QuerySetMixin(results)
results = results.ignore_index("main_foobar_created_at_abcd12345_uniq")
print(results.query)
```

then when I iterate over `results` or print the query, it doesn't work, instead I can see the commented hint which apparently never gets handled and the sql never gets the IGNORE INDEX stuff added:

```
print(results.query)
[...] AND (/*QueryRewrite':index=`main_foobar` IGNORE `main_foobar_created_at_abcd12345_uniq`*/1))
```

Also, when i do `results.explain()` it confirms that the index I want to ignore isn't being ignored.

I'm using approach number 4 in the installation docs (use `add_QuerySetMixin`) but I've tried the other approaches as well and I still have the same results. I've also tried `force_index` and `use_index` and they have the same results.

Am I missing something? Is it broken? Thanks very much! :)


Discussion:
How are your settings laid out? If you're running the `append()` at some point after django has first read the `INSTALLED_APPS` setting, it might not actually be installed, which stops the query rewriting hook being installed.
Ah sorry, normally I've added it as a part of the initial `INSTALLED_APPS` list that's at the top of `settings.py` (the normal way you're supposed to do it). I also experimented with doing it as an `append()` which is what I pasted into this ticket.

Either way that I did it, I still got the same result. I've just added it as a part of `INSTALLED_APPS` and it's still not rewriting the query.
Can you check with version 3.5.0 as 3.6.0 chagned the way query rewriting worked? https://github.com/adamchainz/django-mysql/blob/master/HISTORY.rst#360-2020-06-09 
Hmm ok this is interesting, so as you suggested, I downgraded from 3.7.0 to 3.5.0 and now it's working.

However, when I print result.query, I still don't see the rewritten query - it still has the commented hint in it.

Also when I look at result.explain(), it still says that the wrong index is being used.

However the speed difference is undeniable; so I conclude that 3.5.0 is correctly ignoring the index.

Further testing: If I comment out the line that does the ignore_index(), the query takes 20+ seconds. If I add that line back in to ignore the wrong index that the mysql optimizer chooses then the query takes a few hundred milliseconds.

So it does seem that something's not quite right with whatever 3.6.0 introduced/changed.

Also, is there any way that explain() can correctly show the index that was used? Because I use it to monitor various queries and their times to identify if there needs to be other composite indexes added depending on what people search for.

Thanks for helping!

> However, when I print result.query, I still don't see the rewritten query - it still has the commented hint in it.

This is expected. The query rewriting is done between django's sql compiler and the database execution, the `.query` property only shows the output from the sql compiler.

> Also when I look at result.explain(), it still says that the wrong index is being used.

What database and version are you using? Possibly upgrading will help here.

> So it does seem that something's not quite right with whatever 3.6.0 introduced/changed.

I am looking into this. Can you check too? The code is in https://github.com/adamchainz/django-mysql/blob/master/src/django_mysql/apps.py#L19 and should be fairly comprehensible.
> The query rewriting is done between django's sql compiler and the database execution

I suspected this might have been the case, no problem!


>> Also when I look at result.explain(), it still says that the wrong index is being used.

> What database and version are you using? Possibly upgrading will help here.

I'm using MySQL 5.6 on macOS (installed via brew). I'm going to test on a different system with 5.7 just in case.

> I am looking into this. Can you check too?

I'll have another look, but I did already spend a few hours when I was first encountering this issue, digging through that code and the code in query.py etc. I could see the `rewrite_hook` being added to `connection.execute_wrappers` but I found the code in Django which -I think- executed the wrappers and the `connection.execute_wrappers` list was empty at that point, so it never ran anything.


Resolving patch:
diff --git a/src/django_mysql/apps.py b/src/django_mysql/apps.py
--- a/src/django_mysql/apps.py
+++ b/src/django_mysql/apps.py
@@ -1,5 +1,6 @@
 from django.apps import AppConfig
 from django.conf import settings
+from django.db.backends.signals import connection_created
 from django.utils.translation import ugettext_lazy as _
 
 from django_mysql.checks import register_checks
@@ -22,14 +23,8 @@ def add_database_instrumentation(self):
         ):  # pragma: no cover
             return
         for _alias, connection in mysql_connections():
-            # Rather than use the documented API of the `execute_wrapper()`
-            # context manager, directly insert the hook. This is done because:
-            # 1. Deleting the context manager closes it, so it's not possible
-            # to enter it here and not exit it, unless we store it forever in
-            # some variable.
-            # 2. We want to be idempotent and only install the hook once.
-            if rewrite_hook not in connection.execute_wrappers:  # pragma: no branch
-                connection.execute_wrappers.append(rewrite_hook)
+            install_rewrite_hook(connection)
+        connection_created.connect(install_rewrite_hook)
 
     def add_lookups(self):
         from django.db.models import CharField, TextField
@@ -43,6 +38,20 @@ def add_lookups(self):
         TextField.register_lookup(Soundex)
 
 
+def install_rewrite_hook(connection, **kwargs):
+    """
+    Rather than use the documented API of the `execute_wrapper()` context
+    manager, directly insert the hook. This is done because:
+    1. Deleting the context manager closes it, so it's not possible to enter it
+       here and not exit it, unless we store it forever in some variable.
+    2. We want to be idempotent and only install the hook once.
+    """
+    if connection.vendor != "mysql":
+        return
+    if rewrite_hook not in connection.execute_wrappers:  # pragma: no branch
+        connection.execute_wrappers.insert(0, rewrite_hook)
+
+
 def rewrite_hook(execute, sql, params, many, context):
     if (
         getattr(settings, "DJANGO_MYSQL_REWRITE_QUERIES", False)


Test patch:
diff --git a/tests/testapp/test_dynamicfield.py b/tests/testapp/test_dynamicfield.py
--- a/tests/testapp/test_dynamicfield.py
+++ b/tests/testapp/test_dynamicfield.py
@@ -5,7 +5,7 @@
 import mariadb_dyncol
 import pytest
 from django.core import serializers
-from django.db import connection, connections
+from django.db import connection
 from django.db.migrations.writer import MigrationWriter
 from django.db.models import CharField, Transform
 from django.test import TestCase
@@ -13,6 +13,7 @@
 from django_mysql.models import DynamicField
 from django_mysql.utils import connection_is_mariadb
 from tests.testapp.models import DynamicModel, SpeclessDynamicModel, TemporaryModel
+from tests.testapp.utils import mock_mysql_version
 
 
 class DynColTestCase(TestCase):
@@ -287,8 +288,7 @@ class ValidDynamicModel1(TemporaryModel):
 
     wrapper_path = "django.db.backends.mysql.base.DatabaseWrapper"
 
-    @mock.patch.object(connections["default"], "mysql_version", new=(5, 5, 3))
-    @mock.patch.object(connections["other"], "mysql_version", new=(5, 5, 3))
+    @mock_mysql_version(default=(5, 5, 3), other=(5, 5, 3))
     def test_mariadb_old_version(self):
         class ValidDynamicModel2(TemporaryModel):
             field = DynamicField()
@@ -298,8 +298,7 @@ class ValidDynamicModel2(TemporaryModel):
         assert errors[0].id == "django_mysql.E013"
         assert "MariaDB 10.0.1+ is required" in errors[0].msg
 
-    @mock.patch.object(connections["default"], "mysql_version", new=(5, 5, 3))
-    @mock.patch.object(connections["other"], "mysql_version", new=(10, 0, 1))
+    @mock_mysql_version(default=(5, 5, 3), other=(10, 0, 1))
     def test_mariadb_one_conn_old_version(self):
         class ValidDynamicModel3(TemporaryModel):
             field = DynamicField()
diff --git a/tests/testapp/test_jsonfield.py b/tests/testapp/test_jsonfield.py
--- a/tests/testapp/test_jsonfield.py
+++ b/tests/testapp/test_jsonfield.py
@@ -5,7 +5,7 @@
 import pytest
 from django.core import serializers
 from django.core.serializers.json import DjangoJSONEncoder
-from django.db import connection, connections
+from django.db import connection
 from django.db.models import F
 from django.test import TestCase
 
@@ -13,7 +13,7 @@
 from django_mysql.models import JSONField
 from django_mysql.utils import connection_is_mariadb
 from tests.testapp.models import JSONModel, TemporaryModel
-from tests.testapp.utils import print_all_queries
+from tests.testapp.utils import mock_mysql_version
 
 
 class JSONFieldTestCase(TestCase):
@@ -342,11 +342,10 @@ def test_has_keys_invalid_type(self):
         assert "only works with Sequences" in str(excinfo.value)
 
     def test_has_keys(self):
-        with print_all_queries():
-            assert list(JSONModel.objects.filter(attrs__has_keys=["a", "c"])) == [
-                self.objs[1],
-                self.objs[2],
-            ]
+        assert list(JSONModel.objects.filter(attrs__has_keys=["a", "c"])) == [
+            self.objs[1],
+            self.objs[2],
+        ]
 
     def test_has_keys_2(self):
         assert list(JSONModel.objects.filter(attrs__has_keys=["l"])) == [self.objs[4]]
@@ -584,8 +583,7 @@ class InvalidJSONModel3(TemporaryModel):
         assert errors[0].id == "django_mysql.E016"
         assert "MySQL 5.7+ is required" in errors[0].msg
 
-    @mock.patch.object(connections["default"], "mysql_version", new=(5, 5, 3))
-    @mock.patch.object(connections["other"], "mysql_version", new=(5, 5, 1))
+    @mock_mysql_version(default=(5, 5, 3), other=(5, 5, 3))
     def test_mysql_old_version(self):
         class InvalidJSONModel4(TemporaryModel):
             field = JSONField()
@@ -595,8 +593,7 @@ class InvalidJSONModel4(TemporaryModel):
         assert errors[0].id == "django_mysql.E016"
         assert "MySQL 5.7+ is required" in errors[0].msg
 
-    @mock.patch.object(connections["default"], "mysql_version", new=(5, 5, 3))
-    @mock.patch.object(connections["other"], "mysql_version", new=(5, 7, 1))
+    @mock_mysql_version(default=(5, 5, 3), other=(5, 7, 1))
     def test_mysql_one_conn_old_version(self):
         class InvalidJSONModel5(TemporaryModel):
             field = JSONField()
diff --git a/tests/testapp/test_models.py b/tests/testapp/test_models.py
--- a/tests/testapp/test_models.py
+++ b/tests/testapp/test_models.py
@@ -4,6 +4,7 @@
 
 import pytest
 from django.contrib.contenttypes.models import ContentType
+from django.db import DEFAULT_DB_ALIAS, connections
 from django.db.models.query import QuerySet
 from django.template import Context, Template
 from django.test import TestCase
@@ -330,6 +331,20 @@ def test_force_index_table_name_doesnt_exist_ignored(self):
         assert " FORCE INDEX " not in cap.query
 
 
+class QueryHintNewConnectionTests(TestCase):
+    @classmethod
+    def setUpClass(cls):
+        # Force a new connection to be created, in order to check that the
+        # rewrite hook is installed on it.
+        del connections[DEFAULT_DB_ALIAS]
+        super().setUpClass()
+
+    def test_label_forced_new_connection(self):
+        with CaptureLastQuery() as cap:
+            list(Author.objects.label("QueryHintTests.test_label").all())
+        assert cap.query.startswith("SELECT /*QueryHintTests.test_label*/ ")
+
+
 class FoundRowsTests(TestCase):
     def setUp(self):
         super().setUp()
diff --git a/tests/testapp/utils.py b/tests/testapp/utils.py
--- a/tests/testapp/utils.py
+++ b/tests/testapp/utils.py
@@ -1,4 +1,5 @@
 from contextlib import contextmanager
+from unittest import mock
 
 import pytest
 from django.db import DEFAULT_DB_ALIAS, connection, connections
@@ -80,3 +81,13 @@ def fetchall_dicts(cursor):
     for row in cursor.fetchall():
         rows.append(dict(zip(columns, row)))
     return rows
+
+
+@contextmanager
+def mock_mysql_version(*, default, other):
+    mock_default = mock.patch.object(
+        connections["default"], "mysql_version", new=default
+    )
+    mock_other = mock.patch.object(connections["other"], "mysql_version", new=other)
+    with mock_default, mock_other:
+        yield