Repository: django-tastypie/django-tastypie

Issue:
With MySQL, 0.9.12-alpha, the famous 'Failed to install index for tastypie.ApiKey model: Specified key was too long; max key length is 767 bytes'
This may not be worth addressing. My personal preference would be for Django to just not support MySQL. However...

Tastypie creates the field 'key' as a varchar(256) in the tastypie_apikey table. If you're using a database which is in utf8mb4, the length of the varchar can be 4 \* 256 bytes (excluding whatever storage engine constants), hence MySQL can't actually create a full-length index as 1024 > 767.

There's actually a bug report in Django here: https://code.djangoproject.com/ticket/18392
so probably not worth thinking about. 

For reference, you also do end up with an index being created despite the warning if you use syncdb instead of South, but with South it will stop your migration and you have to work around.  If it gets created it will just be length 191 (as 4\* 191 = 764, the largest it can be while staying less that 768).

Mostly submitting this just so that other people who run into this can verify what's up.

As noted it's not strictly a bug with tastypie, but is a bug to be aware of if using tastypie and certain MySQL configs.

To replicate:
1. Use MySQL.
2. Set the default collation for whatever schema you're using for Django to utf8mb4.
3. Run South migration or regular syncdb after installing the django-tastypie 0.9.12-alpha.

Meh. I'd label this 'really low priority.'



Discussion:
A more appropriate Django bug report may be this: https://code.djangoproject.com/ticket/16460

The fix would be to create an index using a 767-char prefix of the column, as described on http://dev.mysql.com/doc/refman/5.5/en/create-index.html.

Sure, the MySQL idiosyncrasy is annoying, but should be addressed since, well, MySQL is kind of big :/

We noticed this bug after attempting to use 0.9.12-alpha on one of our machines and will probably just edit models.py and remove the index from tastipe.ApiKey for now. I'll try to see how the post_syncdb signal could be used to address this.

Is a varchar(256) really needed for the key or will varchar(255) suffice?

My 2nd paragraph is probably needlessly unclear. 

Varchar columns can obviously be all kinds of lengths - varchar(120), varchar(333), whatever. And a appropriate index with total coverage will probably need as many 'significant bytes' as the max number of bytes/char \* num chars, since that will encompass the maximum length of the column value in bytes following mysql rules.

Whether you actually need a full coverage index is another question. It might not be particularly necessary if you find that your dataset is consistently using much less than the full possible length of strings you can put in the column. I imagine this would be something worth benchmarking. Let's say you did have a mb4 column of length 191. If you were constantly putting strings into that were the maximum length possible (191 bytes), it'd be good to have the full length index. On the other hand if 90% of your strings were under 110 bytes, then maybe there's no reason not to use a partial index. You'd have to compare and benchmark and know your data I guess. I'm not knowledgeable enough about MySQL to know anyway.

Yes, the shorter the index, the greater the performance penalty. However, in this case the penalty would be negligible as we can still use 767 bytes for the MySQL index (255 characters in utf8). On the other hand, regardless of the penalty, this issue would have to be addressed for MySQL, _so long as the penalty would not extend to other model backends_, which it should not since the idea is to create the prefix index only if the model backend is mysql. Moreover, a tiny performance penalty is better than no index at all (which is what is currently happening).

What I am wondering is whether the tastypie.ApiKey field actually needs to be 256 chars (which would obviously be the case if the key length is 256 chars). If not, then the field should just be changed to varchar(255) which is a very common length for varchar fields.

Another thing to note is that no matter what prefix is used, one could potentially use some character encoding which would require more than 767 bytes for the index (well, in practice utf32 may be the largest). But we should at least ensure that things work with the basic utf8, which takes a max of 3 bytes per character, so varchar(255) is the limit. Why utf8? Because Django expects the database to support utf8 (and nothing more, such as utf8mb4) by default (see https://docs.djangoproject.com/en/dev/ref/databases/).

Thus, changing the field to varchar(255), if feasible, would address the issue for utf8. But for a generic solution, we need the post_syncdb signal to explicitly limit the index length to 767 bytes.

@adityar7 That seems cleaner to me: the current implementation uses the python `hmac` module with SHA-1, which will produce a maximum of 40 hexadecimal bytes:

https://github.com/toastdriven/django-tastypie/blob/master/tastypie/models.py#L51

Bumping that all the way up to SHA-256 will only require 64 hex bytes:

```
>>> import hmac, hashlib
>>> len(hmac.new("foobar", digestmod=hashlib.sha1).hexdigest())
40
>>> len(hmac.new("foobar", digestmod=hashlib.sha256).hexdigest())
64
```

Since these are also just hex strings it'd be nice if there's a way to declare the column's encoding as ASCII since we're not using Unicode at all but I doubt there's a way to do that without a migration library which wouldn't end up being way more annoying than simply changing the length.

Since tastypie is not yet 1.0 but hopefully will reach that point soon, I'd like to get @toastdriven's opinion on this design change. I'm inclined to drop it down to `VARCHAR(128)` so it's not an issue even for MySQL `utf8mb4` and with the knowledge that anyone needing more than 128 characters could at the very least pick a more efficient encoding than hexadecimal.

Is there a workaround for this error? It seems to break functionality for anything else that checks the consistency of the databases like `loaddata` `dumpdata` for example. 

Also, this seems to be happening with Utf8_general_ci collation as well. 

Yeah it'll happen with all unicode.

One "workaround" involves changing a line in the models.py file in your local tastypie installation. You can find the installation path like this in the python interpreter:

```
import tastypie
tastypie.__file__
```

Search for the line

`key = models.CharField(max_length=256, blank=True, default='', db_index=True)`

and change it to

`key = models.CharField(max_length=256, blank=True, default='')`

This was the resulting SQL for table creation, in my case:

```
CREATE TABLE `tastypie_apikey` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `user_id` int(11) NOT NULL,
  `key` varchar(256) NOT NULL,
  `created` datetime NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `user_id` (`user_id`),
  KEY `tastypie_apikey_c0d4be93` (`key`(255)),
  CONSTRAINT `user_id_refs_id_1138959b` FOREIGN KEY (`user_id`) REFERENCES `directory_user` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
```

So I guess, just changing this:

```
  `key` varchar(256) NOT NULL,
```

Into this:

```
  `key` varchar(255) NOT NULL,
```

Would solve the problem for databases/tables using UTF-8.

Okay, since this is a persistent annoyance for a number of people: does anyone have an objection to my changing `key` to a `varchar(128)` as discussed in https://github.com/toastdriven/django-tastypie/issues/757#issuecomment-15026308?

(CC: @toastdriven, @georgedorn, @vbabiy, @numan, @codysoyland, @mcroydon, @joshbohde, @issackelly)

I just ran into this after upgrading to MariaDB 10 from mySql 5


Resolving patch:
diff --git a/tastypie/models.py b/tastypie/models.py
--- a/tastypie/models.py
+++ b/tastypie/models.py
@@ -32,7 +32,7 @@ def save(self, *args, **kwargs):
     from tastypie.compat import AUTH_USER_MODEL
     class ApiKey(models.Model):
         user = models.OneToOneField(AUTH_USER_MODEL, related_name='api_key')
-        key = models.CharField(max_length=256, blank=True, default='', db_index=True)
+        key = models.CharField(max_length=128, blank=True, default='', db_index=True)
         created = models.DateTimeField(default=now)
 
         def __unicode__(self):


Test patch: