Repository: WeblateOrg/weblate

Issue:
Upgrade to 4.5 (from 4.4.2) - specified key was too long; max key length is 3072 bytes
Sorry, it's me again with upgrade issue... It is not a happy process for me...


_**1. First run of upgrade process**_

**Specified key was too long; max key length is 3072 bytes**
error when appling 
0110_auto_20210120_0845
migration.

Full traceback in the next post.

I google that the reason may be my collation:
utf8mb4_0900_ai_ci
Which collation should I set (I can do it manually) for this table trans_variant (and default for weblate) for safe work with all languages?

_**2. Second run of upgrade process**_

**Table 'trans_variant_defining_units' already exists**
error when appling 
0110_auto_20210120_0845
migration.

Full traceback is in the post after next.

What should I remove from 
0110_auto_20210120_0845
after fix the issue to continue migration process?

_Thanks in advance..._


Discussion:
First run.

```
 [X] 0105_auto_20201006_1254
 [X] 0106_remove_unit_content_hash
 [ ] 0107_component_links
 [ ] 0108_move_mailing
 [ ] 0109_remove_project_mail
 [ ] 0110_auto_20210120_0845
 [ ] 0111_index_explanation
 [ ] 0112_component_new_unit
 [ ] 0113_component_new_unit
 [ ] 0114_auto_20210129_1239
 [ ] 0115_auto_20210201_1305
 [ ] 0116_migrate_glossaries
 [ ] 0117_remove_change_glossary_term
 [ ] 0118_fixup_glossary_changes
 [ ] 0119_auto_20210206_1141
 [ ] 0120_glossary_name
 [ ] 0121_remove_component_glossary_name
utils
 [X] 0001_alter_role
weblate_auth
 [X] 0001_initial
 [X] 0002_auto_20180507_1540_squashed_0011_auto_20180509_0739_squashed_0007_group_components (6 squashed migrations)
 [X] 0008_auto_20200611_1232
 [X] 0009_migrate_componentlist
 [X] 0010_migrate_componentlist
 [X] 0011_unique_case_insensitive
 [X] 0012_auto_20200729_1200
 [X] 0013_rename_sources_group
weblate_language_data
 (no migrations)
wladmin
 [X] 0001_squashed_0008_auto_20191011_0816 (8 squashed migrations)
Weblate migrate: Press enter to continue

Operations to perform:
  Apply all migrations: accounts, addons, admin, auth, authtoken, checks, configuration, contenttypes, fonts, gitexport, glossary, lang, memory, metrics, screenshots, sessions, social_django, trans, utils, weblate_auth, wladmin
Running migrations:
  Applying accounts.0011_auto_20210106_1903... OK
  Applying accounts.0012_profile_auto_watch... OK
  Applying accounts.0013_auto_20210217_1653... OK
  Applying trans.0107_component_links... OK
  Applying trans.0108_move_mailing... OK
  Applying trans.0109_remove_project_mail... OK
  Applying trans.0110_auto_20210120_0845...Traceback (most recent call last):
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1071, 'Specified key was too long; max key length is 3072 bytes')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/drive/weblate-env/bin/weblate", line 8, in <module>
    sys.exit(main())
  File "/drive/weblate-env/lib/python3.8/site-packages/weblate/runner.py", line 34, in main
    utility.execute()
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 243, in handle
    post_migrate_state = executor.migrate(
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 236, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 571, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type,
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 721, in _alter_field
    self.execute(
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 142, in execute
    cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1071, 'Specified key was too long; max key length is 3072 bytes')
Weblate collectstatic: Press enter to continue

```
Second run.

```
 [X] 0108_move_mailing
 [X] 0109_remove_project_mail
 [ ] 0110_auto_20210120_0845
 [ ] 0111_index_explanation
 [ ] 0112_component_new_unit
 [ ] 0113_component_new_unit
 [ ] 0114_auto_20210129_1239
 [ ] 0115_auto_20210201_1305
 [ ] 0116_migrate_glossaries
 [ ] 0117_remove_change_glossary_term
 [ ] 0118_fixup_glossary_changes
 [ ] 0119_auto_20210206_1141
 [ ] 0120_glossary_name
 [ ] 0121_remove_component_glossary_name
utils
 [X] 0001_alter_role
weblate_auth
 [X] 0001_initial
 [X] 0002_auto_20180507_1540_squashed_0011_auto_20180509_0739_squashed_0007_group_components (6 squashed migrations)
 [X] 0008_auto_20200611_1232
 [X] 0009_migrate_componentlist
 [X] 0010_migrate_componentlist
 [X] 0011_unique_case_insensitive
 [X] 0012_auto_20200729_1200
 [X] 0013_rename_sources_group
weblate_language_data
 (no migrations)
wladmin
 [X] 0001_squashed_0008_auto_20191011_0816 (8 squashed migrations)
Weblate migrate: Press enter to continue

Operations to perform:
  Apply all migrations: accounts, addons, admin, auth, authtoken, checks, configuration, contenttypes, fonts, gitexport, glossary, lang, memory, metrics, screenshots, sessions, social_django, trans, utils, weblate_auth, wladmin
Running migrations:
  Applying trans.0110_auto_20210120_0845...Traceback (most recent call last):
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.OperationalError: (1050, "Table 'trans_variant_defining_units' already exists")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/drive/weblate-env/bin/weblate", line 8, in <module>
    sys.exit(main())
  File "/drive/weblate-env/lib/python3.8/site-packages/weblate/runner.py", line 34, in main
    utility.execute()
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 243, in handle
    post_migrate_state = executor.migrate(
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/migration.py", line 124, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/migrations/operations/fields.py", line 104, in database_forwards
    schema_editor.add_field(
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/mysql/schema.py", line 89, in add_field
    super().add_field(model, field)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 453, in add_field
    return self.create_model(field.remote_field.through)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 324, in create_model
    self.execute(sql, params or None)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 142, in execute
    cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/drive/weblate-env/lib/python3.8/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
    res = self._query(query)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
    db.query(q)
  File "/drive/weblate-env/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
    _mysql.connection.query(self, query)
django.db.utils.OperationalError: (1050, "Table 'trans_variant_defining_units' already exists")
Weblate collectstatic: Press enter to continue

```
See our documentation for recommended setup with MySQL: https://docs.weblate.org/en/latest/admin/install.html#mysql-and-mariadb

You will have to manually drop `trans_variant_defining_units` table before retrying. This is caused by MySQL not supporting DDL transactions.
This issue looks more like a support question than an issue. We strive to answer these reasonably fast, but [purchasing the support subscription](https://weblate.org/support/) is not only more responsible and faster for your business but also makes Weblate stronger. In case your question is already answered, [making a donation](https://weblate.org/donate/) is the right way to say thank you!
I have MySQL 8.0.22.

As far as I google, the following parameters are most significant for the mentioned issue.

https://docs.weblate.org/en/latest/admin/install.html#mysql-and-mariadb
> innodb_large_prefix=1
> innodb_file_format=Barracuda
> innodb_file_per_table=1

Weblate requires **MySQL at least 5.7.8** or MariaDB at least 10.2.7.

https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-0.html
> Important Change; InnoDB: The following InnoDB file format configuration options **were deprecated in MySQL 5.7.7 and are now removed**:
> innodb_file_format
> innodb_file_format_check
> innodb_file_format_max
> innodb_large_prefix

PS. For the table drop hint - thank you.

PPS. I empirically found that varchar(577) is the maximal size for now. May be this information is useles...
> In case your question is already answered, [making a donation](https://weblate.org/donate/) is the right way to say thank you!

I did before. And I will do.
> Important Change; InnoDB: The following InnoDB file format configuration options were deprecated in MySQL 5.7.7 and are now removed:

Hmm, the docs probably could use some updates for most recent MySQL versions, patches are welcome :-).

Anyway back to the original issue - the problem is that the field is used in compound index, and it seems to be too big for your MySQL. On the other side it works well in our CI environment (which runs MySQL 8 and MariaDB 10.5, both from official Docker containers without any configuration change).

> PPS. I empirically found that varchar(577) is the maximal size for now. May be this information is useles...

That matches my calculation. We can probably change this, but I'd like to figure out what is the difference between [MySQL container](https://hub.docker.com/_/mysql) and your environment, and whether it's worth of recommending such configuration.

Resolving patch:
diff --git a/weblate/trans/migrations/0110_auto_20210120_0845.py b/weblate/trans/migrations/0110_auto_20210120_0845.py
--- a/weblate/trans/migrations/0110_auto_20210120_0845.py
+++ b/weblate/trans/migrations/0110_auto_20210120_0845.py
@@ -22,7 +22,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name="variant",
             name="key",
-            field=models.CharField(max_length=768),
+            field=models.CharField(max_length=576),
         ),
         migrations.AlterField(
             model_name="variant",
diff --git a/weblate/trans/migrations/0122_auto_20210228_1846.py b/weblate/trans/migrations/0122_auto_20210228_1846.py
new file mode 100644
--- /dev/null
+++ b/weblate/trans/migrations/0122_auto_20210228_1846.py
@@ -0,0 +1,18 @@
+# Generated by Django 3.1.4 on 2021-02-28 17:46
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("trans", "0121_remove_component_glossary_name"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="variant",
+            name="key",
+            field=models.CharField(max_length=576),
+        ),
+    ]
diff --git a/weblate/trans/models/variant.py b/weblate/trans/models/variant.py
--- a/weblate/trans/models/variant.py
+++ b/weblate/trans/models/variant.py
@@ -27,7 +27,7 @@ class Variant(models.Model):
     variant_regex = RegexField(max_length=190, blank=True)
     # This really should be a TextField, but it does not work with unique
     # index and MySQL
-    key = models.CharField(max_length=768)
+    key = models.CharField(max_length=576)
     defining_units = models.ManyToManyField("Unit", related_name="defined_variants")
 
     class Meta:


Test patch:
diff --git a/weblate/settings_test.py b/weblate/settings_test.py
--- a/weblate/settings_test.py
+++ b/weblate/settings_test.py
@@ -35,7 +35,7 @@
     default_user = "root"
     DATABASES["default"]["OPTIONS"] = {
         "init_command": (
-            "SET NAMES utf8, "
+            "SET NAMES utf8mb4, "
             "wait_timeout=28800, "
             "default_storage_engine=INNODB, "
             'sql_mode="STRICT_TRANS_TABLES"'