Repository: WeblateOrg/weblate

Issue:
Migration from 3.9.1 to 3.10 fails with AttributeError: 'Comment' object has no attribute '__getstate__'
**Describe the bug**
After upgrading to 3.10 (from 3.9.1) applying trans.0053_unitdata fails with an error.

```
  Applying trans.0053_unitdata...Traceback (most recent call last):
  File "/home/ubuntu/weblateenv/bin/weblate", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/weblate/runner.py", line 34, in main
    execute_from_command_line(argv)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/ubuntu/weblateenv/local/lib/python2.7/site-packages/weblate/trans/migrations/0053_unitdata.py", line 24, in migrate_unitdata
    state = obj.__getstate__()
AttributeError: 'Comment' object has no attribute '__getstate__'
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
```

**To Reproduce**
Steps to reproduce the behavior:
1. have 3.9.1 installed
1. upgrade to 3.10 via pip
1. apply changes to the settings.py file
1. run `weblate migrate`
1. see the error (https://sentry.io/share/issue/8e9bee713ce74fc9b53d800b504dcfa5/)

**Expected behavior**
no error and a working migration

**Logs**
https://sentry.io/share/issue/8e9bee713ce74fc9b53d800b504dcfa5/

**Server configuration and status**
output of `weblate list_versions`:
```
 * Weblate: 3.10
 * Django: 1.11.11
 * siphashc: 1.2
 * Whoosh: 2.7.4
 * translate-toolkit: 2.4.0
 * lxml: 4.2.1
 * Pillow: 6.2.1
 * bleach: 3.1.0
 * six: 1.13.0
 * python-dateutil: 2.8.1
 * social-auth-core: 3.2.0
 * social-auth-app-django: 3.1.0
 * django-crispy-forms: 1.8.1
 * oauthlib: 3.1.0
 * django-compressor: 2.3
 * djangorestframework: 3.9.0
 * django-appconf: 1.0.3
 * user-agents: 2.0
 * filelock: 3.0.12
 * setuptools: 39.0.1
 * jellyfish: 0.6.1
 * openpyxl: 2.6.4
 * celery: 4.3.0
 * kombu: 4.6.7
 * celery-batches: 0.2
 * translation-finder: 1.7
 * html2text: 2019.8.11
 * pycairo: 1.18.2
 * pygobject: 3.34.0
 * diff-match-patch: 20181111
 * requests: 2.22.0
 * django-redis: 4.10.0
 * hiredis: 1.0.1
 * sentry_sdk: 0.13.5
 * Cython: 0.29.14
 * misaka: 2.1.1
 * GitPython: 2.1.14
 * Python: 2.7.12
 * Git: 2.7.4
 * phply: 1.2.4
 * chardet: 3.0.4
 * ruamel.yaml: 0.16.5
 * Database backends: django.db.backends.mysql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: smtp.mailgun.org
 * Celery: redis://127.0.0.1:6379/0, redis://127.0.0.1:6379/0, regular
 * Platform: Linux 4.4.0-1049-aws (x86_64)
```

output of `weblate check --deploy`:
```
SystemCheckError: System check identified some issues:

ERRORS:
?: (weblate.E006) Please migrate your database to use PostgreSQL. Support for other database backends will be dropped in Weblate 4.0 currently sheduled on April 2020.
	HINT: https://docs.weblate.org/en/weblate-3.10/admin/install.html#production-database
?: (weblate.W023) Please upgrade your installation to Python 3. Python 2 support will be dropped in Weblate 4.0 currently sheduled on April 2020.
	HINT: https://docs.weblate.org/en/weblate-3.10/admin/upgrade.html#py3

System check identified 2 issues (7 silenced).
```


Not quite sure how to fix this.
Thanks in advance!



Discussion:
Just checked if I can work around this error with clearing the comments table, but than it fails for `Suggestion`. If I clear suggestions it fails for `Check`
I did some queries in my MySQL db that do the migrations, as a workaround.
After that I ran the migration successfully.

Note for someone with the same error: we use 'en' as every source language and you should not just paste this query into your db!

```
UPDATE `trans_comment`
INNER JOIN trans_unit ON trans_unit.content_hash=trans_comment.content_hash
INNER JOIN trans_translation ON trans_translation.language_id=trans_comment.language_id AND trans_unit.translation_id=trans_translation.id
SET trans_comment.unit_id=trans_unit.id;

UPDATE `trans_suggestion`
INNER JOIN trans_unit ON trans_unit.content_hash=trans_suggestion.content_hash
INNER JOIN trans_translation ON trans_translation.language_id=trans_suggestion.language_id AND trans_unit.translation_id=trans_translation.id
SET trans_suggestion.unit_id=trans_unit.id;

UPDATE `checks_check`
INNER JOIN trans_unit ON trans_unit.content_hash=checks_check.content_hash
INNER JOIN trans_translation ON trans_translation.language_id=checks_check.language_id AND trans_unit.translation_id=trans_translation.id
SET checks_check.unit_id=trans_unit.id;

UPDATE `checks_check`
INNER JOIN trans_unit ON trans_unit.content_hash=checks_check.content_hash
INNER JOIN trans_translation ON trans_unit.translation_id=trans_translation.id AND trans_translation.language_code="en"
SET checks_check.unit_id=trans_unit.id
WHERE checks_check.language_id IS NULL;
```
Apparently, we lack migrations tests on Python 2.7. This method was added in Django 2.0, so it could not work there.
Thank you for your report, the issue you have reported has just been fixed. 

* In case you see a problem with the fix, please comment on this issue. 
* In case you see similar problem, please open separate issue.
* If you are happy with the outcome, consider supporting Weblate by [donating](https://weblate.org/donate/).
I have a similar problem with the same setup and Python 2.7
```
Applying trans.0053_unitdata...Traceback (most recent call last):
  File "./manage.py", line 33, in <module>
    main()
  File "/opt/weblate/weblate/weblate/runner.py", line 34, in main
    execute_from_command_line(argv)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/opt/weblate/weblate/weblate/trans/migrations/0053_unitdata.py", line 30, in migrate_unitdata
    if model.objects.using(db_alias).filter(unit=unit, **state).exists():
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/query.py", line 787, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/query.py", line 805, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1250, in add_q
    clause, _ = self._add_q(q_object, self.used_aliases)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1276, in _add_q
    allow_joins=allow_joins, split_subq=split_subq,
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1154, in build_filter
    lookups, parts, reffed_expression = self.solve_lookup_type(arg)
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1034, in solve_lookup_type
    _, field, _, lookup_parts = self.names_to_path(lookup_splitted, self.get_meta())
  File "/opt/weblate/.local/lib/python2.7/site-packages/django/db/models/sql/query.py", line 1352, in names_to_path
    "Choices are: %s" % (name, ", ".join(available)))
django.core.exceptions.FieldError: Cannot resolve keyword '_project_cache' into field. Choices are: change, comment, content_hash, id, language, language_id, project, project_id, timestamp, unit, unit_id, user, user_id
```
@tsipizic which version did you try? 3.10.1 contains some fixes for migration, that weren't working on 3.10.0
I am trying with latest version 3.10.1
Apparently the fix doesn't work well in some situations :-(
The good thing is that I could add tests to reproduce this - https://cloud.drone.io/WeblateOrg/weblate/1462/8/3, looking into fix now...

Resolving patch:
diff --git a/weblate/trans/migrations/0053_unitdata.py b/weblate/trans/migrations/0053_unitdata.py
--- a/weblate/trans/migrations/0053_unitdata.py
+++ b/weblate/trans/migrations/0053_unitdata.py
@@ -2,7 +2,25 @@
 
 from django.db import migrations
 
-MODELS = [("trans", "Comment"), ("trans", "Suggestion"), ("checks", "Check")]
+MODELS = [
+    (
+        ("trans", "Comment"),
+        ("content_hash", "project", "language", "comment", "user", "timestamp"),
+    ),
+    (
+        ("trans", "Suggestion"),
+        (
+            "content_hash",
+            "project",
+            "language",
+            "target",
+            "user",
+            "userdetails",
+            "timestamp",
+        ),
+    ),
+    (("checks", "Check"), ("content_hash", "project", "language", "check", "ignore")),
+]
 
 
 def migrate_unitdata(apps, schema_editor):
@@ -10,7 +28,8 @@ def migrate_unitdata(apps, schema_editor):
 
     Unit = apps.get_model("trans", "Unit")
 
-    for model in (apps.get_model(*args) for args in MODELS):
+    for model_args, fields in MODELS:
+        model = apps.get_model(*model_args)
         # Create new objects for each related unit
         for obj in model.objects.using(db_alias).filter(unit=None).iterator():
             units = Unit.objects.using(db_alias).filter(
@@ -22,14 +41,15 @@ def migrate_unitdata(apps, schema_editor):
             else:
                 units = units.filter(translation__language=obj.language)
             # Using __getstate__ would be cleaner, but needs Django 2.0
-            state = dict(obj.__dict__)
-            del state["_state"]
-            del state["id"]
-            del state["unit_id"]
+            state = {field: getattr(obj, field) for field in fields}
             for unit in units:
                 if model.objects.using(db_alias).filter(unit=unit, **state).exists():
                     continue
-                model.objects.using(db_alias).create(unit=unit, **state)
+                created = model.objects.using(db_alias).create(unit=unit, **state)
+                # Migrate suggestion votes
+                if model_args == ("trans", "Suggestion"):
+                    for vote in obj.vote_set.all():
+                        created.vote_set.create(user=vote.user, value=vote.value)
 
         # Remove old objects without unit link
         model.objects.using(db_alias).filter(unit=None).delete()


Test patch: