Repository: getsentry/sentry

Issue:
I have an issue when I import the export json file
## Important Details

On-Premise w/ Docker, version 9.1.2

## Description

I have two servers with a sentry. There are several projects on the first server, and I would like to copy these projects to the clear second server.
I use the export/import commands. The export command works fine. However, when I run the import command, I get an error.

## Steps to Reproduce

1. Run command 'sentry export sentry_export.json' on the first server
2. Run command 'sentry import sentry_export.json' on the second server
3. Get an error

Good items to include here include:

`Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/backup.py", line 21, in import_
    for obj in serializers.deserialize("json", src, stream=True, use_natural_keys=True):
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/json.py", line 76, in Deserializer
    six.reraise(DeserializationError, DeserializationError(e), sys.exc_info()[2])
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/json.py", line 70, in Deserializer
    for obj in PythonDeserializer(objects, **options):
  File "/usr/local/lib/python2.7/site-packages/django/core/serializers/python.py", line 140, in Deserializer
    data[field.name] = field.to_python(field_value)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/models/fields/array.py", line 56, in to_python
    value = json.loads(value)
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/json.py", line 111, in loads
    return _default_decoder.decode(value)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
django.core.serializers.base.DeserializationError: Expecting value: line 1 column 2 (char 1)`

### What you expected to happen

Import works fine too

### Possible Solution

[If you have an idea on how this could be solved include that detail here.]



Discussion:
Can you use the command we list at our docs to make sure the export command actually produces valid JSON without logging information in between and try again: https://develop.sentry.dev/self-hosted/backup/#quick-backup
> Can you use the command we list at our docs to make sure the export command actually produces valid JSON without logging information in between and try again: https://develop.sentry.dev/self-hosted/backup/#quick-backup

Thank you for your answer, but I don't quite understand how I can run a command from your list if my sentry instance is already running on my server. Could you tell me more about this?
> Can you use the command we list at our docs to make sure the export command actually produces valid JSON without logging information in between and try again: https://develop.sentry.dev/self-hosted/backup/#quick-backup

I was able to do as you recently wrote, I executed the command without logging information. I also checked my json file and it looks valid. However, I still got the error, but a slightly different error.

```
sentry import /tmp/sentry_export.json

10:26:15 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.

10:26:19 [INFO] sentry.plugins.github: apps-not-configured

Traceback (most recent call last):

​ File "/usr/local/bin/sentry", line 8, in <module>

​ ​ ​ sys.exit(main())

​ File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main

​ ​ ​ cli(prog_name=get_prog(), obj={}, max_content_width=100)

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__

​ ​ ​ return self.main(*args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main

​ ​ ​ rv = self.invoke(ctx)

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke

​ ​ ​ return _process_result(sub_ctx.command.invoke(sub_ctx))

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke

​ ​ ​ return ctx.invoke(self.callback, **ctx.params)

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke

​ ​ ​ return callback(*args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func

​ ​ ​ return f(get_current_context(), *args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner

​ ​ ​ return ctx.invoke(f, *args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke

​ ​ ​ return callback(*args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/backup.py", line 22, in import_

​ ​ ​ obj.save()

​ File "/usr/local/lib/python2.7/site-packages/django/core/serializers/base.py", line 164, in save

​ ​ ​ models.Model.save_base(self.object, using=using, raw=True)

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 573, in save_base

​ ​ ​ updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 635, in _save_table

​ ​ ​ forced_update)

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 679, in _do_update

​ ​ ​ return filtered._update(values) > 0

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 510, in _update

​ ​ ​ return query.get_compiler(self.db).execute_sql(None)

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 980, in execute_sql

​ ​ ​ cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)

​ File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql

​ ​ ​ cursor.execute(sql, params)

​ File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 446, in execute

​ ​ ​ return real_execute(self, sql, params)

​ File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute

​ ​ ​ return self.cursor.execute(sql, params)

​ File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__

​ ​ ​ six.reraise(dj_exc_type, dj_exc_value, traceback)

​ File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute

​ ​ ​ return self.cursor.execute(sql, params)

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner

​ ​ ​ raise_the_exception(self.db, e)

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner

​ ​ ​ return func(self, *args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner

​ ​ ​ return func(self, *args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner

​ ​ ​ six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner

​ ​ ​ return func(self, sql, *args, **kwargs)

​ File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 74, in execute

​ ​ ​ return self.cursor.execute(sql, clean_bad_params(params))

django.db.utils.IntegrityError: IntegrityError('duplicate key value violates unique constraint "sentry_option_key_uniq"\nDETAIL:​ Key (key)=(beacon.anonymous) already exists.\n',)

SQL: UPDATE "sentry_option" SET "key" = %s, "value" = %s, "last_updated" = %s WHERE "sentry_option"."id" = %s

Unhandled exception in thread started by <bound method Thread.__bootstrap of <Thread(Thread-1, stopped daemon 139977913902848)>>

Traceback (most recent call last):

​ File "/usr/local/lib/python2.7/threading.py", line 774, in __bootstrap

​ ​ ​ self.__bootstrap_inner()

​ File "/usr/local/lib/python2.7/threading.py", line 814, in __bootstrap_inner

​ ​ ​ (self.name, _format_exc()))

​ File "/usr/local/lib/python2.7/traceback.py", line 241, in format_exc

​ ​ ​ etype, value, tb = sys.exc_info()

AttributeError: 'NoneType' object has no attribute 'exc_info'
```
I've stumbled upon the same import issue (with a few more described in [on-premise repo's issue](https://github.com/getsentry/onpremise/issues/863#issuecomment-781425336)).

In short: auth token scopes aren't serialized properly, instead they're simply converted to list repr string like `"['scope1', 'scope2']"` instead of `"[\"scope1\", \"scope2\"]"`. No idea why they aren't serialized into json list in the first place but this change (`'` to `\"` so properly encoded JSON string is a value for `scope` field) let's the restore process pass.
I think this was incorrectly closed so reopening.

Resolving patch:
diff --git a/src/sentry/db/models/fields/array.py b/src/sentry/db/models/fields/array.py
--- a/src/sentry/db/models/fields/array.py
+++ b/src/sentry/db/models/fields/array.py
@@ -1,3 +1,5 @@
+import ast
+
 from django.db import models
 
 from sentry.db.models.utils import Creator
@@ -36,9 +38,26 @@ def db_type(self, connection):
     def get_internal_type(self):
         return "TextField"
 
+    def get_prep_value(self, value):
+        """Iterate over each item in the array, and run it
+        through the `get_prep_value` of this array's type.
+        """
+        # If no valid value was given, return an empty list.
+        if not value:
+            return []
+
+        # Appropriately coerce each individual value within
+        # our array.
+        return [self.of.get_prep_value(item) for item in value]
+
     def to_python(self, value):
         if not value:
             value = []
         if isinstance(value, str):
-            value = json.loads(value)
+            try:
+                value = json.loads(value)
+            except json.JSONDecodeError:
+                # This is to accomodate the erronous exports pre 21.4.0
+                # See getsentry/sentry#23843 for more details
+                value = ast.literal_eval(value)
         return map(self.of.to_python, value)


Test patch: