Repository: CTFd/CTFd

Issue:
Importing CTFd-exported ZIP gives DateTime error
<!--
If this is a bug report please fill out the template below.

If this is a feature request please describe the behavior that you'd like to see.
-->

**Environment**:

  - CTFd Version/Commit: master
  - Operating System: Debian Testing
  - Web Browser and Version: N/A

**What happened?**
In trying to import a ZIP file that was previously exported with CTFd, an error is thrown stating "SQLite DateTime type only accepts Python"

**What did you expect to happen?**
I expected the data to import successfully

**How to reproduce your issue**
sudo python import.py ~/CTF-DATA.zip

**Any associated stack traces or error logs**
```
> sudo python import.py ~/CTF-DATA.zip 
/*\ CTFd has updated and must update the database! /*\
/*\ Please backup your database before proceeding! /*\
/*\ CTFd maintainers are not responsible for any data loss! /*\
Run database migrations (Y/N)y
 * Loaded module, <module 'CTFd.plugins.challenges' from '/var/www/ctfd/CTFd/plugins/challenges/__init__.pyc'>
 * Loaded module, <module 'CTFd.plugins.keys' from '/var/www/ctfd/CTFd/plugins/keys/__init__.pyc'>
Traceback (most recent call last):
  File "import.py", line 9, in <module>
    import_ctf(sys.argv[1])
  File "/var/www/ctfd/CTFd/utils.py", line 802, in import_ctf
    table.insert(entry)
  File "/usr/local/lib/python2.7/dist-packages/dataset/table.py", line 80, in insert
    res = self.db.executable.execute(self.table.insert(row))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
    None, None)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
    context = constructor(dialect, self, conn, *args)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 623, in _init_compiled
    param.append(processors[key](compiled_params[key]))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", line 610, in process
    raise TypeError("SQLite DateTime type only accepts Python "
sqlalchemy.exc.StatementError: (exceptions.TypeError) SQLite DateTime type only accepts Python datetime and date objects as input. [SQL: u'INSERT INTO teams (name, email, password, website, affiliation, country, bracket, banned, verified, admin, joined) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)']
```



Discussion:
How old is your CTFd instance? 

This should be resolved as of https://github.com/CTFd/CTFd/issues/246

EDIT: I just realized you said master :) 

EDIT2: Although the line numbers don't seem to line up... Are you sure you're running master? 
Upon further inspection, the version of CTFd that the data was exported from was version 1.0.2. Do you think the database will upgrade cleanly to the latest version?

EDIT: Sorry! I didn't mean to post on the old issue
Assuming the export has the alembic_version saved (meaning it is a full export) CTFd should have enough information to insert everything and then update itself. 

It is still simplest to only import into a completely empty database but the basic idea is that CTFd should import data, then realize that it needs to migrate, and then once you apply migrations it should update to latest. 
Sorry, I am making this really difficult. That stacktrace was from 1.0.5. Here is one from master

```
> sudo python import.py ~/CTF-DATA.zip 
/*\ CTFd has updated and must update the database! /*\
/*\ Please backup your database before proceeding! /*\
/*\ CTFd maintainers are not responsible for any data loss! /*\
Run database migrations (Y/N)y
 * Loaded module, <module 'CTFd.plugins.challenges' from '/var/www/ctfd/CTFd/plugins/challenges/__init__.pyc'>
 * Loaded module, <module 'CTFd.plugins.keys' from '/var/www/ctfd/CTFd/plugins/keys/__init__.pyc'>
Traceback (most recent call last):
  File "import.py", line 9, in <module>
    import_ctf(sys.argv[1])
  File "/var/www/ctfd/CTFd/utils.py", line 804, in import_ctf
    table.insert(entry)
  File "/usr/local/lib/python2.7/dist-packages/dataset/table.py", line 80, in insert
    res = self.db.executable.execute(self.table.insert(row))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
    None, None)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
    exc_info
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
    context = constructor(dialect, self, conn, *args)
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 623, in _init_compiled
    param.append(processors[key](compiled_params[key]))
  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/dialects/sqlite/base.py", line 610, in process
    raise TypeError("SQLite DateTime type only accepts Python "
sqlalchemy.exc.StatementError: (exceptions.TypeError) SQLite DateTime type only accepts Python datetime and date objects as input. [SQL: u'INSERT INTO teams (name, email, password, website, affiliation, country, bracket, banned, verified, admin, joined) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)']
```
Can you open up the zip file and give me an example of what the importer is messing up on? 

CTFd should understand datetimes and try to convert them into real datetimes for import. 

Alternatively, you can just not use SQLite.
Also, I looked into alembic_version.json, and it is a blank file.

```
     {
    5       "id": 1,
    6       "name": "<redacted>",
    7       "email": "<redacted>",
    8       "password": "<redacted>",
    9       "website": null,
   10       "affiliation": null,
   11       "country": null,
   12       "bracket": null,
   13       "banned": false,
   14       "verified": false,
   15       "admin": true,
   16       "joined": "2017-08-21T19:26:42.037472"
   17     }
```
If alembic_version is blank then its likely the import was not a full one or some other issue happened. 

Seems like the issue is the microseconds not being accepted. The proper format to accept this would probably be

```
v = datetime.datetime.strptime(v, '%Y-%m-%dT%H:%M:%S.%f')
```

I guess this is coming from a MySQL database and then you're trying to import it into SQLite? For simplicity I would just use MySQL for now so you can update your schema. 
You could also try to update the date format string to the one I posted above in `utils.py` to see if that allows you to import. That should work if you really want to use SQLite. 
Thank you for the help! I think there was some miscommunication between me and another admin about how this was setup. I will try the suggestions above.
CTFd exports should be able to go between databases. They should be viewed as a single portable file that can go between CTFd instances. 

I think there's enough here to warrant some kind of fix so I'll try to reproduce and come up with some general solution. 
I doubled checked, and we definitely ran with SQLite. The production deployment ran on Debian 8.9 with version 3.8.7.1 of SQLite3. The exported data will not import into MySQL because of the following error:

```
sqlalchemy.exc.IntegrityError: (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`ctfd`.`keys`, CONSTRAINT `keys_ibfk_1` FOREIGN KEY (`chal`) REFERENCES `challenges` (`id`))') [SQL: u'INSERT INTO `keys` (chal, key_type, flag, data) VALUES (%s, %s, %s, %s)'] [parameters: (29, 0, u'challenge-flag-here', None)]
```
In that case I would probably see if you can modify the date format, or downgrade to 1.0.2, import, and then upgrade CTFd to master and run migrations. 

Resolving patch:
diff --git a/CTFd/utils.py b/CTFd/utils.py
--- a/CTFd/utils.py
+++ b/CTFd/utils.py
@@ -32,7 +32,7 @@
 from socket import timeout
 from werkzeug.utils import secure_filename
 
-from CTFd.models import db, WrongKeys, Pages, Config, Tracking, Teams, Files, ip2long, long2ip
+from CTFd.models import db, Challenges, WrongKeys, Pages, Config, Tracking, Teams, Files, ip2long, long2ip
 
 from datafreeze.format import SERIALIZERS
 from datafreeze.format.fjson import JSONSerializer, JSONEncoder
@@ -793,6 +793,14 @@ def export_ctf(segments=None):
             result_file.seek(0)
             backup_zip.writestr('db/{}.json'.format(item), result_file.read())
 
+    # Guarantee that alembic_version is saved into the export
+    if 'metadata' not in segments:
+        result = db['alembic_version'].all()
+        result_file = six.BytesIO()
+        datafreeze.freeze(result, format='ctfd', fileobj=result_file)
+        result_file.seek(0)
+        backup_zip.writestr('db/alembic_version.json', result_file.read())
+
     # Backup uploads
     upload_folder = os.path.join(os.path.normpath(app.root_path), get_config('UPLOAD_FOLDER'))
     for root, dirs, files in os.walk(upload_folder):
@@ -863,16 +871,23 @@ def import_ctf(backup, segments=None, erase=False):
                 elif item == 'pages':
                     saved = json.loads(data)
                     for entry in saved['results']:
+                        # Support migration c12d2a1b0926_add_draft_and_title_to_pages
                         route = entry['route']
+                        title = entry.get('title', route.title())
                         html = entry['html']
+                        draft = entry.get('draft', False)
+                        auth_required = entry.get('auth_required', False)
                         page = Pages.query.filter_by(route=route).first()
                         if page:
                             page.html = html
                         else:
-                            page = Pages(route, html)
+                            page = Pages(title, route, html, draft=draft, auth_required=auth_required)
                             db.session.add(page)
                         db.session.commit()
 
+    teams_base = db.session.query(db.func.max(Teams.id)).scalar() or 0
+    chals_base = db.session.query(db.func.max(Challenges.id)).scalar() or 0
+
     for segment in segments:
         group = groups[segment]
         for item in group:
@@ -888,11 +903,31 @@ def import_ctf(backup, segments=None, erase=False):
                     if get_config('SQLALCHEMY_DATABASE_URI').startswith('sqlite'):
                         for k, v in entry.items():
                             if isinstance(v, six.string_types):
-                                try:
+                                match = re.match(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d", v)
+                                if match:
+                                    entry[k] = datetime.datetime.strptime(v, '%Y-%m-%dT%H:%M:%S.%f')
+                                    continue
+                                match = re.match(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}", v)
+                                if match:
                                     entry[k] = datetime.datetime.strptime(v, '%Y-%m-%dT%H:%M:%S')
-                                except ValueError as e:
-                                    pass
-                    table.insert(entry)
+                                    continue
+                    for k, v in entry.items():
+                        if k == 'chal' or k == 'chalid':
+                            entry[k] += chals_base
+                        if k == 'team' or k == 'teamid':
+                            entry[k] += teams_base
+
+                    if item == 'teams':
+                        table.insert_ignore(entry, ['email'])
+                    elif item == 'keys':
+                        # Support migration 2539d8b5082e_rename_key_type_to_type
+                        key_type = entry.get('key_type', None)
+                        if key_type is not None:
+                            entry['type'] = key_type
+                            del entry['key_type']
+                        table.insert(entry)
+                    else:
+                        table.insert(entry)
             else:
                 continue
 
@@ -914,6 +949,6 @@ def import_ctf(backup, segments=None, erase=False):
             os.makedirs(dirname)
 
         source = backup.open(f)
-        target = file(full_path, "wb")
+        target = open(full_path, "wb")
         with source, target:
             shutil.copyfileobj(source, target)


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -3,7 +3,7 @@
 
 from tests.helpers import *
 from CTFd.models import ip2long, long2ip
-from CTFd.utils import get_config, set_config, override_template, sendmail, verify_email, ctf_started, ctf_ended, export_ctf
+from CTFd.utils import get_config, set_config, override_template, sendmail, verify_email, ctf_started, ctf_ended, export_ctf, import_ctf
 from CTFd.utils import register_plugin_script, register_plugin_stylesheet
 from CTFd.utils import base64encode, base64decode
 from CTFd.utils import check_email_format
@@ -11,6 +11,7 @@
 from freezegun import freeze_time
 from mock import patch, Mock
 import json
+import os
 import requests
 import six
 
@@ -348,9 +349,49 @@ def test_export_ctf():
         output = json.loads(output)
         app.db.session.commit()
         backup = export_ctf()
-        backup.seek(0)
+
         with open('export.zip', 'wb') as f:
             f.write(backup.getvalue())
+        os.remove('export.zip')
+    destroy_ctfd(app)
+
+
+def test_import_ctf():
+    """Test that CTFd can import a CTF"""
+    app = create_ctfd()
+    # TODO: Unrelated to an in-memory database, imports in a test environment are not working with SQLite...
+    if app.config['SQLALCHEMY_DATABASE_URI'].startswith('sqlite') is False:
+        with app.app_context():
+            base_user = 'user'
+            for x in range(10):
+                user = base_user + str(x)
+                user_email = user + "@ctfd.io"
+                gen_team(app.db, name=user, email=user_email)
+
+            for x in range(10):
+                chal = gen_challenge(app.db, name='chal_name{}'.format(x))
+                gen_flag(app.db, chal=chal.id, flag='flag')
+
+            app.db.session.commit()
+
+            backup = export_ctf()
+
+            with open('export.zip', 'wb') as f:
+                f.write(backup.read())
+        destroy_ctfd(app)
+
+        app = create_ctfd()
+        with app.app_context():
+            import_ctf('export.zip')
+
+            app.db.session.commit()
+
+            print(Teams.query.count())
+            print(Challenges.query.count())
+
+            assert Teams.query.count() == 11
+            assert Challenges.query.count() == 10
+            assert Keys.query.count() == 10
     destroy_ctfd(app)