Repository: CTFd/CTFd

Issue:
CSV hints with commas get split on imports
It seems not possible to supply a hint string that itself contains a comma. Example:
```
"friendly comma-free hint,https://url-hint-also-friendly,""hint, that contains comma"""
```
This would create four hints on import:
* friendly comma-free hint
* https://url-hint-also-friendly
* "hint
*  that contains comma"

but the expected result should be
* friendly comma-free hint
* https://url-hint-also-friendly
* "hint, that contains comma"

Concrete example to test this with:

```
name,description,category,value,type,state,max_attempts,flags,tags,hints,type_data
API-only XSS,"Perform a <i>persisted</i> XSS attack with <code>&lt;iframe src=\""javascript:alert(&#x60;xss&#x60;)\""&gt;</code> without using the frontend application at all. <em>(This challenge is <strong>not available</strong> on Heroku!)</em> (Difficulty Level: 3)",XSS,450,standard,visible,0,f94113c07b6e82fa8ba90369088818711423838a,Danger Zone,"You need to work with the server-side API directly. Try different HTTP verbs on different entities exposed through the API.,https://pwning.owasp-juice.shop/part2/xss.html#perform-a-persisted-xss-attack-without-using-the-frontend-application-at-all,""<pre><code>ngAfterViewInit () {\n    const products = this.productService.search('')\n    const quantities = this.quantityService.getAll()\n    forkJoin([quantities, products]).subscribe(([quantities, products]) => {\n      const dataTable: TableEntry[] = []\n      this.tableData = products\n      this.trustProductDescription(products)\n      for (const product of products) {\n        dataTable.push({\n          name: product.name,\n          price: product.price,\n          deluxePrice: product.deluxePrice,\n          id: product.id,\n          image: product.image,\n          description: product.description\n        })\n      }\n      for (const quantity of quantities) {\n        const entry = dataTable.find((dataTableEntry) => {\n          return dataTableEntry.id === quantity.ProductId\n        })\n        if (entry === undefined) {\n          continue\n        }\n        entry.quantity = quantity.quantity\n      }\n      this.dataSource = new MatTableDataSource<TableEntry>(dataTable)\n      for (let i = 1; i <= Math.ceil(this.dataSource.data.length / 12); i++) {\n        this.pageSizeOptions.push(i * 12)\n      }\n      this.paginator.pageSizeOptions = this.pageSizeOptions\n      this.dataSource.paginator = this.paginator\n      this.gridDataSource = this.dataSource.connect()\n      this.resultsLength = this.dataSource.data.length\n      this.filterTable()\n      this.routerSubscription = this.router.events.subscribe(() => {\n        this.filterTable()\n      })\n      if (window.innerWidth < 2600) {\n        this.breakpoint = 4\n        if (window.innerWidth < 1740) {\n          this.breakpoint = 3\n          if (window.innerWidth < 1280) {\n            this.breakpoint = 2\n            if (window.innerWidth < 850) {\n              this.breakpoint = 1\n            }\n          }\n        }\n      } else {\n        this.breakpoint = 6\n      }\n      this.cdRef.detectChanges()\n    }, (err) => console.log(err))\n  }\n\n  trustProductDescription (tableData: any[]) {\n    for (let i = 0; i < tableData.length; i++) {\n      tableData[i].description = this.sanitizer.bypassSecurityTrustHtml(tableData[i].description)\n    }\n  }</code></pre>""",
```

After import this is what you get as hints:
<img width="345" alt="image" src="https://github.com/CTFd/CTFd/assets/3531020/ea9ba142-a826-4ec1-86fe-1f96809f5a87">



Discussion:
💡 This one has a pretty easy workaround: Replacing `,` (Comma) with `٬` (Arabic Thousands Separator Unicode char) in all hints before creating the CSV for CTFd import. Works fine, looks like a comma, doesn't hurt unless the comma would be relevant for something that needs to be copy-pasted.

<img width="286" alt="image" src="https://github.com/CTFd/CTFd/assets/3531020/4c91d389-6d06-4967-9205-5c07107e72b4">

Unfortunately my hints are copy-paste - i.e partial code blocks, so need the literal comma.
I think that this along with https://github.com/CTFd/CTFd/issues/2038 can be solved by allowing the Hints CSV column to contain JSON. This idea is already used in other parts of the CSV however it's intended for more unstructured data. 

I think likely we can detect whether or not the field is valid JSON and then either load it using the string behavior or load it as JSON. 

I do think that the CSV importer cannot fully replicate challenge.yml so it's also plausible that we should consider not even implementing this.

Resolving patch:
diff --git a/CTFd/utils/csv/__init__.py b/CTFd/utils/csv/__init__.py
--- a/CTFd/utils/csv/__init__.py
+++ b/CTFd/utils/csv/__init__.py
@@ -364,15 +364,32 @@ def load_challenges_csv(dict_reader):
         db.session.commit()
 
         if flags:
-            flags = [flag.strip() for flag in flags.split(",")]
-            for flag in flags:
-                f = Flags(
-                    type="static",
-                    challenge_id=challenge.id,
-                    content=flag,
-                )
-                db.session.add(f)
-                db.session.commit()
+            try:
+                # Allow for column to contain JSON for more flexible data entry
+                flags = json.loads(flags)
+                for flag in flags:
+                    type = flag.get("type", "static")
+                    content = flag.get("content", "")
+                    data = flag.get("data", None)
+                    f = Flags(
+                        challenge_id=challenge.id,
+                        type=type,
+                        content=content,
+                        data=data,
+                    )
+                    db.session.add(f)
+                    db.session.commit()
+
+            except json.JSONDecodeError:
+                flags = [flag.strip() for flag in flags.split(",")]
+                for flag in flags:
+                    f = Flags(
+                        type="static",
+                        challenge_id=challenge.id,
+                        content=flag,
+                    )
+                    db.session.add(f)
+                    db.session.commit()
 
         if tags:
             tags = [tag.strip() for tag in tags.split(",")]
@@ -385,14 +402,28 @@ def load_challenges_csv(dict_reader):
                 db.session.commit()
 
         if hints:
-            hints = [hint.strip() for hint in hints.split(",")]
-            for hint in hints:
-                h = Hints(
-                    challenge_id=challenge.id,
-                    content=hint,
-                )
-                db.session.add(h)
-                db.session.commit()
+            try:
+                # Allow for column to contain JSON for more flexible data entry
+                hints = json.loads(hints)
+                for hint in hints:
+                    content = hint.get("content", "")
+                    cost = hint.get("cost", 0)
+                    h = Hints(
+                        challenge_id=challenge.id,
+                        content=content,
+                        cost=cost,
+                    )
+                    db.session.add(h)
+                    db.session.commit()
+            except json.JSONDecodeError:
+                hints = [hint.strip() for hint in hints.split(",")]
+                for hint in hints:
+                    h = Hints(
+                        challenge_id=challenge.id,
+                        content=hint,
+                    )
+                    db.session.add(h)
+                    db.session.commit()
     if errors:
         return errors
     return True


Test patch:
diff --git a/tests/admin/test_csv.py b/tests/admin/test_csv.py
--- a/tests/admin/test_csv.py
+++ b/tests/admin/test_csv.py
@@ -1,6 +1,6 @@
 import io
 
-from CTFd.models import Challenges, Teams, Users
+from CTFd.models import Challenges, Flags, Hints, Teams, Users
 from CTFd.utils.crypto import verify_password
 from tests.helpers import create_ctfd, destroy_ctfd, gen_challenge, login_as_user
 
@@ -30,8 +30,8 @@ def test_import_csv_works():
 team1,team1@examplectf.com,password
 team2,team2@examplectf.com,password"""
 
-    CHALLENGES_CSV = b"""name,category,description,value,flags,tags,hints
-challenge1,category1,description1,100,"flag1,flag2,flag3","tag1,tag2,tag3","hint1,hint2,hint3"""
+    CHALLENGES_CSV = b'''name,category,description,value,flags,tags,hints
+challenge1,category1,description1,100,"flag1,flag2,flag3","tag1,tag2,tag3","hint1,hint2,hint3"'''
 
     app = create_ctfd()
     with app.app_context():
@@ -91,3 +91,53 @@ def test_import_csv_works():
         assert len(challenge.hints) == 3
 
     destroy_ctfd(app)
+
+
+def test_import_challenge_csv_with_json():
+
+    CHALLENGES_CSV = b'''name,category,description,value,flags,tags,hints
+challenge1,category1,description1,100,"[{""type"": ""static"", ""content"": ""flag1"", ""data"": ""case_insensitive""}, {""type"": ""regex"", ""content"": ""(.*)"", ""data"": ""case_insensitive""}, {""type"": ""static"", ""content"": ""flag3""}]","tag1,tag2,tag3","[{""content"": ""hint1"", ""cost"": 10}, {""content"": ""hint2"", ""cost"": 20}, {""content"": ""hint3"", ""cost"": 30}]"'''
+
+    app = create_ctfd()
+    with app.app_context():
+        client = login_as_user(app, name="admin", password="password")
+
+        with client.session_transaction() as sess:
+            data = {
+                "csv_type": "challenges",
+                "csv_file": (io.BytesIO(CHALLENGES_CSV), "challenges.csv"),
+                "nonce": sess.get("nonce"),
+            }
+
+        client.post("/admin/import/csv", data=data, content_type="multipart/form-data")
+        assert Challenges.query.count() == 1
+        challenge = Challenges.query.filter_by(id=1).first()
+        assert challenge.name == "challenge1"
+        assert challenge.category == "category1"
+        assert challenge.description == "description1"
+        assert challenge.value == 100
+        assert len(challenge.flags) == 3
+        assert len(challenge.tags) == 3
+        assert len(challenge.hints) == 3
+
+        for i in range(1, 4):
+            h = Hints.query.filter_by(id=i).first()
+            assert h.cost == i * 10
+            assert h.content == f"hint{i}"
+
+        f = Flags.query.filter_by(id=1).first()
+        assert f.type == "static"
+        assert f.content == "flag1"
+        assert f.data == "case_insensitive"
+
+        f = Flags.query.filter_by(id=2).first()
+        assert f.type == "regex"
+        assert f.content == "(.*)"
+        assert f.data == "case_insensitive"
+
+        f = Flags.query.filter_by(id=3).first()
+        assert f.type == "static"
+        assert f.content == "flag3"
+        assert f.data is None
+
+    destroy_ctfd(app)