PrimeIntellect/SWE-rebench-V2-Filtered-Easy-Verified huggingface dataset:
Repository: CTFd/CTFd (python)
Issue: Hidden challenges can be answered without being unhidden
If attacker obtains the key of Hidden Challenge using some method, attacker can answer Hidden Challenge.

It is possible by sending Hidden Challenge's key and id to `/chal`.
Hidden Challenge's id is a sequential number, so it can be searched by brute force.

Measures
- If id of Hidden Challeng is specified, do not perform processing.
- Complicate id.

Pull request description: Hidden challenges now return 404 and can't be solved
Closes #431 

Patch:
diff --git a/CTFd/challenges.py b/CTFd/challenges.py
index 20cb1210..e27f3749 100644
--- a/CTFd/challenges.py
+++ b/CTFd/challenges.py
@@ -306,6 +306,8 @@ def chal(chalid):
         print("[{0}] {1} submitted {2} with kpm {3}".format(*data))
 
         chal = Challenges.query.filter_by(id=chalid).first_or_404()
+        if chal.hidden:
+            abort(404)
         chal_class = get_chal_class(chal.type)
 
         # Anti-bruteforce / submitting keys too quickly


Test patch:
diff --git a/tests/helpers.py b/tests/helpers.py
index c3a00eee..e614363d 100644
--- a/tests/helpers.py
+++ b/tests/helpers.py
@@ -84,8 +84,10 @@ def get_scores(user):
     return scores['standings']
 
 
-def gen_challenge(db, name='chal_name', description='chal_description', value=100, category='chal_category', type='standard'):
+def gen_challenge(db, name='chal_name', description='chal_description', value=100, category='chal_category', type='standard', hidden=False):
     chal = Challenges(name, description, value, category)
+    if hidden:
+        chal.hidden = hidden
     db.session.add(chal)
     db.session.commit()
     return chal
diff --git a/tests/user/test_challenges.py b/tests/user/test_challenges.py
index a7b5e36b..7c96ba54 100644
--- a/tests/user/test_challenges.py
+++ b/tests/user/test_challenges.py
@@ -349,3 +349,27 @@ def test_that_view_challenges_unregistered_works():
         data = json.loads(data)
         assert data['status'] == -1
     destroy_ctfd(app)
+
+
+def test_hidden_challenge_is_unsolveable():
+    """Test that hidden challenges return 404 and do not insert a solve or wrong key"""
+    app = create_ctfd()
+    with app.app_context():
+        register_user(app)
+        client = login_as_user(app)
+        chal = gen_challenge(app.db, hidden=True)
+        flag = gen_flag(app.db, chal=chal.id, flag='flag')
+        with client.session_transaction() as sess:
+            data = {
+                "key": 'flag',
+                "nonce": sess.get('nonce')
+            }
+        r = client.post('/chal/{}'.format(chal.id), data=data)
+        assert r.status_code == 404
+
+        solves = Solves.query.all()
+        assert len(solves) == 0
+
+        wrong_keys = WrongKeys.query.all()
+        assert len(wrong_keys) == 0
+    destroy_ctfd(app)