internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: CTFd/CTFd

Issue:
CSV for teams+members+fields
We should have this CSV format to export but for some reason we don't. Should be an easy implementation. 


Discussion:

Resolving patch:
diff --git a/CTFd/utils/csv/__init__.py b/CTFd/utils/csv/__init__.py
--- a/CTFd/utils/csv/__init__.py
+++ b/CTFd/utils/csv/__init__.py
@@ -216,6 +216,66 @@ def dump_teams_with_fields_csv():
     return output
 
 
+def dump_teams_with_members_fields_csv():
+    temp = StringIO()
+    writer = csv.writer(temp)
+
+    team_fields = TeamFields.query.all()
+    team_field_ids = [f.id for f in team_fields]
+    team_field_names = [f.name for f in team_fields]
+
+    user_fields = UserFields.query.all()
+    user_field_ids = [f.id for f in user_fields]
+    user_field_names = [f.name for f in user_fields]
+
+    user_header = [
+        f"member_{column.name}" for column in Users.__mapper__.columns
+    ] + user_field_names
+
+    header = (
+        [column.name for column in Teams.__mapper__.columns]
+        + team_field_names
+        + user_header
+    )
+    writer.writerow(header)
+
+    responses = Teams.query.all()
+
+    for curr in responses:
+        team_field_entries = {f.field_id: f.value for f in curr.field_entries}
+        team_field_values = [
+            team_field_entries.get(f_id, "") for f_id in team_field_ids
+        ]
+
+        team_row = [
+            getattr(curr, column.name) for column in Teams.__mapper__.columns
+        ] + team_field_values
+
+        writer.writerow(team_row)
+
+        for member in curr.members:
+            padding = [""] * len(team_row)
+
+            user_field_entries = {f.field_id: f.value for f in member.field_entries}
+            user_field_values = [
+                user_field_entries.get(f_id, "") for f_id in user_field_ids
+            ]
+            user_row = [
+                getattr(member, column.name) for column in Users.__mapper__.columns
+            ] + user_field_values
+            writer.writerow(padding + user_row)
+
+    temp.seek(0)
+
+    # In Python 3 send_file requires bytes
+    output = BytesIO()
+    output.write(temp.getvalue().encode("utf-8"))
+    output.seek(0)
+    temp.close()
+
+    return output
+
+
 def dump_database_table(tablename):
     # TODO: It might make sense to limit dumpable tables. Config could potentially leak sensitive information.
     model = get_class_by_tablename(tablename)
@@ -334,4 +394,5 @@ def load_challenges_csv(dict_reader):
     "scoreboard": dump_scoreboard_csv,
     "users+fields": dump_users_with_fields_csv,
     "teams+fields": dump_teams_with_fields_csv,
+    "teams+members+fields": dump_teams_with_members_fields_csv,
 }


Test patch:
Repository: CTFd/CTFd

Issue:
Fix SAFE_MODE
SAFE_MODE isn't working from config.ini apparently at the moment. 


Discussion:

Resolving patch:
diff --git a/CTFd/config.py b/CTFd/config.py
--- a/CTFd/config.py
+++ b/CTFd/config.py
@@ -195,6 +195,8 @@ class ServerConfig(object):
 
     HTML_SANITIZATION: bool = process_boolean_str(empty_str_cast(config_ini["optional"]["HTML_SANITIZATION"], default=False))
 
+    SAFE_MODE: bool = process_boolean_str(empty_str_cast(config_ini["optional"]["SAFE_MODE"], default=False))
+
     if DATABASE_URL.startswith("sqlite") is False:
         SQLALCHEMY_ENGINE_OPTIONS = {
             "max_overflow": int(empty_str_cast(config_ini["optional"]["SQLALCHEMY_MAX_OVERFLOW"], default=20)),  # noqa: E131
diff --git a/CTFd/plugins/__init__.py b/CTFd/plugins/__init__.py
--- a/CTFd/plugins/__init__.py
+++ b/CTFd/plugins/__init__.py
@@ -202,6 +202,8 @@ def init_plugins(app):
             module = importlib.import_module(module, package="CTFd.plugins")
             module.load(app)
             print(" * Loaded module, %s" % module)
+    else:
+        print("SAFE_MODE is enabled. Skipping plugin loading.")
 
     app.jinja_env.globals.update(get_admin_plugin_menu_bar=get_admin_plugin_menu_bar)
     app.jinja_env.globals.update(get_user_page_menu_bar=get_user_page_menu_bar)


Test patch:
Repository: CTFd/CTFd

Issue:
Health check fails on / despite app being online
<!--
If this is a bug report please fill out the template below.

If this is a feature request please describe the behavior that you'd like to see.
-->

**Environment**:

- CTFd Version/Commit: Latest Docker
- Operating System: Latest Docker
- Web Browser and Version: N/A

**What happened?**
When running the application in Kubernetes for the first time, the `/` url 302 redirects to `/setup` which causes the Health chek to fail 

**What did you expect to happen?**

Application would return `200` on `/` [KB](https://cloud.google.com/load-balancing/docs/health-check-concepts#criteria-protocol-http:~:text=HTTP%20200%20(OK)%20status.)

**How to reproduce your issue**

Create application in GKE using below health check for `backendConfig`

**Any associated stack traces or error logs**

```text
1. 172.17.2.1 - - [03/Nov/2022:22:14:15 +0000] "GET /setup HTTP/1.1" 200 15200 "-" "kube-probe/1.24"
2. 35.191.12.160 - - [03/Nov/2022:22:14:20 +0000] "GET / HTTP/1.1" 302 219 "-" "GoogleHC/1.0"
3. 35.191.12.162 - - [03/Nov/2022:22:14:20 +0000] "GET / HTTP/1.1" 302 219 "-" "GoogleHC/1.0"
4. 35.191.12.164 - - [03/Nov/2022:22:14:20 +0000] "GET / HTTP/1.1" 302 219 "-" "GoogleHC/1.0"
```

Line 1 shows the k8 health check that follows redirects 

Line 2-4 shows the Google HC that does not, and fails.


** What I would like to see **

A `/health` or `/halthz` page that returns `http status 200` when the application is running and can connect to the DB regardless of if a user has completed the setup 

[KB on /healthz](https://kubernetes.io/docs/reference/using-api/health-checks/)

---

Let me know if you would like some additional details around this, or more details/ better explanation! 



Discussion:
Thanks for reporting this. If you have the time you could probably solve this by adding "views.healthcheck" to this area here: https://github.com/CTFd/CTFd/blob/a085d0922a6e730c5c479869637a0335fe108bb7/CTFd/utils/initialization/__init__.py#L200.

Resolving patch:
diff --git a/CTFd/utils/initialization/__init__.py b/CTFd/utils/initialization/__init__.py
--- a/CTFd/utils/initialization/__init__.py
+++ b/CTFd/utils/initialization/__init__.py
@@ -198,6 +198,7 @@ def needs_setup():
                 "views.integrations",
                 "views.themes",
                 "views.files",
+                "views.healthcheck",
             ):
                 return
             else:


Test patch:
Repository: CTFd/CTFd

Issue:
Controlling static content like robots.txt
Some users seem to wish to be able to control things like their robots.txt. Is this something that we could implement in CTFd? Perhaps as a type of Page with a custom reponse type? 


Discussion:
I think instead of adding a page we should just add a special section in the config to handle robots.txt. I don't think most people want to upload arbitrary files with custom media types but they do want to control robots.txt. 

Resolving patch:
diff --git a/CTFd/utils/initialization/__init__.py b/CTFd/utils/initialization/__init__.py
--- a/CTFd/utils/initialization/__init__.py
+++ b/CTFd/utils/initialization/__init__.py
@@ -204,6 +204,7 @@ def needs_setup():
                 "views.themes",
                 "views.files",
                 "views.healthcheck",
+                "views.robots",
             ):
                 return
             else:
diff --git a/CTFd/views.py b/CTFd/views.py
--- a/CTFd/views.py
+++ b/CTFd/views.py
@@ -2,7 +2,15 @@
 
 from flask import Blueprint, abort
 from flask import current_app as app
-from flask import redirect, render_template, request, send_file, session, url_for
+from flask import (
+    make_response,
+    redirect,
+    render_template,
+    request,
+    send_file,
+    session,
+    url_for,
+)
 from flask.helpers import safe_join
 from jinja2.exceptions import TemplateNotFound
 from sqlalchemy.exc import IntegrityError
@@ -514,3 +522,11 @@ def healthcheck():
     if check_config() is False:
         return "ERR", 500
     return "OK", 200
+
+
+@views.route("/robots.txt")
+def robots():
+    text = get_config("robots_txt", "User-agent: *\nDisallow: /admin\n")
+    r = make_response(text, 200)
+    r.mimetype = "text/plain"
+    return r


Test patch:
diff --git a/tests/test_views.py b/tests/test_views.py
--- a/tests/test_views.py
+++ b/tests/test_views.py
@@ -451,3 +451,19 @@ def test_user_can_access_files_if_view_after_ctf():
             rmdir(directory)
 
     destroy_ctfd(app)
+
+
+def test_robots_txt():
+    """Does the robots.txt page work"""
+    app = create_ctfd()
+    with app.app_context():
+        with app.test_client() as client:
+            r = client.get("/robots.txt")
+            assert r.status_code == 200
+            assert r.get_data(as_text=True) == "User-agent: *\nDisallow: /admin\n"
+        set_config("robots_txt", "testing")
+        with app.test_client() as client:
+            r = client.get("/robots.txt")
+            assert r.status_code == 200
+            assert r.get_data(as_text=True) == "testing"
+    destroy_ctfd(app)
Repository: CTFd/CTFd

Issue:
HTML_SANITIZATION controllable from config panel
We should maybe make HTML_SANITIZATION controlable from the admin panel so that Admins can decide how they want to deal with HTML. 

Of course this maybe could be done another way, it's just the general idea about the configuration value. 


Discussion:

Resolving patch:
diff --git a/CTFd/admin/__init__.py b/CTFd/admin/__init__.py
--- a/CTFd/admin/__init__.py
+++ b/CTFd/admin/__init__.py
@@ -48,7 +48,7 @@
     db,
 )
 from CTFd.utils import config as ctf_config
-from CTFd.utils import get_config, set_config
+from CTFd.utils import get_app_config, get_config, set_config
 from CTFd.utils.csv import dump_csv, load_challenges_csv, load_teams_csv, load_users_csv
 from CTFd.utils.decorators import admins_only
 from CTFd.utils.exports import background_import_ctf
@@ -190,7 +190,14 @@ def config():
     except ValueError:
         pass
 
-    return render_template("admin/config.html", themes=themes, **configs)
+    force_html_sanitization = get_app_config("HTML_SANITIZATION")
+
+    return render_template(
+        "admin/config.html",
+        themes=themes,
+        **configs,
+        force_html_sanitization=force_html_sanitization
+    )
 
 
 @admin.route("/admin/reset", methods=["GET", "POST"])
diff --git a/CTFd/utils/config/pages.py b/CTFd/utils/config/pages.py
--- a/CTFd/utils/config/pages.py
+++ b/CTFd/utils/config/pages.py
@@ -36,7 +36,11 @@ def format_variables(content):
 
 def build_html(html, sanitize=False):
     html = format_variables(html)
-    if current_app.config["HTML_SANITIZATION"] is True or sanitize is True:
+    if (
+        current_app.config["HTML_SANITIZATION"] is True
+        or bool(get_config("html_sanitization")) is True
+        or sanitize is True
+    ):
         html = sanitize_html(html)
     return html
 
@@ -44,7 +48,11 @@ def build_html(html, sanitize=False):
 def build_markdown(md, sanitize=False):
     html = markdown(md)
     html = format_variables(html)
-    if current_app.config["HTML_SANITIZATION"] is True or sanitize is True:
+    if (
+        current_app.config["HTML_SANITIZATION"] is True
+        or bool(get_config("html_sanitization")) is True
+        or sanitize is True
+    ):
         html = sanitize_html(html)
     return html
 


Test patch:
Repository: CTFd/CTFd

Issue:
Challenge Preview Improvements
Challenge Preview should probably preview in the context of a full page. Primarily because it's easier to theme this. 


Discussion:
I think the solution here is to be able to link to what the public page would look like by linking directly to the public page. 

The issue is fundamentally that the challenges page renders a lot of UI but there still isn't a standard entrypoint to render a challenge. Even if we were to just render the content into a faked challenges.html, we don't really have great control of the JS to jump in and render everything. Especially because we aren't exporting anything. 

I don't think this can be solved with out a lot of JS rewrites....
I don't think this is even worth tackling without CTFd.js. Pushing until then. 

Resolving patch:
diff --git a/CTFd/admin/challenges.py b/CTFd/admin/challenges.py
--- a/CTFd/admin/challenges.py
+++ b/CTFd/admin/challenges.py
@@ -3,7 +3,10 @@
 from CTFd.admin import admin
 from CTFd.models import Challenges, Flags, Solves
 from CTFd.plugins.challenges import CHALLENGE_CLASSES, get_chal_class
+from CTFd.schemas.tags import TagSchema
 from CTFd.utils.decorators import admins_only
+from CTFd.utils.security.signing import serialize
+from CTFd.utils.user import get_current_team, get_current_user
 
 
 @admin.route("/admin/challenges")
@@ -71,6 +74,43 @@ def challenges_detail(challenge_id):
     )
 
 
+@admin.route("/admin/challenges/preview/<int:challenge_id>")
+@admins_only
+def challenges_preview(challenge_id):
+    challenge = Challenges.query.filter_by(id=challenge_id).first_or_404()
+    chal_class = get_chal_class(challenge.type)
+    user = get_current_user()
+    team = get_current_team()
+
+    files = []
+    for f in challenge.files:
+        token = {
+            "user_id": user.id,
+            "team_id": team.id if team else None,
+            "file_id": f.id,
+        }
+        files.append(url_for("views.files", path=f.location, token=serialize(token)))
+
+    tags = [
+        tag["value"] for tag in TagSchema("user", many=True).dump(challenge.tags).data
+    ]
+
+    content = render_template(
+        chal_class.templates["view"].lstrip("/"),
+        solves=None,
+        solved_by_me=False,
+        files=files,
+        tags=tags,
+        hints=challenge.hints,
+        max_attempts=challenge.max_attempts,
+        attempts=0,
+        challenge=challenge,
+    )
+    return render_template(
+        "admin/challenges/preview.html", content=content, challenge=challenge
+    )
+
+
 @admin.route("/admin/challenges/new")
 @admins_only
 def challenges_new():


Test patch:
Repository: CTFd/CTFd

Issue:
Make core-beta the default choice during setup
The core theme is effectively deprecated for us so it doesn't make much sense to allow for new core theme installs after 3.6.


Discussion:

Resolving patch:
diff --git a/CTFd/forms/setup.py b/CTFd/forms/setup.py
--- a/CTFd/forms/setup.py
+++ b/CTFd/forms/setup.py
@@ -11,7 +11,6 @@
 from wtforms.fields.html5 import EmailField
 from wtforms.validators import InputRequired
 
-from CTFd.constants.themes import DEFAULT_THEME
 from CTFd.forms import BaseForm
 from CTFd.forms.fields import SubmitField
 from CTFd.utils.config import get_themes
@@ -69,7 +68,8 @@ class SetupForm(BaseForm):
         _l("Theme"),
         description=_l("CTFd Theme to use. Can be changed later."),
         choices=list(zip(get_themes(), get_themes())),
-        default=DEFAULT_THEME,
+        ## TODO: Replace back to DEFAULT_THEME (aka core) in CTFd 4.0
+        default="core-beta",
         validators=[InputRequired()],
     )
     theme_color = HiddenField(


Test patch:
Repository: CTFd/CTFd

Issue:
Test Translations & Support Spanish
We need to test translations before release and make sure we support Spanish


Discussion:

Resolving patch:
diff --git a/CTFd/constants/languages.py b/CTFd/constants/languages.py
--- a/CTFd/constants/languages.py
+++ b/CTFd/constants/languages.py
@@ -5,12 +5,16 @@ class Languages(str, RawEnum):
     ENGLISH = "en"
     GERMAN = "de"
     POLISH = "pl"
+    SPANISH = "es"
+    CHINESE = "zh"
 
 
 LANGUAGE_NAMES = {
     "en": "English",
     "de": "Deutsch",
     "pl": "Polski",
+    "es": "Español",
+    "zh": "中文",
 }
 
 SELECT_LANGUAGE_LIST = [("", "")] + [


Test patch:
Repository: CTFd/CTFd

Issue:
Setup flow should always use the core theme
This just simplifies theme development. I think would be easier to do after CTFd.js but being optimistic. 


Discussion:
Thinking about this I don't think this is as big a deal as I used to think it is. With theme fallback this shouldn't be as big an issue as it was at one point. 

Resolving patch:
diff --git a/CTFd/__init__.py b/CTFd/__init__.py
--- a/CTFd/__init__.py
+++ b/CTFd/__init__.py
@@ -276,7 +276,7 @@ def set_sqlite_pragma(dbapi_connection, connection_record):
             utils.set_config("ctf_version", __version__)
 
         if not utils.get_config("ctf_theme"):
-            utils.set_config("ctf_theme", DEFAULT_THEME)
+            utils.set_config("ctf_theme", "core-beta")
 
         update_check(force=True)
 
diff --git a/CTFd/views.py b/CTFd/views.py
--- a/CTFd/views.py
+++ b/CTFd/views.py
@@ -297,8 +297,6 @@ def setup():
 
             return redirect(url_for("views.static_html"))
         try:
-            if bool(get_config("ctf_theme")) is False:
-                set_config("ctf_theme", "core-beta")
             return render_template("setup.html", state=serialize(generate_nonce()))
         except TemplateNotFound:
             # Set theme to default and try again


Test patch:
Repository: CTFd/CTFd

Issue:
Dynamic challenges do not show a Next Challenge
<!--
If this is a bug report please fill out the template below.

If this is a feature request please describe the behavior that you'd like to see.
-->

**Environment**:

- CTFd Version/Commit: 3.6.0/8ead306f8b57c059192cd8b137f37ee41a078a41
- Operating System: All
- Web Browser and Version: All

**What happened?**

TLDR: *dynamic* challenges do not serve `next_id` to the frontend.

**How to reproduce your issue**

1. I created two challenges A and B with dynamic scoring.
2. I opened the admin configuration for challenge A.
3. I clicked "Next"
4. I selected challenge B from the dropdown.
5. I clicked the "Save" button.
6. The input field is empty.

**What did you expect to happen?**

The input field shows "Challenge B".

**Any associated stack traces or error logs**

The issue arises from the lack of `next_id` field in API responses for dynamic challenges [here](https://github.com/CTFd/CTFd/blob/8ead306f8b57c059192cd8b137f37ee41a078a41/CTFd/plugins/dynamic_challenges/__init__.py#L60-L89).



Discussion:

Resolving patch:
diff --git a/CTFd/plugins/dynamic_challenges/__init__.py b/CTFd/plugins/dynamic_challenges/__init__.py
--- a/CTFd/plugins/dynamic_challenges/__init__.py
+++ b/CTFd/plugins/dynamic_challenges/__init__.py
@@ -75,6 +75,7 @@ def read(cls, challenge):
             "minimum": challenge.minimum,
             "description": challenge.description,
             "connection_info": challenge.connection_info,
+            "next_id": challenge.next_id,
             "category": challenge.category,
             "state": challenge.state,
             "max_attempts": challenge.max_attempts,


Test patch:
Repository: CTFd/CTFd

Issue:
Expose unix_time_to_utc to Jinja
I have a need for `unix_time_to_utc` in the Jinja template but it isn't currently exposed. Ultimately it's to convert between an Epoch timestamp to ISO 8601. 


Discussion:

Resolving patch:
diff --git a/CTFd/utils/initialization/__init__.py b/CTFd/utils/initialization/__init__.py
--- a/CTFd/utils/initialization/__init__.py
+++ b/CTFd/utils/initialization/__init__.py
@@ -20,7 +20,7 @@
     is_setup,
 )
 from CTFd.utils.config.pages import get_pages
-from CTFd.utils.dates import isoformat, unix_time, unix_time_millis
+from CTFd.utils.dates import isoformat, unix_time, unix_time_millis, unix_time_to_utc
 from CTFd.utils.events import EventManager, RedisEventManager
 from CTFd.utils.humanize.words import pluralize
 from CTFd.utils.modes import generate_account_url, get_mode_as_word
@@ -53,6 +53,7 @@ def init_template_filters(app):
     app.jinja_env.filters["markdown"] = markdown
     app.jinja_env.filters["unix_time"] = unix_time
     app.jinja_env.filters["unix_time_millis"] = unix_time_millis
+    app.jinja_env.filters["unix_time_to_utc"] = unix_time_to_utc
     app.jinja_env.filters["isoformat"] = isoformat
     app.jinja_env.filters["pluralize"] = pluralize
 


Test patch:
Repository: CTFd/CTFd

Issue:
Getting dynamic challenges by ID does not return decay function
When getting a dynamic challenge from `GET /api/v1/challenges/<challenge-id>`, the challenge does not return its decay function.

This seems to be caused by [this](https://github.com/CTFd/CTFd/blob/master/CTFd/plugins/dynamic_challenges/__init__.py#L60-L90) function which does not include `challenge.function`.

This seems be have been missed out when implementing different decay functions for dynamic challenges. This does not have any impact on normal usage of CTFd, but I see no reason not to include the function in the response.



Discussion:

Resolving patch:
diff --git a/CTFd/plugins/dynamic_challenges/__init__.py b/CTFd/plugins/dynamic_challenges/__init__.py
--- a/CTFd/plugins/dynamic_challenges/__init__.py
+++ b/CTFd/plugins/dynamic_challenges/__init__.py
@@ -73,6 +73,7 @@ def read(cls, challenge):
             "initial": challenge.initial,
             "decay": challenge.decay,
             "minimum": challenge.minimum,
+            "function": challenge.function,
             "description": challenge.description,
             "connection_info": challenge.connection_info,
             "next_id": challenge.next_id,


Test patch:
Repository: python-pillow/Pillow

Issue:
Needless call to dpkg-architecture -qDEB_HOST_MULTIARCH
setup.py calls `dpkg-architecture -qDEB_HOST_MULTIARCH` twice, but does not redirect stderr to /dev/null in the first call, which results in:

```
sh: dpkg-architecture: command not found
```

Please delete first call to `dpkg-architecture -qDEB_HOST_MULTIARCH`.



Discussion:
Thanks, can you send a pull request?

Please just commit the change.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -453,8 +453,6 @@ def add_multiarch_paths(self):
         tmpfile = os.path.join(self.build_temp, 'multiarch')
         if not os.path.exists(self.build_temp):
             os.makedirs(self.build_temp)
-        ret = os.system('dpkg-architecture -qDEB_HOST_MULTIARCH > %s' %
-                        tmpfile)
         ret = os.system(
             'dpkg-architecture -qDEB_HOST_MULTIARCH > %s 2> /dev/null' %
             tmpfile)


Test patch:
Repository: python-pillow/Pillow

Issue:
Loading grayscale images ignores transparency
If an image contains a palette paletteputalpha is called and the transparency merged. Nothing happens for grayscale images. If I try to convert it into RGBA and save it, the transparency is gone.

```
i = Image.open('e:\\temp\\testbild\\l_rns.png')
i = i.convert('RGBA')
i.save('e:\\temp\\testbild\\test.png')
```

![l_rns](https://f.cloud.github.com/assets/3226483/291521/e7f892ea-9307-11e2-9328-9368665a9c7c.png)



Discussion:

Resolving patch:
diff --git a/PIL/Image.py b/PIL/Image.py
--- a/PIL/Image.py
+++ b/PIL/Image.py
@@ -713,6 +713,16 @@ def convert(self, mode=None, data=None, dither=None,
         if dither is None:
             dither = FLOYDSTEINBERG
 
+        # fake a P-mode image, otherwise the transparency will get lost as there is
+        # currently no other way to convert transparency into an RGBA image
+        if self.mode == "L" and mode == "RGBA" and "transparency" in self.info:
+            from PIL import ImagePalette
+            self.mode = "P"
+            bytePalette = bytes([i//3 for i in range(768)])
+            self.palette = ImagePalette.raw("RGB", bytePalette)
+            self.palette.dirty = 1
+            self.load()
+
         try:
             im = self.im.convert(mode, dither)
         except ValueError:


Test patch: