Repository: Metaculus/metaculus
Pull request #2457: Feat/1233-discrete-questions/backend
Author: login: lsabor
Created: 2025-03-23T19:52:47Z | Merged: 2025-03-28T21:51:02Z

Backend changes for setting up the new Discrete Question type.
Front end changes will be in separate PRs.

This track supersedes #2454 as that implementation treated discrete as the same as numeric. But keeping the old one as reference.


Diff:
diff --git a/notifications/services.py b/notifications/services.py
index ab9861037c..644bce6447 100644
--- a/notifications/services.py
+++ b/notifications/services.py
@@ -145,7 +145,7 @@ def format_value(self, value, change: bool = False):
         if self.question.type in ("multiple_choice", "binary"):
             return f"{round(value * 100, 2)}%"
 
-        if self.question.type == "date":
+        if self.question.type == Question.QuestionType.DATE:
             if change:
                 # value is a timedelta in seconds
                 difference = timedelta(seconds=value)
@@ -172,7 +172,10 @@ def format_value(self, value, change: bool = False):
                 "%Y-%m-%d"
             )
 
-        if self.question.type == "numeric":
+        if self.question.type in [
+            Question.QuestionType.NUMERIC,
+            Question.QuestionType.DISCRETE,
+        ]:
             return format_value_unit(abbreviated_number(value), self.question.unit)
 
         return value
@@ -400,7 +403,7 @@ def get_email_context_group(cls, notifications: list[Notification]):
             serialized_notifications.append(
                 {
                     **params,
-                    "comments": preview_comments[:cls.comments_to_display],
+                    "comments": preview_comments[: cls.comments_to_display],
                     "comments_count": comments_count,
                     "read_more_count": read_more_count if read_more_count > 0 else 0,
                 }
diff --git a/questions/migrations/0021_question_inbound_outcome_count_and_more.py b/questions/migrations/0021_question_inbound_outcome_count_and_more.py
new file mode 100644
index 0000000000..fb3aa31467
--- /dev/null
+++ b/questions/migrations/0021_question_inbound_outcome_count_and_more.py
@@ -0,0 +1,104 @@
+# Generated by Django 5.1.7 on 2025-03-23 20:00
+
+import django_better_admin_arrayfield.models.fields
+from django.db import migrations, models
+
+
+def reverse_migrate(apps, schema_editor):
+    Question = apps.get_model("questions", "Question")
+    Question.objects.filter(type="discrete").update(type="numeric")
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("questions", "0020_question_spot_scoring_time"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="question",
+            name="inbound_outcome_count",
+            field=models.IntegerField(
+                blank=True,
+                help_text="For Discrete only.\n        Number of possible outcomes NOT including out of bounds values.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="forecast",
+            name="continuous_cdf",
+            field=django_better_admin_arrayfield.models.fields.ArrayField(
+                base_field=models.FloatField(), max_length=201, null=True, size=None
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="open_lower_bound",
+            field=models.BooleanField(
+                blank=True,
+                help_text="For Continuous only.\n        If there can be a resolution below the range_min.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="open_upper_bound",
+            field=models.BooleanField(
+                blank=True,
+                help_text="For Continuous only.\n        If there can be a resolution above the range_max.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="range_max",
+            field=models.FloatField(
+                blank=True,
+                help_text="For Continuous only.\n        Maximum inbound value. For Discrete, this is 1/2 a unit's width above the\n        displayed upper bound.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="range_min",
+            field=models.FloatField(
+                blank=True,
+                help_text="For Continuous only.\n        Minimum inbound value. For Discrete, this is 1/2 a unit's width below the\n        displayed lower bound.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="spot_scoring_time",
+            field=models.DateTimeField(
+                blank=True,
+                help_text="Time when spot scores are evaluated.\n        If not set, defaults to cp reveal time.",
+                null=True,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="type",
+            field=models.CharField(
+                choices=[
+                    ("binary", "Binary"),
+                    ("multiple_choice", "Multiple Choice"),
+                    ("numeric", "Numeric"),
+                    ("date", "Date"),
+                    ("discrete", "Discrete"),
+                ],
+                max_length=20,
+            ),
+        ),
+        migrations.AlterField(
+            model_name="question",
+            name="zero_point",
+            field=models.FloatField(
+                blank=True,
+                help_text="For Continuous only. NOT for Discrete.\n        If logaritmically scaled, the value of the zero point.",
+                null=True,
+            ),
+        ),
+        migrations.RunPython(migrations.RunPython.noop, reverse_migrate),
+    ]
diff --git a/questions/models.py b/questions/models.py
index c5e615f442..e44fa35d92 100644
--- a/questions/models.py
+++ b/questions/models.py
@@ -17,7 +17,7 @@
     from posts.models import Post
     from scoring.models import Score, ArchivedScore
 
-CDF_SIZE = 201
+DEFAULT_INBOUND_OUTCOME_COUNT = 200
 
 
 class QuestionQuerySet(QuerySet):
@@ -61,9 +61,10 @@ class Question(TimeStampedModel, TranslatedModel):  # type: ignore
     # Common fields
     class QuestionType(models.TextChoices):
         BINARY = "binary"
+        MULTIPLE_CHOICE = "multiple_choice"
         NUMERIC = "numeric"
         DATE = "date"
-        MULTIPLE_CHOICE = "multiple_choice"
+        DISCRETE = "discrete"
 
     type = models.CharField(max_length=20, choices=QuestionType.choices)
     resolution = models.TextField(null=True, blank=True)
@@ -128,26 +129,58 @@ class QuestionType(models.TextChoices):
         null=True,
         blank=True,
         help_text="""Time when spot scores are evaluated.
-        If not set, defaults to spot_scoring time.""",
+        If not set, defaults to cp reveal time.""",
     )
 
-    # continuous range fields
-    range_max = models.FloatField(null=True, blank=True)
-    range_min = models.FloatField(null=True, blank=True)
-    zero_point = models.FloatField(null=True, blank=True)
-    open_upper_bound = models.BooleanField(null=True, blank=True)
-    open_lower_bound = models.BooleanField(null=True, blank=True)
+    # continuous fields
+    range_min = models.FloatField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        Minimum inbound value. For Discrete, this is 1/2 a unit's width below the
+        displayed lower bound.""",
+    )
+    range_max = models.FloatField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        Maximum inbound value. For Discrete, this is 1/2 a unit's width above the
+        displayed upper bound.""",
+    )
+    zero_point = models.FloatField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only. NOT for Discrete.
+        If logaritmically scaled, the value of the zero point.""",
+    )
+    open_upper_bound = models.BooleanField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        If there can be a resolution above the range_max.""",
+    )
+    open_lower_bound = models.BooleanField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        If there can be a resolution below the range_min.""",
+    )
+    inbound_outcome_count = models.IntegerField(
+        null=True,
+        blank=True,
+        help_text="""For Discrete only.
+        Number of possible outcomes NOT including out of bounds values.""",
+    )
+    unit = models.CharField(max_length=25, blank=True)
 
     # list of multiple choice option labels
     options = ArrayField(models.CharField(max_length=200), blank=True, null=True)
-    group_variable = models.CharField(blank=True, null=False)
-
-    unit = models.CharField(max_length=25, blank=True)
 
     # Legacy field that will be removed
     possibilities = models.JSONField(null=True, blank=True)
 
     # Group
+    group_variable = models.CharField(blank=True, null=False)
     label = models.TextField(blank=True, null=False)
     group: "GroupOfQuestions" = models.ForeignKey(
         "GroupOfQuestions",
@@ -166,6 +199,27 @@ def save(self, **kwargs):
         if self.resolution is not None and self.resolution.strip() == "":
             self.resolution = None
 
+        # Some simple field enforcement
+        if self.type not in [
+            self.QuestionType.DATE,
+            self.QuestionType.NUMERIC,
+            self.QuestionType.DISCRETE,
+        ]:
+            self.range_min = None
+            self.range_max = None
+            self.open_upper_bound = False
+            self.open_lower_bound = False
+            self.unit = ""
+        if self.type not in [
+            self.QuestionType.DATE,
+            self.QuestionType.NUMERIC,
+        ]:
+            self.zero_point = None
+        if self.type != self.QuestionType.DISCRETE:
+            self.inbound_outcome_count = None
+        if self.type != self.QuestionType.MULTIPLE_CHOICE:
+            self.options = None
+
         return super().save(**kwargs)
 
     def _get_post_rel(self):
@@ -244,6 +298,13 @@ def get_global_leaderboard_dates(
         return None
 
 
+QUESTION_CONTINUOUS_TYPES = [
+    Question.QuestionType.NUMERIC,
+    Question.QuestionType.DATE,
+    Question.QuestionType.DISCRETE,
+]
+
+
 class Conditional(TimeStampedModel):
     condition = models.ForeignKey(
         Question, related_name="conditional_conditions", on_delete=models.PROTECT
@@ -326,7 +387,7 @@ class Forecast(models.Model):
     continuous_cdf = ArrayField(
         models.FloatField(),
         null=True,
-        size=CDF_SIZE,
+        max_length=DEFAULT_INBOUND_OUTCOME_COUNT + 1,
     )
     # binary prediction
     probability_yes = models.FloatField(null=True)
@@ -413,7 +474,9 @@ class AggregateForecast(models.Model):
     method = models.CharField(max_length=200, choices=AggregationMethod.choices)
     start_time = models.DateTimeField(db_index=True)
     end_time = models.DateTimeField(null=True, db_index=True)
-    forecast_values = ArrayField(models.FloatField(), max_length=CDF_SIZE)
+    forecast_values = ArrayField(
+        models.FloatField(), max_length=DEFAULT_INBOUND_OUTCOME_COUNT + 1
+    )
     forecaster_count = models.IntegerField(null=True)
     interval_lower_bounds = ArrayField(models.FloatField(), null=True)
     centers = ArrayField(models.FloatField(), null=True)
@@ -440,11 +503,11 @@ def __repr__(self):
         )
 
     def get_cdf(self) -> list[float] | None:
-        if len(self.forecast_values) == CDF_SIZE:
+        if len(self.forecast_values) == DEFAULT_INBOUND_OUTCOME_COUNT + 1:
             return self.forecast_values
 
     def get_pmf(self) -> list[float]:
-        if len(self.forecast_values) == CDF_SIZE:
+        if len(self.forecast_values) == DEFAULT_INBOUND_OUTCOME_COUNT + 1:
             cdf = self.forecast_values
             pmf = [cdf[0]]
             for i in range(1, len(cdf)):
diff --git a/questions/serializers.py b/questions/serializers.py
index a327226ba6..6f5aa04665 100644
--- a/questions/serializers.py
+++ b/questions/serializers.py
@@ -11,6 +11,8 @@
 from questions.constants import ResolutionType
 from questions.models import Forecast
 from questions.models import (
+    DEFAULT_INBOUND_OUTCOME_COUNT,
+    QUESTION_CONTINUOUS_TYPES,
     Question,
     Conditional,
     GroupOfQuestions,
@@ -63,6 +65,7 @@ class Meta:
             "unit",
             "open_upper_bound",
             "open_lower_bound",
+            "inbound_outcome_count",
             "scaling",
             "group_rank",
         )
@@ -117,6 +120,7 @@ class Meta:
             "zero_point",
             "open_upper_bound",
             "open_lower_bound",
+            "inbound_outcome_count",
             "options",
             "group_variable",
             "label",
@@ -287,7 +291,7 @@ class Meta:
 
     def get_quartiles(self, forecast: Forecast):
         question = forecast.question
-        if question.type in [Question.QuestionType.DATE, Question.QuestionType.NUMERIC]:
+        if question.type in QUESTION_CONTINUOUS_TYPES:
             return get_scaled_quartiles_from_cdf(forecast.continuous_cdf, question)
 
     def get_scaling(self, forecast: Forecast):
@@ -511,15 +515,24 @@ def continuous_validation(self, continuous_cdf, question: Question):
             ],
             10,
         )
-        if len(continuous_cdf) != 201:
-            errors += "continuous_cdf must have 201 values.\n"
-        min_diff = 0.01 / 200  # 0.00005
+        if len(continuous_cdf) != question.inbound_outcome_count + 1:
+            errors += (
+                f"continuous_cdf for question {question.id} must "
+                f"have {question.inbound_outcome_count + 1} values.\n"
+            )
+        min_diff = np.round(
+            0.01 / question.inbound_outcome_count, 10
+        )  # 0.00005 by default
         if not all(inbound_pmf >= min_diff):
             errors += (
                 "continuous_cdf must be increasing by at least "
                 f"{min_diff} at every step.\n"
             )
-        max_diff = 0.59  # derived empirically from slider positions
+        # max diff for default CDF is derived empirically from slider positions
+        # TODO: make this lower and scale with inbound_outcome_count
+        max_diff = (
+            0.59 if len(continuous_cdf) == DEFAULT_INBOUND_OUTCOME_COUNT + 1 else 1
+        )
         if not all(inbound_pmf <= max_diff):
             errors += (
                 "continuous_cdf must be increasing by no more than "
@@ -856,7 +869,7 @@ def validate_question_resolution(question: Question, resolution: str) -> str:
             )
         return resolution
 
-    if question.type == Question.QuestionType.NUMERIC:
+    if question.type in [Question.QuestionType.NUMERIC, Question.QuestionType.DISCRETE]:
         resolution = serializers.FloatField().run_validation(resolution)
         range_min = question.range_min
         range_max = question.range_max
diff --git a/questions/services.py b/questions/services.py
index 7290602b2f..5228661e82 100644
--- a/questions/services.py
+++ b/questions/services.py
@@ -16,6 +16,7 @@
 from projects.models import Project
 from questions.constants import ResolutionType
 from questions.models import (
+    QUESTION_CONTINUOUS_TYPES,
     Question,
     GroupOfQuestions,
     Conditional,
@@ -125,7 +126,7 @@ def build_question_forecasts_for_user(
             forecasts_data["medians"].append(0)
         elif question.type == "binary":
             forecasts_data["medians"].append(forecast.probability_yes)
-        elif question.type in ["numeric", "date"]:
+        elif question.type in QUESTION_CONTINUOUS_TYPES:
             forecasts_data["medians"].append(
                 percent_point_function(forecast.continuous_cdf, 50)
             )
@@ -662,7 +663,7 @@ def create_forecast(
         probability_yes=probability_yes,
         probability_yes_per_category=probability_yes_per_category,
         distribution_input=(
-            distribution_input if question.type in ["date", "numeric"] else None
+            distribution_input if question.type in QUESTION_CONTINUOUS_TYPES else None
         ),
         post=post,
         **kwargs,
@@ -835,7 +836,7 @@ def cp_sorting_key(q: Question):
                 return 0
             if q.type == "binary":
                 return agg.forecast_values[1]
-            if q.type in ("numeric", "date"):
+            if q.type in QUESTION_CONTINUOUS_TYPES:
                 return unscaled_location_to_scaled_location(agg.centers[0], q)
             if q.type == "multiple_choice":
                 return max(agg.forecast_values)
diff --git a/scoring/score_math.py b/scoring/score_math.py
index 15f298be81..5188508399 100644
--- a/scoring/score_math.py
+++ b/scoring/score_math.py
@@ -5,7 +5,12 @@
 from django.db.models import Prefetch
 from scipy.stats.mstats import gmean
 
-from questions.models import AggregateForecast, Forecast, Question
+from questions.models import (
+    QUESTION_CONTINUOUS_TYPES,
+    AggregateForecast,
+    Forecast,
+    Question,
+)
 from questions.types import AggregationMethod
 from scoring.models import Score
 from users.models import User
@@ -198,7 +203,7 @@ def evaluate_forecasts_peer_accuracy(
                     * (gm.num_forecasters / (gm.num_forecasters - 1))
                     * np.log(pmf[resolution_bucket] / gm.pmf[resolution_bucket])
                 )
-                if question_type in ["numeric", "date"]:
+                if question_type in QUESTION_CONTINUOUS_TYPES:
                     score /= 2
                 interval_scores.append(score)
             else:
@@ -256,7 +261,7 @@ def evaluate_forecasts_peer_spot_forecast(
                 * (gm.num_forecasters / (gm.num_forecasters - 1))
                 * np.log(pmf[resolution_bucket] / gm.pmf[resolution_bucket])
             )
-            if question_type in ["numeric", "date"]:
+            if question_type in QUESTION_CONTINUOUS_TYPES:
                 forecast_score /= 2
             forecast_scores.append(ForecastScore(forecast_score, 1.0))
         else:
diff --git a/utils/the_math/aggregations.py b/utils/the_math/aggregations.py
index 7069bdf0af..ad9c769cca 100644
--- a/utils/the_math/aggregations.py
+++ b/utils/the_math/aggregations.py
@@ -16,7 +16,12 @@
 import numpy as np
 from django.db.models import Q
 
-from questions.models import Question, Forecast, AggregateForecast
+from questions.models import (
+    QUESTION_CONTINUOUS_TYPES,
+    Question,
+    Forecast,
+    AggregateForecast,
+)
 from questions.types import AggregationMethod
 from scoring.reputation import (
     get_reputations_at_time,
@@ -122,7 +127,7 @@ def calculate_aggregation_entry(
 ) -> AggregateForecast:
     weights = np.array(weights) if weights is not None else None
     if (
-        question_type in ["date", "numeric"]
+        question_type in QUESTION_CONTINUOUS_TYPES
         or method == AggregationMethod.SINGLE_AGGREGATION
     ):
         aggregation = AggregateForecast(
diff --git a/utils/the_math/formulas.py b/utils/the_math/formulas.py
index 50cb2a4cb6..dd0bf1dff0 100644
--- a/utils/the_math/formulas.py
+++ b/utils/the_math/formulas.py
@@ -37,9 +37,9 @@ def string_location_to_scaled_location(
         return question.range_min - 1.0
     if string_location == "above_upper_bound":
         return question.range_max + 1.0
-    if question.type == "date":
+    if question.type == Question.QuestionType.DATE:
         return datetime.fromisoformat(string_location).timestamp()
-    # question.type == "numeric"
+    # question.type in [Question.QuestionType.NUMERIC, Question.QuestionType.DISCRETE]
     return float(string_location)
 
 
@@ -55,9 +55,9 @@ def scaled_location_to_string_location(
         return "below_lower_bound"
     if scaled_location > question.range_max:
         return "above_upper_bound"
-    if question.type == "date":
+    if question.type == Question.QuestionType.DATE:
         return datetime.fromtimestamp(scaled_location, tz=timezone.utc).isoformat()
-    # question.type == "numeric"
+    # question.type in [Question.QuestionType.NUMERIC, Question.QuestionType.DISCRETE]
     return str(scaled_location)
 
 
@@ -121,10 +121,10 @@ def unscaled_location_to_bucket_index(
     if unscaled_location < 0:
         return 0
     if unscaled_location > 1:
-        return 201
+        return question.inbound_outcome_count + 1
     if unscaled_location == 1:
-        return 200
-    return max(int(unscaled_location * 200 + 1 - 1e-10), 1)
+        return question.inbound_outcome_count
+    return max(int(unscaled_location * question.inbound_outcome_count + 1 - 1e-10), 1)
 
 
 def unscaled_location_to_string_location(
@@ -142,9 +142,9 @@ def bucket_index_to_unscaled_location(bucket_index: int, question: "Question") -
     # continuous
     if bucket_index <= 0:
         return -1
-    if bucket_index >= 201:
+    if bucket_index >= question.inbound_outcome_count + 1:
         return 2
-    return bucket_index / 200 - 1 / 400
+    return (bucket_index - 1 / 2) / (question.inbound_outcome_count + 1)
 
 
 def string_location_to_unscaled_location(


Reviews:
user: login: hlbmtc
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T10:51:36Z
user: login: hlbmtc
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T10:58:05Z
user: login: hlbmtc
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T10:59:18Z
user: login: lsabor
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-28T19:38:48Z

Review comments:
diff_hunk: @@ -128,22 +129,54 @@ class QuestionType(models.TextChoices):
         null=True,
         blank=True,
         help_text="""Time when spot scores are evaluated.
-        If not set, defaults to spot_scoring time.""",
+        If not set, defaults to cp reveal time.""",
     )
 
     # continuous range fields
-    range_max = models.FloatField(null=True, blank=True)
-    range_min = models.FloatField(null=True, blank=True)
-    zero_point = models.FloatField(null=True, blank=True)
-    open_upper_bound = models.BooleanField(null=True, blank=True)
-    open_lower_bound = models.BooleanField(null=True, blank=True)
+    range_min = models.FloatField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        Minimum inbound value. For Discrete, this is 1/2 a unit's width below the
+        displayed lower bound.""",
+    )
+    range_max = models.FloatField(
+        null=True,
+        blank=True,
+        help_text="""For Continuous only.
+        Maximum inbound value. For Discrete, this is 1/2 a unit's width above the
+        displayed upper bound.""",
+    )
+    zero_point = models.FloatField(
path: questions/models.py
user: login: hlbmtc
body: Ideally, we could add a low-level primitive validation inside the `save` method to ensure that non-empty fields align with the expected structure for the given question type

created_at: 2025-03-28T10:51:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -256,7 +260,11 @@ def evaluate_forecasts_peer_spot_forecast(
                 * (gm.num_forecasters / (gm.num_forecasters - 1))
                 * np.log(pmf[resolution_bucket] / gm.pmf[resolution_bucket])
             )
-            if question_type in ["numeric", "date"]:
+            if question_type in [
path: scoring/score_math.py
user: login: hlbmtc
body: Hm, I'm thinking it might be more convenient to introduce a separate constant, like `QUESTION_CONTINUOUS_TYPES`, to store this list of three types for better readability and reuse. Wdyt? 
created_at: 2025-03-28T10:58:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -256,7 +260,11 @@ def evaluate_forecasts_peer_spot_forecast(
                 * (gm.num_forecasters / (gm.num_forecasters - 1))
                 * np.log(pmf[resolution_bucket] / gm.pmf[resolution_bucket])
             )
-            if question_type in ["numeric", "date"]:
+            if question_type in [
path: scoring/score_math.py
user: login: lsabor
body: Sure. I'd ideally like for "continuous" to just be the question type and have a subtype field since the only difference is how the units work...
But this is a good solution for the time being.
created_at: 2025-03-28T19:38:48Z
author_association: CONTRIBUTOR

PR comments:
user: login: sentry[bot]
created_at: 2025-03-29T17:17:20Z
body: ## Suspect Issues
This pull request was deployed and Sentry observed the following issues:

- ‼️ **TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'** `/api/questions/forecast/` [View Issue](https://metaculus.sentry.io/issues/6481093596/?referrer=github-pr-bot)
- ‼️ **NameError: name 'Question' is not defined** `resolve_question_and_send_notifications` [View Issue](https://metaculus.sentry.io/issues/6484462289/?referrer=github-pr-bot)
- ‼️ **TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'** `resolve_question_and_send_notifications` [View Issue](https://metaculus.sentry.io/issues/6490269258/?referrer=github-pr-bot)

<sub>Did you find this useful? React with a 👍 or 👎</sub>
author_association: NONE