Repository: tournesol-app/tournesol

Issue:
[back, front] refactor: Harmonize all API and Serializers (then:feat: Rate later list display information about the number of comparisons)
As a user I want to see how many times I have compared the videos when I look at my rate later list so that I can easily chose which videos to remove from my rate later list.

Display the information similarly as in the card used on the comparison page.

This probably will be best implemented by updating the backend endpoint to include the extra info


Discussion:
See the plan described in the PR https://github.com/tournesol-app/tournesol/pull/1055
Here is the JSON the API could return.

The new `poll_metadata` key contains all data related to the poll, like the total number of comparisons by all contributors. It can also be called `collective_metadata` but poll is more accurate as it matches the related back end's model.

The `user_metadata` key contains all data related to the logged user. Having different keys for collective and and personal data will allow the front end to: (1) know what kind of data are displayed ; (2) display both collective and personal data at the same time.

The goal of such representation is to avoid adding unrelated keys at the root of the JSON, making the variable types managed by the front end unnecessarily complex. Here the `entity` object has always the same structure, `poll_metadata` and `user_metadata` too.

```json
{
  "entity": {
    "uid": "string",
    "type": "video",
    "metadata": {
      "additionalProp1": "string",
      "additionalProp2": "string",
      "additionalProp3": "string"
    }
  },
  "poll_metadata": {
    "name": "string",
    "tournesol_score": 0,
    "n_comparisons": 0,
    "n_contributors": 0
  },
  "user_metadata": {
    "n_comparisons": 0,
    "n_contributors": 0
  },
  "rate_later_metadata": {
    "created_at": "2022-11-08T13:47:39.480Z"
  }
}
```

API also returning criteria scores could use the following JSON:

```json
{
  "entity": {
    "uid": "string",
    "type": "video",
    "metadata": {
      "additionalProp1": "string",
      "additionalProp2": "string",
      "additionalProp3": "string"
    }
  },
  "poll_metadata": {
    "name": "string",
    "tournesol_score": 0,
    "n_comparisons": 0,
    "n_contributors": 0,
    "criteria_scores": []
  },
  "user_metadata": {
    "n_comparisons": 0,
    "n_contributors": 0,
    "criteria_scores": []
  }
}
```

By following this rule (no wild key at the root of the JSON object), we can also simplify the `/entities/` API which currently add the `polls` key directly inside the `entity` object.

**current payload**

(the `tournesol_score` is embedded in the `entity` object and the polls don't contain ratings metadata)

```json
{
  "uid": "string",
  "type": "video",
  "metadata": {
    "additionalProp1": "string",
    "additionalProp2": "string",
    "additionalProp3": "string"
  },
  "tournesol_score": 0,
  "polls": [
    {
      "name": "string",
      "criteria_scores": [
        {
          "criteria": "string",
          "score": 0
        }
      ]
    }
  ]
}
```

**suggested payload**

(the `entity` is delimited and compatible with all front end's components using entities as props, the poll metadata are delimited for the same reasons)

```json
{
  "entity": {
    "uid": "string",
    "type": "video",
    "metadata": {
      "additionalProp1": "string",
      "additionalProp2": "string",
      "additionalProp3": "string"
    }
  },
  "polls": [
    {
      "name": "string",
      "tournesol_score": 0,
      "n_comparisons": 0,
      "n_contributors": 0,
      "criteria_scores": [
        {
          "criteria": "string",
          "score": 0
        }
      ]
    }
  ]
}
```

I just remove the labels Good first issue and Hacktoberfest as the proposed rework requires a global view of the current API's architecture.
User metadata is also poll specific. The distinction is rather at the level or global/community/aggregated vs. user/personal ? Not poll vs. user.
Apart from this, I really like the proposal to simplify the keys at the root of the entity JSON representations and allowing API routes to include the root entries or not depending on what is requested. 

I would even go further and impose fixed types for each of the root entries. For example by having one entry `global_metadata` and one `global_criteria_scores` to avoid changing the type of the global_metadata entry
You have `n_contributors` in entry `user_metadata`. Is it a typo or something that I don't understand correctly?
Thanks @GresilleSiffle for the details, I think it's a step in the good direction.

> User metadata is also poll specific. The distinction is rather at the level or global/community/aggregated vs. user/personal ? Not poll vs. user.

I agree. "poll_metadata" and "user_metadata" could bring more confusion. For example, a "tournesol_score" (contrary to "name") is not a property of a `Poll`, it's part of an `EntityPollRating`. And keeping this distinction clear will also make the implementation of the serializers easier.

Also, about naming, I would not use "_metadata" as a suffix for keys. It does not seem like to good description to me. Either "user_metadata" represents the user and could be named "user", or its name should be more explicit.

So based on the first suggestion, I would suggest something like:
```js
{
  "entity": {
    "uid": "string",
    "type": "video",
    "metadata": {
      "additionalProp1": "string",
      "additionalProp2": "string",
      "additionalProp3": "string"
    }
  },
  "poll": {
  // represents an instance of `Poll`
    "name": "string",
  },
  "global": { // or "global_rating", or  "poll_rating" ?
  // represents an instance of `EntityPollRating`, could be extended with `"criteria_scores"` in some contexts
    "tournesol_score": 0,
    "n_comparisons": 0,
    "n_contributors": 0
  },
  "individual": { // or "contributor_rating" ?
   // represents an instance of `ContributorRating`, could be extended with `"criteria_scores"` in some contexts
    "n_comparisons": 0,
  },
  "rate_later_metadata": {
    "created_at": "2022-11-08T13:47:39.480Z"
  }
}
```

Thank you!

You're right, using the suffix `_metadata` is not that good.

I like `collective_rating` instead of `global` / `global_rating` / `poll_rating`. The `_rating` might disappear the day we will save more information than just ratings, but keeping it right now make it more explicit tha simply `collective` or `global`.

If `individual` represents the contributor rating, maybe we can use the name `contributor_rating` directly. We can try to re-use  the existing `ContributorRatingSerializer` or a similar read-only oriented serializer. 

**edit:** finally i prefer `invididual_rating` rather than `contributor_rating` as the JSON representation differs enough from the `ContributorRatingSerializer` (not entity field, no poll field, no last_compared_at field, etc.)

Based on the last suggestion, what do you think of this?

```json
{
  "entity": {
    "uid": "string",
    "type": "video",
    "metadata": {
      "additionalProp1": "string",
      "additionalProp2": "string",
      "additionalProp3": "string"
    }
  },
  "poll": {
    "name": "string",
  },
  "invididual_rating": {
    "is_public": true,
    "n_comparisons": 0
  },
  "collective_rating": {
    "tournesol_score": 0,
    "n_comparisons": 0,
    "n_contributors": 0
  },
  "rate_later_metadata": {
    "created_at": "2022-11-08T13:47:39.480Z"
  }
}
```
It looks like we will be able to re-use this new structure in the contributor ratings API. The only difference is the field `rate_later_metadata` which transforms into a `contributor_rating_metadata` to provide new fields like `last_compared_at`.
Merging this PR will help to address the current issue: https://github.com/tournesol-app/tournesol/pull/1149/files

The PR will ensure the models `EntityPollRating`s contain data the can be returned in the `collective_rating` field of the rate-later JSON. It's not directly related to the issue, but it will avoid doing two refactor in a row of the same serializer.

Resolving patch:
diff --git a/backend/tournesol/serializers/contributor_recommendations.py b/backend/tournesol/serializers/contributor_recommendations.py
--- a/backend/tournesol/serializers/contributor_recommendations.py
+++ b/backend/tournesol/serializers/contributor_recommendations.py
@@ -1,6 +1,3 @@
-from drf_spectacular.utils import extend_schema_field, extend_schema_serializer
-from rest_framework.serializers import SerializerMethodField
-
 from tournesol.models.ratings import ContributorRating
 from tournesol.serializers.criteria_score import ContributorCriteriaScoreSerializer
 from tournesol.serializers.poll import IndividualRatingSerializer, RecommendationSerializer
@@ -15,41 +12,14 @@ class Meta:
         read_only_fields = fields
 
 
-@extend_schema_serializer(
-    exclude_fields=[
-        # legacy fields have been moved to "entity", "invidual_rating", "collective_rating", etc.
-        "uid",
-        "type",
-        "n_comparisons",
-        "n_contributors",
-        "metadata",
-        "total_score",
-        "tournesol_score",
-        "criteria_scores",
-        "unsafe",
-        "is_public",
-    ]
-)
 class ContributorRecommendationsSerializer(RecommendationSerializer):
     """
     An entity recommended by a user.
     """
-
-    is_public = SerializerMethodField()
-    criteria_scores = SerializerMethodField()
     individual_rating = IndividualRatingWithScoresSerializer(
         source="single_contributor_rating",
         read_only=True,
     )
 
     class Meta(RecommendationSerializer.Meta):
-        fields = RecommendationSerializer.Meta.fields + ["is_public", "individual_rating"]
-
-    @extend_schema_field(ContributorCriteriaScoreSerializer(many=True))
-    def get_criteria_scores(self, obj):
-        return ContributorCriteriaScoreSerializer(
-            obj.single_contributor_rating.criteria_scores, many=True
-        ).data
-
-    def get_is_public(self, obj) -> bool:
-        return obj.single_contributor_rating.is_public
+        fields = RecommendationSerializer.Meta.fields + ["individual_rating"]
diff --git a/backend/tournesol/serializers/poll.py b/backend/tournesol/serializers/poll.py
--- a/backend/tournesol/serializers/poll.py
+++ b/backend/tournesol/serializers/poll.py
@@ -1,4 +1,3 @@
-from drf_spectacular.utils import extend_schema_serializer
 from rest_framework import serializers
 from rest_framework.serializers import IntegerField, ModelSerializer
 
@@ -79,32 +78,7 @@ class RecommendationMetadataSerializer(serializers.Serializer):
     total_score = serializers.FloatField(read_only=True, allow_null=True)
 
 
-@extend_schema_serializer(
-    exclude_fields=[
-        # legacy fields have been moved to "entity", "collective_rating", etc.
-        "uid",
-        "type",
-        "n_comparisons",
-        "n_contributors",
-        "metadata",
-        "total_score",
-        "tournesol_score",
-        "criteria_scores",
-        "unsafe",
-    ]
-)
 class RecommendationSerializer(ModelSerializer):
-    # pylint: disable=duplicate-code
-    n_comparisons = serializers.IntegerField(source="rating_n_ratings")
-    n_contributors = serializers.IntegerField(source="rating_n_contributors")
-    criteria_scores = EntityCriteriaScoreSerializer(many=True)
-    # TODO: the field total_score is the only field in this serializer that
-    # on the parameters of an api request. Should it be treated differently?
-    total_score = serializers.FloatField()
-    unsafe = UnsafeStatusSerializer(
-        source="single_poll_rating", allow_null=True, default=None, read_only=True
-    )
-
     entity = RelatedEntitySerializer(source="*", read_only=True)
     collective_rating = ExtendedCollectiveRatingSerializer(
         source="single_poll_rating",
@@ -121,15 +95,6 @@ class RecommendationSerializer(ModelSerializer):
     class Meta:
         model = Entity
         fields = [
-            "uid",
-            "type",
-            "n_comparisons",
-            "n_contributors",
-            "metadata",
-            "total_score",
-            "tournesol_score",
-            "criteria_scores",
-            "unsafe",
             "entity",
             "collective_rating",
             "entity_contexts",


Test patch:
diff --git a/backend/tournesol/tests/test_api_contributor_recommendations.py b/backend/tournesol/tests/test_api_contributor_recommendations.py
--- a/backend/tournesol/tests/test_api_contributor_recommendations.py
+++ b/backend/tournesol/tests/test_api_contributor_recommendations.py
@@ -112,7 +112,7 @@ def test_recommendations_privacy_auth(self):
 
         results = response.data["results"]
         for entity in results:
-            self.assertEqual(entity["is_public"], True)
+            self.assertEqual(entity["individual_rating"]["is_public"], True)
 
         # The collective metadata `n_comparisons` and `n_contributors` must
         # be present in the response of the public personal reco. endpoint.
diff --git a/backend/tournesol/tests/test_api_polls.py b/backend/tournesol/tests/test_api_polls.py
--- a/backend/tournesol/tests/test_api_polls.py
+++ b/backend/tournesol/tests/test_api_polls.py
@@ -135,13 +135,6 @@ def test_anon_can_list_recommendations(self):
         results = response.data["results"]
         self.assertEqual(len(results), 3)
 
-        # Legacy fields
-        self.assertEqual(results[0]["tournesol_score"], 44.0)
-        self.assertEqual(results[1]["tournesol_score"], 33.0)
-        self.assertEqual(results[2]["tournesol_score"], 22.0)
-        self.assertEqual(results[0]["type"], "video")
-
-        # New fields
         self.assertEqual(results[0]["collective_rating"]["tournesol_score"], 44.0)
         self.assertEqual(results[1]["collective_rating"]["tournesol_score"], 33.0)
         self.assertEqual(results[2]["collective_rating"]["tournesol_score"], 22.0)
@@ -297,13 +290,12 @@ def test_anon_can_list_reco_with_contexts_unsafe(self):
         results = response.data["results"]
 
         self.assertEqual(len(results), initial_safe_results_nbr - 1)
-        for result in results:
-            self.assertNotEqual(result["uid"], self.video_4.uid)
+        self.assertNotIn(self.video_4.uid, [res["entity"]["uid"] for res in results])
 
         response = self.client.get("/polls/videos/recommendations/?unsafe=true")
         vid4 = response.data["results"][0]
 
-        self.assertEqual(vid4["uid"], self.video_4.uid)
+        self.assertEqual(vid4["entity"]["uid"], self.video_4.uid)
         self.assertEqual(vid4["collective_rating"]["unsafe"]["status"], True)
         self.assertEqual(len(vid4["collective_rating"]["unsafe"]["reasons"]), 1)
         self.assertEqual(
@@ -318,7 +310,7 @@ def test_anon_can_list_reco_with_contexts_poll_specific(self):
         """
         response = self.client.get("/polls/videos/recommendations/")
         initial_safe_results_nbr = len(response.data["results"])
-        self.assertEqual(response.data["results"][0]["uid"], self.video_4.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], self.video_4.uid)
 
         other_poll = Poll.objects.create(name="other")
         EntityContext.objects.create(
@@ -344,7 +336,7 @@ def test_anon_can_list_reco_with_contexts_poll_specific(self):
         results = response.data["results"]
 
         self.assertEqual(len(results), initial_safe_results_nbr)
-        self.assertEqual(response.data["results"][0]["uid"], self.video_4.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], self.video_4.uid)
         self.assertEqual(response.data["results"][0]["entity_contexts"], [])
 
     def test_ignore_score_attached_to_another_poll(self):
@@ -434,7 +426,7 @@ def test_anon_can_list_videos_filtered_by_metadata_single(self):
         )
         self.assertEqual(resp.status_code, status.HTTP_200_OK)
         self.assertEqual(
-            [video["uid"] for video in resp.data["results"]],
+            [video["entity"]["uid"] for video in resp.data["results"]],
             [self.video_3.uid, self.video_2.uid],
         )
 
@@ -483,7 +475,7 @@ def test_anon_can_list_videos_filtered_by_metadata_mixed(self):
         )
         self.assertEqual(resp.status_code, status.HTTP_200_OK)
         self.assertEqual(
-            [video["uid"] for video in resp.data["results"]],
+            [video["entity"]["uid"] for video in resp.data["results"]],
             [self.video_3.uid, self.video_2.uid],
         )
 
@@ -535,7 +527,7 @@ def test_anon_can_list_videos_filtered_by_duration_lt(self):
 
         results = response.data["results"]
         self.assertEqual(len(results), 1)
-        self.assertEqual(results[0]["metadata"]["duration"], 10)
+        self.assertEqual(results[0]["entity"]["metadata"]["duration"], 10)
 
         response = self.client.get(
             "/polls/videos/recommendations/?metadata[duration:lte:int]=120"
@@ -766,18 +758,6 @@ def test_auth_can_read(self):
         data = response.data
         self.assertEqual(response.status_code, 200)
 
-        # Legacy fields
-        self.assertEqual(data["tournesol_score"], -2)
-        self.assertEqual(data["n_comparisons"], 8)
-        self.assertEqual(data["n_contributors"], 4)
-        self.assertEqual(data["unsafe"], {
-            "status": True,
-            "reasons": ["insufficient_tournesol_score"],
-        })
-        self.assertIn("total_score", data)
-        self.assertIn("criteria_scores", data)
-
-        # New fields
         self.assertEqual(data["collective_rating"]["tournesol_score"], -2)
         self.assertEqual(data["collective_rating"]["n_comparisons"], 8)
         self.assertEqual(data["collective_rating"]["n_contributors"], 4)
@@ -794,18 +774,6 @@ def test_anon_can_read(self):
         data = response.data
         self.assertEqual(response.status_code, 200)
 
-        # Legacy fields
-        self.assertEqual(data["tournesol_score"], -2)
-        self.assertEqual(data["n_comparisons"], 8)
-        self.assertEqual(data["n_contributors"], 4)
-        self.assertEqual(data["unsafe"], {
-            "status": True,
-            "reasons": ["insufficient_tournesol_score"],
-        })
-        self.assertIn("total_score", data)
-        self.assertIn("criteria_scores", data)
-
-        # New fields
         self.assertEqual(data["collective_rating"]["tournesol_score"], -2)
         self.assertEqual(data["collective_rating"]["n_comparisons"], 8)
         self.assertEqual(data["collective_rating"]["n_contributors"], 4)
@@ -1027,14 +995,6 @@ def test_low_sum_trust_scores_excluded_from_recommendations(self):
         results = response.data["results"]
         self.assertEqual(len(results), 1)
 
-        # Legacy fields
-        self.assertEqual(results[0]["uid"], self.video_2.uid)
-        self.assertEqual(results[0]["unsafe"], {
-            "status": False,
-            "reasons": []
-        })
-
-        # New fields
         self.assertEqual(
             results[0]["entity"]["uid"],
             self.video_2.uid,
diff --git a/backend/tournesol/tests/test_text_search.py b/backend/tournesol/tests/test_text_search.py
--- a/backend/tournesol/tests/test_text_search.py
+++ b/backend/tournesol/tests/test_text_search.py
@@ -414,9 +414,10 @@ def test_sorting_depends_on_total_score(self):
         self.assertEqual(len(response.data["results"]), response.data["count"])
 
         sorted_scores = sorted(scores, reverse=True)
+        results = response.data["results"]
         for index, score in enumerate(sorted_scores):
             entity = entities[scores.index(score)]
-            self.assertEqual(response.data["results"][index]["uid"], entity.uid)
+            self.assertEqual(results[index]["entity"]["uid"], entity.uid)
 
     def test_public_contributor_recommendations_sorting_depends_on_total_score(self):
         """
@@ -442,9 +443,10 @@ def test_public_contributor_recommendations_sorting_depends_on_total_score(self)
         self.assertEqual(len(response.data["results"]), response.data["count"])
 
         sorted_scores = sorted(scores, reverse=True)
+        results = response.data["results"]
         for index, score in enumerate(sorted_scores):
             entity = entities[scores.index(score)]
-            self.assertEqual(response.data["results"][index]["uid"], entity.uid)
+            self.assertEqual(results[index]["entity"]["uid"], entity.uid)
 
     def test_private_contributor_recommendations_sorting_depends_on_total_score(self):
         """
@@ -471,9 +473,10 @@ def test_private_contributor_recommendations_sorting_depends_on_total_score(self
         self.assertEqual(len(response.data["results"]), response.data["count"])
 
         sorted_scores = sorted(scores, reverse=True)
+        results = response.data["results"]
         for index, score in enumerate(sorted_scores):
             entity = entities[scores.index(score)]
-            self.assertEqual(response.data["results"][index]["uid"], entity.uid)
+            self.assertEqual(results[index]["entity"]["uid"], entity.uid)
 
     def test_videos_fields_weights(self):
         """
@@ -503,8 +506,8 @@ def test_videos_fields_weights(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], self.setup_entity.uid)
-        self.assertEqual(response.data["results"][1]["uid"], other_entity.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], self.setup_entity.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], other_entity.uid)
 
     def test_query_weights(self):
         """
@@ -542,8 +545,8 @@ def test_query_weights(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], entity_2.uid)
-        self.assertEqual(response.data["results"][1]["uid"], entity_1.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], entity_2.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], entity_1.uid)
 
         response = self.client.get(
             self._make_url(query) + f"&weights[{criteria_1}]=10&weights[{criteria_2}]=50",
@@ -552,8 +555,8 @@ def test_query_weights(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], entity_1.uid)
-        self.assertEqual(response.data["results"][1]["uid"], entity_2.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], entity_1.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], entity_2.uid)
 
     def test_multiple_matches(self):
         """
@@ -580,8 +583,8 @@ def test_multiple_matches(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], relevant_entity.uid)
-        self.assertEqual(response.data["results"][1]["uid"], less_relevant_entity.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], relevant_entity.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], less_relevant_entity.uid)
 
     def test_public_contributor_recommendations_multiple_matches(self):
         """
@@ -609,8 +612,8 @@ def test_public_contributor_recommendations_multiple_matches(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], relevant_entity.uid)
-        self.assertEqual(response.data["results"][1]["uid"], less_relevant_entity.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], relevant_entity.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], less_relevant_entity.uid)
 
     def test_private_contributor_recommendations_multiple_matches(self):
         """
@@ -639,8 +642,8 @@ def test_private_contributor_recommendations_multiple_matches(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 2)
-        self.assertEqual(response.data["results"][0]["uid"], relevant_entity.uid)
-        self.assertEqual(response.data["results"][1]["uid"], less_relevant_entity.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], relevant_entity.uid)
+        self.assertEqual(response.data["results"][1]["entity"]["uid"], less_relevant_entity.uid)
 
     def test_multiple_words(self):
         """
@@ -666,7 +669,7 @@ def test_multiple_words(self):
 
         self.assertEqual(response.status_code, status.HTTP_200_OK)
         self.assertEqual(response.data["count"], 1)
-        self.assertEqual(response.data["results"][0]["uid"], relevant_entity.uid)
+        self.assertEqual(response.data["results"][0]["entity"]["uid"], relevant_entity.uid)
 
     def test_case_insensitive(self):