Repository: liam-hq/liam
Pull request #1241: 🔧 Fixed review scores to be calculated by system from prompts
Author: login: FunamaYukina
Labels: name: Review effort 3/5
Created: 2025-04-08T09:14:53Z | Merged: 2025-04-08T12:04:02Z

## Why is this change needed?
<!-- Please explain briefly why this change is necessary -->

In the current review prompt, there were cases where the review scores and the feedback (review issues) were inconsistent.
In this pull request, I’ve modified the process so that the score is no longer calculated by the LLM, but instead computed separately by the system.
Critical issues are counted as -3 points, and Warnings as -1 point.

## What would you like reviewers to focus on?
<!-- What specific aspects are you requesting review for? -->

## Testing Verification
<!-- Please describe how you verified these changes in your local environment using text/images/video -->

✅ Preview branch check
Scores were calculated and stored correctly.
https://supabase.com/dashboard/project/lfreuhpknrzwtphomppx/editor/29930

Category | Overall Score | Severity
-- | -- | --
MIGRATION_SAFETY | 7 | CRITICAL
DATA_INTEGRITY | 9 | WARNING
PERFORMANCE_IMPACT | 10 | POSITIVE
PROJECT_RULES_CONSISTENCY | 10 | POSITIVE
SECURITY_OR_SCALABILITY | 9 | WARNING

Testing of the langfuse playground also confirmed that multiple feedbacks are returned for a single category.

```
    {
      "kind": "Data Integrity",
      "severity": "CRITICAL",
      "description": "The repositoryName column is redundant and can lead to inconsistencies.",
      "suggestion": "Remove the repositoryName column from the User table.",
      "suggestionSnippets": [
        {
          "filename": "frontend/packages/db/schema/schema.sql",
          "snippet": "ALTER TABLE \"public\".\"User\" DROP COLUMN \"repositoryName\";"
        }
      ]
    },
    {
      "kind": "Data Integrity",
      "severity": "WARNING",
      "description": "The email column lacks a UNIQUE constraint, allowing duplicate entries.",
      "suggestion": "Add a UNIQUE constraint on the email column.",
      "suggestionSnippets": [
        {
          "filename": "frontend/packages/db/schema/schema.sql",
          "snippet": "ALTER TABLE ONLY \"public\".\"User\" ADD CONSTRAINT \"User_email_key\" UNIQUE (\"email\");"
        }
      ]
    },
```
## What was done
<!-- This section will be filled by PR-Agent when the Pull Request is opened -->

### 🤖 Generated by PR Agent at 313093937a98ce564f3c84d33ba20837d09162e2

- Replaced LLM-based score calculation with system-based computation.
- Introduced severity-based score deductions for review feedback.
- Enhanced feedback grouping and handling for better categorization.
- Updated prompt and schema to align with new scoring logic.


## Detailed Changes
<!-- This section will be filled by PR-Agent when the Pull Request is opened -->

<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>processSaveReview.ts</strong><dd><code>Implemented system-based score computation with severity deductions</code></dd></summary>
<hr>

frontend/packages/jobs/src/functions/processSaveReview.ts

<li>Replaced LLM-based score calculation with system-based computation.<br> <li> Added severity-based deductions for feedback scores.<br> <li> Improved feedback grouping by category for score calculation.<br> <li> Added fallback for unknown categories in mapping logic.


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1241/files#diff-79d3de5007a5d073d01c220caa60a46b00eedc32994413db8fc47c213e72e459">+61/-9</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>reviewSchema.ts</strong><dd><code>Exported SeverityEnum and updated schema for scoring</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/packages/jobs/src/prompts/generateReview/reviewSchema.ts

<li>Exported <code>SeverityEnum</code> for use in score computation logic.<br> <li> Adjusted schema to align with new scoring and feedback requirements.


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1241/files#diff-67f415413057cfc6db882bf55d5978df834ba9ffe9a6faa798ac62e45813d6a8">+1/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Documentation</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>generateReview.ts</strong><dd><code>Updated prompt for system-based scoring and feedback consistency</code></dd></summary>
<hr>

frontend/packages/jobs/src/prompts/generateReview/generateReview.ts

<li>Updated prompt to reflect system-based score computation.<br> <li> Removed LLM-based score deduction logic from the prompt.<br> <li> Added requirements for feedback consistency across categories.<br> <li> Enhanced feedback severity and suggestion guidelines.


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1241/files#diff-bc327c5835be24c334bb5b64878901f0a7a7f7aab54d96897cb8f0a326d9aa6b">+22/-18</a>&nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

## Additional Notes
<!-- Any additional information for reviewers -->

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about Qodo Merge usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>


Diff:
diff --git a/frontend/packages/jobs/src/functions/processSaveReview.ts b/frontend/packages/jobs/src/functions/processSaveReview.ts
index 489ab1f63b..21f3fa64db 100644
--- a/frontend/packages/jobs/src/functions/processSaveReview.ts
+++ b/frontend/packages/jobs/src/functions/processSaveReview.ts
@@ -1,6 +1,10 @@
 import { createClient } from '../libs/supabase'
+import { SeverityEnum } from '../prompts/generateReview/reviewSchema'
 import type { ReviewResponse } from '../types'
 
+// Use SeverityEnum values type
+type Severity = keyof typeof SeverityEnum.enum
+
 export const processSaveReview = async (
   payload: ReviewResponse,
 ): Promise<{ success: boolean }> => {
@@ -40,14 +44,57 @@ export const processSaveReview = async (
       )
     }
 
-    // create review scores
-    const reviewScores = payload.review.scores.map((score) => ({
-      overallReviewId: overallReview.id,
-      overallScore: score.value,
-      category: mapCategoryEnum(score.kind),
-      reason: score.reason,
-      updatedAt: now,
-    }))
+    // Calculate scores based on feedback severity
+    // Group feedbacks by category
+    const feedbacksByCategory: Record<string, typeof payload.review.feedbacks> =
+      {}
+
+    // Use for...of instead of forEach
+    for (const feedback of payload.review.feedbacks) {
+      const kind = feedback.kind || 'Unknown'
+      if (!feedbacksByCategory[kind]) {
+        feedbacksByCategory[kind] = []
+      }
+      feedbacksByCategory[kind].push(feedback)
+    }
+
+    // Create review scores
+    const reviewScores = Object.entries(feedbacksByCategory).map(
+      ([category, feedbacks]) => {
+        // Start with 10 points
+        let score = 10
+
+        // Calculate deductions based on feedback severity
+        // Use for...of instead of forEach
+        for (const feedback of feedbacks) {
+          const severity = feedback.severity as Severity
+          if (severity === SeverityEnum.enum.CRITICAL) {
+            score -= 3
+          } else if (severity === SeverityEnum.enum.WARNING) {
+            score -= 1
+          }
+          // No deduction for POSITIVE feedback (no points subtracted)
+        }
+
+        // Ensure minimum score is 0
+        score = Math.max(0, score)
+
+        // Find the reason from the original scores if available
+        const originalScore = payload.review.scores?.find(
+          (s) => s.kind === category,
+        )
+        const reason =
+          originalScore?.reason || 'Score calculated based on feedback severity'
+
+        return {
+          overallReviewId: overallReview.id,
+          overallScore: score,
+          category: mapCategoryEnum(category),
+          reason,
+          updatedAt: now,
+        }
+      },
+    )
 
     const { error: reviewScoresError } = await supabase
       .from('ReviewScore')
@@ -149,10 +196,15 @@ const mapCategoryEnum = (
     'Performance Impact': 'PERFORMANCE_IMPACT',
     'Project Rules Consistency': 'PROJECT_RULES_CONSISTENCY',
     'Security or Scalability': 'SECURITY_OR_SCALABILITY',
+    // Add fallback for unknown category
+    Unknown: 'MIGRATION_SAFETY', // Default to MIGRATION_SAFETY for unknown categories
   }
   const result = mapping[category]
   if (!result) {
-    throw new Error(`Invalid category: ${category}`)
+    console.warn(
+      `Unknown category: ${category}, defaulting to MIGRATION_SAFETY`,
+    )
+    return 'MIGRATION_SAFETY'
   }
   return result
 }
diff --git a/frontend/packages/jobs/src/prompts/generateReview/generateReview.ts b/frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
index ceb18a0160..9becf7ed28 100644
--- a/frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
+++ b/frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
@@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
+  - "reason": An explanation justifying the score provided, including an overview of identified issues.
+
+- Based on the findings, create an array of identified feedback in the "feedbacks" field. Each feedback item must include:
   - "kind": The feedback category, matching one of the categories used in the scores.
-  - "severity": For each feedback item, assign a severity value:
-    - Use "CRITICAL" for issues that caused major point deductions (2-4 points)
-    - Use "WARNING" for issues that caused minor point deductions (1-2 points)
-    - Use "POSITIVE" to highlight improvements, best practices, or beneficial changes in the schema design. When assigning a score of 10 in any category, at least one POSITIVE feedback must be included to justify the score.
-  - "description": A clear and precise explanation of the feedback. If the severity is POSITIVE, describe what is improved and why it is beneficial.
+  - "severity": Assign a severity value:
+    - Use "CRITICAL" for major issues.
+    - Use "WARNING" for minor issues.
+    - Use "POSITIVE" to highlight improvements in the schema design.
+  - IMPORTANT FEEDBACK REQUIREMENTS:
+    1. For each category, you MUST include AT LEAST one feedback item.
+    2. For any category, include at least one feedback item with severity "WARNING" or "CRITICAL" for any identified issue.
+    3. For any category, include at least one feedback item with severity "POSITIVE" to highlight beneficial changes.
+  - "description": A clear and precise explanation of the feedback.
   - "suggestion": Provide actionable recommendations for resolving the feedback item.
     - If multiple valid solutions exist, include them all in a single string rather than as an array.
       - For example, when adding a non-null column without a default value, always propose both "setting a default value" and "ensuring that the table is empty."
@@ -51,6 +48,7 @@ Your JSON-formatted response must contain:
     - "filename": The filename of the file that needs to be applied.
     - "snippet": The snippet of the file that needs to be applied.
       - For example, if DEFAULT value is needed for a column, the snippet should include the statement with the DEFAULT value.
+
 - A concise overall review in the "bodyMarkdown" field that follows these rules:
   - IMPORTANT:
     - Write no more than 3 sentences and approximately 70 words in total
@@ -71,8 +69,14 @@ Evaluation Criteria Details:
   - **Scalability:** Evaluates the potential for performance degradation due to large-scale data processing, query delays, transaction conflicts, database locks, etc., as the system expands. This is a general perspective for evaluation.
 - **Project Rules Consistency:** This evaluation item represents project-specific requirements. Checks whether schema changes comply with project documents or existing schema rules (e.g., use of specific prefixes, naming conventions, etc.). If project-specific rules are not provided, this evaluation may be omitted.
 
-Ensure your response strictly adheres to the provided JSON schema.
-**Your output must be raw JSON only. Do not include any markdown code blocks or extraneous formatting.**
+Before finalizing your response, perform these self-checks:
+
+**Before finalizing your response, perform these self-checks:**
+1. Ensure that each category has at least one feedback item in "feedbacks".
+2. For every identified issue, ensure there is a corresponding feedback item with the correct severity.
+3. Ensure that there is at least one "POSITIVE" feedback item in "feedbacks" across all categories.
+
+**Your output must be raw JSON only. Do not include any markdown code blocks or extraneous formatting.****
 `
 
 export const USER_PROMPT = `Pull Request Description:
diff --git a/frontend/packages/jobs/src/prompts/generateReview/reviewSchema.ts b/frontend/packages/jobs/src/prompts/generateReview/reviewSchema.ts
index 50361af2a3..e89ac466a9 100644
--- a/frontend/packages/jobs/src/prompts/generateReview/reviewSchema.ts
+++ b/frontend/packages/jobs/src/prompts/generateReview/reviewSchema.ts
@@ -8,7 +8,7 @@ const KindEnum = enumType({
   'Security or Scalability': 'Security or Scalability',
 })
 
-const SeverityEnum = enumType({
+export const SeverityEnum = enumType({
   CRITICAL: 'CRITICAL',
   WARNING: 'WARNING',
   POSITIVE: 'POSITIVE',


Reviews:
user: login: MH4GF
body: Thanks!!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:38:26Z
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T11:44:14Z
user: login: MH4GF
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:45:46Z
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T11:49:47Z

Review comments:
diff_hunk: @@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
path: frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
user: login: MH4GF
body: If this is the notation, shouldn't it be erased?

```suggestion
```
created_at: 2025-04-08T11:34:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
+  - "reason": An explanation justifying the score provided, including an overview of identified issues. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
+
+- Based on the findings, create an array of identified feedback in the "feedbacks" field. Each feedback item must include:
   - "kind": The feedback category, matching one of the categories used in the scores.
-  - "severity": For each feedback item, assign a severity value:
-    - Use "CRITICAL" for issues that caused major point deductions (2-4 points)
-    - Use "WARNING" for issues that caused minor point deductions (1-2 points)
-    - Use "POSITIVE" to highlight improvements, best practices, or beneficial changes in the schema design. When assigning a score of 10 in any category, at least one POSITIVE feedback must be included to justify the score.
-  - "description": A clear and precise explanation of the feedback. If the severity is POSITIVE, describe what is improved and why it is beneficial.
+  - "severity": Assign a severity value:
+    - Use "CRITICAL" for major issues.
+    - Use "WARNING" for minor issues.
+    - Use "POSITIVE" to highlight improvements in the schema design.
path: frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
user: login: MH4GF
body: 💭 With this notation, it looked like `POSITIVE` would be used if there was a problem, but I think it's ok if it's tested and ok.
created_at: 2025-04-08T11:38:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
+  - "reason": An explanation justifying the score provided, including an overview of identified issues. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
+
+- Based on the findings, create an array of identified feedback in the "feedbacks" field. Each feedback item must include:
   - "kind": The feedback category, matching one of the categories used in the scores.
-  - "severity": For each feedback item, assign a severity value:
-    - Use "CRITICAL" for issues that caused major point deductions (2-4 points)
-    - Use "WARNING" for issues that caused minor point deductions (1-2 points)
-    - Use "POSITIVE" to highlight improvements, best practices, or beneficial changes in the schema design. When assigning a score of 10 in any category, at least one POSITIVE feedback must be included to justify the score.
-  - "description": A clear and precise explanation of the feedback. If the severity is POSITIVE, describe what is improved and why it is beneficial.
+  - "severity": Assign a severity value:
+    - Use "CRITICAL" for major issues.
+    - Use "WARNING" for minor issues.
+    - Use "POSITIVE" to highlight improvements in the schema design.
path: frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
user: login: FunamaYukina
body: The langfuse test returned feedback as intended.
```json
    {
      "kind": "Performance Impact",
      "severity": "POSITIVE",
      "description": "The introduction of sequences for ID management is a good practice for performance.",
      "suggestion": "No suggestions needed."
    },
```

If the output is not an improved point but a point that needs to be corrected, I will try to correct it!🔧
created_at: 2025-04-08T11:44:14Z
author_association: MEMBER
diff_hunk: @@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
+  - "reason": An explanation justifying the score provided, including an overview of identified issues. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
+
+- Based on the findings, create an array of identified feedback in the "feedbacks" field. Each feedback item must include:
   - "kind": The feedback category, matching one of the categories used in the scores.
-  - "severity": For each feedback item, assign a severity value:
-    - Use "CRITICAL" for issues that caused major point deductions (2-4 points)
-    - Use "WARNING" for issues that caused minor point deductions (1-2 points)
-    - Use "POSITIVE" to highlight improvements, best practices, or beneficial changes in the schema design. When assigning a score of 10 in any category, at least one POSITIVE feedback must be included to justify the score.
-  - "description": A clear and precise explanation of the feedback. If the severity is POSITIVE, describe what is improved and why it is beneficial.
+  - "severity": Assign a severity value:
+    - Use "CRITICAL" for major issues.
+    - Use "WARNING" for minor issues.
+    - Use "POSITIVE" to highlight improvements in the schema design.
path: frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
user: login: MH4GF
body: Thanks for checking!! I understand.
created_at: 2025-04-08T11:45:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,23 +24,20 @@ Your JSON-formatted response must contain:
     - Performance Impact
     - Project Rules Consistency
     - Security or Scalability
-  - "value": A numeric score from 0 to 10, using a deduction-based approach:
-    - Start with a perfect score of 10 (indicating no issues)
-    - Deduct points based on the severity and number of issues identified in each category
-    - Consider the cumulative impact of multiple issues
-    - A score of 0-3 indicates critical, unfixable problems
-    - A score of 4-6 indicates significant issues requiring attention
-    - A score of 7-9 indicates minor issues or concerns
-    - A score of 10 indicates perfect design with no issues
-  - "reason": An explanation justifying the score provided, including what deductions were made and why. If no issues were found in a category, explicitly state that a score of 10 was given because no issues were identified.
-
-- Based on the scores you've assigned, create an array of identified feedback in the "feedbacks" field, each including:
+  - "value": This field will be calculated by the system. Only provide the "kind" and "reason" fields.
path: frontend/packages/jobs/src/prompts/generateReview/generateReview.ts
user: login: FunamaYukina
body: You are right! I removed the unnecessary parts!🙏
https://github.com/liam-hq/liam/pull/1241/commits/8031eacb2e9eb6115a443c5b56c6e66d6a6477aa
I tested it with langfuse and it was fine.✅

```
{
  "scores": [
    {
      "kind": "Migration Safety",
      "reason": "The migration introduces a User table but lacks explicit transaction handling to ensure atomicity. This could risk partial updates if the migration is interrupted."
    },
```
created_at: 2025-04-08T11:49:47Z
author_association: MEMBER

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-04-08T09:14:57Z
body: ###  ⚠️  No Changeset found

Latest commit: 8031eacb2e9eb6115a443c5b56c6e66d6a6477aa

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. **If these changes should result in a version bump, you need to add a changeset.**

<details><summary>This PR includes no changesets</summary>

  When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

</details>

[Click here to learn what changesets are, and how to add one](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add a changeset to this PR](https://github.com/liam-hq/liam/new/update-review-prompt-to-output-properly?filename=.changeset/nine-flowers-kiss.md&value=---%0A%22%40liam-hq%2Fjobs%22%3A%20patch%0A---%0A%0A%F0%9F%94%A7%20Fixed%20review%20scores%20to%20be%20calculated%20by%20system%20from%20prompts%0A)


author_association: NONE
user: login: vercel[bot]
created_at: 2025-04-08T09:15:03Z
body: [vc]: #Xh8dqZYkiSC6br3P5XkIimkcBWbvnywpDioZCD1JaFA=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJsaWFtLWRvY3MiLCJyb290RGlyZWN0b3J5IjoiZnJvbnRlbmQvYXBwcy9kb2NzIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL3JvdXRlLTA2LWNvcmUvbGlhbS1kb2NzL2R6Y01YVlM1ajNXczl4S2VMVExqeTlkaVRXVkUiLCJwcmV2aWV3VXJsIjoibGlhbS1kb2NzLWdpdC11cGRhdGUtcmV2aWV3LXByb21wdC10by1vdXRwLWFhMjQ3Yi1yb3V0ZS0wNi1jb3JlLnZlcmNlbC5hcHAiLCJuZXh0Q29tbWl0U3RhdHVzIjoiSUdOT1JFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiIifX0seyJuYW1lIjoibGlhbS1lcmQtc2FtcGxlIiwicm9vdERpcmVjdG9yeSI6ImZyb250ZW5kL2FwcHMvZXJkLXNhbXBsZSIsImluc3BlY3RvclVybCI6Imh0dHBzOi8vdmVyY2VsLmNvbS9yb3V0ZS0wNi1jb3JlL2xpYW0tZXJkLXNhbXBsZS9DNG05Uk1YTHR2dXdoOUxoNnFIS3o1V0t4MUZEIiwicHJldmlld1VybCI6ImxpYW0tZXJkLXNhbXBsZS1naXQtdXBkYXRlLXJldmlldy1wcm9tcHQtdC02ZjQyNDctcm91dGUtMDYtY29yZS52ZXJjZWwuYXBwIiwibmV4dENvbW1pdFN0YXR1cyI6IkRFUExPWUVEIiwibGl2ZUZlZWRiYWNrIjp7InJlc29sdmVkIjowLCJ1bnJlc29sdmVkIjowLCJ0b3RhbCI6MCwibGluayI6ImxpYW0tZXJkLXNhbXBsZS1naXQtdXBkYXRlLXJldmlldy1wcm9tcHQtdC02ZjQyNDctcm91dGUtMDYtY29yZS52ZXJjZWwuYXBwIn19LHsibmFtZSI6ImxpYW0tYXBwIiwicm9vdERpcmVjdG9yeSI6ImZyb250ZW5kL2FwcHMvYXBwIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL3JvdXRlLTA2LWNvcmUvbGlhbS1hcHAvMlplbmJaN1N2clNxb0pRb0pmdVlaQmVjYXJrdyIsInByZXZpZXdVcmwiOiJsaWFtLWFwcC1naXQtdXBkYXRlLXJldmlldy1wcm9tcHQtdG8tb3V0cHUtZDRjYzc5LXJvdXRlLTA2LWNvcmUudmVyY2VsLmFwcCIsIm5leHRDb21taXRTdGF0dXMiOiJERVBMT1lFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJsaWFtLWFwcC1naXQtdXBkYXRlLXJldmlldy1wcm9tcHQtdG8tb3V0cHUtZDRjYzc5LXJvdXRlLTA2LWNvcmUudmVyY2VsLmFwcCJ9fV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **liam-app** | ✅ Ready ([Inspect](https://vercel.com/route-06-core/liam-app/2ZenbZ7SvrSqoJQoJfuYZBecarkw)) | [Visit Preview](https://liam-app-git-update-review-prompt-to-outpu-d4cc79-route-06-core.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-app-git-update-review-prompt-to-outpu-d4cc79-route-06-core.vercel.app?via=pr-comment-feedback-link) | Apr 8, 2025 11:54am |
| **liam-erd-sample** | ✅ Ready ([Inspect](https://vercel.com/route-06-core/liam-erd-sample/C4m9RMXLtvuwh9Lh6qHKz5WKx1FD)) | [Visit Preview](https://liam-erd-sample-git-update-review-prompt-t-6f4247-route-06-core.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-erd-sample-git-update-review-prompt-t-6f4247-route-06-core.vercel.app?via=pr-comment-feedback-link) | Apr 8, 2025 11:54am |

<details><summary>1 Skipped Deployment</summary>

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **liam-docs** | ⬜️ Ignored ([Inspect](https://vercel.com/route-06-core/liam-docs/dzcMXVS5j3Ws9xKeLTLjy9diTWVE)) | [Visit Preview](https://liam-docs-git-update-review-prompt-to-outp-aa247b-route-06-core.vercel.app) |  | Apr 8, 2025 11:54am |
</details>




author_association: NONE
user: login: supabase[bot]
created_at: 2025-04-08T09:15:25Z
body: [supa]:lfreuhpknrzwtphomppx
**Updates to [Preview Branch (update-review-prompt-to-output-properly) ↗︎](https://supabase.com/dashboard/project/lfreuhpknrzwtphomppx)**

| Deployments | Status                      | Updated                   |
| ----------- | --------------------------- | ------------------------- |
| Database    | ✅ | Tue, 08 Apr 2025 11:48:45 UTC |
| Services    | ✅  | Tue, 08 Apr 2025 11:48:45 UTC  |
| APIs        | ✅      | Tue, 08 Apr 2025 11:48:45 UTC      |

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

| Tasks          | Status                       | Updated                    |
| -------------- | ---------------------------- | -------------------------- |
| Configurations | ✅ | Tue, 08 Apr 2025 11:48:46 UTC |
| Migrations     | ✅ | Tue, 08 Apr 2025 11:48:46 UTC |
| Seeding        | ✅   | Tue, 08 Apr 2025 11:48:46 UTC   |
| Edge Functions | ✅  | Tue, 08 Apr 2025 11:48:46 UTC  |

<hr/>




View logs for this [Workflow Run ↗︎](https://supabase.com/dashboard/project/lfreuhpknrzwtphomppx/branches).
Learn more about [Supabase for Git ↗︎](https://supabase.com/docs/guides/platform/branching).

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-08T09:16:24Z
body: frontend/packages/prompt-test result:

View results: https://app.promptfoo.dev/eval/f:1935c40b-8534-4af4-952d-b0f39fe39f7e

✅️ Promptfoo test succeeded

| ✅️ Successes | ❌️ Failures | ⚠️ Errors |
| --- | --- | --- |
| 9 | 0 | 0 |
<!-- Sticky Pull Request Comment -->
author_association: CONTRIBUTOR
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-04-08T11:17:49Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 3 🔵🔵🔵⚪⚪</td></tr>
<tr><td>🧪&nbsp;<strong>PR contains tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1241/files#diff-79d3de5007a5d073d01c220caa60a46b00eedc32994413db8fc47c213e72e459R53-R59'><strong>Error Handling</strong></a>

The function doesn't handle the case where payload.review.feedbacks might be undefined or empty, which could cause runtime errors.
</summary>

```typescript
for (const feedback of payload.review.feedbacks) {
  const kind = feedback.kind || 'Unknown'
  if (!feedbacksByCategory[kind]) {
    feedbacksByCategory[kind] = []
  }
  feedbacksByCategory[kind].push(feedback)
}
```

</details>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1241/files#diff-79d3de5007a5d073d01c220caa60a46b00eedc32994413db8fc47c213e72e459R199-R201'><strong>Default Category</strong></a>

Using 'MIGRATION_SAFETY' as the default category for unknown categories might lead to misleading review results. Consider using a more neutral default or improving error handling.
</summary>

```typescript
  // Add fallback for unknown category
  Unknown: 'MIGRATION_SAFETY', // Default to MIGRATION_SAFETY for unknown categories
}
```

</details>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1241/files#diff-bc327c5835be24c334bb5b64878901f0a7a7f7aab54d96897cb8f0a326d9aa6bR72-R77'><strong>Duplicate Check</strong></a>

There's a duplicated self-check instruction in the prompt which could confuse the model and affect the output quality.
</summary>

```typescript
Before finalizing your response, perform these self-checks:

**Before finalizing your response, perform these self-checks:**
1. Ensure that each category has at least one feedback item in "feedbacks".
2. For every identified issue, ensure there is a corresponding feedback item with the correct severity.
3. Ensure that there is at least one "POSITIVE" feedback item in "feedbacks" across all categories.
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-04-08T11:18:27Z
body: ## PR Code Suggestions ✨

<!-- 3130939 -->

Explore these optional code suggestions:

<table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=1>Possible issue</td>
<td>



<details><summary>Handle undefined feedbacks</summary>

___


**The code doesn't handle the case where <code>payload.review.feedbacks</code> might be <br>undefined. This could cause a runtime error if the payload structure is <br>incomplete.**

[frontend/packages/jobs/src/functions/processSaveReview.ts [47-59]](https://github.com/liam-hq/liam/pull/1241/files#diff-79d3de5007a5d073d01c220caa60a46b00eedc32994413db8fc47c213e72e459R47-R59)

```diff
 // Calculate scores based on feedback severity
 // Group feedbacks by category
 const feedbacksByCategory: Record<string, typeof payload.review.feedbacks> =
   {}
 
 // Use for...of instead of forEach
-for (const feedback of payload.review.feedbacks) {
+for (const feedback of (payload.review.feedbacks || [])) {
   const kind = feedback.kind || 'Unknown'
   if (!feedbacksByCategory[kind]) {
     feedbacksByCategory[kind] = []
   }
   feedbacksByCategory[kind].push(feedback)
 }
```

- [ ] **Apply this suggestion** <!-- /improve --apply_suggestion=0 -->

<details><summary>Suggestion importance[1-10]: 8</summary>

__

Why: This suggestion addresses a potential runtime error by adding a null check for payload.review.feedbacks. This is an important defensive programming practice that prevents the application from crashing if the payload structure is incomplete.


</details></details></td><td align=center>Medium

</td></tr><tr><td rowspan=1>General</td>
<td>



<details><summary>Fix typo in prompt</summary>

___


**There's a typo in the prompt text - an extra asterisk at the end of the line <br>which could cause confusion or parsing issues.**

[frontend/packages/jobs/src/prompts/generateReview/generateReview.ts [79]](https://github.com/liam-hq/liam/pull/1241/files#diff-bc327c5835be24c334bb5b64878901f0a7a7f7aab54d96897cb8f0a326d9aa6bR79-R79)

```diff
-**Your output must be raw JSON only. Do not include any markdown code blocks or extraneous formatting.****
+**Your output must be raw JSON only. Do not include any markdown code blocks or extraneous formatting.**
```

- [ ] **Apply this suggestion** <!-- /improve --apply_suggestion=1 -->

<details><summary>Suggestion importance[1-10]: 3</summary>

__

Why: The suggestion correctly identifies and fixes a typo (extra asterisk) in the prompt text. While this is a minor issue that likely wouldn't cause functional problems, fixing it improves code quality and prevents potential confusion.


</details></details></td><td align=center>Low

</td></tr>
<tr><td align="center" colspan="2">

- [ ] Update <!-- /improve --more_suggestions=true -->

</td><td></td></tr></tbody></table>
author_association: CONTRIBUTOR
user: login: FunamaYukina
created_at: 2025-04-08T11:58:43Z
body: Thank you for the review!!🌟
author_association: MEMBER