Repository: liam-hq/liam
Pull request #1803: Implement a new job to analyze connected repositories
Author: login: FunamaYukina
Labels: name: Review effort 4/5
Created: 2025-05-28T08:12:12Z | Merged: 2025-05-30T07:45:15Z

## Why is this change needed?
<!-- Please explain briefly why this change is necessary -->

I have created a job that parses and vectorizes files in the README and docs folders that may be relevant to the db design when users link repositories.

## What would you like reviewers to focus on?
<!-- What specific aspects are you requesting review for? -->

## Testing Verification
<!-- Please describe how you verified these changes in your local environment using text/images/video -->

at Local environment

When creating a new project
<img width="400" src="https://github.com/user-attachments/assets/4cf9d9ad-f8df-4fbb-9f15-b5b002f93946">

analyzeRepositoryTask is executed as a job.
![ss 3366](https://github.com/user-attachments/assets/65a83201-dfaa-4861-987a-d369e27bdc66)

![ss 3365](https://github.com/user-attachments/assets/0c74398d-3107-4f3f-a483-a2f7f41f29fc)


The document then stores a vectorized record of the contents of the parsed file.

![ss 3368](https://github.com/user-attachments/assets/0a1f0270-b9db-42c7-81d8-05d3aff08252)


## What was done
<!-- This section will be filled by PR-Agent when the Pull Request is opened -->

### 🤖 Generated by PR Agent at a6859d61694e254c0aa673c6464a8171c00da07f

- Introduces automated repository analysis job for new projects
  - Scans essential and documentation files, vectorizes content
  - Stores vectorized data in Supabase for future retrieval
- Adds new dependencies for GitHub App authentication and API access
- Integrates background analysis trigger into project creation flow


## Detailed Changes
<!-- This section will be filled by PR-Agent when the Pull Request is opened -->

<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>processRepositoryAnalysis.ts</strong><dd><code>Add repository analysis and vectorization logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts

<li>Implements repository analysis logic for scanning and vectorizing repo <br>files<br> <li> Fetches essential and documentation files, splits and embeds content<br> <li> Handles error logging and batch processing for vector storage<br> <li> Provides main entry point for repository analysis workflow


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8">+584/-0</a>&nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>jobs.ts</strong><dd><code>Register and expose repository analysis task</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/packages/jobs/src/trigger/jobs.ts

<li>Adds <code>analyzeRepositoryTask</code> for running repository analysis as a task<br> <li> Integrates new analysis logic into Trigger.dev task system<br> <li> Logs results and errors from repository analysis


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-2058cacc0f2a83e932ddde6ece0f30766b197a7f2d42b3506be73b24adf87c4c">+26/-0</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>index.ts</strong><dd><code>Export analyzeRepositoryTask from jobs package</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/packages/jobs/src/index.ts

<li>Exports <code>analyzeRepositoryTask</code> from jobs package<br> <li> Makes repository analysis task available for import


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-ddccb56671237ba74d151c0df79840792328f66f24b6951805ec716603a1b6d4">+1/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>addProject.ts</strong><dd><code>Trigger repository analysis after project creation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/apps/app/components/ProjectNewPage/components/InstallationSelector/actions/addProject.ts

<li>Triggers repository analysis task after project-repository mapping<br> <li> Handles errors from analysis trigger without failing project creation<br> <li> Integrates background job into project onboarding flow


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-b585286000fe7cbf250ab8a67b9b3236b668fd51b49d11375d0e15036beb9c0e">+16/-0</a>&nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
  <td>
    <details>
      <summary><strong>package.json</strong><dd><code>Add GitHub API dependencies for repository analysis</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

frontend/packages/jobs/package.json

<li>Adds <code>@octokit/auth-app</code> and <code>@octokit/rest</code> as dependencies<br> <li> Updates package manifest for GitHub API integration


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-48a7b87d9434bd217a31485a9ae887b9bb96ea0d08436de3a3218972adccee8f">+2/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
      <summary><strong>pnpm-lock.yaml</strong><dd><code>Update lockfile for new GitHub dependencies</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

pnpm-lock.yaml

<li>Locks new dependencies for <code>@octokit/auth-app</code> and <code>@octokit/rest</code><br> <li> Updates dependency tree for jobs package


</details>


  </td>
  <td><a href="https://github.com/liam-hq/liam/pull/1803/files#diff-32824c984905bb02bc7ffcef96a77addd1f1602cff71a11fbbfdd7f53ee026bb">+8/-2</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

## Additional Notes
<!-- Any additional information for reviewers -->

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about Qodo Merge usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>


Diff:
diff --git a/frontend/apps/app/components/ProjectNewPage/components/InstallationSelector/actions/addProject.ts b/frontend/apps/app/components/ProjectNewPage/components/InstallationSelector/actions/addProject.ts
index e81073395b..382c62eaed 100644
--- a/frontend/apps/app/components/ProjectNewPage/components/InstallationSelector/actions/addProject.ts
+++ b/frontend/apps/app/components/ProjectNewPage/components/InstallationSelector/actions/addProject.ts
@@ -2,6 +2,7 @@
 
 import { createClient } from '@/libs/db/server'
 import { urlgen } from '@/libs/routes'
+import { analyzeRepositoryTask } from '@liam-hq/jobs'
 import { redirect } from 'next/navigation'
 
 export const addProject = async (formData: FormData) => {
@@ -63,5 +64,20 @@ export const addProject = async (formData: FormData) => {
     throw new Error('Failed to create project-repository mapping')
   }
 
+  // Trigger repository analysis in the background
+  try {
+    await analyzeRepositoryTask.trigger({
+      projectId: project.id,
+      repositoryId: repository.id,
+      repositoryOwner,
+      repositoryName,
+      installationId: Number(installationId),
+      organizationId,
+    })
+  } catch (error) {
+    // Log the error but don't fail the project creation
+    console.error('Failed to trigger repository analysis:', error)
+  }
+
   redirect(urlgen('projects/[projectId]', { projectId: `${project.id}` }))
 }
diff --git a/frontend/packages/jobs/package.json b/frontend/packages/jobs/package.json
index a3d48a74a2..45831114f8 100644
--- a/frontend/packages/jobs/package.json
+++ b/frontend/packages/jobs/package.json
@@ -12,6 +12,8 @@
     "@liam-hq/db": "workspace:*",
     "@liam-hq/db-structure": "workspace:*",
     "@liam-hq/github": "workspace:*",
+    "@octokit/auth-app": "7.2.1",
+    "@octokit/rest": "21.1.1",
     "@sentry/node": "9.19.0",
     "@trigger.dev/build": "3.3.17",
     "@trigger.dev/sdk": "3.3.17",
diff --git a/frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts b/frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
new file mode 100644
index 0000000000..5109de9f04
--- /dev/null
+++ b/frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
@@ -0,0 +1,606 @@
+import { Document } from '@langchain/core/documents'
+import { OpenAIEmbeddings } from '@langchain/openai'
+import { getFileContent } from '@liam-hq/github'
+import { logger } from '@trigger.dev/sdk/v3'
+import { createClient } from '../libs/supabase'
+
+export interface RepositoryAnalysisPayload {
+  projectId: string
+  repositoryId: string
+  repositoryOwner: string
+  repositoryName: string
+  installationId: number
+  organizationId: string
+}
+
+interface DocumentFile {
+  path: string
+  content: string
+  type: 'essential' | 'docs'
+}
+
+interface RepositoryDocumentMetadata {
+  source: 'repository_analysis'
+  type: 'essential' | 'docs'
+  project_id: string
+  repository_id: string
+  file_path: string
+  file_type: string
+  repository_owner: string
+  repository_name: string
+  analyzed_at: string
+  organization_id?: string
+  updated_at: string
+  chunk_index?: number
+  chunk_total?: number
+  section_title?: string
+}
+
+// Essential files that must be checked
+const ESSENTIAL_FILES = ['README.md', 'ABOUT.md']
+
+// Documentation directories to search
+const DOC_DIRECTORIES = ['docs', 'documents']
+
+// Keywords to look for in documentation files
+const DOC_KEYWORDS = [
+  'architecture',
+  'design',
+  'overview',
+  'schema',
+  'migration',
+]
+
+/**
+ * Get repository file tree using GitHub Tree API
+ */
+const getRepositoryFileTree = async (
+  repositoryOwner: string,
+  repositoryName: string,
+  installationId: number,
+  errors: string[],
+): Promise<string[]> => {
+  try {
+    // Import Octokit and auth
+    const { Octokit } = await import('@octokit/rest')
+    const { createAppAuth } = await import('@octokit/auth-app')
+
+    // Initialize Octokit with GitHub App authentication
+    const octokit = new Octokit({
+      authStrategy: createAppAuth,
+      auth: {
+        appId: process.env['GITHUB_APP_ID'],
+        privateKey: process.env['GITHUB_PRIVATE_KEY']?.replace(/\\n/g, '\n'),
+        installationId,
+      },
+    })
+
+    // Get repository tree
+    const { data: tree } = await octokit.git.getTree({
+      owner: repositoryOwner,
+      repo: repositoryName,
+      tree_sha: 'HEAD',
+      recursive: 'true',
+    })
+
+    // Filter for relevant documentation files
+    const relevantFiles = tree.tree
+      .filter((item) => {
+        if (item.type !== 'blob' || !item.path) return false
+
+        const path = item.path.toLowerCase()
+
+        // Check if file is in docs or documents directory
+        const isInDocDir = DOC_DIRECTORIES.some((dir) =>
+          path.startsWith(`${dir}/`),
+        )
+
+        // Check if file contains relevant keywords (more flexible matching)
+        const hasRelevantKeyword = DOC_KEYWORDS.some((keyword) =>
+          path.includes(keyword.toLowerCase()),
+        )
+
+        // Check if file has relevant extension
+        const hasRelevantExt = /\.(md|txt|rst)$/i.test(path)
+
+        // Include files that match directory + keyword + extension criteria
+        return isInDocDir && hasRelevantKeyword && hasRelevantExt
+      })
+      .map((item) => item.path)
+      .filter((path): path is string => path !== undefined)
+
+    return relevantFiles
+  } catch (error) {
+    const errorMessage = getErrorMessage(
+      error,
+      'Failed to get repository file tree',
+    )
+    errors.push(errorMessage)
+    // Fallback to empty array if Tree API fails
+    return []
+  }
+}
+
+/**
+ * Get error message from any error type
+ */
+const getErrorMessage = (error: unknown, context: string): string => {
+  return `${context}: ${error instanceof Error ? error.message : 'Unknown error'}`
+}
+
+/**
+ * Fetch a single file from the repository
+ */
+const fetchFile = async (
+  repositoryFullName: string,
+  filePath: string,
+  installationId: number,
+  fileType: DocumentFile['type'],
+  errors: string[],
+): Promise<DocumentFile | null> => {
+  try {
+    const { content } = await getFileContent(
+      repositoryFullName,
+      filePath,
+      'main', // Default branch
+      installationId,
+    )
+
+    if (content) {
+      return { path: filePath, content, type: fileType }
+    }
+
+    return null
+  } catch (error) {
+    const errorMessage = getErrorMessage(
+      error,
+      `Failed to fetch ${fileType} file ${filePath}`,
+    )
+    errors.push(errorMessage)
+    return null
+  }
+}
+
+/**
+ * Split text into chunks with overlap
+ */
+const splitTextIntoChunks = (
+  text: string,
+  chunkSize = 1000,
+  overlap = 200,
+): string[] => {
+  if (text.length <= chunkSize) {
+    return [text]
+  }
+
+  const chunks: string[] = []
+  let start = 0
+
+  while (start < text.length) {
+    let end = start + chunkSize
+
+    // If this is not the last chunk, try to find a good breaking point
+    if (end < text.length) {
+      // Look for paragraph breaks first
+      const paragraphBreak = text.lastIndexOf('\n\n', end)
+      if (paragraphBreak > start + chunkSize / 2) {
+        end = paragraphBreak
+      } else {
+        // Look for sentence breaks
+        const sentenceBreak = text.lastIndexOf('.', end)
+        if (sentenceBreak > start + chunkSize / 2) {
+          end = sentenceBreak + 1
+        } else {
+          // Look for line breaks
+          const lineBreak = text.lastIndexOf('\n', end)
+          if (lineBreak > start + chunkSize / 2) {
+            end = lineBreak
+          }
+        }
+      }
+    }
+
+    chunks.push(text.slice(start, end).trim())
+
+    // Move start position with overlap
+    start = end - overlap
+    if (start >= text.length) break
+  }
+
+  return chunks.filter((chunk) => chunk.length > 0)
+}
+
+/**
+ * Extract section title from markdown content
+ */
+const extractSectionTitle = (chunk: string): string | undefined => {
+  const lines = chunk.split('\n')
+  for (const line of lines) {
+    const trimmed = line.trim()
+    if (trimmed.startsWith('#')) {
+      return trimmed.replace(/^#+\s*/, '')
+    }
+  }
+  return undefined
+}
+
+/**
+ * Convert DocumentFile to LangChain Documents with chunking
+ */
+const createLangChainDocuments = (
+  file: DocumentFile,
+  projectId: string,
+  repositoryId: string,
+  repositoryOwner: string,
+  repositoryName: string,
+  organizationId: string,
+): Document[] => {
+  const now = new Date().toISOString()
+
+  // Split content into chunks
+  const chunks = splitTextIntoChunks(file.content)
+
+  return chunks.map((chunk, index) => {
+    const sectionTitle = extractSectionTitle(chunk)
+
+    const metadata: RepositoryDocumentMetadata = {
+      source: 'repository_analysis',
+      type: file.type,
+      project_id: projectId,
+      repository_id: repositoryId,
+      file_path: file.path,
+      file_type: file.type,
+      repository_owner: repositoryOwner,
+      repository_name: repositoryName,
+      analyzed_at: now,
+      organization_id: organizationId,
+      updated_at: now,
+      chunk_index: index,
+      chunk_total: chunks.length,
+      ...(sectionTitle && { section_title: sectionTitle }),
+    }
+
+    return new Document({
+      pageContent: chunk,
+      metadata,
+    })
+  })
+}
+
+/**
+ * Validate OpenAI API key
+ */
+const validateOpenAIKey = (): string => {
+  const openAIApiKey = process.env['OPENAI_API_KEY']
+
+  if (!openAIApiKey) {
+    throw new Error(
+      'Valid OpenAI API key is required for generating embeddings',
+    )
+  }
+
+  return openAIApiKey
+}
+
+/**
+ * Insert a single document into Supabase
+ */
+const insertDocumentToSupabase = async (
+  supabase: ReturnType<typeof createClient>,
+  documentContent: string,
+  metadata: RepositoryDocumentMetadata,
+  vector: number[],
+): Promise<boolean> => {
+  // @ts-ignore - Type inconsistencies with Supabase schema definitions
+  const { error } = await supabase.from('documents').insert([
+    {
+      content: documentContent,
+      metadata,
+      embedding: vector,
+      updated_at: metadata.updated_at,
+      organization_id: metadata.organization_id || '',
+    },
+  ])
+
+  if (error) {
+    const errorMessage = `Document insertion failed: ${error.message} ${error.code} ${error.details}`
+    logger.error(errorMessage)
+    throw new Error(errorMessage)
+  }
+
+  return true
+}
+
+/**
+ * Process a batch of documents
+ */
+const processBatch = async (
+  batch: Document[],
+  embeddings: OpenAIEmbeddings,
+  supabase: ReturnType<typeof createClient>,
+  errors: string[],
+): Promise<number> => {
+  try {
+    // Generate embeddings for the batch
+    const texts = batch.map((doc) => doc.pageContent)
+    const metadatas = batch.map(
+      (doc) => doc.metadata as RepositoryDocumentMetadata,
+    )
+    const vectors = await embeddings.embedDocuments(texts)
+
+    // Insert each document into Supabase
+    let processed = 0
+    for (let j = 0; j < vectors.length; j++) {
+      const vector = vectors[j]
+      const metadata = metadatas[j]
+      const text = texts[j]
+
+      if (!metadata) {
+        logger.warn(`Skipping document ${j} due to missing metadata`)
+        continue
+      }
+
+      if (!vector) {
+        logger.warn(`Skipping document ${j} due to missing vector`)
+        continue
+      }
+
+      // Ensure text is always a valid string
+      const documentContent = (() => {
+        if (text === null || text === undefined) {
+          return ''
+        }
+        return typeof text === 'string' ? text : String(text)
+      })()
+
+      try {
+        await insertDocumentToSupabase(
+          supabase,
+          documentContent,
+          metadata,
+          vector,
+        )
+        processed++
+      } catch (insertError) {
+        const errorMessage = getErrorMessage(
+          insertError,
+          `Failed to insert document ${j}`,
+        )
+        errors.push(errorMessage)
+      }
+    }
+
+    return processed
+  } catch (error) {
+    const errorMessage = getErrorMessage(error, 'Failed to process batch')
+    logger.error(errorMessage)
+    errors.push(errorMessage)
+    return 0
+  }
+}
+
+/**
+ * Save documents to vector store with embeddings
+ */
+const saveDocumentsToVectorStore = async (
+  documents: Document[],
+  errors: string[],
+): Promise<number> => {
+  try {
+    if (documents.length === 0) {
+      logger.log('No documents to save')
+      return 0
+    }
+
+    // Validate OpenAI API key
+    const openAIApiKey = validateOpenAIKey()
+
+    // Initialize OpenAI embeddings
+    const embeddings = new OpenAIEmbeddings({ openAIApiKey })
+    const supabase = createClient()
+
+    // Process documents in batches to avoid overwhelming the API
+    const BATCH_SIZE = 10
+    let totalProcessed = 0
+
+    for (let i = 0; i < documents.length; i += BATCH_SIZE) {
+      const batch = documents.slice(i, i + BATCH_SIZE)
+      const processed = await processBatch(batch, embeddings, supabase, errors)
+      totalProcessed += processed
+    }
+
+    return totalProcessed
+  } catch (error) {
+    const errorMessage = getErrorMessage(
+      error,
+      'Failed to save documents to vector store',
+    )
+    logger.error(errorMessage)
+    errors.push(errorMessage)
+    return 0
+  }
+}
+
+/**
+ * Fetch essential files from repository
+ */
+const fetchEssentialFiles = async (
+  repositoryFullName: string,
+  installationId: number,
+  errors: string[],
+): Promise<{ files: DocumentFile[]; count: number }> => {
+  const files: DocumentFile[] = []
+  let count = 0
+
+  for (const fileName of ESSENTIAL_FILES) {
+    const file = await fetchFile(
+      repositoryFullName,
+      fileName,
+      installationId,
+      'essential',
+      errors,
+    )
+
+    if (file) {
+      files.push(file)
+      count++
+    }
+  }
+
+  return { files, count }
+}
+
+/**
+ * Fetch documentation files from repository
+ */
+const fetchDocumentationFiles = async (
+  repositoryOwner: string,
+  repositoryName: string,
+  repositoryFullName: string,
+  installationId: number,
+  errors: string[],
+): Promise<{ files: DocumentFile[]; count: number }> => {
+  const files: DocumentFile[] = []
+  let count = 0
+
+  // Get documentation files using Tree API
+  const docFilePaths = await getRepositoryFileTree(
+    repositoryOwner,
+    repositoryName,
+    installationId,
+    errors,
+  )
+
+  for (const filePath of docFilePaths) {
+    const file = await fetchFile(
+      repositoryFullName,
+      filePath,
+      installationId,
+      'docs',
+      errors,
+    )
+
+    if (file) {
+      files.push(file)
+      count++
+    }
+  }
+
+  return { files, count }
+}
+
+/**
+ * Process and save documents to vector store
+ */
+const processAndSaveDocuments = async (
+  documentFiles: DocumentFile[],
+  projectId: string,
+  repositoryId: string,
+  repositoryOwner: string,
+  repositoryName: string,
+  organizationId: string,
+  errors: string[],
+): Promise<{ processedFiles: number; totalChunks: number }> => {
+  // Convert to LangChain Documents with chunking
+  const langChainDocuments = documentFiles.flatMap((file) =>
+    createLangChainDocuments(
+      file,
+      projectId,
+      repositoryId,
+      repositoryOwner,
+      repositoryName,
+      organizationId,
+    ),
+  )
+
+  // Save to vector store with embeddings
+  const processedFiles = await saveDocumentsToVectorStore(
+    langChainDocuments,
+    errors,
+  )
+
+  return { processedFiles, totalChunks: langChainDocuments.length }
+}
+
+/**
+ * FIXME: Duplicate data prevention needed
+ *
+ * Current implementation allows duplicate documents to be saved if this function
+ * runs multiple times for the same repository. This can happen due to:
+ * - Job failures and retries
+ * - Manual re-analysis of the same repository
+ * - System errors causing process interruption and restart
+ *
+ * Proposed solutions:
+ * 1. Add content hash to documents table for true duplicate detection
+ * 2. Implement upsert logic with unique constraints
+ * 3. Add pre-processing cleanup of existing documents
+ *
+ * Priority: Medium - Should be addressed before production scaling
+ */
+
+/**
+ * Retrieve documents from repository and save to vector store
+ */
+export const processRepositoryAnalysis = async (
+  payload: RepositoryAnalysisPayload,
+): Promise<{ processedFiles: number; errors: string[] }> => {
+  const {
+    projectId,
+    repositoryId,
+    repositoryOwner,
+    repositoryName,
+    installationId,
+    organizationId,
+  } = payload
+
+  logger.log(`🔍 Scanning repository: ${repositoryOwner}/${repositoryName}`)
+
+  const repositoryFullName = `${repositoryOwner}/${repositoryName}`
+  const errors: string[] = []
+
+  try {
+    const documentFiles: DocumentFile[] = []
+
+    // 1. Fetch essential files (README.md, ABOUT.md)
+    const { files: essentialFiles, count: essentialCount } =
+      await fetchEssentialFiles(repositoryFullName, installationId, errors)
+    documentFiles.push(...essentialFiles)
+
+    // 2. Fetch documentation files
+    const { files: docFiles, count: docsCount } = await fetchDocumentationFiles(
+      repositoryOwner,
+      repositoryName,
+      repositoryFullName,
+      installationId,
+      errors,
+    )
+    documentFiles.push(...docFiles)
+
+    // 3. Process and save documents
+    const { processedFiles, totalChunks } = await processAndSaveDocuments(
+      documentFiles,
+      projectId,
+      repositoryId,
+      repositoryOwner,
+      repositoryName,
+      organizationId,
+      errors,
+    )
+
+    // Summary log
+    logger.log(`📊 Repository scan completed:
+  ✅ Found files: ${documentFiles.length}
+  ❌ Missing files: ${errors.length}
+  📁 Essential: ${essentialCount}
+  📚 Docs: ${docsCount}
+  🧩 Total chunks: ${totalChunks}
+  💾 Processed: ${processedFiles}`)
+
+    return { processedFiles, errors }
+  } catch (error) {
+    const errorMessage = getErrorMessage(error, 'Repository analysis failed')
+    logger.error(errorMessage)
+    throw new Error(errorMessage)
+  }
+}
diff --git a/frontend/packages/jobs/src/index.ts b/frontend/packages/jobs/src/index.ts
index 59c2547366..3746901778 100644
--- a/frontend/packages/jobs/src/index.ts
+++ b/frontend/packages/jobs/src/index.ts
@@ -18,6 +18,7 @@ export {
 export {
   helloWorld,
   generateKnowledgeFromFeedbackTask,
+  analyzeRepositoryTask,
 } from './trigger/jobs'
 
 export * from './prompts'
diff --git a/frontend/packages/jobs/src/trigger/jobs.ts b/frontend/packages/jobs/src/trigger/jobs.ts
index 3356b70e8d..11f949c809 100644
--- a/frontend/packages/jobs/src/trigger/jobs.ts
+++ b/frontend/packages/jobs/src/trigger/jobs.ts
@@ -6,6 +6,10 @@ import {
   processGenerateDocsSuggestion,
 } from '../functions/processGenerateDocsSuggestion'
 import { processGenerateSchemaOverride } from '../functions/processGenerateSchemaOverride'
+import {
+  type RepositoryAnalysisPayload,
+  processRepositoryAnalysis,
+} from '../functions/processRepositoryAnalysis'
 import type {
   GenerateSchemaOverridePayload,
   OverallReview,
@@ -13,6 +17,28 @@ import type {
 } from '../types'
 import { helloWorldTask } from './helloworld'
 
+export const analyzeRepositoryTask = task({
+  id: 'analyze-repository',
+  run: async (payload: RepositoryAnalysisPayload) => {
+    logger.log('Executing repository analysis task:', { payload })
+
+    const result = await processRepositoryAnalysis(payload)
+
+    logger.log('Repository analysis completed:', {
+      processedFiles: result.processedFiles,
+      errorCount: result.errors.length,
+    })
+
+    if (result.errors.length > 0) {
+      logger.warn('Repository analysis completed with errors:', {
+        errors: result.errors,
+      })
+    }
+
+    return result
+  },
+})
+
 export const generateDocsSuggestionTask = task({
   id: 'generate-docs-suggestion',
   run: async (payload: {
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 1ab9abb093..3b646b63c7 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -747,6 +747,12 @@ importers:
       '@liam-hq/github':
         specifier: workspace:*
         version: link:../github
+      '@octokit/auth-app':
+        specifier: 7.2.1
+        version: 7.2.1
+      '@octokit/rest':
+        specifier: 21.1.1
+        version: 21.1.1
       '@sentry/node':
         specifier: 9.19.0
         version: 9.19.0
@@ -21154,7 +21160,7 @@ snapshots:
       isstream: 0.1.2
       jsonwebtoken: 9.0.2
       mime-types: 2.1.35
-      retry-axios: 2.6.0(axios@1.9.0(debug@4.4.1))
+      retry-axios: 2.6.0(axios@1.9.0)
       tough-cookie: 4.1.4
     transitivePeerDependencies:
       - supports-color
@@ -23831,7 +23837,7 @@ snapshots:
       onetime: 7.0.0
       signal-exit: 4.1.0
 
-  retry-axios@2.6.0(axios@1.9.0(debug@4.4.1)):
+  retry-axios@2.6.0(axios@1.9.0):
     dependencies:
       axios: 1.9.0(debug@4.4.1)
 


Reviews:
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-28T08:31:14Z
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-28T09:04:20Z
user: login: hoshinotsuyoshi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-29T22:52:11Z
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-30T03:14:14Z
user: login: hoshinotsuyoshi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-30T03:47:08Z
user: login: hoshinotsuyoshi
body: LGTM!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-30T03:47:16Z
user: login: FunamaYukina
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-30T03:55:08Z

Review comments:
diff_hunk: @@ -0,0 +1,584 @@
+import { Document } from '@langchain/core/documents'
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: FunamaYukina
body: This file handles the following.

```mermaid
graph TD
  A[Start: processRepositoryAnalysis Job] --> B["Fetch essential files:<br>- README.md<br>- ABOUT.md<br>- Only at root level"]
  A --> C["Fetch docs via GitHub Tree API:<br>- Search /docs and /documents folders<br>- Target: .md, .txt, .rst files<br>- Must match keywords:<br>architecture, design, overview, schema, migration"]

  B --> D["Add to documentFiles"]
  C --> D

  D --> E["Chunk & convert to LangChain Documents<br>- Use custom splitTextIntoChunks<br>- 1000 chars with 200 overlap<br>- Attach metadata: path, source, repo_id, chunk_index"]

  E --> F["Check OpenAI API Key<br>- Throw error if missing"]
  F --> G["Initialize OpenAIEmbeddings"]
  G --> H["Embed each batch<br>- Up to 10 documents per batch"]
  H --> I["Insert into Supabase<br>- Store content, metadata, embedding<br>- Handle Supabase errors properly"]

  I --> J["Log & return summary<br>- Found files, chunks, processed count"]


```
created_at: 2025-05-28T08:31:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,584 @@
+import { Document } from '@langchain/core/documents'
+import { OpenAIEmbeddings } from '@langchain/openai'
+import { getFileContent } from '@liam-hq/github'
+import { logger } from '@trigger.dev/sdk/v3'
+import { createClient } from '../libs/supabase'
+
+export interface RepositoryAnalysisPayload {
+  projectId: string
+  repositoryId: string
+  repositoryOwner: string
+  repositoryName: string
+  installationId: number
+  organizationId: string
+}
+
+interface DocumentFile {
+  path: string
+  content: string
+  type: 'essential' | 'docs'
+}
+
+interface RepositoryDocumentMetadata {
+  source: 'repository_analysis'
+  type: 'essential' | 'docs'
+  project_id: string
+  repository_id: string
+  file_path: string
+  file_type: string
+  repository_owner: string
+  repository_name: string
+  analyzed_at: string
+  organization_id?: string
+  updated_at: string
+  chunk_index?: number
+  chunk_total?: number
+  section_title?: string
+}
+
+// Essential files that must be checked
+const ESSENTIAL_FILES = ['README.md', 'ABOUT.md']
+
+// Documentation directories to search
+const DOC_DIRECTORIES = ['docs', 'documents']
+
+// Keywords to look for in documentation files
+const DOC_KEYWORDS = [
+  'architecture',
+  'design',
+  'overview',
+  'schema',
+  'migration',
+]
+
+/**
+ * Get repository file tree using GitHub Tree API
+ */
+const getRepositoryFileTree = async (
+  repositoryOwner: string,
+  repositoryName: string,
+  installationId: number,
+  errors: string[],
+): Promise<string[]> => {
+  try {
+    // Import Octokit and auth
+    const { Octokit } = await import('@octokit/rest')
+    const { createAppAuth } = await import('@octokit/auth-app')
+
+    // Initialize Octokit with GitHub App authentication
+    const octokit = new Octokit({
+      authStrategy: createAppAuth,
+      auth: {
+        appId: process.env['GITHUB_APP_ID'],
+        privateKey: process.env['GITHUB_PRIVATE_KEY']?.replace(/\\n/g, '\n'),
+        installationId,
+      },
+    })
+
+    // Get repository tree
+    const { data: tree } = await octokit.git.getTree({
+      owner: repositoryOwner,
+      repo: repositoryName,
+      tree_sha: 'HEAD',
+      recursive: 'true',
+    })
+
+    // Filter for relevant documentation files
+    const relevantFiles = tree.tree
+      .filter((item) => {
+        if (item.type !== 'blob' || !item.path) return false
+
+        const path = item.path.toLowerCase()
+
+        // Check if file is in docs or documents directory
+        const isInDocDir = DOC_DIRECTORIES.some((dir) =>
+          path.startsWith(`${dir}/`),
+        )
+
+        // Check if file contains relevant keywords (more flexible matching)
+        const hasRelevantKeyword = DOC_KEYWORDS.some((keyword) =>
+          path.includes(keyword.toLowerCase()),
+        )
+
+        // Check if file has relevant extension
+        const hasRelevantExt = /\.(md|txt|rst)$/i.test(path)
+
+        // Include files that match directory + keyword + extension criteria
+        return isInDocDir && hasRelevantKeyword && hasRelevantExt
+      })
+      .map((item) => item.path)
+      .filter((path): path is string => path !== undefined)
+
+    return relevantFiles
+  } catch (error) {
+    const errorMessage = getErrorMessage(
+      error,
+      'Failed to get repository file tree',
+    )
+    errors.push(errorMessage)
+    // Fallback to empty array if Tree API fails
+    return []
+  }
+}
+
+/**
+ * Get error message from any error type
+ */
+const getErrorMessage = (error: unknown, context: string): string => {
+  return `${context}: ${error instanceof Error ? error.message : 'Unknown error'}`
+}
+
+/**
+ * Fetch a single file from the repository
+ */
+const fetchFile = async (
+  repositoryFullName: string,
+  filePath: string,
+  installationId: number,
+  fileType: DocumentFile['type'],
+  errors: string[],
+): Promise<DocumentFile | null> => {
+  try {
+    const { content } = await getFileContent(
+      repositoryFullName,
+      filePath,
+      'main', // Default branch
+      installationId,
+    )
+
+    if (content) {
+      return { path: filePath, content, type: fileType }
+    }
+
+    return null
+  } catch (error) {
+    const errorMessage = getErrorMessage(
+      error,
+      `Failed to fetch ${fileType} file ${filePath}`,
+    )
+    errors.push(errorMessage)
+    return null
+  }
+}
+
+/**
+ * Split text into chunks with overlap
+ */
+const splitTextIntoChunks = (
+  text: string,
+  chunkSize = 1000,
+  overlap = 200,
+): string[] => {
+  if (text.length <= chunkSize) {
+    return [text]
+  }
+
+  const chunks: string[] = []
+  let start = 0
+
+  while (start < text.length) {
+    let end = start + chunkSize
+
+    // If this is not the last chunk, try to find a good breaking point
+    if (end < text.length) {
+      // Look for paragraph breaks first
+      const paragraphBreak = text.lastIndexOf('\n\n', end)
+      if (paragraphBreak > start + chunkSize / 2) {
+        end = paragraphBreak
+      } else {
+        // Look for sentence breaks
+        const sentenceBreak = text.lastIndexOf('.', end)
+        if (sentenceBreak > start + chunkSize / 2) {
+          end = sentenceBreak + 1
+        } else {
+          // Look for line breaks
+          const lineBreak = text.lastIndexOf('\n', end)
+          if (lineBreak > start + chunkSize / 2) {
+            end = lineBreak
+          }
+        }
+      }
+    }
+
+    chunks.push(text.slice(start, end).trim())
+
+    // Move start position with overlap
+    start = end - overlap
+    if (start >= text.length) break
+  }
+
+  return chunks.filter((chunk) => chunk.length > 0)
+}
+
+/**
+ * Extract section title from markdown content
+ */
+const extractSectionTitle = (chunk: string): string | undefined => {
+  const lines = chunk.split('\n')
+  for (const line of lines) {
+    const trimmed = line.trim()
+    if (trimmed.startsWith('#')) {
+      return trimmed.replace(/^#+\s*/, '')
+    }
+  }
+  return undefined
+}
+
+/**
+ * Convert DocumentFile to LangChain Documents with chunking
+ */
+const createLangChainDocuments = (
+  file: DocumentFile,
+  projectId: string,
+  repositoryId: string,
+  repositoryOwner: string,
+  repositoryName: string,
+  organizationId: string,
+): Document[] => {
+  const now = new Date().toISOString()
+
+  // Split content into chunks
+  const chunks = splitTextIntoChunks(file.content)
+
+  return chunks.map((chunk, index) => {
+    const sectionTitle = extractSectionTitle(chunk)
+
+    const metadata: RepositoryDocumentMetadata = {
+      source: 'repository_analysis',
+      type: file.type,
+      project_id: projectId,
+      repository_id: repositoryId,
+      file_path: file.path,
+      file_type: file.type,
+      repository_owner: repositoryOwner,
+      repository_name: repositoryName,
+      analyzed_at: now,
+      organization_id: organizationId,
+      updated_at: now,
+      chunk_index: index,
+      chunk_total: chunks.length,
+      ...(sectionTitle && { section_title: sectionTitle }),
+    }
+
+    return new Document({
+      pageContent: chunk,
+      metadata,
+    })
+  })
+}
+
+/**
+ * Validate OpenAI API key
+ */
+const validateOpenAIKey = (): string => {
+  const openAIApiKey = process.env['OPENAI_API_KEY']
+
+  if (!openAIApiKey) {
+    throw new Error(
+      'Valid OpenAI API key is required for generating embeddings',
+    )
+  }
+
+  return openAIApiKey
+}
+
+/**
+ * Insert a single document into Supabase
+ */
+const insertDocumentToSupabase = async (
+  supabase: ReturnType<typeof createClient>,
+  documentContent: string,
+  metadata: RepositoryDocumentMetadata,
+  vector: number[],
+): Promise<boolean> => {
+  // @ts-ignore - Type inconsistencies with Supabase schema definitions
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: FunamaYukina
body:  I ran into a type mismatch: LangChain's `Document.metadata` is typed as `Record<string, any>`, while Supabase expects our stricter `RepositoryDocumentMetadata`.

I had two options:

1. Use `as` for casting
2. Use `@ts-ignore` to bypass the type check

I chose `@ts-ignore` , but if there is a better way, please let me know!🙏
created_at: 2025-05-28T09:04:20Z
author_association: MEMBER
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: hoshinotsuyoshi
body: 
I'm not very familiar with the implementation details, so apologies if I'm misunderstanding anything.

-------

@FunamaYukina 

Is this function intended to be called periodically for a single project?

I'm a bit concerned that analyzing the same repository multiple times might create duplicate documents.
Would it be safe to run this function multiple times for the same project?

If that's not currently the case, I think it would be helpful to add a comment explaining the intended usage—e.g., whether it's expected to support periodic execution in the future.

created_at: 2025-05-29T22:52:02Z
author_association: MEMBER
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: FunamaYukina
body: You're absolutely right. If the same repository is added twice, it will result in duplicate content being stored.
To handle this, I think the best approach would be to add a `content_hash` column to the `documents` table so we can detect duplicates.
However, since that might be a slightly larger change, I’ve just left a comment for now.
https://github.com/liam-hq/liam/pull/1803/commits/7f9c578d006045ad8bd0dcb68872ab8f207aeb30
Thanks a lot for the helpful advice!🙏
created_at: 2025-05-30T03:14:14Z
author_association: MEMBER
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: hoshinotsuyoshi
body: > To handle this, I think the best approach would be to add a content_hash column to the documents table so we can detect duplicates.

That sounds like a good approach! Adding a comment as well makes sense — I agree.

Just to clarify, what I was concerned about wasn't so much adding the same project multiple times, but rather the case where the embedding information becomes outdated compared to the actual documents in the Git repository.
That said, the comment you've added looks good to me!

created_at: 2025-05-30T03:47:08Z
author_association: MEMBER
path: frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts
user: login: FunamaYukina
body: > what I was concerned about wasn't so much adding the same project multiple times, but rather the case where the embedding information becomes outdated compared to the actual documents in the Git repository.

Oh, I see! I may need the ability to synchronize (update records in documents) in the future🤔. Thank you!
created_at: 2025-05-30T03:55:07Z
author_association: MEMBER

PR comments:
user: login: changeset-bot[bot]
created_at: 2025-05-28T08:12:16Z
body: ###  ⚠️  No Changeset found

Latest commit: db9f5ea175147cc0cbc0c8a7b77a9c295e8afffa

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. **If these changes should result in a version bump, you need to add a changeset.**

<details><summary>This PR includes no changesets</summary>

  When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

</details>

[Click here to learn what changesets are, and how to add one](https://github.com/changesets/changesets/blob/main/docs/adding-a-changeset.md).

[Click here if you're a maintainer who wants to add a changeset to this PR](https://github.com/liam-hq/liam/new/add-repository-analysis-job?filename=.changeset/plenty-ties-fry.md&value=---%0A%22%40liam-hq%2Fapp%22%3A%20patch%0A%22%40liam-hq%2Fjobs%22%3A%20patch%0A---%0A%0AImplement%20a%20new%20job%20to%20analyze%20connected%20repositories%0A)


author_association: NONE
user: login: vercel[bot]
created_at: 2025-05-28T08:12:18Z
body: [vc]: #o0PTkK6lKgJXk9FM+VfE2PsUFu5mcBIHpeR0gjdJ+HU=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJsaWFtLWRvY3MiLCJyb290RGlyZWN0b3J5IjoiZnJvbnRlbmQvYXBwcy9kb2NzIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2xpYW1ieC9saWFtLWRvY3MvRHpSbmprRHZFdWVBS3hDd1JUQlFqODlESzlNdiIsInByZXZpZXdVcmwiOiJsaWFtLWRvY3MtZ2l0LWFkZC1yZXBvc2l0b3J5LWFuYWx5c2lzLWpvYi1saWFtYngudmVyY2VsLmFwcCIsIm5leHRDb21taXRTdGF0dXMiOiJERVBMT1lFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJsaWFtLWRvY3MtZ2l0LWFkZC1yZXBvc2l0b3J5LWFuYWx5c2lzLWpvYi1saWFtYngudmVyY2VsLmFwcCJ9fSx7Im5hbWUiOiJsaWFtLWVyZC1zYW1wbGUiLCJyb290RGlyZWN0b3J5IjoiZnJvbnRlbmQvYXBwcy9lcmQtc2FtcGxlIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2xpYW1ieC9saWFtLWVyZC1zYW1wbGUvRkNDYldYdjFTY1RVOGI1SGlkNnZ2ZnlzUlJHNCIsInByZXZpZXdVcmwiOiJsaWFtLWVyZC1zYW1wbGUtZ2l0LWFkZC1yZXBvc2l0b3J5LWFuYWx5c2lzLWpvYi1saWFtYngudmVyY2VsLmFwcCIsIm5leHRDb21taXRTdGF0dXMiOiJERVBMT1lFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJsaWFtLWVyZC1zYW1wbGUtZ2l0LWFkZC1yZXBvc2l0b3J5LWFuYWx5c2lzLWpvYi1saWFtYngudmVyY2VsLmFwcCJ9fSx7Im5hbWUiOiJsaWFtLWFwcCIsInJvb3REaXJlY3RvcnkiOiJmcm9udGVuZC9hcHBzL2FwcCIsImluc3BlY3RvclVybCI6Imh0dHBzOi8vdmVyY2VsLmNvbS9saWFtYngvbGlhbS1hcHAvSkFoTmlka2NIYmpxY1FGaEYxRk5hN1dpS3BrQiIsInByZXZpZXdVcmwiOiJsaWFtLWFwcC1naXQtYWRkLXJlcG9zaXRvcnktYW5hbHlzaXMtam9iLWxpYW1ieC52ZXJjZWwuYXBwIiwibmV4dENvbW1pdFN0YXR1cyI6IkRFUExPWUVEIiwibGl2ZUZlZWRiYWNrIjp7InJlc29sdmVkIjowLCJ1bnJlc29sdmVkIjowLCJ0b3RhbCI6MCwibGluayI6ImxpYW0tYXBwLWdpdC1hZGQtcmVwb3NpdG9yeS1hbmFseXNpcy1qb2ItbGlhbWJ4LnZlcmNlbC5hcHAifX0seyJuYW1lIjoibGlhbS1zdG9yeWJvb2siLCJyb290RGlyZWN0b3J5IjoiZnJvbnRlbmQvaW50ZXJuYWwtcGFja2FnZXMvc3Rvcnlib29rIiwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2xpYW1ieC9saWFtLXN0b3J5Ym9vay84R1piQUdRdHJxY3FXS2VkU3g3Nnlhdm9vbXV6IiwicHJldmlld1VybCI6ImxpYW0tc3Rvcnlib29rLWdpdC1hZGQtcmVwb3NpdG9yeS1hbmFseXNpcy1qb2ItbGlhbWJ4LnZlcmNlbC5hcHAiLCJuZXh0Q29tbWl0U3RhdHVzIjoiREVQTE9ZRUQiLCJsaXZlRmVlZGJhY2siOnsicmVzb2x2ZWQiOjAsInVucmVzb2x2ZWQiOjAsInRvdGFsIjowLCJsaW5rIjoibGlhbS1zdG9yeWJvb2stZ2l0LWFkZC1yZXBvc2l0b3J5LWFuYWx5c2lzLWpvYi1saWFtYngudmVyY2VsLmFwcCJ9fV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **liam-app** | ✅ Ready ([Inspect](https://vercel.com/liambx/liam-app/JAhNidkcHbjqcQFhF1FNa7WiKpkB)) | [Visit Preview](https://liam-app-git-add-repository-analysis-job-liambx.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-app-git-add-repository-analysis-job-liambx.vercel.app?via=pr-comment-feedback-link) | May 30, 2025 3:19am |
| **liam-docs** | ✅ Ready ([Inspect](https://vercel.com/liambx/liam-docs/DzRnjkDvEueAKxCwRTBQj89DK9Mv)) | [Visit Preview](https://liam-docs-git-add-repository-analysis-job-liambx.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-docs-git-add-repository-analysis-job-liambx.vercel.app?via=pr-comment-feedback-link) | May 30, 2025 3:19am |
| **liam-erd-sample** | ✅ Ready ([Inspect](https://vercel.com/liambx/liam-erd-sample/FCCbWXv1ScTU8b5Hid6vvfysRRG4)) | [Visit Preview](https://liam-erd-sample-git-add-repository-analysis-job-liambx.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-erd-sample-git-add-repository-analysis-job-liambx.vercel.app?via=pr-comment-feedback-link) | May 30, 2025 3:19am |
| **liam-storybook** | ✅ Ready ([Inspect](https://vercel.com/liambx/liam-storybook/8GZbAGQtrqcqWKedSx76yavoomuz)) | [Visit Preview](https://liam-storybook-git-add-repository-analysis-job-liambx.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/liam-storybook-git-add-repository-analysis-job-liambx.vercel.app?via=pr-comment-feedback-link) | May 30, 2025 3:19am |





author_association: NONE
user: login: supabase[bot]
created_at: 2025-05-28T08:12:26Z
body: [supa]:pbmjrzgmuufcaqlengxj
**Updates to [Preview Branch (add-repository-analysis-job) ↗︎](https://supabase.com/dashboard/project/pbmjrzgmuufcaqlengxj)**

| Deployments | Status                      | Updated                   |
| ----------- | --------------------------- | ------------------------- |
| Database    | ✅ | Fri, 30 May 2025 03:14:43 UTC |
| Services    | ✅  | Fri, 30 May 2025 03:14:43 UTC  |
| APIs        | ✅      | Fri, 30 May 2025 03:14:43 UTC      |

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

| Tasks          | Status                       | Updated                    |
| -------------- | ---------------------------- | -------------------------- |
| Configurations | ✅ | Fri, 30 May 2025 03:14:44 UTC |
| Migrations     | ✅ | Fri, 30 May 2025 03:14:44 UTC |
| Seeding        | ✅   | Fri, 30 May 2025 03:14:44 UTC   |
| Edge Functions | ✅  | Fri, 30 May 2025 03:14:44 UTC  |

<hr/>




View logs for this [Workflow Run ↗︎](https://supabase.com/dashboard/project/pbmjrzgmuufcaqlengxj/branches).
Learn more about [Supabase for Git ↗︎](https://supabase.com/docs/guides/platform/branching).

author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-28T08:13:20Z
body: frontend/packages/prompt-test result:

visit: https://cloud.langfuse.com/project/cm8ii4o5o03fpad078o638g1d/datasets/cm99wciaz070ead07rgjr88ou/runs/cmba8bx0300rnad07eptk93v1

| run items length |
| --- |
| 3 |
<!-- Sticky Pull Request Comment -->
author_association: CONTRIBUTOR
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-05-28T09:05:23Z
body: ## PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

<table>
<tr><td>⏱️&nbsp;<strong>Estimated effort to review</strong>: 4 🔵🔵🔵🔵⚪</td></tr>
<tr><td>🧪&nbsp;<strong>No relevant tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>Security concerns</strong><br><br>

<strong>GitHub App credentials exposure:</strong><br> The code in processRepositoryAnalysis.ts (lines 69-76) directly accesses GitHub App credentials from environment variables. While not directly exposing them in the code, there's no validation that these credentials exist before attempting to use them, which could lead to runtime errors that might expose partial credential information in logs. Consider adding proper validation for these credentials before attempting to use them.</td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R77-R80'><strong>Error Handling</strong></a>

The error handling in the repository analysis job could be improved. Errors are collected but not properly reported back to the user, and the job continues even when critical operations fail.
</summary>

```typescript

// Get repository tree
const { data: tree } = await octokit.git.getTree({
  owner: repositoryOwner,
```

</details>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R273-R282'><strong>API Key Security</strong></a>

The OpenAI API key validation doesn't check for empty strings or malformed keys, only for null/undefined values, which could lead to runtime errors.
</summary>

```typescript
const validateOpenAIKey = (): string => {
  const openAIApiKey = process.env['OPENAI_API_KEY']

  if (!openAIApiKey) {
    throw new Error(
      'Valid OpenAI API key is required for generating embeddings',
    )
  }

  return openAIApiKey
```

</details>

<details><summary><a href='https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R398-R405'><strong>Resource Management</strong></a>

The code processes documents in batches but doesn't implement rate limiting or backoff strategies when API calls fail, which could lead to quota exhaustion.
</summary>

```typescript
const BATCH_SIZE = 10
let totalProcessed = 0

for (let i = 0; i < documents.length; i += BATCH_SIZE) {
  const batch = documents.slice(i, i + BATCH_SIZE)
  const processed = await processBatch(batch, embeddings, supabase, errors)
  totalProcessed += processed
}
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: qodo-free-for-open-source-projects[bot]
created_at: 2025-05-28T09:06:12Z
body: ## PR Code Suggestions ✨

<!-- a6859d6 -->

Explore these optional code suggestions:

<table><thead><tr><td><strong>Category</strong></td><td align=left><strong>Suggestion&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </strong></td><td align=center><strong>Impact</strong></td></tr><tbody><tr><td rowspan=1>Possible issue</td>
<td>



<details><summary>Fix parameter type</summary>

___


**The recursive parameter for getTree should be a boolean value (true), not a <br>string ('true'). This could cause unexpected behavior or API errors when <br>fetching the repository tree.**

[frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts [78-84]](https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R78-R84)

```diff
 // Get repository tree
 const { data: tree } = await octokit.git.getTree({
   owner: repositoryOwner,
   repo: repositoryName,
   tree_sha: 'HEAD',
-  recursive: 'true',
+  recursive: true,
 })
```

- [ ] **Apply / Chat** <!-- /improve --apply_suggestion=0 -->

<details><summary>Suggestion importance[1-10]: 8</summary>

__

Why: This fixes a type error where `recursive` should be a boolean `true` instead of string `'true'`. The GitHub API expects a boolean value and this could cause API errors or unexpected behavior.


</details></details></td><td align=center>Medium

</td></tr><tr><td rowspan=1>Learned<br>best practice</td>
<td>



<details><summary>Validate function input parameters<!-- not_implemented --></summary>

___


**Add input validation at the beginning of the function to ensure all required <br>payload properties are present and properly formatted. This prevents potential <br>runtime errors when accessing undefined properties later in the function.**

[frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts [523-538]](https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R523-R538)

```diff
 export const processRepositoryAnalysis = async (
   payload: RepositoryAnalysisPayload,
 ): Promise<{ processedFiles: number; errors: string[] }> => {
+  if (!payload || !payload.projectId || !payload.repositoryId || 
+      !payload.repositoryOwner || !payload.repositoryName || 
+      !payload.installationId || !payload.organizationId) {
+    throw new Error('Invalid payload: missing required properties');
+  }
+  
   const {
     projectId,
     repositoryId,
     repositoryOwner,
     repositoryName,
     installationId,
     organizationId,
   } = payload;
 
   logger.log(`🔍 Scanning repository: ${repositoryOwner}/${repositoryName}`);
 
   const repositoryFullName = `${repositoryOwner}/${repositoryName}`;
   const errors: string[] = [];
```

- [ ] **Apply / Chat** <!-- /improve --apply_suggestion=1 -->

<details><summary>Suggestion importance[1-10]: 6</summary>

__

Why: 
Relevant best practice - Validate input parameters at the beginning of functions to prevent runtime errors from invalid, undefined, or improperly formatted data.

</details></details></td><td align=center>Low

</td></tr><tr><td rowspan=1>General</td>
<td>



<details><summary>✅ 
Improve null value handling
</summary>

___

<details><summary><b>Suggestion Impact:</b></summary>The commit implemented the suggestion's intent by adding explicit handling for null and undefined values, converting them to empty strings instead of 'null' or 'undefined' strings

code diff:

```diff
-      // Ensure text is always a string
-      const documentContent = typeof text === 'string' ? text : String(text)
+      // Ensure text is always a valid string
+      const documentContent = (() => {
+        if (text === null || text === undefined) {
+          return ''
+        }
+        return typeof text === 'string' ? text : String(text)
+      })()
```

</details>


___


**When converting non-string text to a string, consider handling null and <br>undefined values explicitly to avoid "null" or "undefined" strings being stored <br>in the database. This could affect search quality and user experience.**

[frontend/packages/jobs/src/functions/processRepositoryAnalysis.ts [348-349]](https://github.com/liam-hq/liam/pull/1803/files#diff-e10df7643f8fb2dfda43a26f3f0ffcb020531a853396e5c2eaff06310c565cc8R348-R349)

```diff
-// Ensure text is always a string
-const documentContent = typeof text === 'string' ? text : String(text)
+// Ensure text is always a valid string
+const documentContent = text == null ? '' : typeof text === 'string' ? text : String(text)
```

`[Suggestion processed]`

<details><summary>Suggestion importance[1-10]: 5</summary>

__

Why: This prevents storing "null" or "undefined" strings as document content by converting them to empty strings instead, which improves data quality for the vector store.


</details></details></td><td align=center>Low

</td></tr>
<tr><td align="center" colspan="2">

- [ ] Update <!-- /improve_multi --more_suggestions=true -->

</td><td></td></tr></tbody></table>
author_association: CONTRIBUTOR