Repository: bw-99/MAP
Pull request #12: init : CI
Author: login: seongyeon01
Labels: name: ci/cd
Created: 2025-01-22T11:50:43Z | Merged: 2025-03-06T12:52:13Z

- create ci action

[Tech Spec](https://www.notion.so/TechSpec-CI-Actions-195d2f288baf80d083b2d17463cb8c7b)



Diff:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
new file mode 100644
index 00000000..a9271a84
--- /dev/null
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,32 @@
+name: ci
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  test-code:
+    runs-on: ubuntu-latest
+    steps:
+      # 리포지토리 코드 가져오기
+      - uses: actions/checkout@v2
+
+      # Docker 이미지 가져오기
+      - name: Pull Docker Image
+        run: docker pull bb1702/onepiece:main
+
+      # Docker 컨테이너 실행
+      - name: Run Docker Container
+        run: |
+          docker run -dit --name con --shm-size=20g -v ${{ github.workspace }}:/workspace bb1702/onepiece:main
+
+      # pytest 실행
+      - name: Run Tests
+        run: docker exec con bash -c "cd /workspace && pytest util/test_file.py --root_path=onepiece_rag"
+
+
+      # Docker 컨테이너 정리
+      - name: Cleanup Docker Container
+        if: always()
+        run: docker rm -f con
diff --git a/Dockerfile b/Dockerfile
index afc94bd2..3084d6d9 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -11,5 +11,6 @@ COPY requirements.txt /workspace/
 RUN pip3 install --no-cache-dir -r requirements.txt
 RUN pip3 install flask
 RUN pip3 install flask_socketio
+RUN pip3 install pytest
 
 ENTRYPOINT [ "/bin/bash" ]
diff --git a/conftest.py b/conftest.py
new file mode 100644
index 00000000..55cb4c37
--- /dev/null
+++ b/conftest.py
@@ -0,0 +1,9 @@
+import pytest
+from pathlib import Path
+
+def pytest_addoption(parser):
+    parser.addoption("--root_path", action="store", default=".", help="Root directory path")
+
+@pytest.fixture(scope="session")
+def root_path(pytestconfig):
+    return Path(pytestconfig.getoption("--root_path")).resolve()
diff --git a/util/test_file.py b/util/test_file.py
new file mode 100644
index 00000000..a0361813
--- /dev/null
+++ b/util/test_file.py
@@ -0,0 +1,41 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+
+def load_viztree(root_path, file_name="create_final_viztree.parquet"):
+    file_path = root_path / "output" / file_name
+    assert file_path.exists(), f"Parquet file not found: {file_path}"
+
+    df = pd.read_parquet(file_path)
+    print("Unique types in Parquet file:", df["type"].unique())  
+    return df
+
+def load_entity_types(root_path):
+    yaml_path = root_path / "settings.yaml"
+    assert yaml_path.exists(), f"Settings file not found: {yaml_path}"
+
+    with yaml_path.open("r", encoding="utf-8") as file:
+        config = yaml.safe_load(file)
+        entity_types = config.get("entity_extraction", {}).get("entity_types", [])
+    
+    assert entity_types, "No entity types found in settings.yaml"
+    
+    print("Loaded entity types:", entity_types)
+    return entity_types
+
+def test_parent_no_negative_one_for_entity_types(root_path):
+    df = load_viztree(root_path)
+    entity_types = load_entity_types(root_path)
+
+    print("Filtering DataFrame with entity types:", entity_types)  
+    
+    filtered_df = df[df["type"].isin(entity_types)]
+    
+    print("Filtered DataFrame shape:", filtered_df.shape)  
+
+    if filtered_df.empty:
+        print("No relevant data found after filtering. Skipping test.") 
+        pytest.skip("No relevant data to test in the Parquet file.")
+
+    assert -1 not in filtered_df["parent"].values, "Found -1 in the 'parent' column for specified entity types."


Reviews:
user: login: bw-99
state: APPROVED
author_association: OWNER
submitted_at: 2025-02-05T13:56:56Z
user: login: Genie051
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-06T09:05:40Z
user: login: bw-99
body: TechSpec 작성해서 개요 적는 란에 링크 첨부해주십쇼.
아래 링크 참고.
https://github.com/bw-99/MAP/pull/17#issue-2811003436
state: CHANGES_REQUESTED
author_association: OWNER
submitted_at: 2025-02-07T10:06:03Z
user: login: bw-99
state: CHANGES_REQUESTED
author_association: OWNER
submitted_at: 2025-02-10T02:01:17Z
user: login: bw-99
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-10T02:01:35Z
user: login: TransferHee
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-10T03:45:49Z
user: login: TransferHee
body: CI Review
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-10T03:46:45Z
user: login: Genie051
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-02-10T09:23:09Z
user: login: TransferHee
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-02-23T06:18:42Z
user: login: bw-99
state: COMMENTED
author_association: OWNER
submitted_at: 2025-02-25T01:15:25Z
user: login: bw-99
state: COMMENTED
author_association: OWNER
submitted_at: 2025-03-04T12:14:55Z
user: login: bw-99
state: APPROVED
author_association: OWNER
submitted_at: 2025-03-06T12:41:41Z

Review comments:
path: .github/workflows/ci.yml
user: login: Genie051
body: name: Install pytest에서 
pytest가 이미 설치되어 있는지 확인하고 이미 설치된 상태면 중복 설치 안해도 되게?
created_at: 2025-02-06T09:02:59Z
author_association: COLLABORATOR
path: onepiece_rag/test/test_file.py
user: login: Genie051
body: load_parquet_file 할 때 파일 존재 여부를 먼저 체크할 수 있게?
created_at: 2025-02-06T09:05:36Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+import pytest
+import pandas as pd
+import os
+
+def load_parquet_file():
+    """Load the parquet file for testing."""
+    file_path = "output/create_final_viztree.parquet"  # 데이터 파일 경로
+    if not os.path.exists(file_path):
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def test_parent_no_negative_one_for_doc_or_entity():
+    """Test if 'parent' column does not contain -1 when 'type' is 'doc' or 'entity'."""
+    df = load_parquet_file()
+    
+    # 'type'이 'doc' 또는 'entity'인 경우 필터링
+    filtered_df = df[df["type"].isin(["doc", "entity"])]
path: onepiece_rag/test/test_file.py
user: login: bw-99
body: 이 type은 entity의 type인데,
entity의 type은 [여기서](https://github.com/bw-99/MAP/blob/9d0d265850ed875b80a59e4687bef88bed4fc9b8/onepiece_rag/settings.yaml#L93) 정의되기 때문에,
해당 파일에서 entity type 종류를 읽고, 필터링 하는게 좋을 것 같습니다.
(실제로 지금 entity type은 ("CONCEPT", "RESEARCH DOCUMENT", "ORGANIZATION")입니다)
created_at: 2025-02-10T02:00:51Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,27 @@
+import pytest
+import pandas as pd
+import os
+
+def load_parquet_file():
+    """Load the parquet file for testing."""
+    file_path = "output/create_final_viztree.parquet"  # 데이터 파일 경로
+    if not os.path.exists(file_path):
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def test_parent_no_negative_one_for_doc_or_entity():
+    """Test if 'parent' column does not contain -1 when 'type' is 'doc' or 'entity'."""
+    df = load_parquet_file()
+    
+    # 'type'이 'doc' 또는 'entity'인 경우 필터링
+    filtered_df = df[df["type"].isin(["doc", "entity"])]
path: onepiece_rag/test/test_file.py
user: login: bw-99
body: 수정하고 테크스펙에서도 수정해주세요~
created_at: 2025-02-10T02:01:35Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,27 @@
+import pytest
+import pandas as pd
+import os
+
+def load_parquet_file():
+    """Load the parquet file for testing."""
+    file_path = "output/create_final_viztree.parquet"  # 데이터 파일 경로
+    if not os.path.exists(file_path):
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def test_parent_no_negative_one_for_doc_or_entity():
+    """Test if 'parent' column does not contain -1 when 'type' is 'doc' or 'entity'."""
+    df = load_parquet_file()
+    
+    # 'type'이 'doc' 또는 'entity'인 경우 필터링
+    filtered_df = df[df["type"].isin(["doc", "entity"])]
+    
+    # 필터링된 데이터가 존재하는 경우만 테스트 진행
+    if not filtered_df.empty:
+        assert -1 not in filtered_df["parent"].values, "Found -1 in the 'parent' column for 'doc' or 'entity' types."
+
path: onepiece_rag/test/test_file.py
user: login: TransferHee
body: 필터링된 데이터가 존재하지 않을 때에도 해당 상황을 알 수 있도록 출력하게 추가해주세요!
created_at: 2025-02-10T03:45:49Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,53 @@
+import pytest
+import pandas as pd
+import os
+import yaml
+
+def load_parquet_file():
+    """Load the parquet file for testing."""
+    file_path = "output/create_final_viztree.parquet" 
+    if not os.path.exists(file_path):
+        pytest.fail(f"Parquet file not found: {file_path}")
path: onepiece_rag/test/test_file.py
user: login: bw-99
body: Path 객체를 활용해주세요. path 관련 util 함수를 많이 활용할 수 있어서 편합니다.
아래처럼 사용합니다 @seongyeon01 
`from pathlib import Path`
`file_path = Path("output/create_final_viztree.parquet")`
`file_path.exists()`

created_at: 2025-02-25T01:12:47Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,53 @@
+import pytest
+import pandas as pd
+import os
+import yaml
+
+def load_parquet_file():
+    """Load the parquet file for testing."""
+    file_path = "output/create_final_viztree.parquet" 
+    if not os.path.exists(file_path):
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+        print("Unique types in Parquet file:", df["type"].unique())  # Parquet 데이터 확인
+
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def load_entity_types():
+    yaml_path = "./settings.yaml" 
+    if not os.path.exists(yaml_path):
path: onepiece_rag/test/test_file.py
user: login: bw-99
body: 이것도 path 수정해주세요 @seongyeon01 
created_at: 2025-02-25T01:13:04Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,31 @@
+name: ci
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  test-code:
+    runs-on: ubuntu-latest
+    steps:
+      # 리포지토리 코드 가져오기
+      - uses: actions/checkout@v2
+
+      # Docker 이미지 가져오기
+      - name: Pull Docker Image
+        run: docker pull bb1702/onepiece:main
+
+      # Docker 컨테이너 실행
+      - name: Run Docker Container
+        run: |
+          docker run -dit --name con --shm-size=20g -v ${{ github.workspace }}:/workspace bb1702/onepiece:main
+
+      # pytest 실행
+      - name: Run Tests
+        run: docker exec con bash -c "cd /workspace/onepiece_rag && pytest -s test/test_file.py"
path: .github/workflows/ci.yml
user: login: bw-99
body: onepiece_rag 하드코딩 말고 테스트하는 path를 argument로 넣을 수 있는 형태가 되었음 해요.
가령 `docker exec con bash -c "pytest -m test.test_file --root_path onepiece_rag`
pytest 할 수 있는 Path를 동적으로 설정할 수 있어야 이후에 실험 시 편할 것 같아서요. @seongyeon01 
created_at: 2025-02-25T01:15:18Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+import os
+
+def change_directory(root_path):
+    try:
+        os.chdir(root_path)
+        print(f"Changed working directory to: {root_path}")
+    except Exception as e:
+        pytest.fail(f"Failed to change directory: {e}")
+
+def load_parquet_file(root_path):
+    file_path = root_path / "output/create_final_viztree.parquet"
+    if not file_path.exists():
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+        print("Unique types in Parquet file:", df["type"].unique()) 
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def load_entity_types(root_path):
+    yaml_path = root_path / "settings.yaml"
+    if not yaml_path.exists():
+        pytest.fail(f"Settings file not found: {yaml_path}")
+
+    try:
+        with yaml_path.open("r", encoding="utf-8") as file:
+            config = yaml.safe_load(file)
+            entity_types = config.get("entity_extraction", {}).get("entity_types", [])
+            
+            if not entity_types:
+                pytest.fail("No entity types found in settings.yaml")
+
+            print("Loaded entity types:", entity_types)
+            return entity_types
+    except Exception as e:
+        pytest.fail(f"Failed to load settings file: {e}")
+
+def test_parent_no_negative_one_for_entity_types(root_path):
+    change_directory(root_path)
+    df = load_parquet_file(root_path)
+    entity_types = load_entity_types(root_path)
+
+    print("Filtering DataFrame with entity types:", entity_types) 
+    
+    filtered_df = df[df["type"].isin(entity_types)]
+    
+    print("Filtered DataFrame shape:", filtered_df.shape)  
+    
+    if filtered_df.empty:
+        print("No relevant data found after filtering. Skipping test.") 
+        pytest.skip("No relevant data to test in the Parquet file.")
+
+    assert -1 not in filtered_df["parent"].values, "Found -1 in the 'parent' column for specified entity types."
path: util/test_file.py
user: login: bw-99
body: pytest.fail이나 assert 둘 중 하나로 통일해주세요 @seongyeon01 
created_at: 2025-03-04T12:04:29Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+import os
+
+def change_directory(root_path):
+    try:
+        os.chdir(root_path)
+        print(f"Changed working directory to: {root_path}")
+    except Exception as e:
+        pytest.fail(f"Failed to change directory: {e}")
+
+def load_parquet_file(root_path):
+    file_path = root_path / "output/create_final_viztree.parquet"
+    if not file_path.exists():
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+        print("Unique types in Parquet file:", df["type"].unique()) 
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
path: util/test_file.py
user: login: bw-99
body: 파일 존재 여부를 미리 확인하는데 try-except로 감싸는 이유가 따로 있나요? @seongyeon01 
created_at: 2025-03-04T12:06:10Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+import os
+
+def change_directory(root_path):
+    try:
+        os.chdir(root_path)
+        print(f"Changed working directory to: {root_path}")
+    except Exception as e:
+        pytest.fail(f"Failed to change directory: {e}")
+
+def load_parquet_file(root_path):
+    file_path = root_path / "output/create_final_viztree.parquet"
+    if not file_path.exists():
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+        print("Unique types in Parquet file:", df["type"].unique()) 
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def load_entity_types(root_path):
+    yaml_path = root_path / "settings.yaml"
+    if not yaml_path.exists():
+        pytest.fail(f"Settings file not found: {yaml_path}")
+
+    try:
+        with yaml_path.open("r", encoding="utf-8") as file:
path: util/test_file.py
user: login: bw-99
body: 여기도 위랑 비슷한데 따로 이유가 있는지? @seongyeon01 
created_at: 2025-03-04T12:07:15Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+import os
+
+def change_directory(root_path):
+    try:
+        os.chdir(root_path)
+        print(f"Changed working directory to: {root_path}")
+    except Exception as e:
+        pytest.fail(f"Failed to change directory: {e}")
+
+def load_parquet_file(root_path):
+    file_path = root_path / "output/create_final_viztree.parquet"
path: util/test_file.py
user: login: bw-99
body: 함수 이름이 적절치 않은 것 같습니다.
load_parquet_file일거면 읽어들일 파일명을 argument로 하나 더 받아서 읽어야할 것 같고,

create_final_viztree만 읽을거면 load_viztree 같은 이름이 더 적절하지 않을까 싶네요.
created_at: 2025-03-04T12:13:03Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,59 @@
+import pytest
+import pandas as pd
+from pathlib import Path
+import yaml
+import os
+
+def change_directory(root_path):
+    try:
+        os.chdir(root_path)
+        print(f"Changed working directory to: {root_path}")
+    except Exception as e:
+        pytest.fail(f"Failed to change directory: {e}")
+
+def load_parquet_file(root_path):
+    file_path = root_path / "output/create_final_viztree.parquet"
+    if not file_path.exists():
+        pytest.fail(f"Parquet file not found: {file_path}")
+
+    try:
+        df = pd.read_parquet(file_path)
+        print("Unique types in Parquet file:", df["type"].unique()) 
+    except Exception as e:
+        pytest.fail(f"Failed to load parquet file: {e}")
+    return df
+
+def load_entity_types(root_path):
+    yaml_path = root_path / "settings.yaml"
+    if not yaml_path.exists():
+        pytest.fail(f"Settings file not found: {yaml_path}")
+
+    try:
+        with yaml_path.open("r", encoding="utf-8") as file:
+            config = yaml.safe_load(file)
+            entity_types = config.get("entity_extraction", {}).get("entity_types", [])
+            
+            if not entity_types:
+                pytest.fail("No entity types found in settings.yaml")
+
+            print("Loaded entity types:", entity_types)
+            return entity_types
+    except Exception as e:
+        pytest.fail(f"Failed to load settings file: {e}")
+
+def test_parent_no_negative_one_for_entity_types(root_path):
+    change_directory(root_path)
+    df = load_parquet_file(root_path)
path: util/test_file.py
user: login: bw-99
body: load_parquet_file이 파일 읽을 때 root_path / viztree.parquet으로 읽던데, 그럼 change_directory로 path를 변경할 필요가 없는건 아닐까요
created_at: 2025-03-04T12:14:24Z
author_association: OWNER

PR comments:
user: login: bw-99
created_at: 2025-02-05T13:57:14Z
body: @TransferHee @Genie051 리뷰 좀
author_association: OWNER
user: login: seongyeon01
created_at: 2025-02-09T03:56:39Z
body: > TechSpec 작성해서 개요 적는 란에 링크 첨부해주십쇼. 아래 링크 참고. [#17 (comment)](https://github.com/bw-99/MAP/pull/17#issue-2811003436)

완료
author_association: COLLABORATOR
user: login: bw-99
created_at: 2025-03-06T13:00:55Z
body: (기록용) 현재 test는 아래 두 가지 문제 존재

1. docker image build하고 push하는 workflow 전에 pull 해서 test하는 workflow 먼저 실행됨 -> dependency 문제

2. main branch에 올라간 결과물에서 검사하는데, 문제는 결과물 자체는 무거워서 깃헙에 안올리고 있음

반영해서 담에 적절히 수정해야 될 것 같습니다.

(실패한 workflow link: https://github.com/bw-99/MAP/actions/runs/13699117623)

@seongyeon01 
author_association: OWNER
user: login: bw-99
created_at: 2025-03-07T10:45:22Z
body: api key github에 등록하고, workflow에 indexing 추가하기 (논문은 하나로만)
debugging_data 실행도 해야함
author_association: OWNER