Repository: ydb-platform/ydb
Pull request #16605: Fix workflow update muted_ya.txt 
Author: login: naspirato
Labels: name: not-for-changelog
Created: 2025-04-01T13:40:59Z | Merged: 2025-04-01T14:27:54Z

### Changelog entry <!-- a user-readable short description of the changes that goes to CHANGELOG.md and Release Notes -->

**Conclusion**
The changes made in this pull request improve the error handling, update the workflow schedules, remove deprecated steps, and add new steps to enhance data collection and monitoring. These updates are beneficial for maintaining the repository's quality and efficiency. Consider the suggestions provided to further improve the implementation.

- added handle_github_errors
- added analytics update call to `.github/workflows/update_muted_ya.yml` for data freshnes during wf work
- changed .github/workflows/collect_analytics.yml cron time frequency (1h-> 2h)
- excludet collecting  of analytics `flaky_tests_history_n_runs.py --runs=50`, reason -  deprecated
- added step `Comment PR` for [create_issues_for_muted_tests.yml](https://github.com/ydb-platform/ydb/pull/16605/files#diff-c67c6018dc69b2e9c71e5bd5625c8c7b03bb3f3117a515757a46a26121d9ded4)


### Changelog category <!-- remove all except one -->

* Not for changelog (changelog entry is not required)

### Description for reviewers <!-- (optional) description for those who read this PR -->

...



Diff:
diff --git a/.github/scripts/tests/update_mute_issues.py b/.github/scripts/tests/update_mute_issues.py
index 7ae751272e48..8387f40a0f33 100755
--- a/.github/scripts/tests/update_mute_issues.py
+++ b/.github/scripts/tests/update_mute_issues.py
@@ -19,6 +19,20 @@
 # admin:org
 # project
 
+
+def handle_github_errors(response):
+    if 'errors' in response:
+        for error in response['errors']:
+            if error['type'] == 'INSUFFICIENT_SCOPES':
+                print("Error: Insufficient Scopes")
+                print("Message:", error['message'])
+                raise Exception("Insufficient scopes. Please update your token's scopes.")
+            # Handle other types of errors if necessary
+            else:
+                print("Unknown error type:", error.get('type', 'No type'))
+                print("Message:", error.get('message', 'No message available'))
+                raise Exception("GraphQL Error: " + error.get('message', 'Unknown error'))
+
 def run_query(query, variables=None):
     GITHUB_TOKEN = os.environ["GITHUB_TOKEN"]
     HEADERS = {"Authorization": f"Bearer {GITHUB_TOKEN}", "Content-Type": "application/json"}
@@ -26,6 +40,7 @@ def run_query(query, variables=None):
         'https://api.github.com/graphql', json={'query': query, 'variables': variables}, headers=HEADERS
     )
     if request.status_code == 200:
+        handle_github_errors(request.json())
         return request.json()
     else:
         raise Exception(f"Query failed to run by returning code of {request.status_code}. {query}")
diff --git a/.github/workflows/collect_analytics.yml b/.github/workflows/collect_analytics.yml
index 3e92d0feb6f9..06a40ecba0e6 100644
--- a/.github/workflows/collect_analytics.yml
+++ b/.github/workflows/collect_analytics.yml
@@ -1,7 +1,7 @@
 name: Collect-analytics-run
 on:
   schedule:
-    - cron: "0 * * * *"  # Every 1 h
+    - cron: "0 1-23/2 * * *" #каждые 2 часа в 0 минут, начиная с 1:00 и заканчивая 23:00.
   workflow_dispatch:
     inputs:
       commit_sha:
@@ -43,11 +43,7 @@ jobs:
     - name: Collect test history data with window 10 run release-asan for main
       continue-on-error: true
       run: python3 .github/scripts/analytics/flaky_tests_history_n_runs.py --runs=10 --build_type=release-asan
-    - name: Collect test history data with window 50 run relwithdebinfo for main
-      continue-on-error: true
-      run: python3 .github/scripts/analytics/flaky_tests_history_n_runs.py --runs=50 
-    - name: Collect test history data with window 50 run release-asan for main
-      run: python3 .github/scripts/analytics/flaky_tests_history_n_runs.py --runs=50 --build_type=release-asan
+
       
 
 
diff --git a/.github/workflows/create_issues_for_muted_tests.yml b/.github/workflows/create_issues_for_muted_tests.yml
index a1e7f9271b48..c9aeef7e8728 100644
--- a/.github/workflows/create_issues_for_muted_tests.yml
+++ b/.github/workflows/create_issues_for_muted_tests.yml
@@ -55,4 +55,25 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ env.GH_TOKEN }}
         run: |
-            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=issues_info.txt
\ No newline at end of file
+            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=${{env.GITHUB_WORKSPACE}}/issues_info.txt
+      
+      - name: Comment PR
+        uses: actions/github-script@v7
+        
+        with:
+          github-token: ${{ env.GH_TOKEN }}
+          script: |
+            const fs = require('fs');
+            const path = require('path');
+            
+            const workflowUrl = `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}`;
+            const filePath = path.join(process.env.GITHUB_WORKSPACE, 'issues_info.txt');
+            const bodyText = fs.readFileSync(filePath, 'utf8');
+            const completeBody = `Collected in workflow [#${{ github.run_number }}](${workflowUrl})\n\n${bodyText}`;
+      
+            github.rest.issues.createComment({
+              issue_number: ${{ github.event.pull_request.number || github.event.inputs.pr_number  }},
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: completeBody
+            });
diff --git a/.github/workflows/update_muted_ya.yml b/.github/workflows/update_muted_ya.yml
index 32d8e9505159..99f40222b6c6 100644
--- a/.github/workflows/update_muted_ya.yml
+++ b/.github/workflows/update_muted_ya.yml
@@ -25,14 +25,23 @@ jobs:
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install ydb[yc] PyGithub
+          pip install ydb[yc] PyGithub codeowners pandas
 
       - name: Setup ydb access
         uses: ./.github/actions/setup_ci_ydb_service_account_key_file_credentials
         with:
           ci_ydb_service_account_key_file_credentials: ${{ secrets.CI_YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS }}
+    
+      - name: Collect test history data with window 1 days relwithdebinfo for ${{ env.BASE_BRANCH }}
+        run: python3 .github/scripts/analytics/flaky_tests_history.py --days-window=1 --branch=${{ env.BASE_BRANCH }}
+      
+      - name: Update muted and not muted tests in DB for ${{ env.BASE_BRANCH }}
+        run: python3 .github/scripts/tests/get_muted_tests.py upload_muted_tests --branch=${{ env.BASE_BRANCH }}
+      
+      - name: Update test monitor (how long tests in state) for ${{ env.BASE_BRANCH }}
+        run: python3 .github/scripts/analytics/tests_monitor.py --branch=${{ env.BASE_BRANCH }}
 
-      - name: Update branch with base branch
+      - name: Update branch ${{ env.BRANCH_FOR_PR }}_${{ env.BASE_BRANCH }} with branch ${{ env.BASE_BRANCH }}
         run: |
           git config user.name YDBot
           git config user.email ydbot@ydb.tech
@@ -51,18 +60,18 @@ jobs:
       
           # Attempt to rebase BRANCH_FOR_PR onto BASE_BRANCH
           if ! git rebase origin/${{ env.BASE_BRANCH }} -X theirs; then
-            echo "Rebase failed, resetting branch to match BASE_BRANCH..."
-      
+            echo "Rebase failed, resetting branch to match ${{ env.BASE_BRANCH }}..."
+
             # Abort the rebase process
             git rebase --abort
       
-            # Reset the branch to BASE_BRANCH
+            echo "Reset branch ${{ env.BRANCH_FOR_PR }}_${{ env.BASE_BRANCH }} to origin/${{ env.BASE_BRANCH }}"
             git reset --hard origin/${{ env.BASE_BRANCH }}
             
           fi
           git push origin ${{ env.BRANCH_FOR_PR }}_${{ env.BASE_BRANCH }} --force
         
-      - name: Run the script
+      - name: Run script create_new_muted_ya.py
         run: |
           .github/scripts/tests/create_new_muted_ya.py update_muted_ya
 
@@ -82,6 +91,7 @@ jobs:
           fi
 
       - name: Collect PR description
+        if: env.changes == 'true'
         id: pr_description
         run: |
           PR_BODY=''
@@ -129,17 +139,16 @@ jobs:
           git commit -m "Update muted YA file"
 
       - name: Push changes
+        if: env.changes == 'true'
         uses: ad-m/github-push-action@v0.8.0
         with:
           github_token: ${{ secrets.YDBOT_TOKEN }}
           branch: ${{ env.BRANCH_FOR_PR }}_${{ env.BASE_BRANCH }}
           force: true
 
-      - name: Install PyGithub
-        run: |
-          pip install PyGithub
 
       - name: Create or update PR
+        if: env.changes == 'true'
         id: create_or_update_pr
         env:
           GITHUB_TOKEN: ${{ secrets.YDBOT_TOKEN }}
@@ -148,9 +157,8 @@ jobs:
 
         
       - name: Comment PR
-        continue-on-error: true
         uses: actions/github-script@v7
-        
+        if: env.changes == 'true'
         with:
           github-token: ${{ secrets.YDBOT_TOKEN }}
           script: |
@@ -176,6 +184,7 @@ jobs:
             });
 
       - name: Add reviewers
+        if: env.changes == 'true'
         uses: octokit/request-action@v2.x
         with:
           route: POST /repos/{owner}/{repo}/pulls/{pull_number}/requested_reviewers


Reviews:
user: login: iddqdex
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T14:05:15Z
user: login: naspirato
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T14:08:00Z
user: login: iddqdex
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-01T14:08:59Z

Review comments:
diff_hunk: @@ -55,4 +55,25 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ env.GH_TOKEN }}
         run: |
-            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=issues_info.txt
\ No newline at end of file
+            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=${{env.GITHUB_WORKSPACE}}/issues_info.txt
+      
+      - name: Comment PR
+        uses: actions/github-script@v7
+        
+        with:
+          github-token: ${{ env.GH_TOKEN }}
+          script: |
+            const fs = require('fs');
+            const path = require('path');
+            
+            const workflowUrl = `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}`;
+            const filePath = path.join(process.env.GITHUB_WORKSPACE, 'issues_info.txt');
+            const bodyText = fs.readFileSync(filePath, 'utf8');
+            const completeBody = `Collected in workflow [#${{ github.run_number }}](${workflowUrl})\n\n${bodyText}`;
+      
+            github.rest.issues.createComment({
+              issue_number: ${{ github.event.pull_request.number || github.event.inputs.pr_number  }},
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: completeBody
+            });
path: .github/workflows/create_issues_for_muted_tests.yml
user: login: iddqdex
body: Давай пустую строчку сделаем в конце
```suggestion
            });

```
created_at: 2025-04-01T14:05:14Z
author_association: COLLABORATOR
diff_hunk: @@ -55,4 +55,25 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ env.GH_TOKEN }}
         run: |
-            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=issues_info.txt
\ No newline at end of file
+            python .github/scripts/create_or_update_pr.py append_pr_body --pr_number=${{ github.event.pull_request.number || github.event.inputs.pr_number  }} --body=${{env.GITHUB_WORKSPACE}}/issues_info.txt
+      
+      - name: Comment PR
+        uses: actions/github-script@v7
+        
+        with:
+          github-token: ${{ env.GH_TOKEN }}
+          script: |
+            const fs = require('fs');
+            const path = require('path');
+            
+            const workflowUrl = `https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}`;
+            const filePath = path.join(process.env.GITHUB_WORKSPACE, 'issues_info.txt');
+            const bodyText = fs.readFileSync(filePath, 'utf8');
+            const completeBody = `Collected in workflow [#${{ github.run_number }}](${workflowUrl})\n\n${bodyText}`;
+      
+            github.rest.issues.createComment({
+              issue_number: ${{ github.event.pull_request.number || github.event.inputs.pr_number  }},
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: completeBody
+            });
path: .github/workflows/create_issues_for_muted_tests.yml
user: login: naspirato
body: done
created_at: 2025-04-01T14:08:00Z
author_association: COLLABORATOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-01T13:42:06Z
body: <!-- status pr=16605, validate PR description status -->
:green_circle: `2025-04-01 13:52:28 UTC` The validation of the Pull Request description is successful.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T13:44:44Z
body: <!-- status pr=16605, preset=release-asan, run=39031 -->
:white_circle: `2025-04-01 13:44:43 UTC` Pre-commit [check](https://github.com/ydb-platform/ydb/actions/runs/14197202074/job/39775085554) **linux-x86_64-release-asan** for 1ca4200b8d12bf5bcfbdaff068f316f4842c7bf5 has started.
:white_circle: `2025-04-01 13:44:57 UTC` Artifacts will be uploaded [here](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64-asan/index.html)
:white_circle: `2025-04-01 13:47:08 UTC` ya make is running...
:green_circle: `2025-04-01 13:47:14 UTC` Tests successful.

[Test history](https://nebius.testmo.net/automation/runs/view/135075) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64-asan/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64-asan/try_1/test_bloat/tree_map.html)
| TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED<sup>[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here")</sup> |
| ---: | ---: | ---: | ---: | ---: | ---: |
| [0](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64-asan/try_1/ya-test.html) | 0 | 0 | 0 | 0 | 0 |

:green_circle: `2025-04-01 13:47:20 UTC` Build successful.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T13:44:46Z
body: <!-- status pr=16605, preset=relwithdebinfo, run=39031 -->
:white_circle: `2025-04-01 13:44:46 UTC` Pre-commit [check](https://github.com/ydb-platform/ydb/actions/runs/14197202074/job/39775085537) **linux-x86_64-relwithdebinfo** for 1ca4200b8d12bf5bcfbdaff068f316f4842c7bf5 has started.
:white_circle: `2025-04-01 13:45:00 UTC` Artifacts will be uploaded [here](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64/index.html)
:white_circle: `2025-04-01 13:47:16 UTC` ya make is running...
:green_circle: `2025-04-01 13:47:22 UTC` Tests successful.

[Test history](https://nebius.testmo.net/automation/runs/view/135076) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64/try_1/test_bloat/tree_map.html)
| TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED<sup>[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here")</sup> |
| ---: | ---: | ---: | ---: | ---: | ---: |
| [0](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197202074/ya-x86-64/try_1/ya-test.html) | 0 | 0 | 0 | 0 | 0 |

:green_circle: `2025-04-01 13:47:28 UTC` Build successful.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T14:09:12Z
body: <!-- status pr=16605, preset=relwithdebinfo, run=39042 -->
:white_circle: `2025-04-01 14:09:12 UTC` Pre-commit [check](https://github.com/ydb-platform/ydb/actions/runs/14197829303/job/39777181380) **linux-x86_64-relwithdebinfo** for 145e485e1e4d64d2aab595550702a72f193ab648 has started.
:white_circle: `2025-04-01 14:09:33 UTC` Artifacts will be uploaded [here](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197829303/ya-x86-64/index.html)
:white_circle: `2025-04-01 14:12:16 UTC` ya make is running...
:green_circle: `2025-04-01 14:12:21 UTC` Tests successful.

[Test history](https://nebius.testmo.net/automation/runs/view/135109) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197829303/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197829303/ya-x86-64/try_1/test_bloat/tree_map.html)
| TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED<sup>[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here")</sup> |
| ---: | ---: | ---: | ---: | ---: | ---: |
| [0](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14197829303/ya-x86-64/try_1/ya-test.html) | 0 | 0 | 0 | 0 | 0 |

:green_circle: `2025-04-01 14:12:27 UTC` Build successful.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T14:22:22Z
body: <!-- status pr=16605, preset=relwithdebinfo, run=39048 -->
:white_circle: `2025-04-01 14:22:21 UTC` Pre-commit [check](https://github.com/ydb-platform/ydb/actions/runs/14198140869/job/39778229815) **linux-x86_64-relwithdebinfo** for 145e485e1e4d64d2aab595550702a72f193ab648 has started.
:white_circle: `2025-04-01 14:22:26 UTC` Artifacts will be uploaded [here](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64/index.html)
:white_circle: `2025-04-01 14:24:54 UTC` ya make is running...
:green_circle: `2025-04-01 14:25:00 UTC` Tests successful.

[Test history](https://nebius.testmo.net/automation/runs/view/135125) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64/try_1/test_bloat/tree_map.html)
| TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED<sup>[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here")</sup> |
| ---: | ---: | ---: | ---: | ---: | ---: |
| [0](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64/try_1/ya-test.html) | 0 | 0 | 0 | 0 | 0 |

:green_circle: `2025-04-01 14:25:06 UTC` Build successful.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-04-01T14:24:52Z
body: <!-- status pr=16605, preset=release-asan, run=39048 -->
:white_circle: `2025-04-01 14:24:52 UTC` Pre-commit [check](https://github.com/ydb-platform/ydb/actions/runs/14198140869/job/39778229825) **linux-x86_64-release-asan** for 145e485e1e4d64d2aab595550702a72f193ab648 has started.
:white_circle: `2025-04-01 14:25:06 UTC` Artifacts will be uploaded [here](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64-asan/index.html)
:white_circle: `2025-04-01 14:27:29 UTC` ya make is running...
:green_circle: `2025-04-01 14:27:34 UTC` Tests successful.

[Test history](https://nebius.testmo.net/automation/runs/view/135126) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64-asan/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64-asan/try_1/test_bloat/tree_map.html)
| TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED<sup>[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here")</sup> |
| ---: | ---: | ---: | ---: | ---: | ---: |
| [0](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/14198140869/ya-x86-64-asan/try_1/ya-test.html) | 0 | 0 | 0 | 0 | 0 |

:green_circle: `2025-04-01 14:27:40 UTC` Build successful.
author_association: NONE