Repository: oppia/oppia

Issue:
Backend test flake: Incomplete coverage of create_topological_sort_of_all_services
Example: https://github.com/oppia/oppia/actions/runs/3521701249/jobs/5903838607#step:8:1749

Log:

```
INCOMPLETE COVERAGE (99.0%): scripts.create_topological_sort_of_all_services_test
Name                                                 Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------------------------
scripts/create_topological_sort_of_all_services.py      72      0     40      1    99%   139->138
------------------------------------------------------------------------------------------------
TOTAL                                                   72      0     40      1    99%
```

The topological sort script produces a different order each time, so I suspect that this non-determinism is at the root of this flake. From the code it doesn't look like we actually take advantage of any of the set's special properties, so we could just swap it out for a list:

https://github.com/oppia/oppia/blob/09adf9fd37f1c93564515fc736fb9c657b645dda/scripts/create_topological_sort_of_all_services.py#L73


Discussion:
Has recurred in:

* https://github.com/oppia/oppia/actions/runs/4491467658/jobs/8201613367 (on develop)
* https://github.com/oppia/oppia/pull/17803#issuecomment-1499821807
* https://github.com/oppia/oppia/actions/runs/4633930250/jobs/8199623909
I think we've eliminated most of the non-determinism in the script, except for the filesystem accesses. Perhaps there's a race condition with temporary files being created there?
This test is failing consistently on PRs but not on develop, so a recent change to develop is likely the cause. Possible causes:

* https://github.com/oppia/oppia/pull/17738
* https://github.com/oppia/oppia/pull/17770
@U8NWXD My local enviroment do not show this error after reset previous pr https://github.com/oppia/oppia/commit/d2d6ea70a100dd40aa02df9b4c8620b16fc4d7c0.
Git bisect seems not working. I tried to revert these two PR:
https://github.com/oppia/oppia/pull/17738
https://github.com/oppia/oppia/pull/17770
But still shows the same error. See log in this PR: https://github.com/oppia/oppia/pull/18025
@462702985 just to confirm, you tried reverting our hacky fix (ignoring the coverage error) and still didn't see the error locally. That means that when you take a version of the code that fails on CI, it passes locally. Is that correct?

If so, then I'm not surprised that `git bisect` didn't work--that depends on being able to reproduce the error locally. You might still be able do a manual bisect on CI like this:

1. Check the GitHub Actions log to find a version of the code from before the flake occurred. Open a new PR with that version of the code (you can just revert all commits merged since the identified version of the code). If all goes well, the backend tests should pass (you can delete all the other workflows to speed things up).
2. Now you have a known good code state (the early version where the tests pass) and a known bad code state (the version just before our hacky fix). You can do a binary search (running tests on CI) on the commits between these to states to identify the problematic PR

Note that there's a possibility that (1) will fail because the flake started happening because of some change to GitHub's infrastructure, not a code change.
@U8NWXD Yes, I will try this method.

Resolving patch:
diff --git a/scripts/create_topological_sort_of_all_services.py b/scripts/create_topological_sort_of_all_services.py
--- a/scripts/create_topological_sort_of_all_services.py
+++ b/scripts/create_topological_sort_of_all_services.py
@@ -134,7 +134,7 @@ def main() -> None:
     adj_list, nodes_list = make_graph()
     visit_stack: List[str] = []
     topo_sort_stack: List[str] = []
-
+    nodes_list.sort()
     for unchecked_node in nodes_list:
         if unchecked_node not in visit_stack:
             dfs(unchecked_node, topo_sort_stack, adj_list, visit_stack)


Test patch:
diff --git a/scripts/backend_tests_incomplete_coverage.txt b/scripts/backend_tests_incomplete_coverage.txt
--- a/scripts/backend_tests_incomplete_coverage.txt
+++ b/scripts/backend_tests_incomplete_coverage.txt
@@ -92,7 +92,6 @@ main_test
 scripts.build_test
 scripts.common_test
 scripts.concurrent_task_utils_test
-scripts.create_topological_sort_of_all_services_test
 scripts.install_python_prod_dependencies_test
 scripts.install_third_party_test
 scripts.linters.html_linter_test