Repository: oppia/oppia

Issue:
Background python processes doesn't exit when e2e test quits
**Describe the bug**
When an e2e test closes on its own (either on success or on failure), after the process is stopped, it is seen that some app engine processes stilll remain in the background. If this is done multiple times  (serially, one after the other), these processes accumulate. 
These quickly become quite a memory hog (at one point all of these were using around 12GB RAM!) wiith no direct way of closing them, other than to find their process ID and run `kill <pid>`.

Currently, I am working around this by running `for pid in $(ps -ef | awk '/oppia_tools/ {print $2}'); do kill -9 $pid; done`, which I found online that can kill all these processes.

Screenshots:

Initially, before e2e tests:
![image](https://user-images.githubusercontent.com/22347970/80024614-faa58d00-84fc-11ea-91cf-ead2eb83a87b.png)

After one test:
![image](https://user-images.githubusercontent.com/22347970/80024667-0d1fc680-84fd-11ea-9d38-2e2a67ea13e1.png)

After two tests:
![image](https://user-images.githubusercontent.com/22347970/80024725-21fc5a00-84fd-11ea-9afc-1c59dd13d734.png)

**To Reproduce**
Steps to reproduce the behavior:
 1. Check running python processes by typing `ps aux | grep python` in terminal.
 1. Run any e2e test using `python -m scripts.run_e2e_tests --suite=<suiteName>`
 2. Wait for it to finish, then again type `ps aux | grep python` in terminal.

**Observed behavior**
App engine processes remain in background.

**Expected behavior**
All these should be closed.

**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: Ubuntu 18.04
 - Browser: Chrome
 - Version: 81



Background python processes doesn't exit when e2e test quits
**Describe the bug**
When an e2e test closes on its own (either on success or on failure), after the process is stopped, it is seen that some app engine processes stilll remain in the background. If this is done multiple times  (serially, one after the other), these processes accumulate. 
These quickly become quite a memory hog (at one point all of these were using around 12GB RAM!) wiith no direct way of closing them, other than to find their process ID and run `kill <pid>`.

Currently, I am working around this by running `for pid in $(ps -ef | awk '/oppia_tools/ {print $2}'); do kill -9 $pid; done`, which I found online that can kill all these processes.

Screenshots:

Initially, before e2e tests:
![image](https://user-images.githubusercontent.com/22347970/80024614-faa58d00-84fc-11ea-91cf-ead2eb83a87b.png)

After one test:
![image](https://user-images.githubusercontent.com/22347970/80024667-0d1fc680-84fd-11ea-9d38-2e2a67ea13e1.png)

After two tests:
![image](https://user-images.githubusercontent.com/22347970/80024725-21fc5a00-84fd-11ea-9afc-1c59dd13d734.png)

**To Reproduce**
Steps to reproduce the behavior:
 1. Check running python processes by typing `ps aux | grep python` in terminal.
 1. Run any e2e test using `python -m scripts.run_e2e_tests --suite=<suiteName>`
 2. Wait for it to finish, then again type `ps aux | grep python` in terminal.

**Observed behavior**
App engine processes remain in background.

**Expected behavior**
All these should be closed.

**Desktop (please complete the following information; delete this section if the issue does not arise on desktop):**
 - OS: Ubuntu 18.04
 - Browser: Chrome
 - Version: 81





Discussion:

Resolving patch:
diff --git a/scripts/common.py b/scripts/common.py
--- a/scripts/common.py
+++ b/scripts/common.py
@@ -46,9 +46,10 @@
 RELEASE_BRANCH_NAME_PREFIX = 'release-'
 CURR_DIR = os.path.abspath(os.getcwd())
 OPPIA_TOOLS_DIR = os.path.join(CURR_DIR, os.pardir, 'oppia_tools')
+OPPIA_TOOLS_DIR_ABS_PATH = os.path.abspath(OPPIA_TOOLS_DIR)
 THIRD_PARTY_DIR = os.path.join(CURR_DIR, 'third_party')
 GOOGLE_APP_ENGINE_HOME = os.path.join(
-    OPPIA_TOOLS_DIR, 'google_appengine_1.9.67', 'google_appengine')
+    OPPIA_TOOLS_DIR_ABS_PATH, 'google_appengine_1.9.67', 'google_appengine')
 GOOGLE_CLOUD_SDK_HOME = os.path.join(
     OPPIA_TOOLS_DIR, 'google-cloud-sdk-251.0.0', 'google-cloud-sdk')
 NODE_PATH = os.path.join(OPPIA_TOOLS_DIR, 'node-%s' % NODE_VERSION)


Test patch:
diff --git a/scripts/run_e2e_tests.py b/scripts/run_e2e_tests.py
--- a/scripts/run_e2e_tests.py
+++ b/scripts/run_e2e_tests.py
@@ -150,13 +150,13 @@ def ensure_screenshots_dir_is_removed():
 
 def cleanup():
     """Kill the running subprocesses and server fired in this program."""
-    dev_appserver_path = '%s/dev_appserver.py' % common.GOOGLE_APP_ENGINE_HOME
+    google_app_engine_path = '%s/' % common.GOOGLE_APP_ENGINE_HOME
     webdriver_download_path = '%s/downloads' % WEBDRIVER_HOME_PATH
     if common.is_windows_os():
         # In windows system, the java command line will use absolute path.
         webdriver_download_path = os.path.abspath(webdriver_download_path)
     processes_to_kill = [
-        '.*%s.*' % re.escape(dev_appserver_path),
+        '.*%s.*' % re.escape(google_app_engine_path),
         '.*%s.*' % re.escape(webdriver_download_path)
     ]
     for p in SUBPROCESSES:
diff --git a/scripts/run_e2e_tests_test.py b/scripts/run_e2e_tests_test.py
--- a/scripts/run_e2e_tests_test.py
+++ b/scripts/run_e2e_tests_test.py
@@ -143,12 +143,12 @@ def mock_is_windows_os():
         subprocess_swap = self.swap(run_e2e_tests, 'SUBPROCESSES', [])
 
 
-        dev_appserver_path = '%s/dev_appserver.py' % (
+        google_app_engine_path = '%s/' % (
             common.GOOGLE_APP_ENGINE_HOME)
         webdriver_download_path = '%s/downloads' % (
             run_e2e_tests.WEBDRIVER_HOME_PATH)
         process_pattern = [
-            ('.*%s.*' % re.escape(dev_appserver_path),),
+            ('.*%s.*' % re.escape(google_app_engine_path),),
             ('.*%s.*' % re.escape(webdriver_download_path),)
         ]
         def mock_kill_process_based_on_regex(unused_regex):
@@ -187,12 +187,12 @@ def mock_is_windows_os():
 
         subprocess_swap = self.swap(run_e2e_tests, 'SUBPROCESSES', [])
 
-        dev_appserver_path = '%s/dev_appserver.py' % (
+        google_app_engine_path = '%s/' % (
             common.GOOGLE_APP_ENGINE_HOME)
         webdriver_download_path = '%s/downloads' % (
             run_e2e_tests.WEBDRIVER_HOME_PATH)
         process_pattern = [
-            ('.*%s.*' % re.escape(dev_appserver_path),),
+            ('.*%s.*' % re.escape(google_app_engine_path),),
             ('.*%s.*' % re.escape(webdriver_download_path),)
         ]
         expected_pattern = process_pattern[:]