Repository: phusion/passenger
Pull request #2594: Fix postcondition assertion error for old routing algorithm
Author: login: CamJN
Created: 2025-03-05T21:12:26Z | Merged: 2025-03-13T22:01:41Z

The assert at https://github.com/phusion/passenger/blob/1bf36f17820d3b7f46019bc488283e42b224fe8b/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp#L74 was triggered by the process selected by the old routing algo. Since that process is not routable, it does not uphold the invariant documented here: https://github.com/phusion/passenger/blob/1bf36f17820d3b7f46019bc488283e42b224fe8b/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp#L54-L55 so we need to return NULL if the process is not routable, or change the assert & invariant.

 I'm trying the NULL approach first.


Diff:
diff --git a/src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp b/src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp
index 21ac00fa94..a52f4c9d01 100644
--- a/src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp
+++ b/src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp
@@ -171,7 +171,7 @@ Group::findBestProcessPreferringStickySessionId(unsigned int id) const {
  *
  * If there is no process that can be routed to, then returns nullptr.
  *
- * @post result != nullptr || result.canBeRoutedTo()
+ * @post result == nullptr || result.canBeRoutedTo()
  */
 Process *
 Group::findBestProcess(const ProcessList &processes) const {
diff --git a/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp b/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp
index 56c50b3514..28947055a3 100644
--- a/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp
+++ b/src/agent/Core/ApplicationPool/Group/SessionManagement.cpp
@@ -67,42 +67,57 @@ Group::route(const Options &options) const {
 		if (options.stickySessionId == 0) {
 			if (OXT_LIKELY(useNewRouting())) {
 				process = findBestProcess(enabledProcesses);
+				if (process != nullptr) {
+					assert(process->canBeRoutedTo());
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, true);
+				}
             } else {
 				process = findEnabledProcessWithLowestBusyness();
-			}
-			if (process != nullptr) {
-				assert(process->canBeRoutedTo());
-				return RouteResult(process);
-			} else {
-				return RouteResult(NULL, true);
+				if (process && process->canBeRoutedTo()) {
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, process == nullptr);
+				}
 			}
 		} else {
 			if (OXT_LIKELY(useNewRouting())) {
 				process = findBestProcessPreferringStickySessionId(options.stickySessionId);
+				if (process != nullptr) {
+					if (process->canBeRoutedTo()) {
+						return RouteResult(process);
+					} else {
+						return RouteResult(NULL, false);
+					}
+				} else {
+					return RouteResult(NULL, true);
+				}
 			}else{
 				process = findProcessWithStickySessionIdOrLowestBusyness(options.stickySessionId);
-			}
-			if (process != nullptr) {
-				if (process->canBeRoutedTo()) {
+				if (process && process->canBeRoutedTo()) {
 					return RouteResult(process);
 				} else {
-					return RouteResult(NULL, false);
+					return RouteResult(NULL, process == nullptr);
 				}
-			} else {
-				return RouteResult(NULL, true);
 			}
 		}
 	} else {
 		if (OXT_LIKELY(useNewRouting())) {
 			process = findBestProcess(disablingProcesses);
+			if (process != nullptr) {
+				assert(process->canBeRoutedTo());
+				return RouteResult(process);
+			} else {
+				return RouteResult(NULL, true);
+			}
 		} else {
 			process = findProcessWithLowestBusyness(disablingProcesses);
-		}
-		if (process != nullptr) {
-			assert(process->canBeRoutedTo());
-			return RouteResult(process);
-		} else {
-			return RouteResult(NULL, true);
+			if (process && process->canBeRoutedTo()) {
+				return RouteResult(process);
+			} else {
+				return RouteResult(NULL, process == nullptr);
+			}
 		}
 	}
 }


Reviews:
user: login: FooBarWidget
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T05:26:28Z
user: login: FooBarWidget
body: Ok looks good so far.

Have you tested this manually as well? Test cases:
- When all processes are at full capacity, test that the request gets queued, and waits until one process is ready to accept the next request.
   - Also test with stable-6.0 and verify that it crashes. This verifies that you got the test case right.
   - In the PR branch, test with both the old routing and the new routing algorithm. Verify that both work.
 - Test that sticky sessions still work. When there are two processes, test that a request with a sticky session always ends up at the same process, even if that process is busier and newer than the other.
    - Also test this in combination with the sticky process being at full capacity. Test that the request gets properly queued, and then unqueued when the sticky process becomes ready.
    - Test all combinations with both new and old routing algorithm.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T06:13:35Z
user: login: copilot-pull-request-reviewer[bot]
body: 

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.



state: COMMENTED
author_association: NONE
submitted_at: 2025-03-10T18:23:17Z
user: login: FooBarWidget
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-13T20:44:49Z

Review comments:
diff_hunk: @@ -81,8 +81,13 @@ Group::findProcessWithStickySessionIdOrLowestBusyness(unsigned int id) const {
 
 	if (leastBusyProcessIndex == -1) {
path: src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp
user: login: FooBarWidget
body: I found a bug in the post-condition description for `findBestProcess()`:

```c++
@post result != nullptr || result.canBeRoutedTo()
```

This should be `@post result == nullptr || result.canBeRoutedTo()`, so can you fix that too?
created_at: 2025-03-06T05:23:06Z
author_association: MEMBER
diff_hunk: @@ -81,8 +81,13 @@ Group::findProcessWithStickySessionIdOrLowestBusyness(unsigned int id) const {
 
 	if (leastBusyProcessIndex == -1) {
path: src/agent/Core/ApplicationPool/Group/ProcessListManagement.cpp
user: login: FooBarWidget
body: Good work noticing the contract description for route() which says we guarantee `process->canBeRoutedTo()`, I totally missed that.

Yes it is important to keep adhering to that contract, but I think we don't need to change the functions for the old routing algorithm that much. It's easier to change the code in route() in order to adapt to the differences in contract between the old routing functions and the new ones. Doing so should be less risky because we don't change the previous behavior so much. We've had so many issues lately so I think it's wise to be conservative.

To summarize:

- `findBestProcess()` guarantees `result == nullptr || result.canBeRoutedTo()`
- `findBestProcessPreferringStickySessionId()` guarantees nothing.
- `findEnabledProcessWithLowestBusyness()` and `findProcessWithLowestBusyness()` guarantee `if process is non-empty: result != nullptr` (but not `canBeRoutedTo()`).
- `findProcessWithStickySessionIdOrLowestBusyness()` guarantees nothing.

I suggest code like this which makes it easy to visually scan whether we're using the old code not not:

```c++
if (OXT_LIKELY(useNewRouting())) {
	process = findBestProcess(enabledProcesses);
	if (process != nullptr) {
		assert(process->canBeRoutedTo());
		return RouteResult(process);
	} else {
		return RouteResult(NULL, true);
	}
} else {
	process = findEnabledProcessWithLowestBusyness();
	if (process->canBeRoutedTo()) {
		return RouteResult(process);
	} else {
		return RouteResult(NULL, true);
	}
}
```
created_at: 2025-03-06T05:26:23Z
author_association: MEMBER
diff_hunk: @@ -67,42 +67,57 @@ Group::route(const Options &options) const {
 		if (options.stickySessionId == 0) {
 			if (OXT_LIKELY(useNewRouting())) {
 				process = findBestProcess(enabledProcesses);
+				if (process != nullptr) {
+					assert(process->canBeRoutedTo());
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, true);
+				}
             } else {
 				process = findEnabledProcessWithLowestBusyness();
-			}
-			if (process != nullptr) {
-				assert(process->canBeRoutedTo());
-				return RouteResult(process);
-			} else {
-				return RouteResult(NULL, true);
+				if (process && process->canBeRoutedTo()) {
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, true);
+				}
 			}
 		} else {
 			if (OXT_LIKELY(useNewRouting())) {
 				process = findBestProcessPreferringStickySessionId(options.stickySessionId);
+				if (process != nullptr) {
+					if (process->canBeRoutedTo()) {
+						return RouteResult(process);
+					} else {
+						return RouteResult(NULL, false);
+					}
+				} else {
+					return RouteResult(NULL, true);
+				}
 			}else{
 				process = findProcessWithStickySessionIdOrLowestBusyness(options.stickySessionId);
-			}
-			if (process != nullptr) {
-				if (process->canBeRoutedTo()) {
+				if (process && process->canBeRoutedTo()) {
 					return RouteResult(process);
 				} else {
-					return RouteResult(NULL, false);
+					return RouteResult(NULL, process == nullptr);
 				}
-			} else {
-				return RouteResult(NULL, true);
 			}
 		}
 	} else {
 		if (OXT_LIKELY(useNewRouting())) {
 			process = findBestProcess(disablingProcesses);
+			if (process != nullptr) {
+				assert(process->canBeRoutedTo());
+				return RouteResult(process);
+			} else {
+				return RouteResult(NULL, true);
+			}
 		} else {
 			process = findProcessWithLowestBusyness(disablingProcesses);
-		}
-		if (process != nullptr) {
-			assert(process->canBeRoutedTo());
-			return RouteResult(process);
-		} else {
-			return RouteResult(NULL, true);
+			if (process && process->canBeRoutedTo()) {
+				return RouteResult(process);
+			} else {
+				return RouteResult(NULL, true);
path: src/agent/Core/ApplicationPool/Group/SessionManagement.cpp
user: login: FooBarWidget
body: `true` here should be `process == nullptr`.
created_at: 2025-03-07T06:05:31Z
author_association: MEMBER
diff_hunk: @@ -67,42 +67,57 @@ Group::route(const Options &options) const {
 		if (options.stickySessionId == 0) {
 			if (OXT_LIKELY(useNewRouting())) {
 				process = findBestProcess(enabledProcesses);
+				if (process != nullptr) {
+					assert(process->canBeRoutedTo());
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, true);
+				}
             } else {
 				process = findEnabledProcessWithLowestBusyness();
-			}
-			if (process != nullptr) {
-				assert(process->canBeRoutedTo());
-				return RouteResult(process);
-			} else {
-				return RouteResult(NULL, true);
+				if (process && process->canBeRoutedTo()) {
+					return RouteResult(process);
+				} else {
+					return RouteResult(NULL, true);
path: src/agent/Core/ApplicationPool/Group/SessionManagement.cpp
user: login: FooBarWidget
body: `process` should never be non-NULL here. But since we're being defensive, let's also change the `true` to `process == nullptr`.
created_at: 2025-03-07T06:11:18Z
author_association: MEMBER

PR comments:
user: login: FooBarWidget
created_at: 2025-03-13T06:16:14Z
body: How about manual tests? Tinco can help you with that.
author_association: MEMBER
user: login: CamJN
created_at: 2025-03-13T19:18:34Z
body: I manually tested these cases:

- When all processes are at full capacity, test that the request gets queued, and waits until one process is ready to accept the next request.
  - Also test with stable-6.0 and verify that it crashes. This verifies that you got the test case right.
  - In the PR branch, test with both the old routing and the new routing algorithm. Verify that both work.

6.0.26 with old routing crashes with `ab -n 1000 -c 12 -i 'http://127.0.0.1:3000/'` none of the other cases crash (new routing + 6.0.26, old routing + bugfix branch, new routing + bugfix bbranch).

author_association: MEMBER
user: login: CamJN
created_at: 2025-03-13T19:51:56Z
body: For sticky sessions I did the following:

- started passenger (bugfix branch) with sticky sessions & old routing enabled & 2 app processes
- ran ab against passenger with enough concurrency and requests to get the sticky session cookies of both processes
- checked passenger-status to see which process is older & to check the request count of both processes for later
- sent a tonne of traffic to passenger with the younger process' sticky session cookie set
- used passenger status to make sure none of the requests went to the other process

I found that all of the requests that I directed to a particular process went to that process no matter how busy or how young or old.

I repeated the process with new routing and got the same results.
author_association: MEMBER