Repository: Baldinof/roadrunner-bundle
Pull request #169: Fix a race condition when a kernel reboot is triggered
Author: login: mikael-titinovskii
Created: 2025-05-07T12:24:31Z | Merged: 2025-05-12T10:33:04Z

A race condition is observed when a kernel reboot is triggered leading for some requests to not experience a service reset.

This leads for example to Client B having session data of Client A

### How to reproduce

1. Checkout https://github.com/mikael-titinovskii/sf-rr-bug-demo  on branch main
2. run chmod +x test.sh
3. open 3 terminal windows
4. on terminal 1 run rr serve
5. on terminal 2 run test.sh 1
6. on terminal 3 run test.sh 2

### What to look for 

![ezgif-36328d87dc3d49](https://github.com/user-attachments/assets/0ae1a2c4-bb75-4d2f-a8e7-326f4d602619)

terminal 3 window will indicate that it has session from terminal 2
a json will appear, for example
```
{"session_id":"e2qq7ggu9hlmme96c93kir5dhk","pid":84487,"cr":"2","cs":"1","match":"no ********"} 
```
from time to time cr - client from request will not match cs - client from session 

another experiment you can comment out the block in TestController 
```
        // reset the session
        if (rand(0,10) === 10){
            $session->clear();
            $session->invalidate();
            $session->getFlashBag()->clear();
            $session->migrate(true);
        }

``` 

now the session change is permanent for client 2 

### Other observations: 
- adding more "clients" will make the event less frequent
- increasing the number of workers, for ex 20, will make the event less frequent
- on  > 0.2 rps  the bug is barely noticeable 

### Issue source

i notice that \Symfony\Component\HttpKernel\Kernel::shutdown flips resetServices to false 

\Symfony\Component\HttpKernel\Kernel::handle sets resetServices to true then has that var is overturned to false 

### The fix proposal: 

I've added this fix to reset the services before a reboot

to test the effects, switch to branch ``` with-fix ``` and run composer install and run the procedure again




Diff:
diff --git a/src/Worker/GrpcInvoker.php b/src/Worker/GrpcInvoker.php
index 4a06c02..b802ddb 100644
--- a/src/Worker/GrpcInvoker.php
+++ b/src/Worker/GrpcInvoker.php
@@ -79,6 +79,22 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+            }
+
             $this->kernel->reboot(null);
             /** @var GrpcDependencies */
             $deps = $this->kernel->getContainer()->get(GrpcDependencies::class);
diff --git a/src/Worker/HttpWorker.php b/src/Worker/HttpWorker.php
index 4d19dcb..78c0ffe 100644
--- a/src/Worker/HttpWorker.php
+++ b/src/Worker/HttpWorker.php
@@ -15,6 +15,7 @@
 use Symfony\Component\HttpFoundation\Response;
 use Symfony\Component\HttpKernel\KernelInterface;
 use Symfony\Component\HttpKernel\RebootableInterface;
+use Symfony\Contracts\Service\ResetInterface;
 
 use function Baldinof\RoadRunnerBundle\consumes;
 
@@ -144,6 +145,22 @@ public function start(): void
                 break;
             } finally {
                 if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+                    try {
+                        if ($this->kernel->getContainer()->has('services_resetter')) {
+                            /** @var ResetInterface $resetter */
+                            $resetter = $this->kernel->getContainer()->get('services_resetter');
+                            $resetter->reset();
+                        }
+                    } catch (\Throwable $e) {
+                        $this->logger->error(
+                            \sprintf(
+                                'An error occurred when resetting services: %s',
+                                $e->getMessage()
+                            ),
+                            ['exception' => $e]
+                        );
+                    }
+
                     $this->kernel->reboot(null);
                     /** @var HttpDependencies */
                     $deps = $this->kernel->getContainer()->get(HttpDependencies::class);
diff --git a/tests/Worker/GrpcWorkerTest.php b/tests/Worker/GrpcWorkerTest.php
index f874609..db35ad5 100644
--- a/tests/Worker/GrpcWorkerTest.php
+++ b/tests/Worker/GrpcWorkerTest.php
@@ -237,4 +237,37 @@ public function test_it_reboot_the_kernel_according_to_the_strategy()
         $this->kernel->reboot(null)->shouldHaveBeenCalled();
         $this->assertTrue($rebootedEventFired);
     }
+
+    public function test_it_resets_services_before_reboot(): void
+    {
+        $this->responder = function () use (&$terminated) {
+            yield 'hello';
+
+            $terminated = true;
+        };
+
+        $this->roadrunnerWorker->respond(Argument::any()); // Allow respond() calls
+
+        $this->requests->push(new Payload('', Json::encode([
+            'service' => FakeGrpcService::NAME,
+            'method' => 'fake',
+            'context' => [],
+        ])));
+
+        $resetter = $this->prophesize(\Symfony\Contracts\Service\ResetInterface::class);
+        $resetter->reset()->shouldBeCalledOnce();
+        $this->container->set('services_resetter', $resetter->reveal());
+
+        self::$rebootStrategyReturns = true;
+
+        $rebootedEventFired = false;
+        $this->eventDispatcher->addListener(WorkerKernelRebootedEvent::class, function () use (&$rebootedEventFired) {
+            $rebootedEventFired = true;
+        });
+
+        $this->worker->start();
+
+        $this->kernel->reboot(null)->shouldHaveBeenCalled();
+        $this->assertTrue($rebootedEventFired);
+    }
 }
diff --git a/tests/Worker/HttpWorkerTest.php b/tests/Worker/HttpWorkerTest.php
index c15adce..05387a7 100644
--- a/tests/Worker/HttpWorkerTest.php
+++ b/tests/Worker/HttpWorkerTest.php
@@ -267,4 +267,25 @@ public function test_it_reboot_the_kernel_according_to_the_strategy()
         $this->kernel->reboot(null)->shouldHaveBeenCalled();
         $this->assertTrue($rebootedEventFired);
     }
+
+    public function test_it_resets_services_before_reboot(): void
+    {
+        $this->responder = function () use (&$terminated) {
+            yield new Response('hello', 200, []);
+
+            $terminated = true;
+        };
+
+        $this->httpFoundationWorker->respond(Argument::any()); // Allow respond() calls
+
+        $this->requests->push(Request::create('http://example.org/'));
+
+        $resetter = $this->prophesize(\Symfony\Contracts\Service\ResetInterface::class);
+        $resetter->reset()->shouldBeCalledOnce();
+        $this->container->set('services_resetter', $resetter->reveal());
+
+        self::$rebootStrategyReturns = true;
+        $this->worker->start();
+        $this->kernel->reboot(null)->shouldHaveBeenCalled();
+    }
 }


Reviews:
user: login: Nyholm
body: Thank you for this PR. 

I like that you add tests. I do see the other similar Symfony Runtimes (ie Franken PHP) does not use `Kernel::reboot()`, they just do another `Kernel::boot()` to make sure we reset services. 

After a few changes, I am happy with this PR. 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T13:13:23Z
user: login: mikael-titinovskii
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:00:42Z
user: login: mikael-titinovskii
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:00:55Z
user: login: mikael-titinovskii
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:01:00Z
user: login: Nyholm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:36:42Z
user: login: mikael-titinovskii
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:41:09Z
user: login: Nyholm
body: Lovely. 

With this we can be sure the services are always reset and I cannot find any side-effects. 

Thank you!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-07T15:55:59Z
user: login: Baldinof
body: Finally got some time to try it, it's great, thanks a lot!

And as always, thank you @Nyholm for the help!
state: APPROVED
author_association: OWNER
submitted_at: 2025-05-12T10:32:55Z

Review comments:
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
+            }
+
             $this->kernel->reboot(null);
             /** @var GrpcDependencies */
             $deps = $this->kernel->getContainer()->get(GrpcDependencies::class);
 
             $this->dependencies = $deps;
             $this->dependencies->getEventDispatcher()->dispatch(new WorkerKernelRebootedEvent());
-        } elseif ($this->kernel->getContainer()->has('services_resetter')) {
-            /** @var ResetInterface $resetter */
-            $resetter = $this->kernel->getContainer()->get('services_resetter');
-            $resetter->reset();
path: src/Worker/GrpcInvoker.php
user: login: Nyholm
body: We need to keep this for non-rebootable Kernels and for when we should not reboot. 
created_at: 2025-05-07T13:10:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
path: src/Worker/GrpcInvoker.php
user: login: Nyholm
body: I suggest not to rethrow. We restart for a reason. The reason could be because we know the current `Container` is invalid. So we should not handle it twice. 

I think a log message is enough


created_at: 2025-05-07T13:12:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -144,6 +145,24 @@ public function start(): void
                 break;
             } finally {
                 if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+                    try {
+                        if ($this->kernel->getContainer()->has('services_resetter')) {
+                            /** @var ResetInterface $resetter */
+                            $resetter = $this->kernel->getContainer()->get('services_resetter');
+                            $resetter->reset();
+                        }
+                    } catch (\Throwable $e) {
+                        $this->logger->error(
+                            \sprintf(
+                                'An error occurred when resetting services: %s',
+                                $e->getMessage()
+                            ),
+                            ['exception' => $e]
+                        );
+
+                        throw $e;
path: src/Worker/HttpWorker.php
user: login: Nyholm
body: Same
created_at: 2025-05-07T13:12:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
+            }
+
             $this->kernel->reboot(null);
             /** @var GrpcDependencies */
             $deps = $this->kernel->getContainer()->get(GrpcDependencies::class);
 
             $this->dependencies = $deps;
             $this->dependencies->getEventDispatcher()->dispatch(new WorkerKernelRebootedEvent());
-        } elseif ($this->kernel->getContainer()->has('services_resetter')) {
-            /** @var ResetInterface $resetter */
-            $resetter = $this->kernel->getContainer()->get('services_resetter');
-            $resetter->reset();
path: src/Worker/GrpcInvoker.php
user: login: mikael-titinovskii
body: brought it back , add to top
created_at: 2025-05-07T15:00:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
path: src/Worker/GrpcInvoker.php
user: login: mikael-titinovskii
body: sure, removed throw
created_at: 2025-05-07T15:00:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -144,6 +145,24 @@ public function start(): void
                 break;
             } finally {
                 if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+                    try {
+                        if ($this->kernel->getContainer()->has('services_resetter')) {
+                            /** @var ResetInterface $resetter */
+                            $resetter = $this->kernel->getContainer()->get('services_resetter');
+                            $resetter->reset();
+                        }
+                    } catch (\Throwable $e) {
+                        $this->logger->error(
+                            \sprintf(
+                                'An error occurred when resetting services: %s',
+                                $e->getMessage()
+                            ),
+                            ['exception' => $e]
+                        );
+
+                        throw $e;
path: src/Worker/HttpWorker.php
user: login: mikael-titinovskii
body: same
created_at: 2025-05-07T15:00:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
+            }
+
             $this->kernel->reboot(null);
             /** @var GrpcDependencies */
             $deps = $this->kernel->getContainer()->get(GrpcDependencies::class);
 
             $this->dependencies = $deps;
             $this->dependencies->getEventDispatcher()->dispatch(new WorkerKernelRebootedEvent());
-        } elseif ($this->kernel->getContainer()->has('services_resetter')) {
-            /** @var ResetInterface $resetter */
-            $resetter = $this->kernel->getContainer()->get('services_resetter');
-            $resetter->reset();
path: src/Worker/GrpcInvoker.php
user: login: Nyholm
body: I like that and it is probably fine. But technically it has a side-effect. What if the resetter on old line 91 threw an exception. Now it wont bubble up. 

I fear we need to have our old code (which indeed looks like we duplicated something) to make sure we dont change existing behaviour. 
created_at: 2025-05-07T15:36:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -79,16 +79,30 @@ public function invokeFinalize(): void
         }
 
         if ($this->kernel instanceof RebootableInterface && $this->dependencies->getKernelRebootStrategy()->shouldReboot()) {
+            try {
+                if ($this->kernel->getContainer()->has('services_resetter')) {
+                    /** @var ResetInterface $resetter */
+                    $resetter = $this->kernel->getContainer()->get('services_resetter');
+                    $resetter->reset();
+                }
+            } catch (\Throwable $e) {
+                $this->logger->error(
+                    \sprintf(
+                        'An error occurred when resetting services: %s',
+                        $e->getMessage()
+                    ),
+                    ['exception' => $e]
+                );
+
+                throw $e;
+            }
+
             $this->kernel->reboot(null);
             /** @var GrpcDependencies */
             $deps = $this->kernel->getContainer()->get(GrpcDependencies::class);
 
             $this->dependencies = $deps;
             $this->dependencies->getEventDispatcher()->dispatch(new WorkerKernelRebootedEvent());
-        } elseif ($this->kernel->getContainer()->has('services_resetter')) {
-            /** @var ResetInterface $resetter */
-            $resetter = $this->kernel->getContainer()->get('services_resetter');
-            $resetter->reset();
path: src/Worker/GrpcInvoker.php
user: login: mikael-titinovskii
body: sounds reasonable 
created_at: 2025-05-07T15:41:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: Baldinof
created_at: 2025-05-07T12:56:55Z
body: Thanks a lot for the PR and the reproducer! I'll try to have a look today or tomorrow
author_association: OWNER
user: login: Nyholm
created_at: 2025-05-11T12:13:46Z
body: @Baldinof Can you please approve CI to run?
author_association: CONTRIBUTOR