Stop old runtime when switching runtimes