Repository: dotnet/orleans

Issue #3106: Cannot delete/upgrade Orleans deployed to SF in timely fashion
Created: 2017-06-08T18:58:52Z
Steps to reproduce:

Unpack attached OrleansSF.zip
Set "MainTableStorage" in "ApplicationParameters/Development.xml" and "ClusterConnectionParameters" in "PublishProfiles/Development.xml"
Deploy to Azure SF cluster
Once deployed, try to delete it (or upgrade to other version)

OrleansSF.zip
The deletion or upgrade hangs then. In SF explorer, I can see the following warning:
Unhealthy event: SourceId='System.RAP', Property='IStatelessServiceInstance.CloseDuration', HealthState='Warning', ConsiderWarningAsError=false.

The only thing that helps is to kill the service processes on SF nodes.

Duplicate issue #3099: Orleans hangs whilst shutting down
Created: 2017-06-07T07:49:32Z
I am using TopShelf to host Orleans but when I come to shut it down, either via CTRL+C in the console, or via services when installed. Orleans never shuts down.
What happens is it seems to go through the motions but then hangs.  I just spotted a very similar issue here so double checked that my issue was different by stopping the installed service.  Below is the relevant parts of the log file at the point of the shutdown event.  Note I shut down Orlenas simply by stopping the service.  In the case of running it through VS [in console app] I just press CTRL+C and the same occurs.
To stop Orleans I simply call this.siloHost.ShutdownOrleansSilo();, although I have also tried this.siloHost.StopOrleansSilo(); to no avail.
Edit Note that if I call siloHost.Dispose it shuts down without any fuss. This is a temporary solution for me though as I expect it doesnt stop things gracefully!

[2017-06-07 07:43:04.244 GMT    93	INFO   	100294	SystemStatus	192.168.5.184:22222]	SystemStatus=ShuttingDown
[2017-06-07 07:43:04.244 GMT    93	INFO   	100417	Silo	192.168.5.184:22222]	Silo starting to Shutdown()
[2017-06-07 07:43:04.244 GMT    13	INFO   	100606	MembershipOracle	192.168.5.184:22222]	-ShutDown
[2017-06-07 07:43:04.260 GMT    10	INFO   	102927	Orleans.Runtime.ReminderService.LocalReminderService	192.168.5.184:22222]	Stopping Orleans.Runtime.ReminderService.LocalReminderService grain service
[2017-06-07 07:43:04.260 GMT    93	INFO   	100545	Catalog	192.168.5.184:22222]	DeactivateAllActivations.
[2017-06-07 07:43:04.275 GMT    93	INFO   	100541	Catalog	192.168.5.184:22222]	DeactivateActivations: total 1063 to shutdown, out of them 1063 promptly, 0 later when become idle and 0 are already being destroyed or invalid.
[2017-06-07 07:43:04.369 GMT     5	WARNING	100502	Catalog	192.168.5.184:22222]	UnregisterManyAsync 1 failed.
Exc level 0: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<UnregisterManyAsync>d__108.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<<QueueTask>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Catalog.<FinishDestroyActivations>d__75.MoveNext() [2017-06-07 07:43:04.369 GMT    77	WARNING	100502	Catalog	192.168.5.184:22222]	UnregisterManyAsync 101 failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__108.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Catalog.d__75.MoveNext()
[2017-06-07 07:43:04.369 GMT   106	WARNING	100502	Catalog	192.168.5.184:22222]	UnregisterManyAsync 101 failed.
Exc level 0: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<UnregisterManyAsync>d__108.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<<QueueTask>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Catalog.<FinishDestroyActivations>d__75.MoveNext() [2017-06-07 07:43:04.369 GMT    52	WARNING	100502	Catalog	192.168.5.184:22222]	UnregisterManyAsync 101 failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.d__108.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Orleans.Runtime.Catalog.d__75.MoveNext()
[2017-06-07 07:43:04.369 GMT    42	WARNING	100502	Catalog	192.168.5.184:22222]	UnregisterManyAsync 101 failed.
Exc level 0: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterOrPutInForwardList(IEnumerable1 addresses, UnregistrationCause cause, Int32 hopCount, Dictionary2& forward, List1 tasks, String context) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<UnregisterManyAsync>d__108.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Scheduler.SchedulerExtensions.<>c__DisplayClass1_0.<<QueueTask>b__0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.Catalog.<FinishDestroyActivations>d__75.MoveNext() [2017-06-07 07:43:04.385 GMT    93	INFO   	101302	Orleans.Messaging.Gateway	192.168.5.184:22222]	Recorded closed socket from endpoint 192.168.5.189:49772, client ID *cli/f39c3cb5.	 [2017-06-07 07:43:04.385 GMT    93	INFO   	101302	Orleans.Messaging.Gateway	192.168.5.184:22222]	Recorded closed socket from endpoint 192.168.5.186:49848, client ID *cli/1077f397.	 [2017-06-07 07:43:04.385 GMT    93	INFO   	101302	Orleans.Messaging.Gateway	192.168.5.184:22222]	Recorded closed socket from endpoint 192.168.5.187:49804, client ID *cli/caea6582.	 [2017-06-07 07:43:04.385 GMT    93	INFO   	101302	Orleans.Messaging.Gateway	192.168.5.184:22222]	Recorded closed socket from endpoint 192.168.0.127:55434, client ID *cli/969c71a0.	 [2017-06-07 07:43:04.400 GMT    93	INFO   	100413	Silo	192.168.5.184:22222]	Silo is Stopped()	 [2017-06-07 07:43:04.416 GMT    13	INFO   	100609	MembershipOracle	192.168.5.184:22222]	-KillMyself	 [2017-06-07 07:43:04.416 GMT    35	INFO   	100312	Runtime.Messaging.GatewaySender/GatewaySiloSender_3	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.416 GMT    35	INFO   	100328	Runtime.Messaging.GatewaySender/GatewaySiloSender_3	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.GatewaySender/GatewaySiloSender_3 that runs on managed thread 35	 [2017-06-07 07:43:04.416 GMT    12	INFO   	601201	Runtime.Scheduler.WorkerPoolThread/3	192.168.5.184:22222]	Log code 101201 occurred 5 additional times in the previous 15:16:45.4556572	 [2017-06-07 07:43:04.416 GMT     9	INFO   	601201	Runtime.Scheduler.WorkerPoolThread/0	192.168.5.184:22222]	Log code 101201 occurred 3 additional times in the previous 15:16:45.4556572	 [2017-06-07 07:43:04.416 GMT    12	INFO   	100328	Runtime.Scheduler.WorkerPoolThread/3	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/3 that runs on managed thread 12	 [2017-06-07 07:43:04.416 GMT     9	INFO   	100328	Runtime.Scheduler.WorkerPoolThread/0	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/0 that runs on managed thread 9	 [2017-06-07 07:43:04.416 GMT    13	INFO   	100328	Runtime.Scheduler.WorkerPoolThread/System.5	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/System.5 that runs on managed thread 13	 [2017-06-07 07:43:04.416 GMT    34	INFO   	100312	Runtime.Messaging.GatewaySender/GatewaySiloSender_2	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.416 GMT    34	INFO   	100328	Runtime.Messaging.GatewaySender/GatewaySiloSender_2	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.GatewaySender/GatewaySiloSender_2 that runs on managed thread 34	 [2017-06-07 07:43:04.416 GMT    32	INFO   	100312	Runtime.Messaging.GatewaySender/GatewaySiloSender_0	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.416 GMT    32	INFO   	100328	Runtime.Messaging.GatewaySender/GatewaySiloSender_0	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.GatewaySender/GatewaySiloSender_0 that runs on managed thread 32	 [2017-06-07 07:43:04.416 GMT    10	INFO   	100328	Runtime.Scheduler.WorkerPoolThread/1	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/1 that runs on managed thread 10	 [2017-06-07 07:43:04.416 GMT    33	INFO   	100312	Runtime.Messaging.GatewaySender/GatewaySiloSender_1	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.416 GMT    33	INFO   	100328	Runtime.Messaging.GatewaySender/GatewaySiloSender_1	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.GatewaySender/GatewaySiloSender_1 that runs on managed thread 33	 [2017-06-07 07:43:04.416 GMT    93	INFO   	101203	Scheduler.OrleansTaskScheduler	192.168.5.184:22222]	OrleansTaskScheduler.PrintStatistics(): RunQueue=0, WorkItems=13, Directory: [--System*WorkItemGroup:Name=,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-7:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914@S00000018] CurrentlyExecuting=null> --System*WorkItemGroup:Name=,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-2:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914@S00000019] CurrentlyExecuting=null> --System*WorkItemGroup:Name=/567A1EA9/00000000,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=198; Total processed=198; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-11:Queued=0; Detailed SchedulingContext=<[LowPrioritySystemTarget: S192.168.5.184:22222:234461914/567A1EA9/00000000@S00000000] CurrentlyExecuting=null> --System*WorkItemGroup:Name=Catalog,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=123707; Total processed=123707; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-12:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914Catalog@S0000000e] CurrentlyExecuting=Response S192.168.0.121:22222:234517236DirectoryService@S0000000a->S192.168.5.184:22222:234461914Catalog@S0000000e #601073: global::Orleans.Runtime.IRemoteGrainDirectory:LookupAsync()> --System*WorkItemGroup:Name=ClientObserverRegistrar,WorkGroupStatus=Running. Currently QueuedWorkItems=0; Total EnQueued=281; Total processed=280; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-8:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914ClientObserverRegistrar@S0000000d] CurrentlyExecuting=Transient Rejection (info: Silo S192.168.5.184:22222:234461914 is rejecting message: Request S192.168.5.184:22222:234461914ClientObserverRegistrar@S0000000d->S192.168.0.127:22222:234463135DirectoryService@S0000000a #58024: global::Orleans.Runtime.IRemoteGrainDirectory:RegisterAsync(). Reason = Target S192.168.0.127:22222:234463135 silo is known to be dead) Response S192.168.0.127:22222:234463135DirectoryService@S0000000a->S192.168.5.184:22222:234461914ClientObserverRegistrar@S0000000d #58024: global::Orleans.Runtime.IRemoteGrainDirectory:RegisterAsync()> --System*WorkItemGroup:Name=DeploymentLoadPublisherSystemTarget,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=220686; Total processed=220686; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-4:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914DeploymentLoadPublisherSystemTarget@S00000016] CurrentlyExecuting=Response S192.168.5.184:22222:234461914DeploymentLoadPublisherSystemTarget@S00000016->S192.168.5.184:22222:234461914DeploymentLoadPublisherSystemTarget@S00000016 #601734: global::Orleans.Runtime.IDeploymentLoadPublisher:UpdateRuntimeStatistics()> --System*WorkItemGroup:Name=DirectoryCacheValidator,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=7570; Total processed=7570; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-6:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914DirectoryCacheValidator@S0000000b] CurrentlyExecuting=Response S192.168.0.121:22222:234517236DirectoryCacheValidator@S0000000b->S192.168.5.184:22222:234461914DirectoryCacheValidator@S0000000b #601196: global::Orleans.Runtime.IRemoteGrainDirectory:LookUpMany()> --System*WorkItemGroup:Name=DirectoryService,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=13357; Total processed=13357; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-5:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914DirectoryService@S0000000a] CurrentlyExecuting=Request S192.168.0.121:22222:234517236Catalog@S0000000e->S192.168.5.184:22222:234461914DirectoryService@S0000000a #4051: global::Orleans.Runtime.IRemoteGrainDirectory:LookupAsync()> --System*WorkItemGroup:Name=MembershipOracle,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=8129; Total processed=8129; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-10:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914MembershipOracle@S0000000f] CurrentlyExecuting=Request S192.168.0.121:22222:234517236MembershipOracle@S0000000f->S192.168.5.184:22222:234461914MembershipOracle@S0000000f #4055: global::Orleans.Runtime.IMembershipService:SiloStatusChangeNotification()> --System*WorkItemGroup:Name=ProtocolGateway,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-3:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914ProtocolGateway@S0000001b] CurrentlyExecuting=null> --System*WorkItemGroup:Name=ProviderManagerSystemTarget,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=166267; Total processed=166267; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-13:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914ProviderManagerSystemTarget@S00000013] CurrentlyExecuting=Response S192.168.5.184:22222:234461914*grn/ACD5BA64/00000000+192.168.0.121:22222@234517236@087dd38e->S192.168.5.184:22222:234461914ProviderManagerSystemTarget@S00000013 #601742: global::OrleansDashboard.ISiloGrain:GetRuntimeStatistics()> --System*WorkItemGroup:Name=SiloControl,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=159986; Total processed=159986; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-1:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914SiloControl@S0000000c] CurrentlyExecuting=Request S192.168.5.184:22222:234461914*grn/7483D9D2/00000000@8dcc9fa9->S192.168.5.184:22222:234461914SiloControl@S0000000c #601739: global::Orleans.ISiloControl:GetSimpleGrainStatistics()> --System*WorkItemGroup:Name=TypeManagerId,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=56; Total processed=56; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-9:Queued=0; Detailed SchedulingContext=<[SystemTarget: S192.168.5.184:22222:234461914TypeManagerId@S00000011] CurrentlyExecuting=Request S192.168.0.121:22222:234517236TypeManagerId@S00000011->S192.168.5.184:22222:234461914TypeManagerId@S00000011 #3996: global::Orleans.Runtime.ISiloTypeManager:GetSiloTypeCodeMap()>]	 [2017-06-07 07:43:04.416 GMT    93	INFO   	100528	ActivationDirectory	192.168.5.184:22222]	ActivationDirectory.PrintActivationDirectory(): Size = 0, Directory: []	 [2017-06-07 07:43:04.432 GMT    11	ERROR  	101317	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! Directory.RegisterAsync S192.168.5.184:22222:234461914*cli/f39c3cb5@312c1135 failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() [2017-06-07 07:43:04.432 GMT    17	INFO   	100312	Runtime.Messaging.SiloMessageSender/PingSender	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.432 GMT   140	INFO   	100312	Runtime.Messaging.SiloMessageSender/AppMsgsSender_1	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.432 GMT    17	INFO   	100328	Runtime.Messaging.SiloMessageSender/PingSender	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/PingSender that runs on managed thread 17	 [2017-06-07 07:43:04.432 GMT   140	INFO   	100328	Runtime.Messaging.SiloMessageSender/AppMsgsSender_1	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/AppMsgsSender_1 that runs on managed thread 140	 [2017-06-07 07:43:04.432 GMT    11	ERROR  	101317	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! Directory.RegisterAsync S192.168.5.184:22222:234461914*cli/de3801b4@7b69c8c0 failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() [2017-06-07 07:43:04.432 GMT   112	INFO   	100312	Runtime.Messaging.SiloMessageSender/AppMsgsSender_0	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.432 GMT   112	INFO   	100328	Runtime.Messaging.SiloMessageSender/AppMsgsSender_0	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/AppMsgsSender_0 that runs on managed thread 112	 [2017-06-07 07:43:04.432 GMT   155	INFO   	100312	Runtime.Messaging.SiloMessageSender/AppMsgsSender_3	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.432 GMT   155	INFO   	100328	Runtime.Messaging.SiloMessageSender/AppMsgsSender_3	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/AppMsgsSender_3 that runs on managed thread 155	 [2017-06-07 07:43:04.432 GMT    11	ERROR  	101317	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! Directory.RegisterAsync S192.168.5.184:22222:234461914*cli/97802d8f@93c1e91e failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() [2017-06-07 07:43:04.432 GMT    18	INFO   	601016	Runtime.Messaging.SiloMessageSender/SystemSender	192.168.5.184:22222]	Log code 101016 occurred 105 additional times in the previous 15:05:16.3943287	 [2017-06-07 07:43:04.432 GMT    18	INFO   	100312	Runtime.Messaging.SiloMessageSender/SystemSender	192.168.5.184:22222]	Stop request processed	 [2017-06-07 07:43:04.432 GMT    18	INFO   	100328	Runtime.Messaging.SiloMessageSender/SystemSender	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.SiloMessageSender/SystemSender that runs on managed thread 18	 [2017-06-07 07:43:04.432 GMT    11	ERROR  	101317	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! Directory.RegisterAsync S192.168.5.184:22222:234461914*cli/caea6582@e0c21737 failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() [2017-06-07 07:43:04.432 GMT    93	INFO   	100703	WindowsPerfCountersStatistics	192.168.5.184:22222]	Stopping  Windows perf counter stats collection	 [2017-06-07 07:43:04.432 GMT    11	ERROR  	101317	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! Directory.RegisterAsync S192.168.5.184:22222:234461914*cli/1077f397@8040df4a failed.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() [2017-06-07 07:43:04.432 GMT    11	ERROR  	101319	ClientObserverRegistrar	192.168.5.184:22222]	!!!!!!!!!! OnClientRefreshTimer has thrown an exceptions.	 Exc level 0: System.InvalidOperationException: Grain directory is stopping at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription) at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.<RegisterAsync>d__103.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.OrleansTaskExtentions.<LogException>d__0.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Orleans.Runtime.ClientObserverRegistrar.<OnClientRefreshTimer>d__16.MoveNext() [2017-06-07 07:43:04.432 GMT    11	WARNING	101222	Scheduler.ClientObserverRegistrar.WorkItemGroup	192.168.5.184:22222]	Thread <Runtime.Scheduler.WorkerPoolThread/2, ManagedThreadId=11, Executing WorkItem=System*WorkItemGroup:Name=ClientObserverRegistrar,WorkGroupStatus=Running Executing for 00:00:00.0155840. .> is exiting work loop due to cancellation token. WorkItemGroup: System*WorkItemGroup:Name=ClientObserverRegistrar,WorkGroupStatus=Running, Have 0 work items in the queue.	 [2017-06-07 07:43:04.432 GMT    11	INFO   	100328	Runtime.Scheduler.WorkerPoolThread/2	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Scheduler.WorkerPoolThread/2 that runs on managed thread 11	 [2017-06-07 07:43:04.432 GMT    93	INFO   	100704	SiloLogStatistics	192.168.5.184:22222]	Statistics: ^^^ App.Requests.Latency.Average.Millis=48.000 App.Requests.LatencyHistogram.Millis=[0.0512:0.1023]=120, [0.1024:0.2047]=1965, [0.2048:0.4095]=19653, [0.4096:0.8191]=22834, [0.8192:1.6383]=18441, [1.6384:3.2767]=154123, [3.2768:6.5535]=75644, [6.5536:13.1071]=31398, [13.1072:26.2143]=157760, [26.2144:52.4287]=55163, [52.4288:104.8575]=14569, [104.8576:209.7151]=17669, [209.7152:419.4303]=17742, [419.4304:838.8607]=11249, [838.8608:1677.7215]=2803, [1677.7216:3355.4431]=410, [3355.4432:6710.8863]=62, [6710.8864:13421.7727]=28, [13421.7728:26843.5455]=21, [26843.5456:53687.0911]=90,  App.Requests.TimedOut.Current=86 App.Requests.Total.Requests.Current=601744,      Delta=5024 Catalog.Activation.Collection.NumberOfCollections.Current=924,      Delta=4 Catalog.Activation.Created.Current=26047,      Delta=1678 Catalog.Activation.CurrentCount=0 Catalog.Activation.Destroyed.Current=26047,      Delta=1681 Catalog.Activation.DuplicateActivations.Current=6638,      Delta=406 Catalog.Activation.FailedToActivate.Current=0 Catalog.Activation.NonExistentActivations.Current=75 Catalog.Activation.Shutdown.ViaCollection.Current=10906 Catalog.Activation.Shutdown.ViaDeactivateOnIdle.Current=16,      Delta=13 Catalog.Activation.Shutdown.ViaDirectShutdown.Current=8492,      Delta=1262 ConsistentRing.AverageRangePercentage=100.000 ConsistentRing.MyRange.RingDistance=x100000000 ConsistentRing.MyRange.RingPercentage=100.000 ConsistentRing.Ring=[S192.168.5.184:22222:234461914 -> <MultiRange: Size=x100000000, %Ring=100.000%>] ConsistentRing.RingSize=1 Directory.CacheSize=0 Directory.Lookups.Cache.HitRatio=0.4719871, Delta=0.1503496 Directory.Lookups.Cache.Issued.Current=18545,      Delta=286 Directory.Lookups.Cache.Successes.Current=8753,      Delta=43 Directory.Lookups.Full.Issued.Current=27818,      Delta=2076 Directory.Lookups.Local.Issued.Current=370838,      Delta=3999 Directory.Lookups.Local.Successes.Current=343020,      Delta=1923 Directory.Lookups.LocalDirectory.Issued.Current=378481,      Delta=6435 Directory.Lookups.LocalDirectory.Successes.Current=336530,      Delta=1885 Directory.Lookups.Remote.Received.Current=8162,      Delta=889 Directory.Lookups.Remote.Sent.Current=9792,      Delta=243 Directory.PartitionSize=1056 Directory.Registrations.Issued.Current=1018,      Delta=19 Directory.Registrations.Local.Current=1009,      Delta=13 Directory.Registrations.Remote.Received.Current=14 Directory.Registrations.Remote.Sent.Current=10 Directory.Registrations.SingleAct.Issued.Current=26019,      Delta=1665 Directory.Registrations.SingleAct.Local.Current=23886,      Delta=2717 Directory.Registrations.SingleAct.Remote.Received.Current=10280,      Delta=2420 Directory.Registrations.SingleAct.Remote.Sent.Current=7273,      Delta=158 Directory.Ring=[S192.168.5.184:22222:234461914/C875E2A] Directory.Ring.MyPortion.AverageRingPercentage=100.000 Directory.Ring.MyPortion.RingDistance=0 Directory.Ring.MyPortion.RingPercentage=0.000 Directory.Ring.MyPredecessors=[] Directory.Ring.MySuccessors=[] Directory.Ring.RingSize=1 Directory.UnRegistrations.Issued.Current=30 Directory.UnRegistrations.Local.Current=19452,      Delta=1309 Directory.UnRegistrations.Remote.Received.Current=0 Directory.UnRegistrations.Remote.Sent.Current=0 Directory.UnRegistrationsMany.Issued.Current=272,      Delta=28 Directory.UnRegistrationsMany.Remote.Received.Current=13,      Delta=12 Directory.UnRegistrationsMany.Remote.Sent.Current=3 Directory.Validations.Cache.Received.Current=3 Directory.Validations.Cache.Sent.Current=16,      Delta=1 Dispatcher.NewPlacement.Current=24312,      Delta=2064 Gateway.ConnectedClients.Current=6 Gateway.LoadShedding.Current=0 Gateway.Received.Current=27648,      Delta=927 Gateway.Sent.Current=24088,      Delta=909 Grain.Grains.Implementations.GeocodeGrain.Current=0 Grain.Grains.Implementations.JourneyGrain.Current=0 Grain.Grains.Implementations.ResourceGovernerGrain.Current=0 Grain.Grains.Implementations.TrailerGrain.Current=0 Grain.Grains.Implementations.UnitGrain.Current=0 Grain.Grains.Implementations.VehicleGrain.Current=0 Grain.Orleans.Runtime.Management.ManagementGrain.Current=0 Grain.OrleansDashboard.DashboardGrain.Current=0 Grain.OrleansDashboard.SiloGrain.Current=0 Membership.ActiveCluster=[] Membership.ActiveClusterSize=0 MessageCenter.ReceiveQueueLength=0 MessageCenter.SendQueueLength=0 Messaging.Acceptor.AllocatedSocketEventArgs.Current=8 Messaging.Acceptor.CheckedInSocketEventArgs.Current=37 Messaging.Acceptor.CheckedOutSocketEventArgs.Current=37 Messaging.Acceptor.InUseSocketEventArgs=0 Messaging.Expired.AtDispatch.Current=0 Messaging.Expired.AtInvoke.Current=3 Messaging.Expired.AtReceive.Current=0 Messaging.Expired.AtRespond.Current=3 Messaging.Expired.AtSend.Current=0 Messaging.Pings.Received.S192.168.0.121:22222:234463621.Current=4 Messaging.Pings.Received.S192.168.0.121:22222:234463910.Current=5 Messaging.Pings.Received.S192.168.0.121:22222:234517236.Current=9 Messaging.Pings.Received.S192.168.0.127:22222:234462361.Current=52 Messaging.Pings.Received.S192.168.0.127:22222:234463135.Current=35 Messaging.Pings.ReplyMissed.S192.168.0.121:22222:234461122.Current=3 Messaging.Pings.ReplyMissed.S192.168.0.127:22222:234462361.Current=6 Messaging.Pings.ReplyMissed.S192.168.0.127:22222:234463135.Current=5 Messaging.Pings.ReplyReceived.S192.168.0.121:22222:234463621.Current=3 Messaging.Pings.ReplyReceived.S192.168.0.121:22222:234463910.Current=4 Messaging.Pings.ReplyReceived.S192.168.0.121:22222:234517236.Current=11 Messaging.Pings.ReplyReceived.S192.168.0.127:22222:234462361.Current=53 Messaging.Pings.ReplyReceived.S192.168.0.127:22222:234463135.Current=34 Messaging.Pings.Sent.S192.168.0.121:22222:234461122.Current=3 Messaging.Pings.Sent.S192.168.0.121:22222:234463621.Current=3 Messaging.Pings.Sent.S192.168.0.121:22222:234463910.Current=4 Messaging.Pings.Sent.S192.168.0.121:22222:234517236.Current=11 Messaging.Pings.Sent.S192.168.0.127:22222:234462361.Current=59 Messaging.Pings.Sent.S192.168.0.127:22222:234463135.Current=39 Messaging.Processing.ActivationData.All=0 Messaging.Processing.Dispatcher.Processed.Errors.Direction.OneWay.Current=0 Messaging.Processing.Dispatcher.Processed.Errors.Direction.Request.Current=26 Messaging.Processing.Dispatcher.Processed.Errors.Direction.Response.Current=56 Messaging.Processing.Dispatcher.Processed.Ok.Direction.OneWay.Current=0 Messaging.Processing.Dispatcher.Processed.Ok.Direction.Request.Current=403167,      Delta=4289 Messaging.Processing.Dispatcher.Processed.Ok.Direction.Response.Current=473729,      Delta=3760 Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.OneWay.Current=0 Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Request.Current=6671,      Delta=406 Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Response.Current=0 Messaging.Processing.Dispatcher.Processed.Total.Current=883649,      Delta=8455 Messaging.Processing.Dispatcher.Received.Direction.OneWay.Current=0 Messaging.Processing.Dispatcher.Received.Direction.Request.Current=409838,      Delta=4695 Messaging.Processing.Dispatcher.Received.Direction.Response.Current=473785,      Delta=3760 Messaging.Processing.Dispatcher.Received.OnActivationContext.Current=851760,      Delta=6360 Messaging.Processing.Dispatcher.Received.OnNullContext.Current=31863,      Delta=2095 Messaging.Processing.Dispatcher.Received.Total.Current=883623,      Delta=8455 Messaging.Processing.IGC.Forwarded.Current=6671,      Delta=406 Messaging.Processing.IGC.ReRoute.Current=0 Messaging.Processing.IGC.Resent.Current=0 Messaging.Processing.IMA.Enqueued.ToActivationContex.Current=851760,      Delta=6360 Messaging.Processing.IMA.Enqueued.ToNullContex.Current=31863,      Delta=2095 Messaging.Processing.IMA.Enqueued.ToSystemTargetContex.Current=356659,      Delta=4215 Messaging.Processing.IMA.Received.Current=1240282,      Delta=12670 Messaging.Received.BatchSize.PerSocketDirection.GatewayToClient=0.000 Messaging.Received.BatchSize.PerSocketDirection.SiloToSilo=0.000 Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient= Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo= Messaging.Received.Bytes.Header.Current=18041130,      Delta=1023415 Messaging.Received.Bytes.Total.Current=49783455,      Delta=3194392 Messaging.Received.Direction.OneWay.Current=0 Messaging.Received.Direction.Request.Current=56679,      Delta=3339 Messaging.Received.Direction.Response.Current=21429,      Delta=1080 Messaging.Received.Messages.From.Null.Current=27648,      Delta=927 Messaging.Received.Messages.From.S192.168.0.121:22222:234463621.Current=3054 Messaging.Received.Messages.From.S192.168.0.121:22222:234463910.Current=717 Messaging.Received.Messages.From.S192.168.0.121:22222:234517236.Current=3483 Messaging.Received.Messages.From.S192.168.0.127:22222:234462361.Current=24927 Messaging.Received.Messages.From.S192.168.0.127:22222:234463135.Current=18100 Messaging.Received.Messages.From.S192.168.5.184:22222:234461914.Current=179,      Delta=9 Messaging.Received.Messages.Total.Current=78108,      Delta=4419 Messaging.Received.MessageSizeHistogram.Bytes=[128:255]=2725, [256:511]=47898, [512:1023]=24852, [1024:2047]=2547, [2048:4095]=28, [4096:8191]=24, [8192:16383]=22, [16384:32767]=3, [32768:65535]=1, [524288:1048575]=2, [1048576:2097151]=6,  Messaging.Rejected.Request.Current=123,      Delta=25 Messaging.Rerouted.OneWay.Current=0 Messaging.Rerouted.Request.Current=27648,      Delta=927 Messaging.Rerouted.Response.Current=0 Messaging.Sent.BatchSize.PerSocketDirection.GatewayToClient=0.000 Messaging.Sent.BatchSize.PerSocketDirection.SiloToSilo=0.000 Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient= Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo= Messaging.Sent.Bytes.Header.Current=18904417,      Delta=1058033 Messaging.Sent.Bytes.Total.Current=27676568,      Delta=1562471 Messaging.Sent.Direction.OneWay.Current=0 Messaging.Sent.Direction.Request.Current=26320,      Delta=1089 Messaging.Sent.Direction.Response.Current=51637,      Delta=3311 Messaging.Sent.Dropped.Response.Current=118,      Delta=1 Messaging.Sent.Failed.Request.Current=116,      Delta=25 Messaging.Sent.Failed.Response.Current=118,      Delta=1 Messaging.Sent.LocalMessages.Current=1190236,      Delta=9562 Messaging.Sent.Messages.To.Null.Current=24088,      Delta=909 Messaging.Sent.Messages.To.S192.168.0.121:22222:234463621.Current=3056 Messaging.Sent.Messages.To.S192.168.0.121:22222:234463910.Current=718 Messaging.Sent.Messages.To.S192.168.0.121:22222:234517236.Current=3491 Messaging.Sent.Messages.To.S192.168.0.127:22222:234462361.Current=27061 Messaging.Sent.Messages.To.S192.168.0.127:22222:234463135.Current=19543 Messaging.Sent.Messages.Total.Current=77957,      Delta=4400 Messaging.Sent.MessageSizeHistogram.Bytes=[128:255]=27277, [256:511]=44358, [512:1023]=5252, [1024:2047]=481, [2048:4095]=484, [4096:8191]=44, [8192:16383]=55, [16384:32767]=4, [32768:65535]=1, [65536:131071]=1,  Networking.Sockets.GatewayToClient.Duplex.Closed.Current=32,      Delta=6 Networking.Sockets.GatewayToClient.Duplex.Opened.Current=32,      Delta=1 Networking.Sockets.Silo.Receiving.Closed.Current=5,      Delta=2 Networking.Sockets.Silo.Receiving.Opened.Current=5,      Delta=1 Networking.Sockets.Silo.Sending.Closed.Current=6,      Delta=3 Networking.Sockets.Silo.Sending.Opened.Current=5,      Delta=1 Queues.QueueSize.Instantaneous.Scheduler.LevelOne=0 Reminders.AverageTardiness.Seconds=0 Secs Reminders.NumberOfActiveReminders=0 Reminders.TicksDelivered.Current=0 Runtime.CpuUsage=4.454 Runtime.DOT.NET.ThreadPool.InUse.CompletionPortThreads=0 Runtime.DOT.NET.ThreadPool.InUse.WorkerThreads=1 Runtime.GC.AllocatedBytesInKbPerSec=21040.380 Runtime.GC.GenCollectonCount=gen0=1321, gen1=580, gen2=15 Runtime.GC.GenSizesKb=gen0=210450.40, gen1=52867.30, gen2=615486.60 Runtime.GC.LargeObjectHeapSizeKb=113709.000 Runtime.GC.NumberOfInducedGCs=2.000 Runtime.GC.PercentOfTimeInGC=0.101 Runtime.GC.PromotedFinalizationMemoryFromGen0Kb=20603.020 Runtime.GC.PromotedMemoryFromGen1Kb=0.000 Runtime.GC.TotalMemoryKb=772440 Runtime.IsOverloaded=False Runtime.Memory.AvailableMemoryMb=30154 Runtime.Memory.TotalPhysicalMemoryMb=32768 Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer1.Current=1
Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayAcceptor.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayClientCleanupAgent.Current=1
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewaySender.Current=0,      Delta=-4
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAcceptor.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent.Current=3
Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender.Current=0,      Delta=-4
Runtime.Threads.AsynchAgent.Runtime.Scheduler.WorkerPoolThread.Current=0,      Delta=-5
Runtime.Threads.AsynchAgent.Runtime.Watchdog.Current=1
Runtime.Threads.AsynchAgent.TotalThreadsCreated.Current=23,      Delta=1
Scheduler.NumLongQueueWaitTimes.Current=18
Scheduler.NumLongRunningTurns.Current=681,      Delta=27
Scheduler.WorkItemGroupCount=13
Serialization.BufferPool.AllocatedBuffers.Current=791,      Delta=2
Serialization.BufferPool.BuffersInPool=258
Serialization.BufferPool.CheckedInBuffers.Current=254562,      Delta=13698
Serialization.BufferPool.CheckedOutBuffers.Current=255095,      Delta=13701
Serialization.BufferPool.DroppedBuffers.Current=0
Serialization.BufferPool.DroppedTooLargeBuffers.Current=65,      Delta=2
Serialization.BufferPool.InUse.AllocatedAndNotInPool_Buffers=533
Serialization.BufferPool.InUse.CheckedOutAndNotCheckedIn_Buffers=533
Silo.StartTime=2017-06-06 16:18:34.639 GMT
Storage.Activate.Errors.Current=0
Storage.Activate.Total.Current=0
Storage.Clear.Errors.Current=0
Storage.Clear.Latency=0 Secs
Storage.Clear.Total.Current=0
Storage.Read.Errors.Current=0
Storage.Read.Latency=0 Secs
Storage.Read.Total.Current=0
Storage.Write.Errors.Current=0
Storage.Write.Latency=0 Secs
Storage.Write.Total.Current=0
SystemTarget..Current=3
Watchdog.NumHealthChecks.Current=184,      Delta=1
[2017-06-07 07:43:04.671 GMT   108	INFO   	100000	TypeManager	192.168.5.184:22222]	OnRefreshClusterMapTimer: refresh start
[2017-06-07 07:43:05.124 GMT    20	INFO   	100328	Runtime.Messaging.GatewayClientCleanupAgent	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent that runs on managed thread 20
[2017-06-07 07:43:05.389 GMT    29	INFO   	100328	Runtime.Watchdog	192.168.5.184:22222]	Stopping AsyncAgent Runtime.Watchdog that runs on managed thread 29

Closing comment (2017-06-09T05:36:09Z): I wonder if it could be related to #3106