akka/akka — pull request #2117 by patriknw (2014-04-06 19:48:53+00:00)
Title: =3974 per Persist (serialize) actor refs  with transport info

* The reason for the problem with NoSuchElementException in ClusterSharding was
that actor references were not serialized with full address information. In
certain fail over scenarios the references could not be resolved and therefore
the ShardRegionTerminated did not match corresponding ShardRegionRegistered.
* Wrap serialization with transport information from defaultAddress

There is a great description of how to reproduce the problem in the [ticket](https://www.assembla.com/spaces/akka/tickets/3974).

When debugging I found the following for a failing actor ref with uid 1022170393:

On 2551 it is first registered as expected:

[INFO] [04/06/2014 18:55:25.844] [ClusterSystem-akka.actor.default-dispatcher-14] [akka.tcp://ClusterSystem@127.0.0.1:2551/user/sharding/AuthorListingCoordinator/singleton] # updated: ShardRegionRegistered(Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393])

# serializedActorPath Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393] Information akka.tcp://ClusterSystem@127.0.0.1:2551
### serializedActorPath Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393] => akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393

Recovery on 2552:

[INFO] [04/06/2014 18:56:10.115] [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-27] [LocalActorRefProvider(akka://ClusterSystem)] # resolve of path sequence [/user/sharding/AuthorListing#1022170393] failed

# readResolve path: akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393 => Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393]

[INFO] [04/06/2014 18:56:10.122] [ClusterSystem-akka.actor.default-dispatcher-14] [akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListingCoordinator/singleton] # updated: ShardRegionRegistered(Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393])

### serializedActorPath Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393] => akka://ClusterSystem/user/sharding/AuthorListing#1022170393

[INFO] [04/06/2014 18:56:10.172] [ClusterSystem-akka.actor.default-dispatcher-14] [akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListingCoordinator/singleton] # updated: ShardRegionTerminated(Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393])

Recovery on 2553:

# readResolve path: akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393 => Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393]

[INFO] [04/06/2014 18:56:40.155] [ClusterSystem-akka.persistence.dispatchers.default-replay-dispatcher-29] [LocalActorRefProvider(akka://ClusterSystem)] # resolve of path sequence [/user/sharding/AuthorListing#1022170393] failed

# readResolve path: akka://ClusterSystem/user/sharding/AuthorListing#1022170393 => Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393]

[INFO] [04/06/2014 18:56:40.158] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.tcp://ClusterSystem@127.0.0.1:2553/user/sharding/AuthorListingCoordinator/singleton] # updated: ShardRegionRegistered(Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393])

[INFO] [04/06/2014 18:56:40.162] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.tcp://ClusterSystem@127.0.0.1:2553/user/sharding/AuthorListingCoordinator/singleton] # receiveRecover ShardRegionTerminated Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393] => state before: State(Map(17 -> Actor[akka.tcp://ClusterSystem@127.0.0.1:2551/user/sharding/AuthorListing#1443663680]),Map(Actor[akka.tcp://ClusterSystem@127.0.0.1:2551/user/sharding/AuthorListing#1443663680] -> Vector(17), Actor[akka.tcp://ClusterSystem@127.0.0.1:2552/user/sharding/AuthorListing#1022170393] -> Vector(), Actor[akka://ClusterSystem/user/sharding/AuthorListing#-85570854] -> Vector()),Set())
[INFO] [04/06/2014 18:56:40.162] [ClusterSystem-akka.actor.default-dispatcher-24] [akka.tcp://ClusterSystem@127.0.0.1:2553/user/sharding/AuthorListingCoordinator/singleton] # updated: ShardRegionTerminated(Actor[akka://ClusterSystem/user/sharding/AuthorListing#1022170393])

And boom for that ShardRegionTerminated.