+1 to move the discussion to the mailing list, could you summarize your changes (preferably for both non-tx and tx cases) and send an email to the list?

And now to add some more to this already unwieldy discussion :)

1. Many interceptors check ownership, I don't think that would be a problem. Besides, I think creating a new L1ReadSynchronizer for every read is just as bad for performance as locking the key on every read, so you'd need that check either way.

2. By default L1 invalidations are sent as multicasts, so I'm not sure ISPN-3273 really matters here. BTW, I wonder if we have a check to only send L1 invalidations from one node if the threshold is 0...

3a. Right, for put commands we can't hold the local lock while executing the remote put, or we'll have a deadlock. But I think a shorter lock, held only after the remote put completed (or after the lock on the primary owner was acquired, with txs) should work.

3b. We'd also have an ownership check before, so we'd only serialize the get commands that need to go remotely for the same key. I think it would be almost the same as your solution (although it does have one disadvantage - if the key doesn't exist in the cache, all the get commands will go remotely). The number of L1 writes should be very small compared to the number of L1 reads anyway, otherwise it would be more efficient to get the key from the owner every time.

It would be nice to agree on what guarantees we want to provide for L1 invalidation in non-tx caches, I'm not sure if we can do anything to prevent this scenario:

A initiates a put(k, v1) to the primary owner B
B performs the put(k, v1), invalidates every non-owner and returns
B performs another put(k, v2), invalidating every non-owner
A receives the result from B and puts k=v1 in its L1