Repository: onflow/flow-evm-gateway
Pull request #772: Fix concurrent map access in keystore
Author: login: peterargue
Labels: name: Bugfix
name: Bug
description: Something isn't working
Created: 2025-03-06T19:05:36Z | Merged: 2025-03-07T20:33:05Z

Closes: #???

## Description

The keystore tracks the keys currently in use with a map. Currently, tha map does not have any synchronization, causing concurrent map access. 

______

For contributor use:

- [x] Targeted PR against `master` branch
- [ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
- [x] Code follows the [standards mentioned here](https://github.com/onflow/flow-nft/blob/master/CONTRIBUTING.md#styleguides).
- [x] Updated relevant documentation 
- [x] Re-reviewed `Files changed` in the Github PR explorer
- [x] Added appropriate labels 

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
  - Introduced a new key management system with enhanced structures for managing signing keys and account keys.
  - Added new methods for handling key lifecycle, including locking, unlocking, and notifying transactions and blocks.

- **Bug Fixes**
  - Improved system stability by enhancing synchronization in key management processes. These updates ensure that concurrent operations handle keys reliably even under heavy usage, leading to more consistent performance and reducing potential operational hiccups. Users may notice enhanced performance and fewer glitches during prolonged usage.

- **Chores**
  - Updated project dependencies to include new libraries that enhance functionality and support for key management.
  - Removed obsolete key management structures and streamlined the codebase for better maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->


Diff:
diff --git a/bootstrap/bootstrap.go b/bootstrap/bootstrap.go
index 94b5f549d..d97f18d20 100644
--- a/bootstrap/bootstrap.go
+++ b/bootstrap/bootstrap.go
@@ -32,6 +32,7 @@ import (
 	"github.com/onflow/flow-evm-gateway/services/ingestion"
 	"github.com/onflow/flow-evm-gateway/services/replayer"
 	"github.com/onflow/flow-evm-gateway/services/requester"
+	"github.com/onflow/flow-evm-gateway/services/requester/keystore"
 	"github.com/onflow/flow-evm-gateway/storage"
 	"github.com/onflow/flow-evm-gateway/storage/pebble"
 )
@@ -76,7 +77,7 @@ type Bootstrap struct {
 	events     *ingestion.Engine
 	profiler   *api.ProfileServer
 	db         *pebbleDB.DB
-	keystore   *requester.KeyStore
+	keystore   *keystore.KeyStore
 }
 
 func New(config config.Config) (*Bootstrap, error) {
@@ -219,7 +220,7 @@ func (b *Bootstrap) StartAPIServer(ctx context.Context) error {
 		b.config,
 	)
 
-	accountKeys := make([]*requester.AccountKey, 0)
+	accountKeys := make([]*keystore.AccountKey, 0)
 	if !b.config.IndexOnly {
 		account, err := b.client.GetAccount(ctx, b.config.COAAddress)
 		if err != nil {
@@ -239,15 +240,15 @@ func (b *Bootstrap) StartAPIServer(ctx context.Context) error {
 			if !key.PublicKey.Equals(signer.PublicKey()) {
 				continue
 			}
-			accountKeys = append(accountKeys, &requester.AccountKey{
-				AccountKey: *key,
-				Address:    b.config.COAAddress,
-				Signer:     signer,
-			})
+			accountKeys = append(accountKeys, keystore.NewAccountKey(
+				*key,
+				b.config.COAAddress,
+				signer,
+			))
 		}
 	}
 
-	b.keystore = requester.NewKeyStore(accountKeys)
+	b.keystore = keystore.New(accountKeys)
 
 	evm, err := requester.NewEVM(
 		b.storages.Registers,
diff --git a/go.mod b/go.mod
index 71bee8074..6a28ac20c 100644
--- a/go.mod
+++ b/go.mod
@@ -19,6 +19,7 @@ require (
 	github.com/sethvargo/go-retry v0.2.3
 	github.com/spf13/cobra v1.8.1
 	github.com/stretchr/testify v1.10.0
+	go.uber.org/atomic v1.11.0
 	go.uber.org/ratelimit v0.3.1
 	golang.org/x/exp v0.0.0-20241217172543-b2144cdd0a67
 	golang.org/x/sync v0.10.0
@@ -94,6 +95,8 @@ require (
 	github.com/hashicorp/hcl v1.0.0 // indirect
 	github.com/holiman/bloomfilter/v2 v2.0.3 // indirect
 	github.com/holiman/uint256 v1.3.0 // indirect
+	github.com/huandu/go-clone v1.6.0 // indirect
+	github.com/huandu/go-clone/generic v1.7.2 // indirect
 	github.com/huin/goupnp v1.3.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/ipfs/bbloom v0.0.4 // indirect
@@ -115,6 +118,8 @@ require (
 	github.com/kr/text v0.2.0 // indirect
 	github.com/libp2p/go-buffer-pool v0.1.0 // indirect
 	github.com/libp2p/go-libp2p v0.38.2 // indirect
+	github.com/libp2p/go-libp2p-pubsub v0.13.0 // indirect
+	github.com/libp2p/go-msgio v0.3.0 // indirect
 	github.com/logrusorgru/aurora/v4 v4.0.0 // indirect
 	github.com/magiconair/properties v1.8.7 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
@@ -127,6 +132,7 @@ require (
 	github.com/multiformats/go-base32 v0.1.0 // indirect
 	github.com/multiformats/go-base36 v0.2.0 // indirect
 	github.com/multiformats/go-multiaddr v0.14.0 // indirect
+	github.com/multiformats/go-multiaddr-fmt v0.1.0 // indirect
 	github.com/multiformats/go-multibase v0.2.0 // indirect
 	github.com/multiformats/go-multicodec v0.9.0 // indirect
 	github.com/multiformats/go-multihash v0.2.3 // indirect
@@ -190,7 +196,6 @@ require (
 	go.opentelemetry.io/otel/sdk v1.31.0 // indirect
 	go.opentelemetry.io/otel/trace v1.31.0 // indirect
 	go.opentelemetry.io/proto/otlp v1.3.1 // indirect
-	go.uber.org/atomic v1.11.0 // indirect
 	go.uber.org/multierr v1.11.0 // indirect
 	go.uber.org/zap v1.27.0 // indirect
 	golang.org/x/crypto v0.31.0 // indirect
diff --git a/go.sum b/go.sum
index 3055f70a3..2f593a1bb 100644
--- a/go.sum
+++ b/go.sum
@@ -116,13 +116,17 @@ github.com/consensys/bavard v0.1.13 h1:oLhMLOFGTLdlda/kma4VOJazblc7IM5y5QPd2A/Yj
 github.com/consensys/bavard v0.1.13/go.mod h1:9ItSMtA/dXMAiL7BG6bqW2m3NdSEObYWoH223nGHukI=
 github.com/consensys/gnark-crypto v0.12.1 h1:lHH39WuuFgVHONRl3J0LRBtuYdQTumFSDtJF7HpyG8M=
 github.com/consensys/gnark-crypto v0.12.1/go.mod h1:v2Gy7L/4ZRosZ7Ivs+9SfUDr0f5UlG+EM5t7MPHiLuY=
+github.com/containerd/cgroups v1.1.0 h1:v8rEWFl6EoqHB+swVNjVoCJE8o3jX7e8nqBGPLaDFBM=
+github.com/containerd/cgroups v1.1.0/go.mod h1:6ppBcbh/NOOUU+dMKrykgaBnK9lCIBxHqJDGwsa1mIw=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-semver v0.3.0 h1:wkHLiw0WNATZnSG7epLsujiMCgPAc9xhjJ4tgnAxmfM=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e h1:Wf6HqHfScWJN9/ZjdUKyjop4mf3Qdd+1TvvltAvM3m8=
 github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
+github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8iXXhfZs=
 github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
 github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
 github.com/cpuguy83/go-md2man v1.0.10 h1:BSKMNlYxDvnunlTymqtgONjNnaRV1sTpcovwwjF22jk=
@@ -137,6 +141,8 @@ github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ3
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c h1:pFUpOrbxDR6AkioZ1ySsx5yxlDQZ8stG2b88gTPxgJU=
+github.com/davidlazar/go-crypto v0.0.0-20200604182044-b73af7476f6c/go.mod h1:6UhI8N9EjYm1c2odKpFpAYeR8dsBeM7PtzQhRgxRr9U=
 github.com/deckarep/golang-set/v2 v2.6.0 h1:XfcQbWM1LlMB8BsJ8N9vW5ehnnPVIw0je80NsVHagjM=
 github.com/deckarep/golang-set/v2 v2.6.0/go.mod h1:VAky9rY/yGXJOLEDv3OMci+7wtDpOF4IN+y82NBOac4=
 github.com/decred/dcrd/crypto/blake256 v1.0.1 h1:7PltbUIQB7u/FfZ39+DGa/ShuMyJ5ilcvdfma9wOH6Y=
@@ -167,6 +173,8 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp
 github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
 github.com/ef-ds/deque v1.0.4 h1:iFAZNmveMT9WERAkqLJ+oaABF9AcVQ5AjXem/hroniI=
 github.com/ef-ds/deque v1.0.4/go.mod h1:gXDnTC3yqvBcHbq2lcExjtAcVrOnJCbMcZXmuj8Z4tg=
+github.com/elastic/gosigar v0.14.3 h1:xwkKwPia+hSfg9GqrCUKYdId102m9qTJIIr7egmK/uo=
+github.com/elastic/gosigar v0.14.3/go.mod h1:iXRIGg2tLnu7LBdpqzyQfGDEidKCfWcCMS0WKyPWoMs=
 github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
 github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
 github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
@@ -183,6 +191,10 @@ github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2
 github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
 github.com/fjl/memsize v0.0.2 h1:27txuSD9or+NZlnOWdKUxeBzTAUkWCVh+4Gf2dWFOzA=
 github.com/fjl/memsize v0.0.2/go.mod h1:VvhXpOYNQvB+uIk2RvXzuaQtkQJzzIx6lSBe1xv7hi0=
+github.com/flynn/noise v1.1.0 h1:KjPQoQCEFdZDiP03phOvGi11+SVVhBG2wOWAorLsstg=
+github.com/flynn/noise v1.1.0/go.mod h1:xbMo+0i6+IGbYdJhF31t2eR1BIU0CYc12+BNAKwUTag=
+github.com/francoispqt/gojay v1.2.13 h1:d2m3sFjloqoIUQU3TsHBgj6qg/BVGlTBeHDUmyJnXKk=
+github.com/francoispqt/gojay v1.2.13/go.mod h1:ehT5mTG4ua4581f1++1WLG0vPdaA9HaiDsoyrBGkyDY=
 github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
 github.com/frankban/quicktest v1.14.4/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
@@ -228,9 +240,14 @@ github.com/go-sourcemap/sourcemap v2.1.3+incompatible h1:W1iEw64niKVGogNgBN3ePyL
 github.com/go-sourcemap/sourcemap v2.1.3+incompatible/go.mod h1:F8jJfvm2KbVjc5NqelyYJmf/v5J0dwNLS2mL4sNA1Jg=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/go-task/slim-sprig v0.0.0-20210107165309-348f09dbbbc0/go.mod h1:fyg7847qk6SyHyPtNmDHnmrv/HOrqktSC+C9fM+CJOE=
+github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI=
+github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
+github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
 github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
 github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
+github.com/godbus/dbus/v5 v5.1.0 h1:4KLkAxT3aOY8Li4FRJe/KvhoNFFxo0m6fNuFUO8QJUk=
+github.com/godbus/dbus/v5 v5.1.0/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
 github.com/gofrs/flock v0.8.1 h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw=
 github.com/gofrs/flock v0.8.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU=
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
@@ -298,6 +315,8 @@ github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeN
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
 github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
+github.com/google/gopacket v1.1.19 h1:ves8RnFZPGiFnTS0uPQStjwru6uO6h+nlr9j6fL7kF8=
+github.com/google/gopacket v1.1.19/go.mod h1:iJ8V8n6KS+z2U1A8pUwu8bW5SyEMkXJB8Yo/Vo+TKTo=
 github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
 github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
 github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
@@ -329,10 +348,15 @@ github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5m
 github.com/googleapis/gax-go/v2 v2.12.2 h1:mhN09QQW1jEWeMF74zGR81R30z4VJzjZsfkUhuHF+DA=
 github.com/googleapis/gax-go/v2 v2.12.2/go.mod h1:61M8vcyyXR2kqKFxKrfA22jaA8JGF7Dc8App1U3H6jc=
 github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
+github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
+github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
 github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
 github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.0 h1:Iju5GlWwrvL6UBg4zJJt3btmonfrMlCDdsejg4CZE7c=
 github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0 h1:pRhl55Yx1eC7BZ1N+BBWwnKaMyD8uC+34TLdndZMAKk=
+github.com/grpc-ecosystem/go-grpc-middleware/v2 v2.1.0/go.mod h1:XKMd7iuf/RGPSMJ/U4HP0zS2Z9Fh8Ps9a+6X26m/tmI=
 github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
 github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
 github.com/grpc-ecosystem/grpc-gateway/v2 v2.22.0 h1:asbCHRVmodnJTuQ3qamDwqVOIjwqUPTYmYuemVOx+Ys=
@@ -357,6 +381,8 @@ github.com/holiman/bloomfilter/v2 v2.0.3/go.mod h1:zpoh+gs7qcpqrHr3dB55AMiJwo0iU
 github.com/holiman/uint256 v1.3.0 h1:4wdcm/tnd0xXdu7iS3ruNvxkWwrb4aeBQv19ayYn8F4=
 github.com/holiman/uint256 v1.3.0/go.mod h1:EOMSn4q6Nyt9P6efbI3bueV4e1b3dGlUCXeiRV4ng7E=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
+github.com/huandu/go-assert v1.1.5 h1:fjemmA7sSfYHJD7CUqs9qTwwfdNAx7/j2/ZlHXzNB3c=
+github.com/huandu/go-assert v1.1.5/go.mod h1:yOLvuqZwmcHIC5rIzrBhT7D3Q9c3GFnd0JrPVhn/06U=
 github.com/huandu/go-clone v1.6.0 h1:HMo5uvg4wgfiy5FoGOqlFLQED/VGRm2D9Pi8g1FXPGc=
 github.com/huandu/go-clone v1.6.0/go.mod h1:ReGivhG6op3GYr+UY3lS6mxjKp7MIGTknuU5TbTVaXE=
 github.com/huandu/go-clone/generic v1.7.2 h1:47pQphxs1Xc9cVADjOHN+Bm5D0hNagwH9UXErbxgVKA=
@@ -394,6 +420,8 @@ github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j
 github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus=
 github.com/jackpal/go-nat-pmp v1.0.2/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc=
 github.com/jbenet/go-cienv v0.1.0/go.mod h1:TqNnHUmJgXau0nCzC7kXWeotg3J9W34CUv5Djy1+FlA=
+github.com/jbenet/go-temp-err-catcher v0.1.0 h1:zpb3ZH6wIE8Shj2sKS+khgRvf7T7RABoLk/+KKHggpk=
+github.com/jbenet/go-temp-err-catcher v0.1.0/go.mod h1:0kJRvmDZXNMIiJirNPEYfhpPwbGVtZVWC34vc5WLsDk=
 github.com/jbenet/goprocess v0.1.4 h1:DRGOFReOMqqDNXwW70QkacFW0YN9QnwLV0Vqk+3oU0o=
 github.com/jbenet/goprocess v0.1.4/go.mod h1:5yspPrukOVuOLORacaBi858NqyClJPQxYZlqdZVfqY4=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@@ -416,6 +444,8 @@ github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90
 github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
 github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/koron/go-ssdp v0.0.4 h1:1IDwrghSKYM7yLf7XCzbByg2sJ/JcNOZRXS2jczTwz0=
+github.com/koron/go-ssdp v0.0.4/go.mod h1:oDXq+E5IL5q0U8uSBcoAXzTzInwy5lEgC91HoKtbmZk=
 github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
 github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@@ -436,6 +466,8 @@ github.com/libp2p/go-buffer-pool v0.1.0 h1:oK4mSFcQz7cTQIfqbe4MIj9gLW+mnanjyFtc6
 github.com/libp2p/go-buffer-pool v0.1.0/go.mod h1:N+vh8gMqimBzdKkSMVuydVDq+UV5QTWy5HSiZacSbPg=
 github.com/libp2p/go-cidranger v1.1.0 h1:ewPN8EZ0dd1LSnrtuwd4709PXVcITVeuwbag38yPW7c=
 github.com/libp2p/go-cidranger v1.1.0/go.mod h1:KWZTfSr+r9qEo9OkI9/SIEeAtw+NNoU0dXIXt15Okic=
+github.com/libp2p/go-flow-metrics v0.2.0 h1:EIZzjmeOE6c8Dav0sNv35vhZxATIXWZg6j/C08XmmDw=
+github.com/libp2p/go-flow-metrics v0.2.0/go.mod h1:st3qqfu8+pMfh+9Mzqb2GTiwrAGjIPszEjZmtksN8Jc=
 github.com/libp2p/go-libp2p v0.38.2 h1:9SZQDOCi82A25An4kx30lEtr6kGTxrtoaDkbs5xrK5k=
 github.com/libp2p/go-libp2p v0.38.2/go.mod h1:QWV4zGL3O9nXKdHirIC59DoRcZ446dfkjbOJ55NEWFo=
 github.com/libp2p/go-libp2p-asn-util v0.4.1 h1:xqL7++IKD9TBFMgnLPZR6/6iYhawHKHl950SO9L6n94=
@@ -444,13 +476,25 @@ github.com/libp2p/go-libp2p-kbucket v0.6.3 h1:p507271wWzpy2f1XxPzCQG9NiN6R6lHL9G
 github.com/libp2p/go-libp2p-kbucket v0.6.3/go.mod h1:RCseT7AH6eJWxxk2ol03xtP9pEHetYSPXOaJnOiD8i0=
 github.com/libp2p/go-libp2p-pubsub v0.13.0 h1:RmFQ2XAy3zQtbt2iNPy7Tt0/3fwTnHpCQSSnmGnt1Ps=
 github.com/libp2p/go-libp2p-pubsub v0.13.0/go.mod h1:m0gpUOyrXKXdE7c8FNQ9/HLfWbxaEw7xku45w+PaqZo=
+github.com/libp2p/go-libp2p-testing v0.12.0 h1:EPvBb4kKMWO29qP4mZGyhVzUyR25dvfUIK5WDu6iPUA=
+github.com/libp2p/go-libp2p-testing v0.12.0/go.mod h1:KcGDRXyN7sQCllucn1cOOS+Dmm7ujhfEyXQL5lvkcPg=
 github.com/libp2p/go-msgio v0.3.0 h1:mf3Z8B1xcFN314sWX+2vOTShIE0Mmn2TXn3YCUQGNj0=
 github.com/libp2p/go-msgio v0.3.0/go.mod h1:nyRM819GmVaF9LX3l03RMh10QdOroF++NBbxAb0mmDM=
+github.com/libp2p/go-nat v0.2.0 h1:Tyz+bUFAYqGyJ/ppPPymMGbIgNRH+WqC5QrT5fKrrGk=
+github.com/libp2p/go-nat v0.2.0/go.mod h1:3MJr+GRpRkyT65EpVPBstXLvOlAPzUVlG6Pwg9ohLJk=
+github.com/libp2p/go-netroute v0.2.2 h1:Dejd8cQ47Qx2kRABg6lPwknU7+nBnFRpko45/fFPuZ8=
+github.com/libp2p/go-netroute v0.2.2/go.mod h1:Rntq6jUAH0l9Gg17w5bFGhcC9a+vk4KNXs6s7IljKYE=
+github.com/libp2p/go-reuseport v0.4.0 h1:nR5KU7hD0WxXCJbmw7r2rhRYruNRl2koHw8fQscQm2s=
+github.com/libp2p/go-reuseport v0.4.0/go.mod h1:ZtI03j/wO5hZVDFo2jKywN6bYKWLOy8Se6DrI2E1cLU=
+github.com/libp2p/go-yamux/v4 v4.0.1 h1:FfDR4S1wj6Bw2Pqbc8Uz7pCxeRBPbwsBbEdfwiCypkQ=
+github.com/libp2p/go-yamux/v4 v4.0.1/go.mod h1:NWjl8ZTLOGlozrXSOZ/HlfG++39iKNnM5wwmtQP1YB4=
 github.com/logrusorgru/aurora/v4 v4.0.0 h1:sRjfPpun/63iADiSvGGjgA1cAYegEWMPCJdUpJYn9JA=
 github.com/logrusorgru/aurora/v4 v4.0.0/go.mod h1:lP0iIa2nrnT/qoFXcOZSrZQpJ1o6n2CUf/hyHi2Q4ZQ=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
 github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
+github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd h1:br0buuQ854V8u83wA0rVZ8ttrq5CpaPZdvrK0LP2lOk=
+github.com/marten-seemann/tcp v0.0.0-20210406111302-dfbc87cc63fd/go.mod h1:QuCEs1Nt24+FYQEqAAncTDPJIuGs+LxK1MCiFL25pMU=
 github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
 github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
 github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ=
@@ -465,6 +509,12 @@ github.com/mattn/go-runewidth v0.0.15/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/miekg/dns v1.1.62 h1:cN8OuEF1/x5Rq6Np+h1epln8OiyPWV+lROx9LxcGgIQ=
 github.com/miekg/dns v1.1.62/go.mod h1:mvDlcItzm+br7MToIKqkglaGhlFMHJ9DTNNWONWXbNQ=
+github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b h1:z78hV3sbSMAUoyUMM0I83AUIT6Hu17AWfgjzIbtrYFc=
+github.com/mikioh/tcpinfo v0.0.0-20190314235526-30a79bb1804b/go.mod h1:lxPUiZwKoFL8DUUmalo2yJJUCxbPKtm8OKfqr2/FTNU=
+github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc h1:PTfri+PuQmWDqERdnNMiD9ZejrlswWrCpBEZgWOiTrc=
+github.com/mikioh/tcpopt v0.0.0-20190314235656-172688c1accc/go.mod h1:cGKTAVKx4SxOuR/czcZ/E2RSJ3sfHs8FpHhQ5CWMf9s=
+github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1/go.mod h1:pD8RvIylQ358TN4wwqatJ8rNavkEINozVn9DtGI3dfQ=
+github.com/minio/sha256-simd v0.1.1-0.20190913151208-6de447530771/go.mod h1:B5e1o+1/KgNmWrSQK08Y6Z1Vb5pwIktudl0J58iy0KM=
 github.com/minio/sha256-simd v1.0.1 h1:6kaan5IFmwTNynnKKpDHe6FWHohJOHhCPchzK49dzMM=
 github.com/minio/sha256-simd v1.0.1/go.mod h1:Pz6AKMiUdngCLpeTL/RJY1M9rUuPMYujV5xJjtbRSN8=
 github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
@@ -480,12 +530,14 @@ github.com/mmcloughlin/addchain v0.4.0/go.mod h1:A86O+tHqZLMNO4w6ZZ4FlVQEadcoqky
 github.com/mmcloughlin/profile v0.1.1/go.mod h1:IhHD7q1ooxgwTgjxQYkACGA77oFTDdFVejUS1/tS/qU=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
 github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
+github.com/mr-tron/base58 v1.1.2/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
 github.com/mr-tron/base58 v1.2.0 h1:T/HDJBh4ZCPbU39/+c3rRvE0uKBQlU27+QI8LJ4t64o=
 github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
 github.com/multiformats/go-base32 v0.1.0 h1:pVx9xoSPqEIQG8o+UbAe7DNi51oej1NtK+aGkbLYxPE=
 github.com/multiformats/go-base32 v0.1.0/go.mod h1:Kj3tFY6zNr+ABYMqeUNeGvkIC/UYgtWibDcT0rExnbI=
 github.com/multiformats/go-base36 v0.2.0 h1:lFsAbNOGeKtuKozrtBsAkSVhv1p9D0/qedU9rQyccr0=
 github.com/multiformats/go-base36 v0.2.0/go.mod h1:qvnKE++v+2MWCfePClUEjE78Z7P2a1UV0xHgWc0hkp4=
+github.com/multiformats/go-multiaddr v0.1.1/go.mod h1:aMKBKNEYmzmDmxfX88/vz+J5IU55txyt0p4aiWVohjo=
 github.com/multiformats/go-multiaddr v0.14.0 h1:bfrHrJhrRuh/NXH5mCnemjpbGjzRw/b+tJFOD41g2tU=
 github.com/multiformats/go-multiaddr v0.14.0/go.mod h1:6EkVAxtznq2yC3QT5CM1UTAwG0GTP3EWAIcjHuzQ+r4=
 github.com/multiformats/go-multiaddr-dns v0.4.1 h1:whi/uCLbDS3mSEUMb1MsoT4uzUeZB0N32yzufqS0i5M=
@@ -496,6 +548,7 @@ github.com/multiformats/go-multibase v0.2.0 h1:isdYCVLvksgWlMW9OZRYJEa9pZETFivnc
 github.com/multiformats/go-multibase v0.2.0/go.mod h1:bFBZX4lKCA/2lyOFSAoKH5SS6oPyjtnzK/XTFDPkNuk=
 github.com/multiformats/go-multicodec v0.9.0 h1:pb/dlPnzee/Sxv/j4PmkDRxCOi3hXTz3IbPKOXWJkmg=
 github.com/multiformats/go-multicodec v0.9.0/go.mod h1:L3QTQvMIaVBkXOXXtVmYE+LI16i14xuaojr/H7Ai54k=
+github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew=
 github.com/multiformats/go-multihash v0.2.3 h1:7Lyc8XfX/IY2jWb/gI7JP+o7JEq9hOa7BFvVU9RSh+U=
 github.com/multiformats/go-multihash v0.2.3/go.mod h1:dXgKXCXjBzdscBLk9JkjINiEsCKRVch90MdaGiKsvSM=
 github.com/multiformats/go-multistream v0.6.0 h1:ZaHKbsL404720283o4c/IHQXiS6gb8qAN5EIJ4PN5EA=
@@ -552,11 +605,15 @@ github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9k
 github.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc=
 github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
 github.com/onsi/ginkgo/v2 v2.0.0/go.mod h1:vw5CSIxN1JObi/U8gcbwft7ZxR2dgaR70JSE3/PpL4c=
+github.com/onsi/ginkgo/v2 v2.22.0 h1:Yed107/8DjTr0lKCNt7Dn8yQ6ybuDRQoMGrNFKzMfHg=
+github.com/onsi/ginkgo/v2 v2.22.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
 github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
 github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
 github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE=
 github.com/onsi/gomega v1.18.1/go.mod h1:0q+aL8jAiMXy9hbwj2mr5GziHiwhAIQpFmmtT5hitRs=
+github.com/opencontainers/runtime-spec v1.2.0 h1:z97+pHb3uELt/yiAWD691HNHQIF07bE7dzrbT927iTk=
+github.com/opencontainers/runtime-spec v1.2.0/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
 github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
 github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
 github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 h1:onHthvaw9LFnH4t2DcNVpwGmV9E1BkGknEliJkfwQj0=
@@ -568,6 +625,40 @@ github.com/pierrec/lz4 v2.6.1+incompatible h1:9UY3+iC23yxF0UfGaYrGplQ+79Rg+h/q9F
 github.com/pierrec/lz4 v2.6.1+incompatible/go.mod h1:pdkljMzZIN41W+lC3N2tnIh5sFi+IEE17M5jbnwPHcY=
 github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4=
 github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
+github.com/pion/datachannel v1.5.10 h1:ly0Q26K1i6ZkGf42W7D4hQYR90pZwzFOjTq5AuCKk4o=
+github.com/pion/datachannel v1.5.10/go.mod h1:p/jJfC9arb29W7WrxyKbepTU20CFgyx5oLo8Rs4Py/M=
+github.com/pion/dtls/v2 v2.2.12 h1:KP7H5/c1EiVAAKUmXyCzPiQe5+bCJrpOeKg/L05dunk=
+github.com/pion/dtls/v2 v2.2.12/go.mod h1:d9SYc9fch0CqK90mRk1dC7AkzzpwJj6u2GU3u+9pqFE=
+github.com/pion/ice/v2 v2.3.37 h1:ObIdaNDu1rCo7hObhs34YSBcO7fjslJMZV0ux+uZWh0=
+github.com/pion/ice/v2 v2.3.37/go.mod h1:mBF7lnigdqgtB+YHkaY/Y6s6tsyRyo4u4rPGRuOjUBQ=
+github.com/pion/interceptor v0.1.37 h1:aRA8Zpab/wE7/c0O3fh1PqY0AJI3fCSEM5lRWJVorwI=
+github.com/pion/interceptor v0.1.37/go.mod h1:JzxbJ4umVTlZAf+/utHzNesY8tmRkM2lVmkS82TTj8Y=
+github.com/pion/logging v0.2.2 h1:M9+AIj/+pxNsDfAT64+MAVgJO0rsyLnoJKCqf//DoeY=
+github.com/pion/logging v0.2.2/go.mod h1:k0/tDVsRCX2Mb2ZEmTqNa7CWsQPc+YYCB7Q+5pahoms=
+github.com/pion/mdns v0.0.12 h1:CiMYlY+O0azojWDmxdNr7ADGrnZ+V6Ilfner+6mSVK8=
+github.com/pion/mdns v0.0.12/go.mod h1:VExJjv8to/6Wqm1FXK+Ii/Z9tsVk/F5sD/N70cnYFbk=
+github.com/pion/randutil v0.1.0 h1:CFG1UdESneORglEsnimhUjf33Rwjubwj6xfiOXBa3mA=
+github.com/pion/randutil v0.1.0/go.mod h1:XcJrSMMbbMRhASFVOlj/5hQial/Y8oH/HVo7TBZq+j8=
+github.com/pion/rtcp v1.2.15 h1:LZQi2JbdipLOj4eBjK4wlVoQWfrZbh3Q6eHtWtJBZBo=
+github.com/pion/rtcp v1.2.15/go.mod h1:jlGuAjHMEXwMUHK78RgX0UmEJFV4zUKOFHR7OP+D3D0=
+github.com/pion/rtp v1.8.10 h1:puphjdbjPB+L+NFaVuZ5h6bt1g5q4kFIoI+r5q/g0CU=
+github.com/pion/rtp v1.8.10/go.mod h1:8uMBJj32Pa1wwx8Fuv/AsFhn8jsgw+3rUC2PfoBZ8p4=
+github.com/pion/sctp v1.8.35 h1:qwtKvNK1Wc5tHMIYgTDJhfZk7vATGVHhXbUDfHbYwzA=
+github.com/pion/sctp v1.8.35/go.mod h1:EcXP8zCYVTRy3W9xtOF7wJm1L1aXfKRQzaM33SjQlzg=
+github.com/pion/sdp/v3 v3.0.9 h1:pX++dCHoHUwq43kuwf3PyJfHlwIj4hXA7Vrifiq0IJY=
+github.com/pion/sdp/v3 v3.0.9/go.mod h1:B5xmvENq5IXJimIO4zfp6LAe1fD9N+kFv+V/1lOdz8M=
+github.com/pion/srtp/v2 v2.0.20 h1:HNNny4s+OUmG280ETrCdgFndp4ufx3/uy85EawYEhTk=
+github.com/pion/srtp/v2 v2.0.20/go.mod h1:0KJQjA99A6/a0DOVTu1PhDSw0CXF2jTkqOoMg3ODqdA=
+github.com/pion/stun v0.6.1 h1:8lp6YejULeHBF8NmV8e2787BogQhduZugh5PdhDyyN4=
+github.com/pion/stun v0.6.1/go.mod h1:/hO7APkX4hZKu/D0f2lHzNyvdkTGtIy3NDmLR7kSz/8=
+github.com/pion/transport/v2 v2.2.10 h1:ucLBLE8nuxiHfvkFKnkDQRYWYfp8ejf4YBOPfaQpw6Q=
+github.com/pion/transport/v2 v2.2.10/go.mod h1:sq1kSLWs+cHW9E+2fJP95QudkzbK7wscs8yYgQToO5E=
+github.com/pion/transport/v3 v3.0.7 h1:iRbMH05BzSNwhILHoBoAPxoB9xQgOaJk+591KC9P1o0=
+github.com/pion/transport/v3 v3.0.7/go.mod h1:YleKiTZ4vqNxVwh77Z0zytYi7rXHl7j6uPLGhhz9rwo=
+github.com/pion/turn/v2 v2.1.6 h1:Xr2niVsiPTB0FPtt+yAWKFUkU1eotQbGgpTIld4x1Gc=
+github.com/pion/turn/v2 v2.1.6/go.mod h1:huEpByKKHix2/b9kmTAM3YoX6MKP+/D//0ClgUYR2fY=
+github.com/pion/webrtc/v3 v3.3.5 h1:ZsSzaMz/i9nblPdiAkZoP+E6Kmjw+jnyq3bEmU3EtRg=
+github.com/pion/webrtc/v3 v3.3.5/go.mod h1:liNa+E1iwyzyXqNUwvoMRNQ10x8h8FOeJKL8RkIbamE=
 github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -596,6 +687,14 @@ github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoG
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
 github.com/psiemens/sconfig v0.1.0 h1:xfWqW+TRpih7mXZIqKYTmpRhlZLQ1kbxV8EjllPv76s=
 github.com/psiemens/sconfig v0.1.0/go.mod h1:+MLKqdledP/8G3rOBpknbLh0IclCf4WneJUtS26JB2U=
+github.com/quic-go/qpack v0.5.1 h1:giqksBPnT/HDtZ6VhtFKgoLOWmlyo9Ei6u9PqzIMbhI=
+github.com/quic-go/qpack v0.5.1/go.mod h1:+PC4XFrEskIVkcLzpEkbLqq1uCoxPhQuvK5rH1ZgaEg=
+github.com/quic-go/quic-go v0.48.2 h1:wsKXZPeGWpMpCGSWqOcqpW2wZYic/8T3aqiOID0/KWE=
+github.com/quic-go/quic-go v0.48.2/go.mod h1:yBgs3rWBOADpga7F+jJsb6Ybg1LSYiQvwWlLX+/6HMs=
+github.com/quic-go/webtransport-go v0.8.1-0.20241018022711-4ac2c9250e66 h1:4WFk6u3sOT6pLa1kQ50ZVdm8BQFgJNA117cepZxtLIg=
+github.com/quic-go/webtransport-go v0.8.1-0.20241018022711-4ac2c9250e66/go.mod h1:Vp72IJajgeOL6ddqrAhmp7IM9zbTcgkQxD/YdxrVwMw=
+github.com/raulk/go-watchdog v1.3.0 h1:oUmdlHxdkXRJlwfG0O9omj8ukerm8MEQavSiDTEtBsk=
+github.com/raulk/go-watchdog v1.3.0/go.mod h1:fIvOnLbF0b0ZwkB9YU4mOW9Did//4vPZtDqv66NfsMU=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis=
 github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
@@ -703,6 +802,8 @@ github.com/vmihailenco/msgpack/v4 v4.3.11 h1:Q47CePddpNGNhk4GCnAx9DDtASi2rasatE0
 github.com/vmihailenco/msgpack/v4 v4.3.11/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
 github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
 github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
+github.com/wlynxg/anet v0.0.5 h1:J3VJGi1gvo0JwZ/P1/Yc/8p63SoW98B5dHkYDmpgvvU=
+github.com/wlynxg/anet v0.0.5/go.mod h1:eay5PRQr7fIVAMbTbchTnO9gG65Hg/uYGdc7mguHxoA=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
 github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
@@ -752,9 +853,15 @@ go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/atomic v1.11.0 h1:ZvwS0R+56ePWxUNi+Atn9dWONBPp/AUETXlHW0DxSjE=
 go.uber.org/atomic v1.11.0/go.mod h1:LUxbIzbOniOlMKjJjyPfpl4v+PKK2cNJn91OQbhoJI0=
+go.uber.org/dig v1.18.0 h1:imUL1UiY0Mg4bqbFfsRQO5G4CGRBec/ZujWTvSVp3pw=
+go.uber.org/dig v1.18.0/go.mod h1:Us0rSJiThwCv2GteUN0Q7OKvU7n5J4dxZ9JKUXozFdE=
+go.uber.org/fx v1.23.0 h1:lIr/gYWQGfTwGcSXWXu4vP5Ws6iqnNEIY+F/aFzCKTg=
+go.uber.org/fx v1.23.0/go.mod h1:o/D9n+2mLP6v1EG+qsdT1O8wKopYAsqZasju97SDFCU=
 go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
+go.uber.org/mock v0.5.0 h1:KAMbZvZPyBPWgD14IrIQ38QCyjwpvVVV6K/bHl1IwQU=
+go.uber.org/mock v0.5.0/go.mod h1:ge71pBPLYDk7QIi1LupWxdAykm7KIEFchiOqd6z7qMM=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
 go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
@@ -770,6 +877,7 @@ golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnf
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
diff --git a/services/ingestion/event_subscriber.go b/services/ingestion/event_subscriber.go
index bc9d37084..49a5f05eb 100644
--- a/services/ingestion/event_subscriber.go
+++ b/services/ingestion/event_subscriber.go
@@ -12,6 +12,7 @@ import (
 	"github.com/onflow/flow-evm-gateway/models"
 	errs "github.com/onflow/flow-evm-gateway/models/errors"
 	"github.com/onflow/flow-evm-gateway/services/requester"
+	"github.com/onflow/flow-evm-gateway/services/requester/keystore"
 
 	"github.com/onflow/flow-go-sdk"
 	"github.com/onflow/flow-go-sdk/access"
@@ -35,7 +36,7 @@ type RPCEventSubscriber struct {
 
 	client  *requester.CrossSporkClient
 	chain   flowGo.ChainID
-	keyLock requester.KeyLock
+	keyLock keystore.KeyLock
 	height  uint64
 
 	recovery        bool
@@ -46,7 +47,7 @@ func NewRPCEventSubscriber(
 	logger zerolog.Logger,
 	client *requester.CrossSporkClient,
 	chainID flowGo.ChainID,
-	keyLock requester.KeyLock,
+	keyLock keystore.KeyLock,
 	startHeight uint64,
 ) *RPCEventSubscriber {
 	logger = logger.With().Str("component", "subscriber").Logger()
@@ -173,9 +174,9 @@ func (r *RPCEventSubscriber) subscribe(ctx context.Context, height uint64) <-cha
 					}
 				}
 				for _, evt := range blockEvents.Events {
-					r.keyLock.UnlockKey(evt.TransactionID)
+					r.keyLock.NotifyTransaction(evt.TransactionID)
 				}
-				r.keyLock.Notify(blockEvents.Height)
+				r.keyLock.NotifyBlock(blockEvents.Height)
 
 				eventsChan <- evmEvents
 
diff --git a/services/ingestion/event_subscriber_test.go b/services/ingestion/event_subscriber_test.go
index 86131d19c..650e53175 100644
--- a/services/ingestion/event_subscriber_test.go
+++ b/services/ingestion/event_subscriber_test.go
@@ -13,6 +13,7 @@ import (
 	"github.com/onflow/flow-evm-gateway/models"
 	errs "github.com/onflow/flow-evm-gateway/models/errors"
 	"github.com/onflow/flow-evm-gateway/services/requester"
+	"github.com/onflow/flow-evm-gateway/services/requester/keystore"
 	"github.com/onflow/flow-evm-gateway/services/testutils"
 
 	flowGo "github.com/onflow/flow-go/model/flow"
@@ -43,7 +44,7 @@ func Test_Subscribing(t *testing.T) {
 	)
 	require.NoError(t, err)
 
-	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, requester.NewKeyStore(nil), 1)
+	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, keystore.New(nil), 1)
 
 	events := subscriber.Subscribe(context.Background())
 
@@ -83,7 +84,7 @@ func Test_MissingBlockEvent(t *testing.T) {
 	)
 	require.NoError(t, err)
 
-	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, requester.NewKeyStore(nil), 1)
+	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, keystore.New(nil), 1)
 
 	events := subscriber.Subscribe(context.Background())
 
@@ -185,7 +186,7 @@ func Test_SubscribingWithRetryOnError(t *testing.T) {
 	)
 	require.NoError(t, err)
 
-	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, requester.NewKeyStore(nil), 1)
+	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, keystore.New(nil), 1)
 
 	events := subscriber.Subscribe(context.Background())
 
@@ -248,7 +249,7 @@ func Test_SubscribingWithRetryOnErrorMultipleBlocks(t *testing.T) {
 	)
 	require.NoError(t, err)
 
-	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, requester.NewKeyStore(nil), 1)
+	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, keystore.New(nil), 1)
 
 	events := subscriber.Subscribe(context.Background())
 
@@ -310,7 +311,7 @@ func Test_SubscribingWithRetryOnErrorEmptyBlocks(t *testing.T) {
 	)
 	require.NoError(t, err)
 
-	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, requester.NewKeyStore(nil), 1)
+	subscriber := NewRPCEventSubscriber(zerolog.Nop(), client, flowGo.Previewnet, keystore.New(nil), 1)
 
 	events := subscriber.Subscribe(context.Background())
 
diff --git a/services/requester/key_store.go b/services/requester/key_store.go
deleted file mode 100644
index 9338d4d43..000000000
--- a/services/requester/key_store.go
+++ /dev/null
@@ -1,133 +0,0 @@
-package requester
-
-import (
-	"fmt"
-	"sync"
-
-	flowsdk "github.com/onflow/flow-go-sdk"
-	"github.com/onflow/flow-go-sdk/crypto"
-	"github.com/onflow/flow-go/model/flow"
-)
-
-var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
-
-const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
-
-type AccountKey struct {
-	flowsdk.AccountKey
-
-	mu            sync.Mutex
-	ks            *KeyStore
-	Address       flowsdk.Address
-	Signer        crypto.Signer
-	lastUsedBlock uint64
-}
-
-// Done unlocks a key after use and puts it back into the pool.
-func (k *AccountKey) Done() {
-	k.mu.Lock()
-	defer k.mu.Unlock()
-
-	k.ks.availableKeys <- k
-}
-
-func (k *AccountKey) SetProposerPayerAndSign(
-	tx *flowsdk.Transaction,
-	account *flowsdk.Account,
-) error {
-	if k.Address != account.Address {
-		return fmt.Errorf(
-			"expected address: %v, got address: %v",
-			k.Address,
-			account.Address,
-		)
-	}
-	if k.Index >= uint32(len(account.Keys)) {
-		return fmt.Errorf(
-			"key index: %d exceeds keys length: %d",
-			k.Index,
-			len(account.Keys),
-		)
-	}
-	seqNumber := account.Keys[k.Index].SequenceNumber
-
-	return tx.
-		SetProposalKey(k.Address, k.Index, seqNumber).
-		SetPayer(k.Address).
-		SignEnvelope(k.Address, k.Index, k.Signer)
-}
-
-type KeyLock interface {
-	LockKey(
-		txID flowsdk.Identifier,
-		referenceBlockHeight uint64,
-		key *AccountKey,
-	)
-	UnlockKey(txID flowsdk.Identifier)
-	Notify(blockHeight uint64)
-}
-
-type KeyStore struct {
-	availableKeys chan *AccountKey
-	usedKeys      map[flowsdk.Identifier]*AccountKey
-	size          int
-}
-
-var _ KeyLock = (*KeyStore)(nil)
-
-func NewKeyStore(keys []*AccountKey) *KeyStore {
-	ks := &KeyStore{
-		usedKeys: map[flowsdk.Identifier]*AccountKey{},
-	}
-
-	availableKeys := make(chan *AccountKey, len(keys))
-	for _, key := range keys {
-		key.ks = ks
-		availableKeys <- key
-	}
-	ks.size = len(keys)
-	ks.availableKeys = availableKeys
-
-	return ks
-}
-
-func (k *KeyStore) AvailableKeys() int {
-	return k.size - len(k.usedKeys)
-}
-
-func (k *KeyStore) Take() (*AccountKey, error) {
-	select {
-	case key := <-k.availableKeys:
-		return key, nil
-	default:
-		return nil, ErrNoKeysAvailable
-	}
-}
-
-func (k *KeyStore) LockKey(
-	txID flowsdk.Identifier,
-	referenceBlockHeight uint64,
-	key *AccountKey,
-) {
-	key.mu.Lock()
-	defer key.mu.Unlock()
-
-	key.lastUsedBlock = referenceBlockHeight
-	k.usedKeys[txID] = key
-}
-
-func (k *KeyStore) UnlockKey(txID flowsdk.Identifier) {
-	key, ok := k.usedKeys[txID]
-	if ok && key != nil {
-		key.Done()
-		delete(k.usedKeys, txID)
-	}
-}
-
-func (k *KeyStore) Notify(blockHeight uint64) {
-	for txID, key := range k.usedKeys {
-		if blockHeight-key.lastUsedBlock >= accountKeyBlockExpiration {
-			k.UnlockKey(txID)
-		}
-	}
-}
diff --git a/services/requester/keystore/account_key.go b/services/requester/keystore/account_key.go
new file mode 100644
index 000000000..45acd98bc
--- /dev/null
+++ b/services/requester/keystore/account_key.go
@@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
+
+	return tx.
+		SetProposalKey(k.Address, k.Index, seqNumber).
+		SetPayer(k.Address).
+		SignEnvelope(k.Address, k.Index, k.Signer)
+}
+
+// lock reserves a key for use in a transaction.
+// should only be called by the keystore
+func (k *AccountKey) lock() bool {
+	return k.inUse.CompareAndSwap(false, true)
+}
diff --git a/services/requester/keystore/key_store.go b/services/requester/keystore/key_store.go
new file mode 100644
index 000000000..a64fe6dbf
--- /dev/null
+++ b/services/requester/keystore/key_store.go
@@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.Mutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
+
+// NotifyTransaction unlocks a key after use and puts it back into the pool.
+func (k *KeyStore) NotifyTransaction(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unsafeUnlockKey(txID)
+}
+
+// NotifyBlock is called to notify the KeyStore of a new ingested block height.
+// Pending transactions older than a threshold number of blocks are removed.
+func (k *KeyStore) NotifyBlock(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	for txID, key := range k.usedKeys {
+		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
+			k.unsafeUnlockKey(txID)
+		}
+	}
+}
+
+// unsafeUnlockKey unlocks a key referenced by the transaction ID set during setLockMetadata
+// the caller must hold the keyMu lock
+func (k *KeyStore) unsafeUnlockKey(txID flowsdk.Identifier) {
+	if key, ok := k.usedKeys[txID]; ok {
+		key.Done()
+		delete(k.usedKeys, txID)
+	}
+}
+
+// release puts a key back into the pool.
+func (k *KeyStore) release(key *AccountKey) {
+	k.availableKeys <- key
+}
+
+// setLockMetadata sets the transaction ID for a key reservation.
+// this method is called by the key's SetLockMetadata method
+func (k *KeyStore) setLockMetadata(
+	key *AccountKey,
+	txID flowsdk.Identifier,
+) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+	k.usedKeys[txID] = key
+}
diff --git a/services/requester/keystore/key_store_test.go b/services/requester/keystore/key_store_test.go
new file mode 100644
index 000000000..eb7151ec9
--- /dev/null
+++ b/services/requester/keystore/key_store_test.go
@@ -0,0 +1,212 @@
+package keystore
+
+import (
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	sdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/test"
+	"github.com/onflow/flow-go/utils/unittest"
+	"golang.org/x/sync/errgroup"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestTake(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	keys := make([]*AccountKey, 0, 2)
+	for range 2 {
+		accountKey, signer := keyGenerator.NewWithSigner()
+		keys = append(keys, NewAccountKey(*accountKey, addrGenerator.New(), signer))
+	}
+
+	ks := New(keys)
+
+	t.Run("Take with no metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		// key is reserved
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.True(t, key.inUse.Load())
+
+		key.Done()
+
+		// key is released
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.Equal(t, key, ks.usedKeys[txID])
+
+		ks.NotifyTransaction(txID)
+
+		// keystore and key should be reset
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with expiration", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.Equal(t, key, ks.usedKeys[txID])
+
+		// notify for one block before the expiration block, key should still be reserved
+		ks.NotifyBlock(blockHeight + accountKeyBlockExpiration - 1)
+
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, key, ks.usedKeys[txID])
+
+		// notify for the expiration block
+		ks.NotifyBlock(blockHeight + accountKeyBlockExpiration)
+
+		// keystore and key should be reset
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.False(t, key.inUse.Load())
+	})
+}
+
+func TestKeySigning(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	address := addrGenerator.New()
+	accountKey, signer := keyGenerator.NewWithSigner()
+	accountKey.Index = 0 // the fixture starts from index 1
+	accountKey.SequenceNumber = 42
+
+	ks := New([]*AccountKey{
+		NewAccountKey(*accountKey, address, signer),
+	})
+
+	key, err := ks.Take()
+	require.NoError(t, err)
+
+	account := &sdk.Account{
+		Address: address,
+		Keys:    []*sdk.AccountKey{accountKey},
+	}
+	tx := sdk.NewTransaction()
+
+	err = key.SetProposerPayerAndSign(tx, account)
+	require.NoError(t, err)
+
+	assert.Equal(t, account.Address, tx.ProposalKey.Address)
+	assert.Equal(t, account.Keys[0].Index, tx.ProposalKey.KeyIndex)
+	assert.Equal(t, account.Keys[0].SequenceNumber, tx.ProposalKey.SequenceNumber)
+	assert.Equal(t, account.Address, tx.Payer)
+	assert.NotEmpty(t, tx.EnvelopeSignatures)
+}
+
+func TestConcurrentUse(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	// number of keys to add
+	keyCount := uint32(100)
+
+	// total tx to send
+	totalTxCount := 10_000
+
+	// number of tx to send in parallel
+	// should be less than keyCount otherwise the extra workers will just be spinning waiting for
+	// keys within the test code.
+	concurrentTxCount := 75
+
+	keys := make([]*AccountKey, 0, keyCount)
+	for i := range keyCount {
+		accountKey, signer := keyGenerator.NewWithSigner()
+		key := NewAccountKey(*accountKey, addrGenerator.New(), signer)
+		key.Index = i
+		key.SequenceNumber = uint64(i)
+
+		keys = append(keys, key)
+	}
+
+	ks := New(keys)
+
+	g := errgroup.Group{}
+	g.SetLimit(concurrentTxCount)
+
+	wg := sync.WaitGroup{}
+
+	for i := range totalTxCount {
+		g.Go(func() error {
+			var key *AccountKey
+			for {
+				var err error
+				key, err = ks.Take()
+				if errors.Is(err, ErrNoKeysAvailable) {
+					time.Sleep(100 * time.Microsecond)
+					continue
+				}
+				require.NoError(t, err)
+				break
+			}
+
+			if i%7 == 0 {
+				// cause some keys to be released early simulating errors
+				key.Done()
+				return nil
+			}
+
+			// the rest should be released after a delay
+			txID := identifierFixture()
+			key.SetLockMetadata(txID, uint64(10))
+
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+				time.Sleep(3 * time.Millisecond)
+				ks.NotifyTransaction(txID)
+			}()
+
+			return nil
+		})
+	}
+
+	err := g.Wait()
+	require.NoError(t, err)
+
+	// all the notifier goroutines are finished by now
+	wg.Wait()
+
+	assert.Equal(t, int(keyCount), ks.AvailableKeys(), "all keys should be available")
+	assert.Empty(t, ks.usedKeys, "no keys should be in use")
+}
+
+func identifierFixture() sdk.Identifier {
+	id := unittest.IdentifierFixture()
+	return sdk.BytesToID(id[:])
+}
diff --git a/services/requester/requester.go b/services/requester/requester.go
index a37c758a9..ddf56033b 100644
--- a/services/requester/requester.go
+++ b/services/requester/requester.go
@@ -20,6 +20,7 @@ import (
 	gethCore "github.com/onflow/go-ethereum/core"
 	"github.com/onflow/go-ethereum/core/txpool"
 	"github.com/onflow/go-ethereum/core/types"
+	gethParams "github.com/onflow/go-ethereum/params"
 	"github.com/rs/zerolog"
 	"github.com/sethvargo/go-limiter"
 	"github.com/sethvargo/go-limiter/memorystore"
@@ -30,10 +31,9 @@ import (
 	"github.com/onflow/flow-evm-gateway/metrics"
 	"github.com/onflow/flow-evm-gateway/models"
 	errs "github.com/onflow/flow-evm-gateway/models/errors"
+	"github.com/onflow/flow-evm-gateway/services/requester/keystore"
 	"github.com/onflow/flow-evm-gateway/storage"
 	"github.com/onflow/flow-evm-gateway/storage/pebble"
-
-	gethParams "github.com/onflow/go-ethereum/params"
 )
 
 var (
@@ -106,7 +106,7 @@ type EVM struct {
 	logger        zerolog.Logger
 	blocks        storage.BlockIndexer
 	mux           sync.Mutex
-	keystore      *KeyStore
+	keystore      *keystore.KeyStore
 
 	head              *types.Header
 	evmSigner         types.Signer
@@ -123,7 +123,7 @@ func NewEVM(
 	blocks storage.BlockIndexer,
 	txPool *TxPool,
 	collector metrics.Collector,
-	keystore *KeyStore,
+	keystore *keystore.KeyStore,
 ) (*EVM, error) {
 	logger = logger.With().Str("component", "requester").Logger()
 
@@ -619,11 +619,6 @@ func (e *EVM) buildTransaction(
 		return nil, err
 	}
 
-	accKey, err := e.keystore.Take()
-	if err != nil {
-		return nil, err
-	}
-
 	flowTx := flow.NewTransaction().
 		SetScript(script).
 		SetReferenceBlockID(latestBlock.ID).
@@ -635,10 +630,18 @@ func (e *EVM) buildTransaction(
 		}
 	}
 
+	accKey, err := e.keystore.Take()
+	if err != nil {
+		return nil, err
+	}
+
 	if err := accKey.SetProposerPayerAndSign(flowTx, account); err != nil {
+		accKey.Done()
 		return nil, err
 	}
-	e.keystore.LockKey(flowTx.ID(), latestBlock.Height, accKey)
+
+	// now that the transaction is prepped, store the transaction's metadata
+	accKey.SetLockMetadata(flowTx.ID(), latestBlock.Height)
 
 	e.collector.OperatorBalance(account)
 


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 44ca5b43d1ed6d9d0c37d60ab40d1cca56c0c06d and 6ab1011863b10b67d7fa6936534a3133dd449b24.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `services/requester/key_store.go` (3 hunks)

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Test

</details>

<details>
<summary>🔇 Additional comments (6)</summary><blockquote>

<details>
<summary>services/requester/key_store.go (6)</summary>

`74-75`: **Good addition of an RWMutex for concurrency.**

Introducing `keyMu sync.RWMutex` is a clean approach to protect shared state in `KeyStore`, allowing concurrent reads while ensuring exclusive writes.

---

`96-102`: **Safe use of a read lock for available key calculation.**

Using `k.keyMu.RLock()` and `defer k.keyMu.RUnlock()` ensures that concurrent readers do not block each other when determining the count of available keys.

---

`104-105`: **Channel-based approach for key retrieval looks fine.**

Providing a doc comment clarifies usage. The select on `availableKeys` avoids contention on the map, leveraging Go channels for key reservation.

---

`114-127`: **Proper locking sequence for updating key usage.**

Locking the `AccountKey` before updating `lastUsedBlock`, then protecting `usedKeys` with `keyMu`, correctly segregates locks and prevents overlapping lock scopes. This approach helps avoid direct lock ordering conflicts.

---

`129-135`: **Straightforward unlocked key cleanup.**

Calling `k.unlockKey(txID)` under `k.keyMu.Lock()` ensures consistent map manipulation and safe return of the key to the channel.

---

`137-143`: **Modular key unlocking logic.**

Separating out `unlockKey` clarifies unlocking steps and map cleanup. This method is properly invoked inside protected sections of code.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T19:09:48Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (3)</summary><blockquote>

<details>
<summary>services/requester/requester.go (1)</summary><blockquote>

`109-109`: **Add concurrency consideration.**  
The new `keystore` field allows concurrency usage for key management. Ensure that usage is properly synchronized, especially in combination with the `e.mux` lock for building transactions.

</blockquote></details>
<details>
<summary>services/requester/keystore/key_store_test.go (1)</summary><blockquote>

`100-130`: **Strong coverage of signing flow in `TestKeySigning`.**  
The test ensures that the transaction's fields are properly set and signed. Consider adding negative tests (e.g., invalid address, key index out of range) to bolster robustness.

</blockquote></details>
<details>
<summary>services/requester/keystore/account_key.go (1)</summary><blockquote>

`12-25`: **Unused `mu` field.**  
The embedded `mu` field is never used, leading to a potential code smell. Consider removing it, or utilize it for concurrency if that was the original intent.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 GitHub Check: Lint</summary>

[failure] 15-15: 
field `mu` is unused (unused)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 6ab1011863b10b67d7fa6936534a3133dd449b24 and acd7c6d5311ed2751bd43dbba7ec20bc6ba1a497.

</details>

<details>
<summary>⛔ Files ignored due to path filters (1)</summary>

* `go.sum` is excluded by `!**/*.sum`

</details>

<details>
<summary>📒 Files selected for processing (9)</summary>

* `bootstrap/bootstrap.go` (4 hunks)
* `go.mod` (4 hunks)
* `services/ingestion/event_subscriber.go` (4 hunks)
* `services/ingestion/event_subscriber_test.go` (6 hunks)
* `services/requester/key_store.go` (0 hunks)
* `services/requester/keystore/account_key.go` (1 hunks)
* `services/requester/keystore/key_store.go` (1 hunks)
* `services/requester/keystore/key_store_test.go` (1 hunks)
* `services/requester/requester.go` (5 hunks)

</details>

<details>
<summary>💤 Files with no reviewable changes (1)</summary>

* services/requester/key_store.go

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 GitHub Check: Lint</summary>

<details>
<summary>services/requester/keystore/account_key.go</summary>

[failure] 15-15: 
field `mu` is unused (unused)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Test

</details>

<details>
<summary>🔇 Additional comments (45)</summary><blockquote>

<details>
<summary>services/ingestion/event_subscriber_test.go (6)</summary>

`16-16`: **Import change aligns with new keystore package structure.**

The import change reflects the migration of the KeyStore to a dedicated keystore package, which is part of the solution for fixing concurrent map access.

---

`47-47`: **Instance creation updated to use new keystore package.**

This change replaces `requester.NewKeyStore(nil)` with `keystore.New(nil)` to use the new thread-safe keystore implementation.

---

`87-87`: **Instance creation consistent with keystore migration.**

The update maintains consistency with the new keystore package usage throughout the test cases.

---

`189-189`: **Consistent keystore instantiation in test case.**

Same change as above, ensuring all test cases use the new thread-safe keystore implementation.

---

`252-252`: **Consistent keystore instantiation in test case.**

Same change as above, continuing the pattern of consistent adoption of the new keystore.

---

`314-314`: **Consistent keystore instantiation in final test case.**

Completes the migration to the new keystore implementation across all test cases.

</details>
<details>
<summary>services/ingestion/event_subscriber.go (5)</summary>

`15-15`: **Import change aligns with new keystore package structure.**

The import change reflects the migration of the KeyStore to a dedicated keystore package, which is part of the solution for fixing concurrent map access.

---

`39-39`: **Field type updated to use thread-safe keystore implementation.**

The `keyLock` field has been updated to use the new `keystore.KeyLock` type, which has thread-safe operations to prevent concurrent map access issues.

---

`50-50`: **Parameter type updated for consistency with field type change.**

This change ensures the constructor function parameter matches the updated field type.

---

`177-178`: **Method name changed to be more semantically meaningful.**

The method `UnlockKey` has been renamed to `NotifyTransaction`, which better describes its purpose in the context of transaction processing and improves code readability.

---

`179-179`: **Method name changed to be more semantically meaningful.**

The method `Notify` has been renamed to `NotifyBlock`, which better describes its purpose in the context of block processing and improves code readability.

</details>
<details>
<summary>go.mod (3)</summary>

`22-22`: **Added atomic package to support thread-safe operations.**

The addition of `go.uber.org/atomic` as a direct dependency suggests the new keystore implementation uses atomic operations to ensure thread safety, which is a good approach for fixing concurrent map access issues.

---

`98-99`: **Added new dependencies for enhanced functionality.**

The addition of go-clone libraries suggests the new implementation may be making deep copies of objects to avoid shared state issues, further enhancing thread safety.

---

`121-122`: **Added network-related dependencies.**

These additional dependencies appear to be related to network communication. They may be indirect requirements from other updated packages, or they could be used for improved connection management in a concurrent environment.



Also applies to: 135-135

</details>
<details>
<summary>bootstrap/bootstrap.go (5)</summary>

`35-35`: **Import updated to use new keystore package.**

This change is consistent with the migration of the KeyStore implementation to a dedicated package that addresses concurrent access issues.

---

`80-80`: **Field type updated to use thread-safe keystore implementation.**

The `keystore` field has been updated to use the new `keystore.KeyStore` type, which implements thread-safe operations.

---

`223-223`: **Variable type updated for new keystore implementation.**

This change updates the account keys type to match the new keystore package, ensuring type compatibility.

---

`243-247`: **Constructor function used for creating account keys.**

Rather than directly creating an AccountKey struct, the code now uses a constructor function `keystore.NewAccountKey()`, which is a good practice as it encapsulates the creation logic and allows for any initialization the new implementation might require.

---

`251-251`: **Keystore initialization updated to use new implementation.**

This change replaces `requester.NewKeyStore(accountKeys)` with `keystore.New(accountKeys)` to use the new thread-safe keystore implementation.

</details>
<details>
<summary>services/requester/requester.go (6)</summary>

`23-23`: **No functional issues found in import statement.**

---

`34-34`: **No functional issues found in import statement.**

---

`126-126`: **No issues found in constructor parameter.**

---

`634-634`: **Good practice releasing the key on error.**  
This ensures that the key is freed if adding an argument fails.

---

`640-640`: **Proper cleanup on error during transaction signing.**  
The call to `accKey.Done()` correctly frees the key in case of an error.

---

`644-644`: **Lock metadata set effectively.**  
The call to `SetLockMetadata` ensures the key is locked until it is eventually released.

</details>
<details>
<summary>services/requester/keystore/key_store_test.go (3)</summary>

`18-98`: **Validate key reservation and release logic in `TestTake`.**  
The tests cover scenarios where keys are taken, assigned metadata, and released. The logic for block-based expiration is robust. Consider verifying leftover transaction references to ensure thorough coverage.

---

`132-197`: **Excellent concurrency coverage in `TestConcurrentUse`.**  
It effectively validates that multiple goroutines can safely take and release keys. The approach with `errgroup` and timed sleeps is valuable for concurrency stress testing.

---

`199-202`: **Fixture usage for consistent test data.**  
The `identifierFixture` function centralizes test ID creation and helps maintain consistency.

</details>
<details>
<summary>services/requester/keystore/account_key.go (5)</summary>

`27-39`: **Clean constructor for `AccountKey`.**  
The function clearly initializes concurrency flags (`inUse`, `lastLockedBlock`). Straightforward and concise.

---

`41-47`: **Robust key release with `Done()`.**  
`Done` ensures that the key is only returned to the pool if it was indeed marked as in use, preventing accidental double-release.

---

`49-54`: **Lock metadata maintenance.**  
Storing the reference block height and associating it with the transaction ID helps the keystore manage expiration accurately.

---

`56-81`: **Proper validation in `SetProposerPayerAndSign`.**  
It checks addresses and index bounds. Error messages are descriptive, aiding quick troubleshooting.

---

`83-87`: **Atomic key locking.**  
This ensures only one active reservation of the key, reinforcing concurrency safety within the keystore.

</details>
<details>
<summary>services/requester/keystore/key_store.go (12)</summary>

`11-11`: **Meaningful error handling.**  
`ErrNoKeysAvailable` provides clear feedback when all keys are in use.

---

`13-13`: **Dependence on Flow's default expiration.**  
Ensure that `accountKeyBlockExpiration` matches the intended usage timeframe. This might need revisiting if network parameters change.

---

`15-18`: **Well-defined `KeyLock` interface.**  
It clarifies the responsibilities for transaction and block height notifications.

---

`20-25`: **Thread-safe design for `KeyStore`.**  
Using a channel of available keys plus a map of used keys with a RW mutex is well structured for concurrency.

---

`29-43`: **Initialization in `New` function.**  
Populating `availableKeys` at creation is straightforward and sets up safe concurrent usage.

---

`45-48`: **Efficient retrieval of available key count.**  
Using `len(k.availableKeys)` is direct and avoids extra synchronization overhead.

---

`50-62`: **Correct usage of non-blocking select in `Take()`.**  
Returning `ErrNoKeysAvailable` if the channel is empty aligns with the KeyStore's concurrency design.

---

`64-70`: **Proper key unlocking in `NotifyTransaction`.**  
The RW mutex ensures state consistency while removing the key from `usedKeys`.

---

`72-83`: **Removing stale keys in `NotifyBlock`.**  
This ensures that keys locked too long are returned, preserving system stability.

---

`85-90`: **Synchronized `unlockKey` flow.**  
Calls `Done()` if found in `usedKeys` to ensure the key is re-released.

---

`92-94`: **`release` reintroduces keys to the pool.**  
This method conceptually matches the concurrency approach in `Take()`.

---

`96-103`: **`setLockMetadata` ensures trackability.**  
Associating the key with a specific transaction ID helps the store release it at the correct time.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T00:54:42Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 3**

<details>
<summary>🧹 Nitpick comments (4)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store.go (4)</summary><blockquote>

`11-11`: **Improve error handling with more descriptive message.**

The error message could be more informative by including context about when this might occur or potential remedies.

```diff
-var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available - all keys are currently in use")
```

---

`29-43`: **Consider adding validation for empty keys slice.**

The current implementation doesn't check if the input slice is empty, which could lead to a system with no available keys.

```diff
 func New(keys []*AccountKey) *KeyStore {
+	if len(keys) == 0 {
+		panic("cannot create KeyStore with zero keys")
+	}
 	ks := &KeyStore{
 		usedKeys: map[flowsdk.Identifier]*AccountKey{},
 	}
```

---

`72-83`: **Add metrics for key expirations.**

Consider adding metrics to track how often keys expire due to block height thresholds, as this could indicate issues with transaction processing.

```diff
 func (k *KeyStore) NotifyBlock(blockHeight uint64) {
 	k.keyMu.Lock()
 	defer k.keyMu.Unlock()
 
+	var expiredKeys int
 	for txID, key := range k.usedKeys {
 		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
 			k.unsafeUnlockKey(txID)
+			expiredKeys++
 		}
 	}
+	// Add metrics reporting for expiredKeys
 }
```

---

`1-109`: **Missing graceful shutdown mechanism.**

The KeyStore lacks a mechanism for graceful shutdown. Consider adding a Close() method that releases resources and handles pending operations.

```diff
+// Close gracefully shuts down the KeyStore, releasing any resources.
+// It returns the number of keys that were in use at the time of closing.
+func (k *KeyStore) Close() int {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+	
+	// Count and clear used keys
+	inUseCount := len(k.usedKeys)
+	k.usedKeys = nil
+	
+	// Close and clear available keys channel
+	close(k.availableKeys)
+	// Drain the channel
+	for range k.availableKeys {
+		// Just drain
+	}
+	k.availableKeys = nil
+	
+	return inUseCount
+}
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between acd7c6d5311ed2751bd43dbba7ec20bc6ba1a497 and b4e6b0c1315231fd765673dc3161c1d97803597c.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `services/requester/keystore/key_store.go` (1 hunks)

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Test

</details>

<details>
<summary>🔇 Additional comments (2)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store.go (2)</summary>

`15-18`: **LGTM - Interface design is clear and focused.**

The `KeyLock` interface defines a clean, minimal contract with two well-named methods for transaction and block notifications.

---

`20-25`: **Thread-safe design addresses the concurrent map access issue.**

The `KeyStore` structure uses a combination of a mutex (`keyMu`) and a channel (`availableKeys`) to ensure thread-safe access to keys, properly fixing the concurrent map access issue mentioned in the PR description.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:00:24Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store.go (1)</summary><blockquote>

`20-25`: **Add documentation to struct fields for better clarity.**

The `KeyStore` struct contains fields with no documentation. Adding descriptive comments to each field would improve code readability and maintenance.

```diff
 type KeyStore struct {
+	// availableKeys is a channel of keys that are available for use
 	availableKeys chan *AccountKey
+	// usedKeys maps transaction IDs to the keys they're using
 	usedKeys      map[flowsdk.Identifier]*AccountKey
+	// size is the total number of keys managed by this KeyStore
 	size          int
+	// keyMu protects concurrent access to the usedKeys map
 	keyMu         sync.Mutex
 }
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between b4e6b0c1315231fd765673dc3161c1d97803597c and f4a9e98b5416575c50cdd9858398fbf76baebb73.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `services/requester/keystore/account_key.go` (1 hunks)
* `services/requester/keystore/key_store.go` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* services/requester/keystore/account_key.go

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Test

</details>

<details>
<summary>🔇 Additional comments (5)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store.go (5)</summary>

`29-43`: **LGTM: Good implementation of the constructor.**

The `New` function correctly initializes the `KeyStore` with the provided keys, establishing the bidirectional relationship between keys and the keystore.

---

`50-62`: **Replace panic with error handling in production code.**

Using `panic` in production code can bring down the entire service. Consider logging the error instead and returning a more specific error to the caller.

```diff
 func (k *KeyStore) Take() (*AccountKey, error) {
 	select {
 	case key := <-k.availableKeys:
 		if !key.lock() {
 			// this should never happen and means there's a bug
-			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+			err := fmt.Errorf("internal error: key %d available, but locked - this indicates a bug", key.Index)
+			// Log the error here
+			return nil, err
 		}
 		return key, nil
 	default:
 		return nil, ErrNoKeysAvailable
 	}
 }
```

---

`64-83`: **LGTM: Good implementation of notification methods.**

The notification methods properly handle mutex locking to ensure thread safety when operating on shared state, which is crucial for the concurrent access fix.

---

`94-97`: **Implement non-blocking send in the release method.**

The current implementation could potentially block if the channel is full. Using a non-blocking send with a select statement would prevent potential deadlocks.

```diff
 func (k *KeyStore) release(key *AccountKey) {
-	k.availableKeys <- key
+	select {
+	case k.availableKeys <- key:
+		// Successfully released the key
+	default:
+		// Log a warning: channel is full, which shouldn't happen with current design
+		// This provides a safety mechanism against deadlocks
+	}
 }
```

---

`99-108`: **Add consistency validation in setLockMetadata.**

The method should verify the key isn't already associated with another transaction before adding it to the map.

```diff
 func (k *KeyStore) setLockMetadata(
 	key *AccountKey,
 	txID flowsdk.Identifier,
 ) {
 	k.keyMu.Lock()
 	defer k.keyMu.Unlock()
+	
+	// Check if key is already mapped to a different transaction
+	for existingID, existingKey := range k.usedKeys {
+		if existingKey == key && existingID != txID {
+			// Log warning about key being remapped
+			delete(k.usedKeys, existingID)
+			break
+		}
+	}
+	
 	k.usedKeys[txID] = key
 }
```

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:04:16Z
user: login: peterargue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:33:56Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:34:27Z
user: login: peterargue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:34:44Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:35:10Z
user: login: m-Peter
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:45:39Z
user: login: m-Peter
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:46:34Z
user: login: m-Peter
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:46:58Z
user: login: m-Peter
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:55:01Z
user: login: m-Peter
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:55:16Z
user: login: m-Peter
body: Awesome work :raised_hands: 
Left a few comments, feel free to ignore.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:56:50Z
user: login: peterargue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T17:19:19Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (5)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store_test.go (5)</summary><blockquote>

`137-145`: **Enforce constraint on concurrent transaction count**

The comment states that `concurrentTxCount` should be less than `keyCount`, but this constraint isn't enforced in code. Consider adding a check to prevent potential test issues.

```diff
// number of tx to send in parallel
// should be less than keyCount otherwise the extra workers will just be spinning waiting for
// keys within the test code.
concurrentTxCount := 75
+require.Less(t, concurrentTxCount, int(keyCount), "concurrentTxCount must be less than keyCount")
```

---

`169-173`: **Potential for test flakiness with fixed sleep time**

The retry mechanism with a fixed sleep time could make the test flaky in constrained environments. Consider using exponential backoff or a more reliable synchronization mechanism.

```diff
var key *AccountKey
for {
    var err error
    key, err = ks.Take()
    if errors.Is(err, ErrNoKeysAvailable) {
-       time.Sleep(100 * time.Microsecond)
+       // Use a context with timeout to avoid indefinite spinning
+       select {
+       case <-time.After(100 * time.Microsecond):
+           // Continue the retry loop
+       case <-t.Context().Done():
+           require.Fail(t, "Test timed out waiting for available keys")
+           return nil
+       }
        continue
    }
    require.NoError(t, err)
    break
}
```

---

`132-207`: **Add test for permanent key exhaustion scenario**

The current test verifies concurrent key usage but doesn't test the behavior when all keys are permanently exhausted. Consider adding a specific test case that verifies the system behavior when no keys are available for an extended period.

---

`191-192`: **Fixed sleep time in goroutine**

Using a fixed sleep time can make tests flaky. Consider using a more deterministic approach, like WaitGroups or channels, to synchronize these operations.

```diff
go func() {
    defer wg.Done()
-   time.Sleep(3 * time.Millisecond)
+   // Either use a configurable sleep time for tests
+   // or implement a more deterministic synchronization mechanism
    ks.NotifyTransaction(txID)
}()
```

---

`178-182`: **Add test coverage metrics**

The code intentionally releases some keys early to simulate errors (when i%7==0), but there's no verification that this branch is actually exercised. Consider adding a counter to verify that both code paths are tested.

```diff
+earlyReleaseCount := int32(0)

for i := range totalTxCount {
    g.Go(func() error {
        // ... existing code ...

        if i%7 == 0 {
            // cause some keys to be released early simulating errors
            key.Done()
+           atomic.AddInt32(&earlyReleaseCount, 1)
            return nil
        }

        // ... existing code ...
    })
}

// ... existing code ...

+assert.Greater(t, int(atomic.LoadInt32(&earlyReleaseCount)), 0, "early release code path should be exercised")
assert.Equal(t, int(keyCount), ks.AvailableKeys(), "all keys should be available")
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between f4a9e98b5416575c50cdd9858398fbf76baebb73 and 2416e3e36781dbdbed5fbec41d06772b93fdf93a.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `services/requester/keystore/key_store_test.go` (1 hunks)
* `services/requester/requester.go` (5 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* services/requester/requester.go

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Test

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>services/requester/keystore/key_store_test.go (4)</summary>

`1-207`: **Well-structured test suite for the keystore package**

This test suite thoroughly covers the key functionality of the keystore package, including single-threaded operations, signing, and concurrent access patterns. Good job implementing comprehensive tests that target the concurrent map access issues mentioned in the PR objectives.

---

`59-59`: **Direct access to internal state in assertion**

This assertion directly accesses the internal `usedKeys` map. While acceptable for unit tests, consider if there's a public method that could verify this relationship instead to make tests less dependent on implementation details.

---

`82-82`: **Direct access to internal state in assertion**

Same comment as above - direct access to internal `usedKeys` map.

---

`88-88`: **Direct access to internal state in assertion**

Same comment as above - direct access to internal `usedKeys` map.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T17:27:55Z
user: login: zhangchiqing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T19:03:18Z
user: login: peterargue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T19:16:02Z
user: login: zhangchiqing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T19:33:33Z
user: login: peterargue
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T20:08:51Z
user: login: zhangchiqing
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T20:26:46Z
user: login: zhangchiqing
body: Looks good.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-07T20:27:01Z

Review comments:
diff_hunk: @@ -104,30 +111,46 @@ func (k *KeyStore) Take() (*AccountKey, error) {
 	}
 }
 
+// LockKey locks a key for use in a transaction.
 func (k *KeyStore) LockKey(
 	txID flowsdk.Identifier,
 	referenceBlockHeight uint64,
 	key *AccountKey,
 ) {
 	key.mu.Lock()
-	defer key.mu.Unlock()
-
 	key.lastUsedBlock = referenceBlockHeight
+	key.mu.Unlock()
+
+	k.keyMu.Lock()
 	k.usedKeys[txID] = key
+	k.keyMu.Unlock()
 }
 
+// UnlockKey unlocks a key after use and puts it back into the pool.
 func (k *KeyStore) UnlockKey(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unlockKey(txID)
+}
+
+func (k *KeyStore) unlockKey(txID flowsdk.Identifier) {
 	key, ok := k.usedKeys[txID]
 	if ok && key != nil {
 		key.Done()
 		delete(k.usedKeys, txID)
 	}
 }
 
+// Notify is called to notify the KeyStore of a new ingested block height.
+// This is used to expire locks on keys after the transaction expiry time.
 func (k *KeyStore) Notify(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
 	for txID, key := range k.usedKeys {
 		if blockHeight-key.lastUsedBlock >= accountKeyBlockExpiration {
-			k.UnlockKey(txID)
+			k.unlockKey(txID)
 		}
 	}
 }
path: services/requester/key_store.go
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Potential data race when reading `lastUsedBlock` without locking `key.mu`.**

The `Notify` method locks `k.keyMu` to iterate `usedKeys` but reads `key.lastUsedBlock` outside of `key.mu.Lock()`. In Go, shared data needs uniform locking to avoid race conditions. As a result, updating and reading `lastUsedBlock` under different locks (or no lock) can lead to undefined behavior.


Below is one possible fix: lock each `AccountKey` while checking `lastUsedBlock`. Be mindful of lock ordering to prevent deadlocks—here, we never hold both locks at once in other methods, so it should be safe to nest `key.mu.Lock()` within `k.keyMu.Lock()`:

```diff
 func (k *KeyStore) Notify(blockHeight uint64) {
     k.keyMu.Lock()
     defer k.keyMu.Unlock()

     for txID, key := range k.usedKeys {
+        key.mu.Lock()
         if blockHeight - key.lastUsedBlock >= accountKeyBlockExpiration {
             k.unlockKey(txID)
         }
+        key.mu.Unlock()
     }
 }
```


<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
// Notify is called to notify the KeyStore of a new ingested block height.
// This is used to expire locks on keys after the transaction expiry time.
func (k *KeyStore) Notify(blockHeight uint64) {
	k.keyMu.Lock()
	defer k.keyMu.Unlock()

	for txID, key := range k.usedKeys {
		key.mu.Lock()
		if blockHeight-key.lastUsedBlock >= accountKeyBlockExpiration {
			k.unlockKey(txID)
		}
		key.mu.Unlock()
	}
}
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-06T19:09:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
+
+// NotifyTransaction unlocks a key after use and puts it back into the pool.
+func (k *KeyStore) NotifyTransaction(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unsafeUnlockKey(txID)
+}
+
+// NotifyBlock is called to notify the KeyStore of a new ingested block height.
+// Pending transactions older than a threshold number of blocks are removed.
+func (k *KeyStore) NotifyBlock(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	for txID, key := range k.usedKeys {
+		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
+			k.unsafeUnlockKey(txID)
+		}
+	}
+}
+
+// unsafeUnlockKey unlocks a key referenced by the transaction ID set during setLockMetadata
+// the caller must hold the keyMu lock
+func (k *KeyStore) unsafeUnlockKey(txID flowsdk.Identifier) {
+	if key, ok := k.usedKeys[txID]; ok {
+		key.Done()
+		delete(k.usedKeys, txID)
+	}
+}
+
+// release puts a key back into the pool.
+func (k *KeyStore) release(key *AccountKey) {
+	k.availableKeys <- key
+}
+
+// setLockMetadata sets the transaction ID for a key reservation.
+// this method is called by the key's SetLockMetadata method
+func (k *KeyStore) setLockMetadata(
+	key *AccountKey,
+	txID flowsdk.Identifier,
+) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+	k.usedKeys[txID] = key
+}
path: services/requester/keystore/key_store.go
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Add consistency validation in setLockMetadata.**

The method should verify the key isn't already associated with another transaction before adding it to the map.

```diff
 func (k *KeyStore) setLockMetadata(
 	key *AccountKey,
 	txID flowsdk.Identifier,
 ) {
 	k.keyMu.Lock()
 	defer k.keyMu.Unlock()
+	
+	// Check if key is already mapped to a different transaction
+	for existingID, existingKey := range k.usedKeys {
+		if existingKey == key && existingID != txID {
+			// Log warning about key being remapped
+			delete(k.usedKeys, existingID)
+			break
+		}
+	}
+	
 	k.usedKeys[txID] = key
 }
```


<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
// setLockMetadata sets the transaction ID for a key reservation.
// this method is called by the key's SetLockMetadata method
func (k *KeyStore) setLockMetadata(
	key *AccountKey,
	txID flowsdk.Identifier,
) {
	k.keyMu.Lock()
	defer k.keyMu.Unlock()

	// Check if key is already mapped to a different transaction
	for existingID, existingKey := range k.usedKeys {
		if existingKey == key && existingID != txID {
			// Log warning about key being remapped
			delete(k.usedKeys, existingID)
			break
		}
	}

	k.usedKeys[txID] = key
}
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-07T01:00:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
path: services/requester/keystore/key_store.go
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Replace panic with logging in production code.**

Using `panic` in production code can bring down the entire service. Consider logging the error instead and returning a more specific error to the caller.

```diff
 func (k *KeyStore) Take() (*AccountKey, error) {
 	select {
 	case key := <-k.availableKeys:
 		if !key.lock() {
 			// this should never happen and means there's a bug
-			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+			err := fmt.Errorf("internal error: key %d available, but locked - this indicates a bug", key.Index)
+			// Log the error here
+			return nil, err
 		}
 		return key, nil
 	default:
 		return nil, ErrNoKeysAvailable
 	}
 }
```


<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
// Take reserves a key for use in a transaction.
func (k *KeyStore) Take() (*AccountKey, error) {
	select {
	case key := <-k.availableKeys:
		if !key.lock() {
			// this should never happen and means there's a bug
			err := fmt.Errorf("internal error: key %d available, but locked - this indicates a bug", key.Index)
			// Log the error here
			return nil, err
		}
		return key, nil
	default:
		return nil, ErrNoKeysAvailable
	}
}
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-03-07T01:00:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
+
+// NotifyTransaction unlocks a key after use and puts it back into the pool.
+func (k *KeyStore) NotifyTransaction(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unsafeUnlockKey(txID)
+}
+
+// NotifyBlock is called to notify the KeyStore of a new ingested block height.
+// Pending transactions older than a threshold number of blocks are removed.
+func (k *KeyStore) NotifyBlock(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	for txID, key := range k.usedKeys {
+		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
+			k.unsafeUnlockKey(txID)
+		}
+	}
+}
+
+// unsafeUnlockKey unlocks a key referenced by the transaction ID set during setLockMetadata
+// the caller must hold the keyMu lock
+func (k *KeyStore) unsafeUnlockKey(txID flowsdk.Identifier) {
+	if key, ok := k.usedKeys[txID]; ok {
+		key.Done()
+		delete(k.usedKeys, txID)
+	}
+}
+
+// release puts a key back into the pool.
+func (k *KeyStore) release(key *AccountKey) {
+	k.availableKeys <- key
+}
path: services/requester/keystore/key_store.go
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>🧩 Analysis chain</summary>

**Add protection against channel blocking.**

The `release` method could potentially block if the channel is full, though this shouldn't happen with the current design. Consider adding a safety check or using a non-blocking send with select.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if AccountKey has a mechanism to prevent double release
rg -A 3 -B 3 "func \(\w+ \*AccountKey\) (Done|lock)" --type go
```

Length of output: 998

---



</details>

**Channel Blocking Risk in the `release` Method**

While the protection against double release is handled in `AccountKey.Done` via the `inUse` flag, the `release` method itself still uses a direct channel send:

```go
func (k *KeyStore) release(key *AccountKey) {
	k.availableKeys <- key
}
```

This direct send could block if `availableKeys` is unexpectedly full. Even if this scenario is unlikely given the current design, it's recommended to update this method to use a non-blocking send. For example:

```go
func (k *KeyStore) release(key *AccountKey) {
	select {
	case k.availableKeys <- key:
		// Successfully released the key.
	default:
		// Optionally log a warning or handle the full channel scenario.
	}
}
```

This adjustment ensures that even in unforeseen situations, the code avoids potential deadlocks.

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-03-07T01:00:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
+
+// NotifyTransaction unlocks a key after use and puts it back into the pool.
+func (k *KeyStore) NotifyTransaction(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unsafeUnlockKey(txID)
+}
+
+// NotifyBlock is called to notify the KeyStore of a new ingested block height.
+// Pending transactions older than a threshold number of blocks are removed.
+func (k *KeyStore) NotifyBlock(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	for txID, key := range k.usedKeys {
+		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
+			k.unsafeUnlockKey(txID)
+		}
+	}
+}
+
+// unsafeUnlockKey unlocks a key referenced by the transaction ID set during setLockMetadata
+// the caller must hold the keyMu lock
+func (k *KeyStore) unsafeUnlockKey(txID flowsdk.Identifier) {
+	if key, ok := k.usedKeys[txID]; ok {
+		key.Done()
+		delete(k.usedKeys, txID)
+	}
+}
+
+// release puts a key back into the pool.
+func (k *KeyStore) release(key *AccountKey) {
+	k.availableKeys <- key
+}
path: services/requester/keystore/key_store.go
user: login: peterargue
body: we do not want to drop keys, so this must block
created_at: 2025-03-07T01:33:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
+
+// NotifyTransaction unlocks a key after use and puts it back into the pool.
+func (k *KeyStore) NotifyTransaction(txID flowsdk.Identifier) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	k.unsafeUnlockKey(txID)
+}
+
+// NotifyBlock is called to notify the KeyStore of a new ingested block height.
+// Pending transactions older than a threshold number of blocks are removed.
+func (k *KeyStore) NotifyBlock(blockHeight uint64) {
+	k.keyMu.Lock()
+	defer k.keyMu.Unlock()
+
+	for txID, key := range k.usedKeys {
+		if blockHeight-key.lastLockedBlock.Load() >= accountKeyBlockExpiration {
+			k.unsafeUnlockKey(txID)
+		}
+	}
+}
+
+// unsafeUnlockKey unlocks a key referenced by the transaction ID set during setLockMetadata
+// the caller must hold the keyMu lock
+func (k *KeyStore) unsafeUnlockKey(txID flowsdk.Identifier) {
+	if key, ok := k.usedKeys[txID]; ok {
+		key.Done()
+		delete(k.usedKeys, txID)
+	}
+}
+
+// release puts a key back into the pool.
+func (k *KeyStore) release(key *AccountKey) {
+	k.availableKeys <- key
+}
path: services/requester/keystore/key_store.go
user: login: coderabbitai[bot]
body: @peterargue I understand your requirement completely. The blocking behavior is indeed appropriate here since maintaining the integrity of the key pool is critical. In this design, dropping keys would be worse than blocking, as it could lead to a reduced key pool and potential system inconsistencies.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: peterargue
PR: onflow/flow-evm-gateway#772
File: services/requester/keystore/key_store.go:94-97
Timestamp: 2025-03-07T01:34:25.290Z
Learning: The `release` method in KeyStore must use blocking channel sends to ensure no keys are ever dropped from the pool, even if this means potentially blocking until the channel has space.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-03-07T01:34:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
path: services/requester/keystore/key_store.go
user: login: peterargue
body: this is a bug and we want to find it immediately. this will ensure we're likely to find it during dev/testing
created_at: 2025-03-07T01:34:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+package keystore
+
+import (
+	"fmt"
+	"sync"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go/model/flow"
+)
+
+var ErrNoKeysAvailable = fmt.Errorf("no signing keys available")
+
+const accountKeyBlockExpiration = flow.DefaultTransactionExpiry
+
+type KeyLock interface {
+	NotifyTransaction(txID flowsdk.Identifier)
+	NotifyBlock(blockHeight uint64)
+}
+
+type KeyStore struct {
+	availableKeys chan *AccountKey
+	usedKeys      map[flowsdk.Identifier]*AccountKey
+	size          int
+	keyMu         sync.RWMutex
+}
+
+var _ KeyLock = (*KeyStore)(nil)
+
+func New(keys []*AccountKey) *KeyStore {
+	ks := &KeyStore{
+		usedKeys: map[flowsdk.Identifier]*AccountKey{},
+	}
+
+	availableKeys := make(chan *AccountKey, len(keys))
+	for _, key := range keys {
+		key.ks = ks
+		availableKeys <- key
+	}
+	ks.size = len(keys)
+	ks.availableKeys = availableKeys
+
+	return ks
+}
+
+// AvailableKeys returns the number of keys available for use.
+func (k *KeyStore) AvailableKeys() int {
+	return len(k.availableKeys)
+}
+
+// Take reserves a key for use in a transaction.
+func (k *KeyStore) Take() (*AccountKey, error) {
+	select {
+	case key := <-k.availableKeys:
+		if !key.lock() {
+			// this should never happen and means there's a bug
+			panic(fmt.Sprintf("key %d available, but locked", key.Index))
+		}
+		return key, nil
+	default:
+		return nil, ErrNoKeysAvailable
+	}
+}
path: services/requester/keystore/key_store.go
user: login: coderabbitai[bot]
body: @peterargue I understand your reasoning. Using `panic` in this case is a deliberate choice to catch critical invariant violations early during development and testing, rather than allowing them to go undetected. This "fail fast" approach makes sense for bugs that should never occur in a correctly functioning system.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: peterargue
PR: onflow/flow-evm-gateway#772
File: services/requester/keystore/key_store.go:50-62
Timestamp: 2025-03-07T01:35:09.531Z
Learning: In the flow-evm-gateway codebase, panics are acceptable in scenarios where immediate detection of critical bugs is desired during development and testing, particularly for invariant violations that should never occur in a correctly functioning system (e.g., when a key is available but locked in the keystore implementation).
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-03-07T01:35:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,202 @@
+package keystore
+
+import (
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	sdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/test"
+	"github.com/onflow/flow-go/utils/unittest"
+	"golang.org/x/sync/errgroup"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestTake(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	keys := make([]*AccountKey, 0, 2)
+	for range 2 {
+		accountKey, signer := keyGenerator.NewWithSigner()
+		keys = append(keys, NewAccountKey(*accountKey, addrGenerator.New(), signer))
+	}
+
+	ks := New(keys)
+
+	t.Run("Take with no metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		// key is reserved
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.True(t, key.inUse.Load())
+
+		key.Done()
+
+		// key is released
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
path: services/requester/keystore/key_store_test.go
user: login: m-Peter
body: ```suggestion
		assert.Equal(t, key, ks.usedKeys[txID])
```
created_at: 2025-03-07T09:45:38Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,202 @@
+package keystore
+
+import (
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	sdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/test"
+	"github.com/onflow/flow-go/utils/unittest"
+	"golang.org/x/sync/errgroup"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestTake(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	keys := make([]*AccountKey, 0, 2)
+	for range 2 {
+		accountKey, signer := keyGenerator.NewWithSigner()
+		keys = append(keys, NewAccountKey(*accountKey, addrGenerator.New(), signer))
+	}
+
+	ks := New(keys)
+
+	t.Run("Take with no metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		// key is reserved
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.True(t, key.inUse.Load())
+
+		key.Done()
+
+		// key is released
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
+
+		ks.NotifyTransaction(txID)
+
+		// keystore and key should be reset
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with expiration", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
path: services/requester/keystore/key_store_test.go
user: login: m-Peter
body: ```suggestion
		assert.Equal(t, key, ks.usedKeys[txID])
```
created_at: 2025-03-07T09:46:34Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,202 @@
+package keystore
+
+import (
+	"errors"
+	"sync"
+	"testing"
+	"time"
+
+	sdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/test"
+	"github.com/onflow/flow-go/utils/unittest"
+	"golang.org/x/sync/errgroup"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestTake(t *testing.T) {
+	keyGenerator := test.AccountKeyGenerator()
+	addrGenerator := test.AddressGenerator()
+
+	keys := make([]*AccountKey, 0, 2)
+	for range 2 {
+		accountKey, signer := keyGenerator.NewWithSigner()
+		keys = append(keys, NewAccountKey(*accountKey, addrGenerator.New(), signer))
+	}
+
+	ks := New(keys)
+
+	t.Run("Take with no metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		// key is reserved
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.True(t, key.inUse.Load())
+
+		key.Done()
+
+		// key is released
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with metadata updates", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
+
+		ks.NotifyTransaction(txID)
+
+		// keystore and key should be reset
+		assert.Equal(t, 2, ks.AvailableKeys())
+		assert.Empty(t, ks.usedKeys)
+		assert.False(t, key.inUse.Load())
+	})
+
+	t.Run("Take with expiration", func(t *testing.T) {
+		key, err := ks.Take()
+		require.NoError(t, err)
+
+		blockHeight := uint64(10)
+		txID := identifierFixture()
+		key.SetLockMetadata(txID, blockHeight)
+
+		// key is reserved, and metadata is set
+		assert.Equal(t, 1, ks.AvailableKeys())
+		assert.Len(t, ks.usedKeys, 1)
+		assert.True(t, key.inUse.Load())
+		assert.Equal(t, blockHeight, key.lastLockedBlock.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
+
+		// notify for one block before the expiration block, key should still be reserved
+		ks.NotifyBlock(blockHeight + accountKeyBlockExpiration - 1)
+
+		assert.True(t, key.inUse.Load())
+		assert.NotNil(t, ks.usedKeys[txID])
path: services/requester/keystore/key_store_test.go
user: login: m-Peter
body: ```suggestion
		assert.Equal(t, key, ks.usedKeys[txID])
```
created_at: 2025-03-07T09:46:58Z
author_association: COLLABORATOR
diff_hunk: @@ -631,14 +631,18 @@ func (e *EVM) buildTransaction(
 
 	for _, arg := range args {
 		if err := flowTx.AddArgument(arg); err != nil {
+			accKey.Done()
path: services/requester/requester.go
user: login: m-Peter
body: Instead of adding this `accKey.Done()` call, we could simply move down this piece of code:
```go
accKey, err := e.keystore.Take()
if err != nil {
	return nil, err
}
```
Ideally, just before calling `accKey.SetProposerPayerAndSign`.
created_at: 2025-03-07T09:55:01Z
author_association: COLLABORATOR
diff_hunk: @@ -631,14 +631,18 @@ func (e *EVM) buildTransaction(
 
 	for _, arg := range args {
 		if err := flowTx.AddArgument(arg); err != nil {
+			accKey.Done()
 			return nil, fmt.Errorf("failed to add argument: %s, with %w", arg, err)
 		}
 	}
 
 	if err := accKey.SetProposerPayerAndSign(flowTx, account); err != nil {
+		accKey.Done()
path: services/requester/requester.go
user: login: m-Peter
body: That's neat :100: 
created_at: 2025-03-07T09:55:16Z
author_association: COLLABORATOR
diff_hunk: @@ -631,14 +631,18 @@ func (e *EVM) buildTransaction(
 
 	for _, arg := range args {
 		if err := flowTx.AddArgument(arg); err != nil {
+			accKey.Done()
path: services/requester/requester.go
user: login: peterargue
body: good call
created_at: 2025-03-07T17:19:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
path: services/requester/keystore/account_key.go
user: login: zhangchiqing
body: Q1: What guarantees the sequence number here is consistent with the onchain state on initial use?
Q2: How is sequence number updated after a key is used and released?
created_at: 2025-03-07T18:59:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
path: services/requester/keystore/account_key.go
user: login: peterargue
body: currently, we fetch the account for each tx sent, so we always get the current value. (see https://github.com/onflow/flow-evm-gateway/issues/773)

this works because we lock the key until we know the tx is executed, so we should always get the right value
created_at: 2025-03-07T19:16:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
path: services/requester/keystore/account_key.go
user: login: zhangchiqing
body: There are some challenges to know the state of an account key:
1. the request to AN that sends the signed tx was rate limited or failed
2. the query to AN that querys the status of an tx that was submited was rate limited or failed.
3. the query to AN that querys the status of an account key was rate limited or failed. 

We've ensured that once a key is used, it won't be used again until the query to AN succeed, and from which we got the updated sequence number. 

I just concerned if it's possible that due to the above reasons, and some keys will never be unlocked, and eventually all the keys will be stuck like this and no more key is available. 

I wonder if we need some fallback approach, that go through all the used keys and filter those that hasn't been unlocked for a long time, and check again with AN to adjust their key stat. Suppose after the transaction expiry time, the key state won't change, if those keys are still locked, we should be able to query AN again to confirm the key state, right?

created_at: 2025-03-07T19:33:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
path: services/requester/keystore/account_key.go
user: login: peterargue
body: we do have a fallback for that, though it's fairly coarse.

We set the block height of the reference block used by the transaction
https://github.com/onflow/flow-evm-gateway/pull/772/files#diff-dead54ca0eb128b19b453911c9bc13c64e01b2c50553a574820820564552e39bR644

https://github.com/onflow/flow-evm-gateway/pull/772/files#diff-12a0e0bebbd6c2e5083853ca60776a8b0a3fcd0aa249681b95dd917f9689d779R49-R52

Then, when each block is handled, we trigger the keystore to expire any tx that are older than 600 blocks.
https://github.com/onflow/flow-evm-gateway/pull/772/files#diff-b2a31343f878e6879293b991cc287bceb34bffd590f70fe5d987216043291403R179

https://github.com/onflow/flow-evm-gateway/pull/772/files#diff-4d0c2f3e8fbdcb80962681f8bf741810b15391ed5d0f2d20163e1f8edbd9a78bR74-R83
created_at: 2025-03-07T20:08:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,85 @@
+package keystore
+
+import (
+	"fmt"
+
+	flowsdk "github.com/onflow/flow-go-sdk"
+	"github.com/onflow/flow-go-sdk/crypto"
+	"go.uber.org/atomic"
+)
+
+type AccountKey struct {
+	flowsdk.AccountKey
+
+	ks      *KeyStore
+	Address flowsdk.Address
+	Signer  crypto.Signer
+
+	// lastLockedBlock tracks the block height when this key was last locked
+	lastLockedBlock *atomic.Uint64
+
+	// inUse is set to true when the key is locked for use in a transaction
+	inUse *atomic.Bool
+}
+
+func NewAccountKey(
+	accountKey flowsdk.AccountKey,
+	address flowsdk.Address,
+	signer crypto.Signer,
+) *AccountKey {
+	return &AccountKey{
+		AccountKey:      accountKey,
+		Address:         address,
+		Signer:          signer,
+		lastLockedBlock: atomic.NewUint64(0),
+		inUse:           atomic.NewBool(false),
+	}
+}
+
+// Done releases a key after use.
+func (k *AccountKey) Done() {
+	// make sure the key is only put back on the availableKeys channel once
+	if k.inUse.CompareAndSwap(true, false) {
+		k.ks.release(k)
+	}
+}
+
+// SetLockMetadata sets the transaction ID and reference block height for the transaction the
+// key was used for.
+func (k *AccountKey) SetLockMetadata(txID flowsdk.Identifier, referenceBlockHeight uint64) {
+	k.lastLockedBlock.Store(referenceBlockHeight)
+	k.ks.setLockMetadata(k, txID)
+}
+
+// SetProposerPayerAndSign sets the proposer, payer, and signs the transaction with the key.
+func (k *AccountKey) SetProposerPayerAndSign(
+	tx *flowsdk.Transaction,
+	account *flowsdk.Account,
+) error {
+	if k.Address != account.Address {
+		return fmt.Errorf(
+			"expected address: %v, got address: %v",
+			k.Address,
+			account.Address,
+		)
+	}
+	if k.Index >= uint32(len(account.Keys)) {
+		return fmt.Errorf(
+			"key index: %d exceeds keys length: %d",
+			k.Index,
+			len(account.Keys),
+		)
+	}
+	seqNumber := account.Keys[k.Index].SequenceNumber
path: services/requester/keystore/account_key.go
user: login: zhangchiqing
body: Nice. Thanks for pointing out these logics. It would be great that they could be added to the comment. 

I would suggest to add some comment to say something like keystore avoids keys from being concurrently used by multiple txs with locking and tracking tx status and expiry time, and ensure the correct sequence number is used by  checking with AN before using the key. 
created_at: 2025-03-07T20:26:46Z
author_association: MEMBER

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-03-06T19:05:44Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough

This update involves significant changes to the key management system within the codebase. The `KeyStore` and `AccountKey` implementations have been completely removed, and a new `KeyStore` structure has been introduced in a different package. The `Bootstrap` and `RPCEventSubscriber` structs have been updated to reflect these changes, including modifications to how account keys are handled. Additionally, several new dependencies have been added to the project, and new test files have been created to validate the functionality of the updated key management system.

## Changes

| File Path                           | Change Summary                                                                                     |
|-------------------------------------|----------------------------------------------------------------------------------------------------|
| services/requester/key_store.go     | Removed `KeyStore`, `AccountKey`, and associated methods; eliminated `KeyLock` interface and related functionality. |
| bootstrap/bootstrap.go               | Updated `Bootstrap` struct to use `keystore.KeyStore`; modified `StartAPIServer` to use `keystore.AccountKey`. |
| go.mod                               | Added dependencies: `go.uber.org/atomic`, `github.com/huandu/go-clone`, `github.com/libp2p/go-libp2p-pubsub`, etc.; removed indirect marking for `go.uber.org/atomic`. |
| services/ingestion/event_subscriber.go | Updated `RPCEventSubscriber` to use `keystore.KeyLock`; modified method calls for key locking and notifications. |
| services/ingestion/event_subscriber_test.go | Changed instantiation of `subscriber` to use `keystore.New(nil)` instead of `requester.NewKeyStore(nil)`. |
| services/requester/keystore/account_key.go | Introduced new `AccountKey` struct with associated methods for managing account keys. |
| services/requester/keystore/key_store.go | Introduced new `KeyStore` struct with methods for key management, including `NotifyTransaction` and `NotifyBlock`. |
| services/requester/keystore/key_store_test.go | Added tests for `KeyStore` functionality, including `TestTake`, `TestKeySigning`, and `TestConcurrentUse`. |
| services/requester/requester.go      | Updated `EVM` struct to use `keystore.KeyStore`; modified transaction handling to use new key management methods. |

## Possibly related PRs

- onflow/flow-evm-gateway#734: The changes in the main PR involve the complete removal of the `KeyStore` and `AccountKey` implementations, while the retrieved PR modifies the `accountKeyBlockExpiration` constant within the `KeyStore`, indicating that the two PRs are related through their modifications to the `services/requester/key_store.go` file.

## Suggested labels

`Improvement`, `Chore`, `EVM`

## Suggested reviewers

- m-Peter
- zhangchiqing

## Poem

> In the code's quiet glen, I hop with glee,  
> Keys now guarded securely as can be.  
> With changes made and tests so bright,  
> I dance through functions, a joyful sight.  
> A rabbit's cheer for the code we mend,  
> Hopping through logic, around each bend!  
> 🐇✨

<!-- walkthrough_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpADK2VmgqyjcEj2wAbb5AokAI7Y0riQ4rjedJAARABi8AAeChgM2BQBGGHMaNzoDAzSyPBYANYksogEATGQAO5oyIjYAszqNPRykNiIlDwkNFQURCHoGPSo8EpZ8ABm8NGN1nYYTgJ9AOwbAEwaMLCevPA5rpCz+GnI+LPhCMi2KMgE6LS0AYjImI/NngHe1NFnrgDpBypVqiQADT1BAMWCMdKZSLyXBUBilZBg4pYHqeYEUfDYIjwtCQHJ5YHUSD/dFNWSpWAEjDwABe1Hg+AwezgqBUvVSnmukEQ9LhTNZ7M51JIaFoT3wPAJAiizGQdXU8KYqUR7HyhQ+4QVwM85I07hBSkQDAo8G44ilJVo8AYAKesCpxuW6GO8v80nw3ikt08SB+/BuWrSGV15L1RWhdVhmq+7x8YRK3QwvDoztwKiikHW7okHL4mHo3HwNBmaD8tGopKYGWwdo5XP2ngetHw0kgGCrwu4ojmKJBJESaDE31Gq2Y6z46kY3nwvUQ0KWjudAM6o8ONJIsADSj4MTMAH4L7UM57LdbbfauUYAIKvdRt2veWTQz1a1HwAR4PgfDutiDDeNgSj0I2Bzot4SBhOcfBkM0NoYEQtweiCDxsAMyCFBQeYZnegzwKS6yISalBECURDcncjAwaUcFVCgqTgZaKTzBQbRoRhYSeg8ebDLhwaQAABjkVSUGJhZUAy0JwRgpQ0Sg/EKqSfwkBImBhAA4uoAASLTToKfCIEODBzFuD7rrQByoehzxME6vFVOWLhytCLb1uIvHdmkbBZJKWBLOQ+qIC4X5+iWJDqrxP7umhvY5Eo67jC8LnobkvD4EcALUioJDeIgZoGHEQFnOkxp8EohHFd+WF2C6WDrHGHyLPxII5SWkGQAAqjYAAyXAANoJMkkY6lkZK5G12KghUVRASQAC6AAUsC4Lg3CIBwAD0e3UcCLQaEwzB7ZyszLnUe1XfgdRgFpzDEACDSyHt3A+N4e1bNsACUpUWJAADCLCBbgyCOM4rhGAAklgP5g+wTSiA+4aibYXm9HwQ6DC4IyeOO2btfQkaUBgyABEQHkqZW1biLWkBKLKy60n6/wdIaoms8pvFsHCmBIMwrGiViELcuprzvG60ieGTFAU9CIIBHUBI0KLFR+tj2lo+W7MyogKnLtRDBc2BLgjqJfzBYgCB5OsuB1CQZAa/IeuemLy10agAtJd6qpc2QiWFKJIoMuKbK6+l+BSFQvjs6Rahwbg8hCh7i0QgA5FcQ5UA+a6QMh6QqZSYSTdG02TuFJmfAEhezLMqOSEV8htOI1MdKV+jGOAUBkPQQpoIBxBkMonNneDXC8PwwhN1IMjyM5yiqOoWg6N3JhQDyxSfCFw+kOQefRBP7BcFQdTCk4JzyF0S9UCvmjaLoYCGD3pgGNrzrSHtATBKElB7TBAAfSWgEDQRB8AcAMDEGBBggZPlhiPQ+25L7Q1ThGRKpBEDmjlpg3sJQJABiDIjZg3AojqwCMwGOjM04gjEgAaQqPYCEMk9ZiSfAUQkWRGGyBkscMhJBwY2zOASYWnoxKf31D/IIIQpIUEARUEB4sIEyXmFEOinh2GcOwNwioMkqgUGwGIBMSZWJgQgtEeYRU5RnAqjkDAaBqKuXgEQZkvEsTrjJHgcctizKikZJyCUNkxj0DYMCRQyBEKZh5ipLEIThQuLcY5OSEUxBtgLiBQszssBKHIXQDR4keHMOWvo1ERjcDQneJWCm/4CxRPsY4lSaBtLwH+MqTwetcT0A8egYqCoSjmN6mEw8Niolghmg40g4NL5wnQMgdhLS2lRB4YgMS0IxLQDQOUNZ4lBoXFKDwnZYk+oYB5octK9AxIADkqwjjEgUhhFQ9noj4VkSgsxJxQhhM6eESh5jkHoIjLIaIEIVRibxTppz9mxMWkrJY6wXaUJjosZcvF1TAlUsUUhKp2DBRFo82QxSAj3I7IXDIFVEq0EUuhKJLpVwwkoJ4fsC1Kj1EZegRZ+ZPCZNrIgBUCKsBFWOCUbc65UXoXRfCT0jQ+WWRQVqNy00xIAFEMg3JWU+Tl7SSUAHlY4fgaqgJF2k/CbhdDQT4CTXFWWamXPBItDwX3GQ0qZuoGjIGzCWQkiBPyQEpVECYCMQRLwEI0EggNLBPm8HjfOXMbyiH+HndJ6MiZAU5hVT6ypnSFxmOIaQRgt6yCHH6KhUhaBcDEinYtHCmA6NwDw4UZSxCvPEpI7+v9ZGDAUbIJRXsVEGCgAAWQGCMktyLy3iVmDo02a1SiQAAFQ1q4fWiof1IAABFOQkDWn9FtEjKAlikR2/+8jgGgPDf2odI7FBjrLRWqdqRICzoXUuutPC132AGFYAklZsZWDQLISgT5xj2ESWtXAyR513TqIgWgpQNDQBSZOYJVdl0Lug7B+Dr6shrsoASCge622IGkX/OR3be1gMvTAItvxBHjorVWzwPDnlzpKIMD5hRCMHq/sR49ZGz3KPwGJAd1Hi3GroAxmjkAikQkbYY5t+KiMkc7QAgTfahMieHeE+g4mJ1iQfabK5sUZPLVnYtSAI0VqLu0bo2Qa750meJYp7jR6ZEnvI+e8BGmr3advRJyd06n1zoc0wiEa7NXaCWSQFZO7WK4C4xQQ97a3P8cUZ5qjWnR26fvYF59IXCVhZgFs7da61rWdrbZ6EeGgK7uc4lnjyn3Nqco95yAmWb3ZYC4+vLjmSBrpYzw8DiRYbrrONdTDGhYbTHEFYiglSSCN0yIUAAQjzAyJAXGbW6GxgAbAAFmhOM8ry730JaS7xlLXbmsXta+1nTdG71dZncF3ra6TlnIqENkbY37oTam+wKylBasZn3fV1zpGrtpcE8JnzWWHv+f07ll7oXlprpuTN2Qa1lT7PW5tsI2Bdt7eB1gUH53GupZ7eljTMCYgFvfgIfAVYDG5D2gzpnqJcheagTTuBkbEEHzHtEZoaD0a+ywTghimAsEi3EWziGHPuCs8Z/Lqg3AvOqNaSGDAhDAwmkUDa4RQI6HLeV8z7gpT5NhHdnQ5hLhcBPisLDT9iXpJkmvbQB5nsnNWO8IG0SYkTfs9Vxb8pfr4XZO6NwHyljRHoB4PgNjfRnhiT44MDQvWZLPFJJWRPfBk9e/DRn72eF7UBCuqjS1ts5jpiDZ4Pl6QQ60M0Rn8IUn1gqS6XsOGtfxK24Ig7p3B7XfDMUA1PEUmm/iVQ2+xa+i4Ih0yQKyXSV6CzFj2JSz87U+UA0NhldvCuYb6swX3fNn98kpfE6B8H4oqejelPs/KzWF10rJ9Dm0RF8R6oU6KxE6ShSVlHRmyn7iaT7FinEm3woFPwq3Pzk3KTHwUCUD7Hum6HpXERPyMzqD30OUqlSDRicgCHyj1mAPSjEhP2wL0VYkVX1D2ASAcV8FvxBBKDfFrCCTbHRnQIzhKTOAWF9zDxkAj28hQTXxYAgMux30wN6zWmn1s0QFq3zy4LAUwOkMf0Wl3WhDNXZAhWX1IAdRQPERb0mApjzBrE7iMD1WUF8DH16B0PwXGGsg6T9A42qBUiFEdVdnPXiRkLCDiRcG5XLADWhEQHRThBUhELEToUgJkm4EnFKEcTxCNDoQL2iNiPiN8WDBsIM2v2TgWBKiMHMEjWjTHmTSN08CUHNiTU5CuBuFTQImiAzRaHnxzQiFyILUgASGsVb2LSEI6ArQLwXUgPTxRycwzAMVDwD1NwVz3TlzNyVyD05xUUgEACTCcSfo+dE/QwrAMYqcCY+Y83fFGYhXOYlXBY1rAANQti5S6M8B6IR28JWUgA4AAF4Zpyg1pN9BiKCooAAGYnN3XzMSPve3R3Z3WOaYyY1XY4s3dXZYh/GAh454147dTfcg1Qn4v4kfS5IEgfUE13EHQ4yEgk044THnAdd+CBDQb/bnWBeBfnUeI+egYXa+UXPBbBAwDwWw+acRCk7/DXAsAhIhPEEEWUK/dgvWY1GhG4cyYceYU2JQIccYMgSyaQApRSTRCkloHfICIgPaagFgbNCQAARg0ENONO+Jkk/xdhFMBESM8GYjCCFA7XgACFCUUB8GkA0PsPNRLndE6lQHlP7iVPkFQH7AvjCiKEinSM9ByhEDED2EvzfE5Bv0NSaDQDYCZhIAVOmAYHkHdUVC0g5B6F9ROHKCgmxCdACCnD1ktKwF0xEVEM9E3ErLCCdMoWRmFCbk5C7zKiqnsioQCCCK0ksLAIvibNRgzKzKVNyLDyDCXzYwJFoCMToE9MGRUjEiOlgBOjOj2k3PLGwEOkIDAi3UgCNI0B2w0HNPWQ3K3JYB3OwD3IPLACPPIEOnpINONI2A0G2COWvIEFOlvLggEG4G2EVwgTAEAuAu4DAEzWaAEBPO+JNIAGYLyfyNQbzzoIKQLHzVRqIFQJAELkLLz4l1y0K/ztzmA0w5ggIcgIZsLKKRSKAwBZhmAwh8KTSUKNEbCRTEyKZL4iAsFOpMJDgCRYz0xkAiZywVJ1BIlp1sj1Ab5gzUggJKwk0IVXxsjqR/whhpzS4Zp5BuopgmVwDG5qB0hewKoXQYik43wVTzD9UrDRJv8DdY0y8ogpxbjfR41JzUhpzWzLEKpoyRLUYEwNQ49zhLh+AWoqxflnTxy9YxypwAzFSfLbL8j4EijKjeLSiMyKjhEhRaj00cZGjs0Ac802SoB11MzAzUg3ZXgEcNT5wNBtTdSCA2hTZTzTSUL8UeTFAYcN0qrkqcyMp6rSL/zzpdzxh9ywLnzPBTzzzzTur8BKTeqRNKrvKhrrS9Nfyxq7yHzprUUSBXzD53yNBPzvzFrlraA+q1rqqNq6qtrRrtzMLQLCBnroKWhYL4KkKuqQceqrrVqBrszarIIK1tqnr/xILsLEBcKvrCK90/rrrAagzhqHrjoyLbyKLo0qLuJqBiMwLMaGYpYmKWKvqzT4alreSAb1r5BOt1ylrNSoDmq9S2qTzjTOqFq1oSzFhyzYqxA11fqKaVrSSIAwAjAlMaJQg2w9ohysgQEPq7xGqIFqTadaSkFBdGSr5IyhQxd812Tg1WSZc6FxakoqgpaZbcA5aBArQbRFahMeD+TtdBSyR9dZTDdbTriOCkinl9kNdOjrw6EbArBgZlUpAshHAraFbXdtird0ppLUC0j/bNFMDA7g7Q7cBw7rb/xXcFUm1qh4lxFYLM71gZJMTPdvaXkeDrFs4PaazbDV919BjmMfbD8Njy7SgSVQZoxzJORMpPwECc7LcgJq6YiqAwkk8pMokyC26LSlheV+VBCo8UFngaLZlgQQzwDGNuz4Z/dC7I6S73cEDMTGAPxKYiol6FQp7ZAWMZ7ZyI9axBh/NxF3t9kcCj7a6AgyFPl6BJVxJ0cRxENMBUkHwdlrck7blZgD837w9EVMz9xv7QrrlwHZBVsfbOLcEpc7CnRzVHCq9Zga8/V9Cvar7m6sj3xk5gysUBFwZlynC3LfIsomY5hFtdRsoCRJwpUFR/US4kM0kpR+wZtrJk09Zsd0RkCBHvSqiI1IAo0Y0Sj3byjE08qajEgVLCqvAs1TZSrWiRMOi+DGNI9o89MwQWM/QIcd8m6K7Rjc7xIU6Q72AM7I6CM6sycJbTbORpa07Lai6d9FiVjL6THW7iHLGtjrGxJbG06HGba8SSdjaBKzbPHd6omoCqN/1R6Bhx7ujF7ejVi27TGVMoCLH26RZSGpRrlYpwn7H5akmzsGtXGHwPH2AvHHGYS/HjH9kWVPNCmW0SmSdk6g67Gw6qms6nGQdYnJb3Hzammkn1doFYEyTRaP4XNv46n4nGnEnhmgELVNAlbZmVa+c1aGTUFmTtbWSJcdauSuoCQepogf8bgBTdcrZYHPlplJ8U9xCoDJDhjt1mRvBasenj744f7L7PNlCfmMTKKBFwhQhcDeHeLJVE7W0lniMVmJmEmhn5xNnQgYS1Fw19hUAnLXbdYG4K9RJ/8TCGY0ZXn1n5wZJBBRLDa8RQhdpf7yn+mIn0XXdQy/RChm5LVgWWEqCTCQ4yWZQB4bhSQ3mzGnHxlPNSUdbHgUhDYpJK5rQVwUZ7KoWWI6UPSzF2I1zoBQggFIn/waIjkDWqggFB1QwaIUH0QBn4t1lzWLbjW1A0IAB1DUGwAYVwHVDAVVfDM1w1l1miD14EL11EWQX1/1oCQdCFqIW1jEEB0gp1o19FkNz171yNv18lCgZVUhFOBN1ZApZFOOPwaOsy+JY2bNSfLZ0+nIDMadPBWgaEcKnoVwnvdZ1sKUHp1glOeJarYCAImiUqAo6RjKt20SBRlwJRwuFRtNeooqjR5omy8qyAG5XpWRqouNC0BNad2NfKuduogeRdporR3sJ2OuFKLXW4f0phs0YW9+JTSA7tc9XUs/IBMELnXZ3naRuk5BTmJkrWjBDBtkp8Ec+2zRbwj9ioGE2u+ct0woP3AupF8nSHcEbgmI9EeI6EP5Zg+KOhL4kPKs9KGVC4UiTmTEyJOxTARpCFM/DpkWMiHmAWIiTkGgRITQUlLReEyg6OwuOcOgeZDDODaAk7Sg+KtiCxSJXg0ZCqMOMUQJSOdgtaCSbAMSP6TxMvRlAULmCVjPBA7w6uhi+MOPQ2VxSgC5dAVqg0y49pTytEOdT0f4FiHmaIERudUBllaunoNItyXAHoUqDksprAtE1RWS9g5ghmOCFkXsL4bjsTg/MlzAEOH+wKmOQyqCejsEWyN4EzvWMzw+QcraH0rXFgvwE1WRdIsSZz3AFjOgBN1hUgkoPqXoElILzdcgfe3zTSMNGWFldAPBvoOO3EKrCmYuXiRcVATkX1AIfzhWG00SZpSLK4uJX2cgU1O5sIPMrmsskWXEB5T9Wr/ZLTWUBsLr0dDy7mRoMIVzzoHmP1DbYkGO0mE+uPI+qUAwr5zPBUXoG79psJU7vMQLm3L9H9FcSgf9QDCgYDWgUDVxc7m9WOQHN0ISvUNDYzg0Feg4PrsEIzqWIoeJOEUQDEPiV2ViJQZIVAE1KYLJCiDs4r/DvEHhh8YesHiKPwbLngADCz+JArkrnNKQZcIcB5HmBH+7bWXsdOeQKJDnCmZDdgkbnNFCKSsuL4abm+OWD8Nz3cDp8WUdmR4ord7KqdzK6o2d1Rhd9R093NbRqATAuAqs+6yTatNEh3+LZxhrZ9gvN9mA6D2QGZu34y8LqUTanLR9TAr4lQnj6X8bETr4nL6WH7GDOP/Hj4IIxJPoa0ItAgDQOHw+ezQjj38HfJl9iEH35dP3gPtd8Ao+0Pp7ILF9NEtdDrkrGp4vprRQw6qDz9qje32vp3+vvLL4j9AYFjE7nyNAL7UbYT+Df7GYWbebZhgUBN3HJ77bLIfbP40nT3950v5acvutSv3vmv93FGsP57Rv6Pkf3Ab9XKcHigSHoDEDMDCDdDWP+DABuX2F9cejqDd/0TutLhhzZt9ksUrPfgEAP6y0e+rWPvqfzr6I5uswWYftSH2SxY2c3gEARdjAHe9u+MHftA+wWZPtd+3va7F+x5yq0BchzQDqcBOYgdnw4HHFjk0pzQ5+CWSF2PB0XKIcGWiLMHKAJL7JEueWHUgBoWxSCIAc2hAlEShIBEc5ungepDRycToQ+e7iRaA8hbz1sgo/+OPGtyKhAEtUBYXwiRxmh5AhQXSDpqN2V74dCCtAMABFEbhzRbgBIIkCSC8TscHkmBMLngQi7MgourISXnQhbw/1s8VzDLsKHnyCgbg8XGfIl2MLJcdWr+HwFoUcj61MA63PQct3aQdNgemiCLK0i5RP5/icOObllWDRoYhQq3KaMiA5QZC6kFUfblx02TbJChHWWWC7ktTjIZezPBXr0AUjQoJuz3P0MUJV7RICezKXwnXCW55D2knpb5LMm1aqRBh6QXil8AHbuCkGn/IBm2DF7RJ9k7QzWGgEG4LgIYqBL5ImE1y/0kGDXCZPET65wRG4OZMCBEIY6hpegA8FqPdwOB44vIUKdEDClZS6ViwngJfLdyjKM4UBaEbIeJB0R2CSAz9dEK/VP5GppA1SQ2JkKiQ6JwUDDcZFpyWza848svTYVKBGzzZggzpfni6F8BJ5OGR4Ryt4mSA8wHkPXVrs0KgivA9hN8WItuw6T6DPAq3RKOtweS/cx+AwQHmgG2Gkc5UFqOPOMiCHhAuhRI9dBYPG7oQcoucOUbEVcI3BxkPnUgJCMYFz0My/yWLlgFzwcZm809ExD8n9yGFRB4MAuEXAcik9FwzUMRpbBUCEh+I8o5YelHc4Pc8cJUaRoKhzYnlbOBYFVGqnwAaoeRfCbEKiAQ4Ld/UmvelImHZRjDzMfhaoVMNqF8B9u+vcdrGmN67tTeKaQ9mo0zTW8WiutQPhfEYwVoumcWd5J8iL58CO+6HCAaQOP41iaMdYr7m72bFYD+BnfDzNDhEz28em5/avnUDMyspN8yA/LFIMwGodVMQ4jsTAJP7aYJxPWL5uFh5ExZ+aWQRcV7xXFQ51MfVWARuMH7I4CsqOIrG8VKzHZohVWHNlvyIHYDjxzA08aOPXEjJNxV4qQWjnWFeip+SfP7NNkBwUAXxKHI8W2MOqrizx34xQL+IXSvY12lwnmFjjWyPctsBODfkTkPHED3xFGG7PBIviYkkJ84wrNCIOGwifhByT7BBm+wz9JsYE2bJBN4EDjWxr7OCV+NInu5yJKEpkduiOzD98Jb4mCcOM/HViWR/E7cfT2FF5gJ+04y/gl2/DDZp+//OfgIyByiTBx4k7iVJO0g2guUFaQyZAH9bqpFouQqLDpM4ll9uJBAsWlBIIl6STxAQTFlUDIE0l9mlAlBNQPQScl6B5AGsdC0YGX0iJ7k7ZnbTg7Ap4xq+AKkQ3PQpEhBXyX8NoCSReIsakLLZjCwLF4VWC0eUSD23Iae1eRmsWVhyVra6sLE4kJ1o0OkGWjZkSPKxH1yLDNJSwJUmqcVm2E6JjwIYm0N6hSCikqiK5PVq5EKAOJ+paoUKnrElQej/ioogxq6HXBqgiofgJYFw14hExnSwUB5E6yKSJJTWOU9gkTzZiehlBKogkNXEHjqiv+aMNttoQ6Gx4pejaZaEqMdG6VPQBI+XpyFZ539sY0IGIlD0s4FdTKdcWqJFlrhywgIzZX1L93jLqUyGfdGqaEFBjagK4uAFrtIM1bHCgUUYJEA4OyoysIQ6fTGkkIyniBIWECJwb5Fi5bRBErYeKEaGKzmD9YYaYMMwEs78NLYX02FgXDOEFgHRPpKkB+AY4Zjza95BgpmKiwDdBgokXKIbwpgFJKpeAVpH4P9xbMkpcRXQuEVEhxBroDgddPQnzp0ItmdyTSgICGDS8KoMqSgLGnFIyJYqdoyEb+AJB+BoMVUjiAOzDzjBqU6RbVmqHshMoFQ4w7kTUK+TxUwg9M5cKynk4BJmQSnKUPzEShCxfQLqKGWjPxlVwigSsplnMmxidRCYKjVGELjzDqxJ8RM5aMbNKmsoDhsswyYWRzgKz7RY3d6b6W5j4ATY6RbEa0J1hthoQAPCftcNdRZBLOW0zKgqx6YqQmwMM/3nmM3ZZV5GRYmdgVUt7liSqNvKsZOOxlHTOQE42qcVnAwLpTZEI6AH8Uw5azNEAg0Zk5LElcTXJJACKVX3t7ZTxx9fPaUwgOloRD586Y+bRFPktpz5aRfltwWvnsSlxp6Qieekfmdjt5r8hAabCdaoy8Z7ATGd/N/kIYz5qRXQsApGIxMb5uku+R+LclbMn5QfLwbvPr6GV5+CwCgONFkGxZQJAOWbAAqwWXzO+NkinIQvCkkL8BczEWo5LAXPtBiOzcgd5PpK+TNaNA4DibQly3M4sC5JcsgAJaCMje7tSVvk2xaa4giFkA3JLIOH3D6G/uZVGcUHQyD4kcdSjmXRgm+0+CCLFVCYrMW11bi9dUQtngTxvI886kQpL2P0ZZ548ueXTkwM6Y+KaMnpLBmTOggYNuB/RfhDiiCgPhi8thBVgaNcrFykOpQ4woPT4DBdjFpikxOymjLTsx6eeCehVBwVYzUAzitHhQF7qzyDA29WXATl9wbDvpGAMXggQ2noRXmVcHArXXvqMoPcpKT1A3M0rkBSeII56UC3DQn4Bsn2aDNAESCTZ10O6BSK6FwAJsNAq/TaD/wYCnZWBS+XTHsHhgAFm2MsvoL90MVpdf0PPQGTz3y6fzkhTPQBq0oQJiQelMHFvjum2EhkUCFIgNFzFVGUAZuRUA2EKWrnwyhp9BJGaSCCmjL/CAhK0vdS5i/d25ojAeQ2FQJrl3l/vQ7vJNFFrR5liykbCsoKhbMNlWy3ALukSWaEpRpIXBvg3cJS9q6dw0QLIEeEKs059ARcu9K9H5kR6wUN6ULILn9dUAgKigMCqiBho/c8w/KjmzyIGALCpbawugxNq0M4qfobUoLATlYA3ChDauZ4SRFL8uBesTlfUA1B2LclZiiOUovdwFwR64gNIIml9QItfw44B0jcB5lowAIrSTKP22DGdKpGBvYsYWNyr7tlGFvY9lb3XmVjV2FxIyZkP0bOK+infZCb2KsaW5xIlq/seAvAUtMglsmdYp3yGLXjcFfY+xaYuzVCL3mVfVJmmXSYlLMmhjZNTBNTUlqKlWAf5jkocWVrd+wiu2q0xTWFqYJxahccU2D69NYoWa0BWTirVSsZmD7V+JvBzRiscABAA5vKiRhZAz4aAC+H5MLCLxFAy8NQI/HXgvwDAS6ieOoCARTBEAQCAIDFCdi0AlEduZ+Iut7iQAdsKgQ0t8VNIAAOHbIhQEDfqBAO2DYLQA2AfIdsAATkQo7YAArIhT2xoBEKhpRCohVeB7Y9sUGgQNsD2yvrz176ycOBoYA7ZaACG00nQG2AbA4NhpAQLQD2zobVAaADYKIG2DfEBAJG0NIaTQBYaNg+GpdQID2wkAdsAgb4gwFQ2Gk4N2wFDbMHA3wbQN6GhgChp2yGlxNtAKDRsF/XfFEKcGjTabA3jvrZgSGqDSQCg2/qBAcGvbIaXg3UaGAcGsTa8DM1wbf1iFMzbMAECzANgImtACQFUAbBEK/G99bhus0kBEKoW0Db+sNK0A6N6wMje5tEBWbaA3xUDTsAEAwbZNswGDaSG7gEaIACgVgFepvV3qCysUOgEAn7ivqgAA= -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=onflow/flow-evm-gateway&utm_content=772):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: peterargue
created_at: 2025-03-07T00:58:16Z
body: I noticed some other issues with the locking that made me concerned about deadlocks. I ended up rewriting the key reservation/releasing logic to make the relationship between the key and keystore more explicit
author_association: CONTRIBUTOR