[Investigation] Withdrawal payout transactions getting stuck

<!---
Please read this!

Before opening a new issue, make sure to search for keywords in the issues
filtered by the "T - Bug" label and verify the issue you're about to submit isn't a duplicate.
--->

### Summary

We need to come up with a solution for withdrawal payout transactions getting stuck. It is stressful for our members and puts strain on our admin team to be managing so many withdrawal issues.

Assess regression scope and time constraints.

### Steps to reproduce

It is not known exactly how this can be replicated at present, but one suspected frequent way this would happen is by approving two transactions in quick succession.

### Platform information 

Cross-browser web.

### What is the current *bug* behavior?

Members transaction appears as approved, but is not written to the chain.

### What is the expected *correct* behavior?

Member's transaction should be written to the chain.

### Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output,
logs, and code as it's very hard to read otherwise.)

### Possible fixes

First - good read up on nonce handling here https://ethereum.stackexchange.com/a/2809 - a bit old but the information should still be accurate.

#### Diagnostic steps

###### 1. Monitoring
Log new transactions being dispatched on the backend in Sentry, with their nonce in the log string. This way we could monitor if there are any conflicts occurring, and see the patterns of nonce distribution to confirm or disprove the nonce collision hypothesis.

###### 2. Manually check failed TXs visibility in Infura
Make a note of the txid's of any failed payouts and query the Infura API to see if there is any way we can figure out what has happened - possibly via `eth_getTransactionByHash`.

#### Solutions to evaluate

##### 1. Nonce Cache
If it is found that the issues are with nonce conflicts, its entirely possible that upon deciding to dispatch a transaction with a new nonce, we could cache the nonce we've dispatched last, and increment there - there IS some logic here already but if an issue is found, it's obviously not working. Perhaps this is because pending doesn't include transactions that haven't hit the mem-pool yet, perhaps it's because of a bug in the code, perhaps it's because we're not using a shared instance of the class. 

![image](/uploads/c782da8f55c25c32fa1afeeb49c63e86/image.png)

##### 2. TX queue with Pulsar
A bit weightier a task, and IMO if we're going this route we should probably look to expand out into a Web3 microservice sooner rather than later; but we could refactor transactions out into a transactions queue, that awaits a transactions success status before moving onto the next one. The issue with this would be what happens when and if it gets stuck? Do we build in auto-retry logic?

##### 3. ITX with Infura (or Alchemy equivalent).
[ITX with Infura](https://infura.io/docs/transactions#section/Developing-with-ITX) may not be possible because I've not yet determined whether meta-transactions are possible with our contracts.
Regardless it would offer a solid solution to these kinds of issues AND gas pricing. It should be checked whether Alchemy offers a similar product.

##### 4. Do not allow admins to approve another withdrawal until the last is confirmed.
This is no doubt the option that would slow the admin team down the most, but is possibly the simplest. If we can determine from Infura that a transaction is still pending, we can simply throw an error when trying to approve the next transaction.

For improved UX, we could block the UI from making further requests with some kind of "in progress" indicator, and revoke the in-progress state when the tx is confirmed via sockets.

##### 5. Transaction retrier

We could build a state between `PENDING_APPROVAL` and `CONFIRMED`, "`DISPATCHED`" - a runner could iterate through "`DISPATCHED`" transactions periodically and redispatch any that have not been written to chain, or move them to `CONFIRMED`. We could also move to confirmed separately by listening to changes on the blockchain, similar to how we monitor for `PENDING => PENDING_APPROVAL`.

##### 6. Refactor Contracts
Expensive, difficult, and time-consuming but worth a mention. We could refactor contracts and architect withdrawals a bit differently, for example using a funding pool similar to SKALEs CommunityPool, where-in a user would deposit into a pool that they are free to withdraw from as they like - this pool could be used to pay for approvals. There are other ways that could be architected but that is one idea that comes to mind.

##### 7. Wait for L2 / Sidechain integration.
With L2 and sidechain integration, we should see quicker transaction resolution times which in theory should mean fewer issues.


### Additional resources

This is a known failed TX payout `0x05d3668a98737f7db00ac2d864b94cdb257aa1ce87b67b9e940540ca0e5f503d` from a couple of days ago - when we query Infura - we see nothing is found. 

```
{
    "jsonrpc": "2.0",
    "method": "eth_getTransactionByHash",
    "params": [
        "0x05d3668a98737f7db00ac2d864b94cdb257aa1ce87b67b9e940540ca0e5f503d"
    ],
    "id": 1
}
```

Will return 

```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": null
}
```

In contrast to a known valid and confirmed transaction:

```
{
    "jsonrpc": "2.0",
    "method": "eth_getTransactionByHash",
    "params": [
        "0xbb3a336e3f823ec18197f1e13ee875700f08f03e2cab75f0d0b118dabb44cba0"
    ],
    "id": 1
}
```

```
{
    "jsonrpc": "2.0",
    "id": 1,
    "result": {
        "blockHash": "0xb3b20624f8f0f86eb50dd04688409e5cea4bd02d700bf6e79e9384d47d6a5a35",
        "blockNumber": "0x5bad55",
        "from": "0x398137383b3d25c92898c656696e41950e47316b",
        "gas": "0x1d45e",
        "gasPrice": "0xfa56ea00",
        "hash": "0xbb3a336e3f823ec18197f1e13ee875700f08f03e2cab75f0d0b118dabb44cba0",
        "input": "0xf7d8c88300000000000000000000000000000000000000000000000000000000000cee6100000000000000000000000000000000000000000000000000000000000ac3e1",
        "nonce": "0x18",
        "r": "0x2a378831cf81d99a3f06a18ae1b6ca366817ab4d88a70053c41d7a8f0368e031",
        "s": "0x450d831a05b6e418724436c05c155e0a1b7b921015d0fbc2f667aed709ac4fb5",
        "to": "0x06012c8cf97bead5deae237070f9587f8e7a266d",
        "transactionIndex": "0x11",
        "type": "0x0",
        "v": "0x25",
        "value": "0x1c6bf526340000"
    }
}
```

Story points: 13.0