# Trustworthy Transaction Spreading Using Node Reliability Estimation in IoT Blockchain Networks
## 4 Trustworthy Transaction Spreading
### Determining the Next Hop for Transaction Traverse: Q-Learning  
Q-learning is a reinforcement-learning algorithm with model-free, value-based, and off-policy characteristics. Model-free implies that the learning agent does not know the model configuration of the environment. The agent learns only with the relationship between the input (state, action) and output (reward) of the environment. Value-based algorithms estimate a value function. The value function (\(V(s)\)) is the expected value of the total reward obtained when acting according to the policy in a given state. The value function learns the policies to maximize the value. Q-learning uses a state-action value function to select the most valuable action in the current state. Off-policy has different target and behavior policies. Off-policy has data efficiency because samples experienced with past policies can be used to update the current policies. Q-learning stores the function values (\(q(s,a)\)) in tabular form (i.e., Q-table) and updates them when each state is visited. The rows of the Q-table express states, and the columns express actions. A limitation of the tabular form is the rapid increase in the dimensions of the Q-table when solving complex problems. Q-learning is suitable for relatively simple problems or problems with small state and action dimensions. The proposed trustworthy transaction spreading defines the state as \"the node identification (ID) that spreads the transaction in the current time step.\" When spreading a transaction in a multi-hop manner, only one node attempts transaction spreading for each time step. It is suitable to set the node ID as a state of Q-learning. A simple state definition (i.e., node ID as the state) increases the learning efficiency of Q-learning by reducing the dimension of the state. This action is defined as the hop size for transaction spreading. Because the transaction spreading range of each node is determined by the hop size, the action is defined as selecting a hop size (for example, from one to four hops). The dimension of the action can be significantly reduced by selecting the spreading range (i.e., hop size) as an action, rather than selecting the next node as an action. The row in the Q-table is the node number (\(N\)) constituting the network, and the column is generally a limited number of hop sizes. The operation on the Q-table has a limited complexity of \(O(n)\) that increases in proportion to the number of nodes, even if the network is expanded. Rewards and penalties are granted under the following three conditions: (1) When a transaction reaches the destination node successfully, a fixed reward \(K\) is received. (2) The greater the transaction spread, the higher the reward. \(z_{i}\) represents the ratio of nodes in which a transaction spreads. For example, when a transaction spreads to 50 nodes over 100 network nodes, we calculate \(z_{i}\) as 50%. Note that \(i\) is the index of table updates. The  
Figure 3: Concatenation of Q-Learning and GCN for finding Node Reliability. reward increased significantly as the \(z_{i}\) increased. The reward also increases in proportion to the number of duplicated transactions at each node. Duplicate transactions (\(c_{i}^{(n)}\), \(n\): node index) have a positive effect on the transaction validation. (3) However, if the transaction spreading exceeds 50% of the total nodes, the degree of the positive reward for spreading is greatly reduced. When the spreading rate (\(z_{i}\)) exceeds 50%, the advantage of transaction sharing is drastically reduced and only the communication burden increases. We reduce the reward in proportion to the number of nodes (\(N\)). \[R_{i}=\left\{ \begin{array}{l}K\;when\; transaction\; reach\; destination\; node\\ z_{i}^{2}+\sum_{n=1}^{N}c_{i}^{(n)}\;\;when\;z_{i}\;is\;under\;50\%\\ \left(z_{i}^{2}+\sum_{n=1}^{N}c_{i}^{(n)}\right)/N\;when\;z_{i}\;reach\;50\% \end{array} \right. \tag{1}\]  
The origin or border nodes select the action with the highest reward value.