Repository: trustwallet/blockatlas
Pull request #1091 (2020-05-11 22:58:33)

Enable ethereum pending txs
Changes:
1. Remove pending tx condition
2. Check data field when fill metadata
3. Calculate gas (limit * price) for pending tx
4. Add pending incoming eth/token transfer test

codecov[bot]: # [Codecov](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=h1) Report
> Merging [#1091](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=desc) into [master](https://codecov.io/gh/trustwallet/blockatlas/commit/024bd444442bd518faf52a409231ae75da7f735d&el=desc) will **increase** coverage by `0%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/trustwallet/blockatlas/pull/1091/graphs/tree.svg?width=650&height=150&src=pr&token=jnrjoEQAqv)](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1091   +/-   ##
======================================
  Coverage      37%     37%           
======================================
  Files         157     157           
  Lines        5268    5266    -2     
======================================
  Hits         1989    1989           
+ Misses       3145    3144    -1     
+ Partials      134     133    -1     
```


| [Impacted Files](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [platform/ethereum/blockbook/transaction.go](https://codecov.io/gh/trustwallet/blockatlas/pull/1091/diff?src=pr&el=tree#diff-cGxhdGZvcm0vZXRoZXJldW0vYmxvY2tib29rL3RyYW5zYWN0aW9uLmdv) | `71% <ø> (+1%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=footer). Last update [024bd44...c6cdfc2](https://codecov.io/gh/trustwallet/blockatlas/pull/1091?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

hewigovens: @vikmeup current blockbook api returns pending tx like this
```json
{
    "txid": "0xf1fc796cab0a025775658c41b7a77556528081a4141a3dcc821e4c60d8fcf9cc",
    "vin": [
        {
            "n": 0,
            "addresses": [
                "0x23a80893B3CE3B17598c2077739559B5f13F81bA"
            ],
            "isAddress": true
        }
    ],
    "vout": [
        {
            "value": "0",
            "n": 0,
            "addresses": [
                "0xdAC17F958D2ee523a2206206994597C13D831ec7"
            ],
            "isAddress": true
        }
    ],
    "blockHeight": -1,
    "confirmations": 0,
    "blockTime": 1589251274,
    "value": "0",
    "fees": "0",
    "rbf": true,
    "tokenTransfers": [
        {
            "type": "ERC20",
            "from": "0x23a80893B3CE3B17598c2077739559B5f13F81bA",
            "to": "0x87c574FB2DF0EaA2dAf5fc4a8A16dd3Ce39011B1",
            "token": "0xdAC17F958D2ee523a2206206994597C13D831ec7",
            "name": "Tether USD",
            "symbol": "USDT",
            "decimals": 6,
            "value": "3940610000"
        }
    ],
    "ethereumSpecific": {
        "status": -1,
        "nonce": 0,
        "gasLimit": 100000,
        "gasUsed": null,
        "gasPrice": "30000000000"
    }
}
```
hewigovens: we couldn't get nonce/data/gas, so the type is fallback to contract_call
vikmeup: Can we adjust logic to support this? since we have all the data? 
hewigovens: need to include abi parsing somehow, not a quick fix, also we need to update to latest code to include data field
hewigovens: > Can we adjust logic to support this? since we have all the data?

it turns out we don't need to do abi parsing now, the only issue left is low gas price tx might be dropped (but can be seen in etherscan.io)

Patch:
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index 4aafad0..91d2b99 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -1,1 +1,1 @@
-* @EnoRage @catenocrypt @hewigovens @kolya182 @prazd
+* @EnoRage
diff --git a/cmd/observer_notifier/main.go b/cmd/observer_notifier/main.go
index 623676c..850d712 100644
--- a/cmd/observer_notifier/main.go
+++ b/cmd/observer_notifier/main.go
@@ -55,8 +55,9 @@ func init() {
 
 	logger.Info("maxPushNotificationsBatchLimit ", logger.Params{"limit": maxPushNotificationsBatchLimit})
 
-	go mq.RestoreConnectionWorker(mqHost, mq.RawTransactions, time.Second*10)
+	go mq.FatalWorker(time.Second * 10)
 	go db.RestoreConnectionWorker(database, time.Second*10, pgUri)
+
 	time.Sleep(time.Millisecond)
 }
 
diff --git a/platform/ethereum/blockbook/model.go b/platform/ethereum/blockbook/model.go
index 64d800f..415b529 100644
--- a/platform/ethereum/blockbook/model.go
+++ b/platform/ethereum/blockbook/model.go
@@ -73,4 +73,5 @@ type EthereumSpecific struct {
 	GasLimit *big.Int `json:"gasLimit"`
 	GasUsed  *big.Int `json:"gasUsed"`
 	GasPrice string   `json:"gasPrice"`
+	Data     string   `json:"data,omitempty"`
 }
diff --git a/platform/ethereum/blockbook/model_extension.go b/platform/ethereum/blockbook/model_extension.go
index 611b290..40d96de 100644
--- a/platform/ethereum/blockbook/model_extension.go
+++ b/platform/ethereum/blockbook/model_extension.go
@@ -1,6 +1,8 @@
 package blockbook
 
 import (
+	"math/big"
+
 	"github.com/trustwallet/blockatlas/pkg/blockatlas"
 )
 
@@ -24,6 +26,21 @@ func (t *Transaction) FromAddress() string {
 	return ""
 }
 
+func (t *Transaction) GetFee() string {
+	status, _ := t.EthereumSpecific.GetStatus()
+	if status != blockatlas.StatusPending {
+		return t.Fees
+	}
+
+	gasLimit := t.EthereumSpecific.GasLimit
+	gasPrice, ok := new(big.Int).SetString(t.EthereumSpecific.GasPrice, 10)
+	if gasLimit == nil || !ok {
+		return "0"
+	}
+	fee := new(big.Int).Mul(gasLimit, gasPrice)
+	return fee.String()
+}
+
 func (t *Transaction) ToAddress() string {
 	if len(t.Vout) > 0 && len(t.Vout[0].Addresses) > 0 {
 		return t.Vout[0].Addresses[0]
diff --git a/platform/ethereum/blockbook/transaction.go b/platform/ethereum/blockbook/transaction.go
index b5d1090..e539d54 100644
--- a/platform/ethereum/blockbook/transaction.go
+++ b/platform/ethereum/blockbook/transaction.go
@@ -1,6 +1,8 @@
 package blockbook
 
 import (
+	"strings"
+
 	"github.com/trustwallet/blockatlas/coin"
 	Address "github.com/trustwallet/blockatlas/pkg/address"
 	"github.com/trustwallet/blockatlas/pkg/blockatlas"
@@ -30,9 +32,6 @@ func NormalizePage(srcPage *Page, address, token string, coinIndex uint) blockat
 		normalizedToken = Address.EIP55Checksum(token)
 	}
 	for _, srcTx := range srcPage.Transactions {
-		if srcTx.BlockHeight < 0 {
-			continue // Skip pending tx
-		}
 		tx := normalizeTxWithAddress(&srcTx, normalizedAddr, normalizedToken, coinIndex)
 		txs = append(txs, tx)
 	}
@@ -46,7 +45,7 @@ func normalizeTx(srcTx *Transaction, coinIndex uint) blockatlas.Tx {
 		Coin:     coinIndex,
 		From:     srcTx.FromAddress(),
 		To:       srcTx.ToAddress(),
-		Fee:      blockatlas.Amount(srcTx.Fees),
+		Fee:      blockatlas.Amount(srcTx.GetFee()),
 		Date:     srcTx.BlockTime,
 		Block:    normalizeBlockHeight(srcTx.BlockHeight),
 		Status:   status,
@@ -146,8 +145,25 @@ func fillTransferOrContract(final *blockatlas.Tx, tx *Transaction, coinIndex uin
 		}
 		return
 	}
-	final.Meta = blockatlas.ContractCall{
-		Input: "0x", // FIXME blockbook api doesn't return tx data field
-		Value: tx.Value,
+	data := tx.EthereumSpecific.Data
+	if data == "" {
+		// old node doesn't have data field
+		final.Meta = blockatlas.ContractCall{
+			Input: "0x",
+			Value: tx.Value,
+		}
+	} else {
+		if len(strings.TrimPrefix(data, "0x")) > 0 {
+			final.Meta = blockatlas.ContractCall{
+				Input: data,
+				Value: tx.Value,
+			}
+		} else {
+			final.Meta = blockatlas.Transfer{
+				Value:    blockatlas.Amount(tx.Value),
+				Symbol:   coin.Coins[coinIndex].Symbol,
+				Decimals: coin.Coins[coinIndex].Decimals,
+			}
+		}
 	}
 }


Test patch:
diff --git a/platform/ethereum/blockbook/transaction_test.go b/platform/ethereum/blockbook/transaction_test.go
index 1710dea..8593589 100644
--- a/platform/ethereum/blockbook/transaction_test.go
+++ b/platform/ethereum/blockbook/transaction_test.go
@@ -4,6 +4,7 @@ import (
 	"encoding/json"
 	"testing"
 
+	"github.com/stretchr/testify/assert"
 	"github.com/trustwallet/blockatlas/pkg/blockatlas"
 )
 
@@ -137,7 +138,127 @@ func TestNormalizePage(t *testing.T) {
 							"gasUsed": 29126,
 							"gasPrice": "3000000000"
 						}
-  					  }
+					},
+					{
+						"txid": "0x7a3929f2fad5e61f535ed5c1317f34e739655d582bc1b0161b9869b0957df6af",
+						"vin": [
+							{
+								"n": 0,
+								"addresses": [
+									"0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1"
+								],
+								"isAddress": true
+							}
+						],
+						"vout": [
+							{
+								"value": "567000000000000",
+								"n": 0,
+								"addresses": [
+									"0x47331175b23C2f067204B506CA1501c26731C990"
+								],
+								"isAddress": true
+							}
+						],
+						"blockHash": "0xf08fd4b1d6ace92bf9516bbed37de100025f8b0879a80a92359a08f37e788b95",
+						"blockHeight": 10050786,
+						"confirmations": 43,
+						"blockTime": 1589278824,
+						"value": "567000000000000",
+						"fees": "407799043328112",
+						"ethereumSpecific": {
+							"status": 1,
+							"nonce": 535,
+							"gasLimit": 33000,
+							"gasUsed": 21064,
+							"gasPrice": "19360000158",
+							"data": "0xdeadbeef"
+						}
+					},
+					{
+						"txid": "0xb1a56570bcb072d376630b987bd1f44ecc8f2c20ece52f02c9245296d3e3da39",
+						"vin": [
+							{
+								"n": 0,
+								"addresses": [
+									"0x2A0A572d77F6d6Ce62C6539E679d943824c3b218"
+								],
+								"isAddress": true
+							}
+						],
+						"vout": [
+							{
+								"value": "0",
+								"n": 0,
+								"addresses": [
+									"0xdAC17F958D2ee523a2206206994597C13D831ec7"
+								],
+								"isAddress": true
+							}
+						],
+						"blockHeight": -1,
+						"confirmations": 0,
+						"blockTime": 1589279659,
+						"value": "0",
+						"fees": "0",
+						"rbf": true,
+						"tokenTransfers": [
+							{
+								"type": "ERC20",
+								"from": "0x2A0A572d77F6d6Ce62C6539E679d943824c3b218",
+								"to": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1",
+								"token": "0xdAC17F958D2ee523a2206206994597C13D831ec7",
+								"name": "Tether USD",
+								"symbol": "USDT",
+								"decimals": 6,
+								"value": "23000000"
+							}
+						],
+						"ethereumSpecific": {
+							"status": -1,
+							"nonce": 15647,
+							"gasLimit": 100000,
+							"gasUsed": null,
+							"gasPrice": "28560000000",
+							"data": "0xa9059cbb000000000000000000000000595031ff9bf6e0c1de20349e1377f43934f8951400000000000000000000000000000000000000000000000000000000015ef3c0"
+						}
+					},
+					{
+						"txid": "0xfe7cce9928450e356f3332485e611781e407425b5888b8b2c7c66afaa4787cb1",
+						"vin": [
+							{
+								"n": 0,
+								"addresses": [
+									"0x267be1C1D684F78cb4F6a176C4911b741E4Ffdc0"
+								],
+								"isAddress": true
+							}
+						],
+						"vout": [
+							{
+								"value": "295000000000000000",
+								"n": 0,
+								"addresses": [
+									"0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1"
+								],
+								"isAddress": true
+							}
+						],
+						"blockHeight": -1,
+						"confirmations": 0,
+						"blockTime": 1589287339,
+						"value": "295000000000000000",
+						"fees": "0",
+						"rbf": true,
+						"ethereumSpecific": {
+							"status": -1,
+							"nonce": 1282636,
+							"gasLimit": 30000,
+							"gasUsed": null,
+							"gasPrice": "24255000245",
+							"data": "0x"
+						}
+					}
 					]}`,
 				address:   "0x7d8bf18c7ce84b3e175b339c4ca93aed1dd166f1",
 				token:     "",
@@ -182,18 +303,80 @@ func TestNormalizePage(t *testing.T) {
 						"from": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1",
 						"to": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1"
 					}
-				  }]`,
+				  },{
+					"id": "0x7a3929f2fad5e61f535ed5c1317f34e739655d582bc1b0161b9869b0957df6af",
+					"coin": 60,
+					"from": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1",
+					"to": "0x47331175b23C2f067204B506CA1501c26731C990",
+					"fee": "407799043328112",
+					"date": 1589278824,
+					"block": 10050786,
+					"status": "completed",
+					"sequence": 535,
+					"type": "contract_call",
+					"direction": "outgoing",
+					"memo": "",
+					"metadata": {
+						"input": "0xdeadbeef",
+						"value": "567000000000000"
+					}
+				  },{
+					"id": "0xb1a56570bcb072d376630b987bd1f44ecc8f2c20ece52f02c9245296d3e3da39",
+					"coin": 60,
+					"from": "0x2A0A572d77F6d6Ce62C6539E679d943824c3b218",
+					"to": "0xdAC17F958D2ee523a2206206994597C13D831ec7",
+					"fee": "2856000000000000",
+					"date": 1589279659,
+					"block": 0,
+					"status": "pending",
+					"sequence": 15647,
+					"type": "token_transfer",
+					"direction": "incoming",
+					"memo": "",
+					"metadata": {
+						"name": "Tether USD",
+						"symbol": "USDT",
+						"token_id": "0xdAC17F958D2ee523a2206206994597C13D831ec7",
+						"decimals": 6,
+						"value": "23000000",
+						"from": "0x2A0A572d77F6d6Ce62C6539E679d943824c3b218",
+						"to": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1"
+					}
+				  },{
+					"id": "0xfe7cce9928450e356f3332485e611781e407425b5888b8b2c7c66afaa4787cb1",
+					"coin": 60,
+					"from": "0x267be1C1D684F78cb4F6a176C4911b741E4Ffdc0",
+					"to": "0x7d8bf18C7cE84b3E175b339c4Ca93aEd1dD166F1",
+					"fee": "727650007350000",
+					"date": 1589287339,
+					"block": 0,
+					"status": "pending",
+					"sequence": 1282636,
+					"type": "transfer",
+					"direction": "incoming",
+					"memo": "",
+					"metadata": {
+						"value": "295000000000000000",
+						"symbol": "ETH",
+						"decimals": 18
+					}
+				  }
+				  ]`,
 		},
 	}
 	for _, tt := range tests {
 		var page Page
 		var txPage blockatlas.TxPage
-		_ = json.Unmarshal([]byte(tt.args.srcPage), &page)
-		_ = json.Unmarshal([]byte(tt.want), &txPage)
+		err := json.Unmarshal([]byte(tt.args.srcPage), &page)
+		assert.Nil(t, err)
+		err = json.Unmarshal([]byte(tt.want), &txPage)
+		assert.Nil(t, err)
 		t.Run(tt.name, func(t *testing.T) {
 			got := NormalizePage(&page, tt.args.address, tt.args.token, tt.args.coinIndex)
-			gotJson, _ := json.Marshal(got)
-			gotTxPage, _ := json.Marshal(txPage)
+			gotJson, err := json.Marshal(got)
+			assert.Nil(t, err)
+			gotTxPage, err := json.Marshal(txPage)
+			assert.Nil(t, err)
 			if string(gotJson) != string(gotTxPage) {
 				t.Errorf("NormalizePage() = %v, want %v", string(gotJson), string(gotTxPage))
 			}