Repository: rotki/rotki

Issue:
No support for Bech32 (bc1) addresses
## Problem Definition

I have some BTC balance in a Bech32 address (those that begin with `bc1`).
blockchain.info doesn't support those on the API level.
For example, here is a random address I found on blockchain.info: `bc1qhkje0xfvhmgk6mvanxwy09n45df03tj3h3jtnf`
This can be viewed in the interface: <https://www.blockchain.com/btc/address/bc1qhkje0xfvhmgk6mvanxwy09n45df03tj3h3jtnf>
But the API does not support it: <https://blockchain.info/q/addressbalance/bc1qhkje0xfvhmgk6mvanxwy09n45df03tj3h3jtnf>

The relevant rotkehlchen.log entry is:

```
20/02/2020 14:57:07 CET -- DEBUG:rotkehlchen.api.rest:Request successful response={"result": {"status": "completed", "outcome": {"result": null, "message": "The given string bc1qsvhnjfuhfund4684tylcqnd402jvdr9lt3h7l0 is not a valid BTC address"}}, "message": ""}, status_code=200
```

Maybe the `query_btc_account_balance` function in `blockchain.py`  could be modified to use different APIs depending on address type:

```py
        try:
            if account[0:3] == 'bc1':
                btc_resp = request_get(
                    url='https://api.blockcypher.com/v1/btc/main/addrs/%s/balance' % account,
                )['balance']
            else:
                btc_resp = request_get_direct(
                    url='https://blockchain.info/q/addressbalance/%s' % account,
                    handle_429=True,
                    # If we get a 429 then their docs suggest 10 seconds
                    # https://blockchain.info/q
                    backoff_in_seconds=10,
                )
```



### System Description

```
uname -a
command -v geth && geth version
command -v parity && parity --version
command -v rotkehlchen && rotkehlchen version
[ -d .git ] && git rev-parse HEAD
```
gives
```
Linux alladin 5.4.6-arch3-1 #1 SMP PREEMPT Tue, 24 Dec 2019 04:36:53 +0000 x86_64 GNU/Linux
/usr/bin/geth
Geth
Version: 1.9.9-stable
Git Commit: 017449971e1e9e220efcd97d3313a0e27f47003b
Git Commit Date: 20191206
Architecture: amd64
Protocol Versions: [64 63]
Go Version: go1.13.5
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go
/home/hjort/.local/bin/rotkehlchen
1.1.2.dev0+g42481b5.d20200220
42481b5f2ef28f25c99663a092f5c85157285379
```


Discussion:
Argh this is annoying. Had no idea about this. If they don't support querying of these type of address balances then indeed we need to have yet another api to query as you suggest.

Two questions.

1. Where can I read more about these type of addresses and why they are different?
2. Rotki does not accept the address, right? Or it accepts it but fails at balance query?
1. I'm no expert and didn't really know the difference before looking into this. I looked at <https://wiki.trezor.io/Bech32> and the BIP that introduced Bech32 addresses, <https://en.bitcoin.it/wiki/BIP_0173>

2. Correct. I get a "bell" notification: `Operation failed due to The given string bc1*************************************** is not a valid BTC address`

EDIT: We should also petition blockchain.info to add support. If they are willing to, problem solved.
Hey @hjorthjort would you like to handle this issue and submit a PR? If not don't worry we can also do it but since you discovered it I thought it might be a nice thing for you to contribute into Rotki this way.

What needs to be done is:

1. Add [bech32](https://pypi.org/project/bech32/) to the [rotki requirements](https://github.com/rotki/rotki/blob/develop/requirements.txt#L10).
2. Add a check for bech32 addresses in the  BTC address validator here: https://github.com/rotki/rotki/blob/d0c73f3bed898d815743146e985b6d331703e377/rotkehlchen/chain/bitcoin.py#L6. You would need to use [this function](https://github.com/fiatjaf/bech32/blob/master/bech32/__init__.py#L64) and check if it's a valid bech32 address. If not validation should fail. Essentially BTC logic should be like:
    * Is it a valid bas58 BTC address? Yay! We take it.
    * No? Then let's check for bech32 using the newly added library.
    * Is it a valid bech32 address? Yay! We take it.
    * No? Validation Error. The given string is not a valid BTC address
3. Add what you suggested in your [original comment](https://github.com/rotki/rotki/issues/734#issue-568376859). But since then the code changed. The blockchain class is now the chain manager and the new location is here: https://github.com/rotki/rotki/blob/d0c73f3bed898d815743146e985b6d331703e377/rotkehlchen/chain/manager.py#L186
4. Add also test cases for the new bech32 addresses in this unit test: 
https://github.com/rotki/rotki/blob/d0c73f3bed898d815743146e985b6d331703e377/rotkehlchen/tests/test_blockchain.py#L107-L114

For test cases I would recommend the one you used in your original post and the mainnet examples from [the specification](https://github.com/bitcoin/bips/blob/master/bip-0173.mediawiki#examples)


I'd be happy to. Not until next week the earliest though.

Would you prefer using a single endpoint? Rather than checking what type of address it is and using different APIs, if we support the API that can handle more kinds of addresses, maybe we should just do that.
> Would you prefer using a single endpoint? Rather than checking what type of address it is and using different APIs, if we support the API that can handle more kinds of addresses, maybe we should just do that.

Naah using multiple endpoints is good. In general, should have multiple backups as all these services go offline pretty often.

> I'd be happy to. Not until next week the earliest though.

Okay then depending on how v1.2.0 goes I may either do it myself if everything else is ready by the time we want to release it or push it to the next version and leave it to you. It's really something quick and easy. I listed all steps above :D
Alright! I'm moving to Berlin this week which is why things are a bit chaotic, so do as you like. I'll put it in my backlog for now.
hey @hjorthjort how is the move to Berlin going? Think you can still pick this issue up? We pushed it to the next release
> hey @hjorthjort how is the move to Berlin going? Think you can still pick this issue up? We pushed it to the next release

Pretty rough first week, but I'm excited to dig into this. Getting to know virtualenv right now so I can run as a dev.

Resolving patch:
diff --git a/rotkehlchen/chain/bitcoin.py b/rotkehlchen/chain/bitcoin.py
--- a/rotkehlchen/chain/bitcoin.py
+++ b/rotkehlchen/chain/bitcoin.py
@@ -1,10 +1,23 @@
 from hashlib import sha256
 
 import base58check
+import bech32
 
 
 def is_valid_btc_address(value: str) -> bool:
-    """Validates a bitcoin address for the mainnet
+    return is_valid_base58_address(value) or is_valid_bech32_address(value)
+
+
+def is_valid_bech32_address(value: str) -> bool:
+    """Validates a bitcoin SegWit address for the mainnet
+    """
+
+    decoded = bech32.decode('bc', value)
+    return decoded != (None, None)
+
+
+def is_valid_base58_address(value: str) -> bool:
+    """Validates a bitcoin base58 address for the mainnet
 
     Code is taken from:
     https://github.com/joeblackwaslike/coinaddr/blob/ae35c7ae550a687d9a7c2e0cb090d52edbb29cb5/coinaddr/validation.py#L67-L87
@@ -13,7 +26,11 @@ def is_valid_btc_address(value: str) -> bool:
     if 25 > len(value) > 35:
         return False
 
-    abytes = base58check.b58decode(value)
+    try:
+        abytes = base58check.b58decode(value)
+    except (ValueError):
+        return False
+
     if not abytes[0] in (0x00, 0x05):
         return False
 
diff --git a/rotkehlchen/chain/manager.py b/rotkehlchen/chain/manager.py
--- a/rotkehlchen/chain/manager.py
+++ b/rotkehlchen/chain/manager.py
@@ -34,7 +34,7 @@
     cache_response_timewise,
     protect_with_lock,
 )
-from rotkehlchen.utils.misc import request_get_direct, satoshis_to_btc
+from rotkehlchen.utils.misc import request_get, request_get_direct, satoshis_to_btc
 
 if TYPE_CHECKING:
     from rotkehlchen.chain.ethereum import Ethchain
@@ -210,18 +210,25 @@ def query_btc_account_balance(account: BTCAddress) -> FVal:
         """Queries blockchain.info for the balance of account
 
         May raise:
-        - RemotError if there is a problem querying blockchain.info
+        - RemotError if there is a problem querying blockchain.info or blockcypher
         """
         try:
-            btc_resp = request_get_direct(
-                url='https://blockchain.info/q/addressbalance/%s' % account,
-                handle_429=True,
-                # If we get a 429 then their docs suggest 10 seconds
-                # https://blockchain.info/q
-                backoff_in_seconds=10,
-            )
+            if account.lower()[0:3] == 'bc1':
+                url = f'https://api.blockcypher.com/v1/btc/main/addrs/{account.lower()}/balance'
+                response_data = request_get(url=url)
+                if 'balance' not in response_data:
+                    raise RemoteError(f'Unexpected blockcypher balance response: {response_data}')
+                btc_resp = response_data['balance']
+            else:
+                btc_resp = request_get_direct(
+                    url='https://blockchain.info/q/addressbalance/%s' % account,
+                    handle_429=True,
+                    # If we get a 429 then their docs suggest 10 seconds
+                    # https://blockchain.info/q
+                    backoff_in_seconds=10,
+                )
         except (requests.exceptions.ConnectionError, UnableToDecryptRemoteData) as e:
-            raise RemoteError(f'blockchain.info API request failed due to {str(e)}')
+            raise RemoteError(f'bitcoin external API request failed due to {str(e)}')
 
         return satoshis_to_btc(FVal(btc_resp))  # result is in satoshis
 


Test patch:
diff --git a/rotkehlchen/tests/test_blockchain.py b/rotkehlchen/tests/test_blockchain.py
--- a/rotkehlchen/tests/test_blockchain.py
+++ b/rotkehlchen/tests/test_blockchain.py
@@ -105,10 +105,42 @@ def test_add_remove_account_assure_all_balances_not_always_queried(blockchain):
 
 
 def test_is_valid_btc_address():
+    """Test cases for Bech32 addresses taken from here:
+    https://en.bitcoin.it/wiki/BIP_0173#Test_vectors
+    Note that we do not support test net addresses (starting with 'tb1').
+    """
     assert is_valid_btc_address(UNIT_BTC_ADDRESS1)
     assert is_valid_btc_address(UNIT_BTC_ADDRESS2)
     assert is_valid_btc_address(UNIT_BTC_ADDRESS3)
     assert is_valid_btc_address('18ddjB7HWTVxzvTbLp1nWvaBxU3U2oTZF2')
+    assert is_valid_btc_address('bc1qhkje0xfvhmgk6mvanxwy09n45df03tj3h3jtnf')
+    assert is_valid_btc_address('bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t4')
+    assert is_valid_btc_address('bc1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3qccfmv3')
+    assert is_valid_btc_address('BC1QW508D6QEJXTDG4Y5R3ZARVARY0C5XW7KV8F3T4')
+    assert is_valid_btc_address(
+        'bc1pw508d6qejxtdg4y5r3zarvary0c5xw7kw508d6qejxtdg4y5r3zarvary0c5xw7k7grplx')
+    assert is_valid_btc_address('BC1SW50QA3JX3S')
+    assert is_valid_btc_address('bc1zw508d6qejxtdg4y5r3zarvaryvg6kdaj')
 
     assert not is_valid_btc_address('foo')
     assert not is_valid_btc_address('18ddjB7HWTaxzvTbLp1nWvaixU3U2oTZ1')
+    assert not is_valid_btc_address(
+        'tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3qccfmv3')
+    assert not is_valid_btc_address('bc1qr')
+    assert not is_valid_btc_address(
+        'tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3q0sl5k7')
+    assert not is_valid_btc_address(
+        'tb1qqqqqp399et2xygdj5xreqhjjvcmzhxw4aywxecjdzew6hylgvsesrxh6hy')
+    assert not is_valid_btc_address('tc1qw508d6qejxtdg4y5r3zarvary0c5xw7kg3g4ty')
+    assert not is_valid_btc_address('bc1qw508d6qejxtdg4y5r3zarvary0c5xw7kv8f3t5')
+    assert not is_valid_btc_address('BC13W508D6QEJXTDG4Y5R3ZARVARY0C5XW7KN40WF2')
+    assert not is_valid_btc_address('bc1rw5uspcuh')
+    assert not is_valid_btc_address(
+        'bc10w508d6qejxtdg4y5r3zarvary0c5xw7kw508d6qejxtdg4y5r3zarvary0c5xw7kw5rljs90')
+    assert not is_valid_btc_address('BC1QR508D6QEJXTDG4Y5R3ZARVARYV98GJ9P')
+    assert not is_valid_btc_address(
+        'tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3q0sL5k7')
+    assert not is_valid_btc_address('bc1zw508d6qejxtdg4y5r3zarvaryvqyzf3du')
+    assert not is_valid_btc_address(
+        'tb1qrp33g0q5c5txsp9arysrx4k6zdkfs4nce4xj0gdcccefvpysxf3pjxtptv')
+    assert not is_valid_btc_address('bc1gmk9yu')