Cloudflare Zero Trust で Managed network が動かないときのトラブルシュート

# 目的
Cloudflare Zero Trust で Managed network 機能を試そうとしたときに何が原因で動かなかったのか、トラブルシュートした際のメモです。

# Managed network とは
以下のように Managed network 内の TLS エンドポイントに対して SHA256 フィンガープリントのチェックを自動的に行い、 WARP プロファイルを動的に変える仕組みです。

> - [Network detection and settings profiles for the Cloudflare One agent](https://blog.cloudflare.com/location-aware-warp)
> ユーザーのネットワークに応じて、どのトラフィックをCloudflareに送信し、どのトラフィックをローカルに残すかを柔軟に決定できるようになりました。エンドユーザーは、マネージドネットワークに出入りする際に設定を変更する必要はありません。Cloudflare Oneのデバイスエージェントが自動的に検出し、変更を行います。

例えば、自宅ではフィルタリングを適用する一方で、社内ネットワークでは社内プロキシから出ていくようなネットワークロケーションごとにプロファイルを変える構成が考えられます。

- [Managed networks · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/configure-warp/managed-networks/)

```mermaid:
---
title: How WARP automatically detect Managed networks and change Profiles
---
graph LR

subgraph Home [Remote User - Home WiFi]
subgraph HPC [PC]
HWARP["WARP Profile = Home<br>→ Connected"]
end
end

subgraph Office ["Office User - Corp Intranet (Managed network)"]
direction LR
subgraph OPC [PC]
OWARP["WARP Profile = Office<br>→ Filtering OFF"]
end
TLS[TLS endpoint]
CProxy[Corp Proxy]
OWARP --> |Validate SHA-256 fingerprint|TLS
TLS --> |If Match Then Change Profile|OWARP
OWARP --> CProxy
end

Cloudflare[Cloudflare<br>Zero Trust]
Web[Web App]

HWARP ---> |Encrypted Tunnel|Cloudflare ---> |Proxied Traffic|Web
CProxy ---> |Proxied Traffic|Web
```

# 参考 : Managed network 設定

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/34cef7d4-f8be-c7f8-a752-a5dbd0f22f60.png)

```shell:
cfssl certinfo -domain 192.168.1.1 | jq -r .pem | openssl x509 -noout -fingerprint -sha256 -inform pem | tr -d :
```

```shell:result
sha256 Fingerprint=A3D48B68966ADFCCC9AD0E29D135E99174DA6CEF4AECB648DB758ACFB7AEEA90
```

```shell:
% curl https://192.168.1.1:443 -vvvk -IXGET
*   Trying 192.168.1.1:443...
* Connected to 192.168.1.1 (192.168.1.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=CN; ST=JiangSu; L=NanJing; O=ZTE; CN=192.168.1.1; emailAddress=xiaoming@zte.com
*  start date: Nov 15 01:57:17 2019 GMT
*  expire date: Apr 19 01:57:17 2036 GMT
*  issuer: C=CN; ST=JiangSu; L=NanJing; O=ZTE; emailAddress=xiaoming@zte.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha512WithRSAEncryption
* using HTTP/1.x
> GET / HTTP/1.1
> Host: 192.168.1.1
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: Mini web server 1.0 ZTE corp 2005.
Server: Mini web server 1.0 ZTE corp 2005.
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Connection: close
Connection: close
< Cache-Control: no-cache,no-store
Cache-Control: no-cache,no-store
< Content-Length: 8018
Content-Length: 8018
< Set-Cookie: _TESTCOOKIESUPPORT=1; PATH=/; HttpOnly; Secure
Set-Cookie: _TESTCOOKIESUPPORT=1; PATH=/; HttpOnly; Secure
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
X-XSS-Protection: 1; mode=block

< 
* Closing connection
* TLSv1.2 (OUT), TLS alert, close notify (256):
```

# 動かないときは

`warp-diag` コマンドで出力できるログを見ましょう。

- [Debug logs · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/troubleshooting/warp-logs/)

```shell:
warp-diag
```

```shell:result
warp_diag: Gathering data from WARP service...
warp_diag: Gathering system info and log files (this may take a couple minutes)...
warp_diag: Debugging information stored in: ~/Desktop/warp-debugging-info-20240226-033821.zip
```

`zip` 内の `daemon.log` ファイルで `alternate_networks` としてログが確認できます。

> `daemon.log`	: デバイスとCloudflareのグローバルネットワーク間のすべての通信を含む、WARPクライアントが実行したすべてのアクションの詳細ログ。注：これは最も有用なデバッグログです。

```shell:
unzip -o ~/Desktop/warp-debugging-info-20240226-033821.zip -d ~/Desktop "daemon.log"
cat ~/Desktop/daemon.log | grep alternate
```

```shell:result
Archive:  ~/Desktop/warp-debugging-info-20240220-005555.zip
  inflating: ~/Desktop/daemon.log  
2024-02-20T00:15:36.779Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:15:36.780Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TcpConnectionFailed(Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))
2024-02-20T00:15:37.852Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:15:37.860Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TlsHandshakeFailed(Custom { kind: InvalidData, error: AlertReceived(HandshakeFailure) })))
2024-02-20T00:29:47.737Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:29:47.738Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TcpConnectionFailed(Os { code: 61, kind: ConnectionRefused, message: "Connection refused" })))
2024-02-20T00:29:49.509Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:29:49.515Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TlsHandshakeFailed(Custom { kind: InvalidData, error: AlertReceived(HandshakeFailure) })))
2024-02-20T00:32:46.560Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:32:46.567Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TlsHandshakeFailed(Custom { kind: InvalidData, error: AlertReceived(HandshakeFailure) })))
2024-02-20T00:55:28.578Z DEBUG warp::warp_service::alternate_networks: Running alternate network detection check alternate_networks=[AlternateNetwork { name: "My Office Network", check: Tls { tls_sockaddr: "192.168.1.1:443", sha256: Some([163, 212, 139, 104, 150, 106, 223, 204, 201, 173, 14, 41, 209, 53, 233, 145, 116, 218, 108, 239, 74, 236, 182, 72, 219, 117, 138, 207, 183, 174, 234, 144]) } }]
2024-02-20T00:55:28.584Z DEBUG warp::warp_service::alternate_networks: Did not detect alternate network name=My Office Network reason=FailedToGetFingerprint(FailedToConnect(TlsHandshakeFailed(Custom { kind: InvalidData, error: AlertReceived(HandshakeFailure) })))
```

# `ConnectionRefused` , `TlsHandshakeFailed` は暗号スイート（Cipher suite）を確認

TLS ハンドシェイクにて接続が失敗しているため、 Managed network で使用されている接続ライブラリの暗号スイート(Cipher suite)が対応していない可能性があります。

> - [Managed networks - Supported cipher suites · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/configure-warp/managed-networks/#supported-cipher-suites)
> The WARP client establishes a TLS connection using [Rustls](https://github.com/rustls/rustls). Make sure your TLS endpoint accepts one of the [cipher suites supported by Rustls](https://docs.rs/rustls/0.21.10/src/rustls/suites.rs.html#125-143).

実際に `rustls` を使って確認すると、同様のエラーが確認できます。

```
rustls % cargo run --bin tlsclient-mio -- --http 192.168.1.1
    Finished dev [unoptimized + debuginfo] target(s) in 0.44s
     Running `target/debug/tlsclient-mio --http 192.168.1.1`
TLS error: AlertReceived(HandshakeFailure)
Connection closed
```

以下の暗号スイート(Cipher suite)がサポート対象になります。

- [暗号スイート(Cipher suite)表記に関して(OpenSSLとIANA) - Cisco Community](https://community.cisco.com/t5/tkb-%E3%82%BB%E3%82%AD%E3%83%A5%E3%83%AA%E3%83%86%E3%82%A3-%E3%83%89%E3%82%AD%E3%83%A5%E3%83%A1%E3%83%B3%E3%83%88/%E6%9A%97%E5%8F%B7%E3%82%B9%E3%82%A4%E3%83%BC%E3%83%88-cipher-suite-%E8%A1%A8%E8%A8%98%E3%81%AB%E9%96%A2%E3%81%97%E3%81%A6-openssl%E3%81%A8iana/ta-p/4947423)


| TLS | IANA 表記 | OpenSSL 表記 |
|:-:|:-:|:-:|
| 1.3 | TLS13_AES_256_GCM_SHA384 | TLS_AES_256_GCM_SHA384 |
| 1.3 | TLS13_AES_128_GCM_SHA256 | TLS_AES_128_GCM_SHA256 |
| 1.3 | TLS13_CHACHA20_POLY1305_SHA256 | TLS_CHACHA20_POLY1305_SHA256 |
| 1.2 | TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 | ECDHE-ECDSA-AES256-GCM-SHA384 |
| 1.2 | TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 | ECDHE-ECDSA-AES128-GCM-SHA256 |
| 1.2 | TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 | ECDHE-ECDSA-CHACHA20-POLY1305 |
| 1.2 | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 | ECDHE-RSA-AES256-GCM-SHA384 |
| 1.2 | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 | ECDHE-RSA-AES128-GCM-SHA256 |
| 1.2 | TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 | ECDHE-RSA-CHACHA20-POLY1305 |

以下の `sslscan` を使って暗号スイート(Cipher suite)の対応状況を確認できます。

- [rbsec/sslscan: sslscan tests SSL/TLS enabled services to discover supported cipher suites](https://github.com/rbsec/sslscan)

```shell:Install
brew install sslscan
```

```shell:
sslscan \
--iana-names \
--no-cipher-details \
--no-compression \
--no-fallback \
--no-groups \
--no-renegotiation \
--no-heartbleed \
--no-check-certificate \
192.168.1.1
```

結果としては、サポートされる暗号スイートが含まれていなかったため、TLS ハンドシェイクにて接続が失敗したとわかります。

```shell:result
Version: 2.1.3
OpenSSL 3.2.0 23 Nov 2023

Connected to 192.168.1.1

Testing SSL server 192.168.1.1 on port 443 using SNI name 192.168.1.1

  SSL/TLS Protocols:
SSLv2     disabled
SSLv3     disabled
TLSv1.0   disabled
TLSv1.1   disabled
TLSv1.2   enabled
TLSv1.3   disabled

  Supported Server Cipher(s):
Preferred TLSv1.2  256 bits  TLS_RSA_WITH_AES_256_GCM_SHA384              
Accepted  TLSv1.2  128 bits  TLS_RSA_WITH_AES_128_GCM_SHA256              
Accepted  TLSv1.2  256 bits  TLS_RSA_WITH_AES_256_CBC_SHA256              
Accepted  TLSv1.2  128 bits  TLS_RSA_WITH_AES_128_CBC_SHA256              
Accepted  TLSv1.2  256 bits  TLS_RSA_WITH_AES_256_CBC_SHA                 
Accepted  TLSv1.2  128 bits  TLS_RSA_WITH_AES_128_CBC_SHA        
```

# まとめ

Zero Trust ソリューションでは、さまざまなコンポーネントが関係することがありますが、問題切り分けとしては、まずログを見て適切な窓口に問い合わせる、デバッグ目的のコマンド結果を確認することで、具体的にどこでエラーが発生しているかがわかります。

# 参考 : `openssl` コマンドで確認

- [How to pass cipher list to OpenSSL s_client - Information Security Stack Exchange](https://security.stackexchange.com/questions/93143/how-to-pass-cipher-list-to-openssl-s-client)

```shell:
openssl s_client -cipher TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305 \
-connect 192.168.1.1:443
```

```:つながらない例
Connecting to 192.168.1.1
CONNECTED(00000003)
C0270E44F87F0000:error:0A000410:SSL routines:ssl3_read_bytes:ssl/tls alert handshake failure:ssl/record/rec_layer_s3.c:861:SSL alert number 40
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 257 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
```