Windows における Cloudflared トラブルシューティング

# 目的

Windows Server に `cloudflared` トンネルをインストールしたときにつながらなかった場合の調査ステップをメモしておきます。

- 参考：[Tunnels FAQ · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/faq/cloudflare-tunnels-faq/)
- 参考：[Troubleshooting · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/faq/teams-troubleshooting/#my-tunnel-fails-to-authenticate)

# 通信要件の確認

以下の要件を満たしていることを確認しましょう。

通信経路の間にプロキシ等が入っていないか、も確認しておきましょう。

> - [Ports and IPs · Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/do-more-with-tunnels/ports-and-ips/)
>
> ![image-20230411011642992](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20230411011642992.png)

## `regionX.v2.argotunnel.com` の確認
以下のようなコマンドでの確認のやり方が上記リンク内に掲載されているので、実行して確認します。

```powershell:名前解決
PS C:\Program Files (x86)\cloudflared> Resolve-DnsName -Name _v2-origintunneld._tcp.argotunnel.com SRV

Name                                     Type   TTL   Section    NameTarget                     Priority Weight Port
----                                     ----   ---   -------    ----------                     -------- ------ ----
_v2-origintunneld._tcp.argotunnel.com    SRV    300   Answer     region2.v2.argotunnel.com      2        1      7844
_v2-origintunneld._tcp.argotunnel.com    SRV    300   Answer     region1.v2.argotunnel.com      1        1      7844

Name       : region2.v2.argotunnel.com
QueryType  : A
TTL        : 18999
Section    : Additional
IP4Address : 198.41.200.23

...
```

```powershell:TCP 7844 疎通確認
PS C:\Program Files (x86)\cloudflared> tnc region1.v2.argotunnel.com -port 7844


ComputerName     : region1.v2.argotunnel.com
RemoteAddress    : 198.41.192.7
RemotePort       : 7844
InterfaceAlias   : Ethernet
SourceAddress    : 10.146.0.10
TcpTestSucceeded : True
```

## `api.cloudflare.com` の確認
```powershell
> tnc -port 443 api.cloudflare.com

ComputerName     : api.cloudflare.com
RemoteAddress    : 104.19.193.29
RemotePort       : 443
InterfaceAlias   : Ethernet
SourceAddress    : 10.146.0.10
TcpTestSucceeded : True
```

```powershell
> Invoke-WebRequest -Verbose 'https://api.cloudflare.com/client/v4/ips'
VERBOSE: GET https://api.cloudflare.com/client/v4/ips with 0-byte payload
VERBOSE: received 505-byte response of content type application/json


StatusCode        : 200
StatusDescription : OK
Content           : {"result":{"ipv4_cidrs":["173.245.48.0/20","103.21.244.0/22","103.22.200.0/22","103.31.4.0/22","141.101.64.0/18","108.162.192.0/18","190.93.240.0/20","188.114.96.0/20","197.234.240.0/22","198.41.128.0
                    ...
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    CF-Ray: 7b67fe937a113c14-NRT
                    Vary: Accept-Encoding
                    Content-Length: 505
                    Content-Type: application/json
                    Date: Wed, 12 Apr 2023 02:21:11 GMT
                    ETag: "38f79d050...
Forms             : {}
Headers           : {[Connection, keep-alive], [CF-Ray, 7b67fe937a113c14-NRT], [Vary, Accept-Encoding], [Content-Length, 505]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 505
```

## `update.argotunnel.com` の確認
```powershell
> tnc -port 443 update.argotunnel.com

ComputerName     : update.argotunnel.com
RemoteAddress    : 104.18.24.129
RemotePort       : 443
InterfaceAlias   : Ethernet
SourceAddress    : 10.146.0.10
TcpTestSucceeded : True
```

```powershell
> Invoke-WebRequest -Verbose 'https://update.argotunnel.com?arch=amd64&clientVersion=2023.3.1&os=windows'
VERBOSE: GET https://update.argotunnel.com/?arch=amd64&clientVersion=2023.3.1&os=windows with 0-byte payload
VERBOSE: received 335-byte response of content type application/json; charset=utf-8


StatusCode        : 200
StatusDescription : OK
Content           : {"version":"2023.4.0","url":"https://github.com/cloudflare/cloudflared/releases/download/2023.4.0/cloudflared-windows-amd64.exe","checksum":"f49cde976e628012c9db73e1c8d76081944ecf2297cdafeb78bb13290da
                    ...
RawContent        : HTTP/1.1 200 OK
                    Connection: keep-alive
                    CF-RAY: 7b67ec860eeee011-NRT
                    Content-Length: 335
                    Content-Type: application/json; charset=utf-8
                    Date: Wed, 12 Apr 2023 02:08:52 GMT
                    Server: cloudflare

                    {"...
Forms             : {}
Headers           : {[Connection, keep-alive], [CF-RAY, 7b67ec860eeee011-NRT], [Content-Length, 335], [Content-Type, application/json; charset=utf-8]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 335
```

# インストール時コマンド結果の確認

Cloudflared トンネル作成時に提示されるコマンドを実施した結果を確認します。

![image-20230411012541323](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20230411012541323.png)

正しくインストールされた場合は、以下のような結果となります。

```terminal
C:\Program Files (x86)\cloudflared> cloudflared.exe service install eyJhIjoxxxxx
2023-04-06T16:18:25Z INF Installing cloudflared Windows service
2023-04-06T16:18:25Z INF cloudflared agent service is installed windowsServiceName=Cloudflared
2023-04-06T16:18:25Z INF Agent service for cloudflared installed successfully windowsServiceName=Cloudflared
```

# Windows サービスが「実行中(Running)」であることの確認

通常は `Cloudflared agent` でサービスが登録され、起動されます。

```terminal
compmgmt
```

![image-20230411012726676](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20230411012726676.png)

# イベントログ内のエラー確認

カスタムビュー作成からイベントログ内にエラーがないかを確認します。

```terminal
eventvwr
```

![image-20230411013553541](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20230411013553541.png)

Powershell では以下のように検索・出力ができます。

- [PowerShell でイベントログを検索するのが結構べんりやん-スケ郎のお話](https://www.sukerou.com/2021/09/powershell.html)

```
PS C:\Program Files (x86)\cloudflared> Get-EventLog Application | Where-Object {$_.Source -eq "Cloudflared"}| Format-List


Index              : 617021
EntryType          : Information
InstanceId         : 1
Message            : Cloudflared service arguments: [C:\Program Files (x86)\cloudflared\cloudflared.exe tunnel run --token
                     eyJhIjxxxxx]
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {Cloudflared service arguments: [C:\Program Files (x86)\cloudflared\cloudflared.exe tunnel run --token
                     eyJhIjxxxxx]}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:51:33 AM
TimeWritten        : 4/7/2023 2:51:33 AM
UserName           :

Index              : 617020
EntryType          : Information
InstanceId         : 1
Message            : Cloudflared service starting
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {Cloudflared service starting}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:51:33 AM
TimeWritten        : 4/7/2023 2:51:33 AM
UserName           :

Index              : 617012
EntryType          : Information
InstanceId         : 1
Message            : Cloudflared service arguments: [C:\Program Files (x86)\cloudflared\cloudflared.exe tunnel run --token
                     eyJhIjoxxxxx]
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {Cloudflared service arguments: [C:\Program Files (x86)\cloudflared\cloudflared.exe tunnel run --token
                     eyJhIjoxxxxx]}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:50:48 AM
TimeWritten        : 4/7/2023 2:50:48 AM
UserName           :

Index              : 617011
EntryType          : Information
InstanceId         : 1
Message            : Cloudflared service starting
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {Cloudflared service starting}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:50:48 AM
TimeWritten        : 4/7/2023 2:50:48 AM
UserName           :

Index              : 617000
EntryType          : Information
InstanceId         : 1
Message            : Cloudflared service stopped
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {Cloudflared service stopped}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:50:19 AM
TimeWritten        : 4/7/2023 2:50:19 AM
UserName           :

Index              : 616999
EntryType          : Information
InstanceId         : 1
Message            : cloudflared terminated without error
Category           : (0)
CategoryNumber     : 0
ReplacementStrings : {cloudflared terminated without error}
Source             : Cloudflared
TimeGenerated      : 4/7/2023 2:50:19 AM
TimeWritten        : 4/7/2023 2:50:19 AM
UserName           :
```

# より詳細なデバッグログを入手して調査

以下の手順でレジストリ値を変更することで、 Windows サービス起動時のオプションを変更できます。

- [Configure a remote tunnel - Windows | Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/remote/remote-management/#windows)

`--logfile C:\cloudflared.txt --loglevel debug --transport-loglevel debug` のオプションを追加します。

- [Cloudflared parameters - logfile, loglevel, transport-loglevel | Cloudflare Zero Trust docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/arguments/#logfile)

具体的には以下のようなレジストリ値とし、デバッグレベルのログを取得します。

```
"C:\Program Files (x86)\cloudflared\cloudflared.exe" tunnel --logfile C:\cloudflared.txt --loglevel debug --transport-loglevel debug run --token eyJhIjoxxxxx
```

レジストリ値にサービス起動オプションを追加した後は、サービスを再起動します。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/101361/283f165f-f90c-a007-a3aa-47f4fcb76a72.png)

そうすることで詳細な `debug` ログが確認できるため、問題の特定に役立ちます。

![image-20230411014425166](https://cdn.jsdelivr.net/gh/kyouheicf/image@master/uPic/image-20230411014425166.png)

# 最新バージョンへのアップデート

ソフトウェアの既知のバグを踏んでいる場合は、最新バージョンへのアップデートで解決する場合があります。

- [Releases · cloudflare/cloudflared · GitHub](https://github.com/cloudflare/cloudflared/releases)

```terminal
C:\Program Files (x86)\cloudflared> cloudflared.exe update --version 2023.4.0
2023-04-11T00:31:57Z INF cloudflared has been updated version=2023.4.0
cloudflared has been updated to version 2023.4.0
```

最後にサービスを再起動します。

```terminal
C:\Program Files (x86)\cloudflared> net stop "Cloudflared" & net start "Cloudflared"
.
The Cloudflared agent service was stopped successfully.

The Cloudflared agent service is starting.
The Cloudflared agent service was started successfully.
```

# 一般的な通信エラーの例

`--transport-loglevel debug` のフラグを有効にすると、一般的な通信エラーログが確認できることがあります。

具体的には以下のようなログになりますが、この場合はファイアウォールの出口ルールがあり、Cloudflare Tunnel の通信をブロックしていることが考えられます。

通信経路にある社内のファイアウォールルール・ログを改めて確認し、必要に応じて穴開けをしましょう。

- [Error TLS handshake with edge error: read tcp 10.10.41.50:59594->198.41.200.233:7844: wsarecv: An existing connection was forcibly closed by the remote host - Zero Trust / Cloudflare Tunnel - Cloudflare Community](https://community.cloudflare.com/t/error-tls-handshake-with-edge-error-read-tcp-10-10-41-50-59594-198-41-200-233-wsarecv-an-existing-connection-was-forcibly-closed-by-the-remote-host/243837/4)

```
"level":"error","error":"Get \"https://update.argotunnel.com?arch=amd64&clientVersion=2023.3.1&os=windows\": read tcp 10.x.x.x:XXXXX->104.18.24.129:443: wsarecv: An existing connection was forcibly closed by the remote host.","time":"2023-04-11T04:25:10Z","message":"update check failed"}
...
{"level":"error","ip":"198.41.192.57","connIndex":0,"error":"timeout: no recent network activity","time":"2023-04-11T04:25:15Z","message":"Failed to serve quic connection"}
{"level":"error","ip":"198.41.192.57","connIndex":0,"error":"timeout: no recent network activity","time":"2023-04-11T04:25:15Z","message":"Serve tunnel error"}
...
{"level":"warn","ip":"198.41.192.57","connIndex":0,"time":"2023-04-11T04:25:16Z","message":"If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/\nIf you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network with `quic`."}
...
{"level":"error","ip":"198.41.192.107","connIndex":0,"error":"TLS handshake with edge error: read tcp 10.x.x.x:XXXXX->198.41.192.107:7844: wsarecv: An existing connection was forcibly closed by the remote host.","time":"2023-04-11T04:25:16Z","message":"Unable to establish connection with Cloudflare edge"}
{"level":"error","ip":"198.41.192.107","connIndex":0,"error":"TLS handshake with edge error: read tcp 10.x.x.x:XXXXX->198.41.192.107:7844: wsarecv: An existing connection was forcibly closed by the remote host.","time":"2023-04-11T04:25:16Z","message":"Serve tunnel error"}
```