nginx＋ngx_luaで証明書動的読み込みの常時HTTPS用リバースプロキシ(2) 動的／静的の処理時間を比較してみた

先の記事、

 * **[nginx＋ngx_luaで証明書動的読み込みの常時HTTPS用リバースプロキシを手軽に立ててみる](https://qiita.com/hmatsu47/items/54bfcd599546a5988bd2)**

の続きです。

実際に、9 ドメイン 180 FQDN（Let's Encrypt で 1 週間に 9 ドメインに対して発行できる限界）で試してみました。

※本当は 10 ドメイン作ったのですが、`.nagoya`ドメインの Route 53 への移行で、運悪くお名前.com のネームサーバからの切り替えに時間が掛かってしまい、テストでの利用を断念しました…。

# テストの内容

 * **`curl`コマンド**で対象 FQDN に対してリクエストを送り、**`time_starttransfer`**を計測
 * 動的に SSL/TLS 証明書を読み込むテストでは、1 ドメインあたり 20 FQDN × 10 回を、`cron`を使って 5 ドメイン／ 9 ドメイン並行でリクエストを送る
 * 静的に SSL/TLS 証明書を読み込むテスト（比較用）では、1 FQDN に対して同じ数だけ並行してリクエストを送る
 * 主に**ハンドシェイクに掛かる時間の差**（動的 vs 静的）を計測する
 * 使用インスタンスは AWS EC2 の m4.large（CentOS 7）
 * nginx（OpenResty）と同じインスタンスに Apache で Web サーバを起動（パッケージインストール後、ほぼ初期設定のまま：レスポンスは 403 で Welcome ページが返る状態）

# 結果

| ドメイン（並行）数 | FQDN 数 | 総リクエスト | 静的 (a) | 動的 (b) | 増加率 |
|:-:|:-:|:-:|:-:|:-:|:-:|
| 5 | 100 | 1,000 | 0.306s | 0.318s | 3.95% |
| 9 | 180 | 1,800 | 0.443s | 0.454s | 2.33% |

※静的・動的の数値は 1 リクエストあたりの平均値。

いずれも、**トータルの所要時間は 60 秒（5 ドメイン並行時）強・90秒（9 ドメイン並行時）強**でした。

リクエスト毎の記録は掲示しませんが、特に時間が順に延びていくこともなく、この程度の負荷では**動的処理のオーバーヘッドはごくわずか**であることが分かりました。

---

# 参考／テスト環境の構築について

今回は、以下の流れで環境を用意しました。

## リバースプロキシ／Web サーバ側

### EC2 インスタンスを用意

 * EC2 に割り当てる IAM Role を作成（`AmazonRoute53FullAccess`ポリシー付き）
 * CentOS 7 の AMI から EC2 インスタンスを作成（↑を割り当て）
   * 要 awscli・httpd・openssl
     * `aws configure`でデフォルトリージョンを指定（`aws`コマンドを使うユーザで）
     * `/etc/httpd/conf.d/ssl.conf`をリネーム（`.bak`などに） 
   * OpenResty と Certbot クライアントをインストール
     * https://openresty.org/en/linux-packages.html
     * https://letsencrypt.jp/usage/#Install

### お名前.com でドメイン取得

### EC2 から Route 53 の Public Hosted Zone を登録

```bash:Zone作成
$ cat create_zone_hmatsu47.sh
#!/bin/sh
#aws route53 create-hosted-zone --name hmatsu47.site --caller-reference "20180704-hmatsu47.site"
aws route53 create-hosted-zone --name hmatsu47.com --caller-reference "20180704-hmatsu47.com"
aws route53 create-hosted-zone --name hmatsu47.net --caller-reference "20180704-hmatsu47.net"
aws route53 create-hosted-zone --name hmatsu47.info --caller-reference "20180704-hmatsu47.info"
aws route53 create-hosted-zone --name hmatsu47.org --caller-reference "20180704-hmatsu47.org"
aws route53 create-hosted-zone --name hmatsu47.cloud --caller-reference "20180704-hmatsu47.cloud"
aws route53 create-hosted-zone --name hmatsu47.biz --caller-reference "20180704-hmatsu47.biz"
aws route53 create-hosted-zone --name hmatsu47.work --caller-reference "20180704-hmatsu47.work"
aws route53 create-hosted-zone --name hmatsu47.jp.net --caller-reference "20180704-hmatsu47.jp.net"

$ sh ./create_zone_hmatsu47.sh
```

※`hmatsu47.site`は元から登録済。

各 Zone の NS の内容をお名前.com で DNS（その他）に設定。
登録完了を待つ間に各 Zone のレコードを登録。

```bash:レコード設定
$ cat record_set_site.json
{
  "Changes": [
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "test01.hmatsu47.site.",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "origin.hmatsu47.site."
          }
        ]
      }
    },
    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "test02.hmatsu47.site.",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "origin.hmatsu47.site."
          }
        ]
      }
    },

（中略）

    {
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "test20.hmatsu47.site.",
        "Type": "CNAME",
        "TTL": 60,
        "ResourceRecords": [
          {
            "Value": "origin.hmatsu47.site."
          }
        ]
      }
    }
  ]
}

$ aws route53 change-resource-record-sets --hosted-zone-id /hostedzone/XXXXXXXXXXXXXX --change-batch file://record_set_site.json

（後略）
```

※`origin.hmatsu47.site`の指す先をテスト用の EC2 サーバ（またはそれをターゲット設定した NLB など）のアドレスにしておく。

### 証明書発行

あらかじめ`httpd`を起動しておき、外部から`TCP:80`・`TCP:443`でアクセスできるようゲートウェイやセキュリティグループ等を設定しておく。

```bash:証明書リクエスト
$ cat certbot_site.sh
#!/bin/sh
#certbot certonly --webroot -w /var/www/html/ -d test01.hmatsu47.site --agree-tos -m hmatsu47@gmail.com
certbot certonly --webroot -w /var/www/html/ -d test02.hmatsu47.site --agree-tos -m hmatsu47@gmail.com
（中略）
certbot certonly --webroot -w /var/www/html/ -d test20.hmatsu47.site --agree-tos -m hmatsu47@gmail.com

$ certbot certonly --webroot -w /var/www/html/ -d test01.hmatsu47.site --agree-tos -m hmatsu47@gmail.com
※1 件目は手動操作で。 

$ sh ./certbot_site.sh

（後略）
```

```bash:証明書・秘密鍵確認
# ls -l /etc/letsencrypt/live/
合計 0
drwxr-xr-x. 2 root root 93  7月  4 04:32 test01.hmatsu47.biz
drwxr-xr-x. 2 root root 93  7月  4 04:30 test01.hmatsu47.cloud
drwxr-xr-x. 2 root root 93  7月  4 04:04 test01.hmatsu47.com
drwxr-xr-x. 2 root root 93  7月  4 04:10 test01.hmatsu47.info
drwxr-xr-x. 2 root root 93  7月  4 04:43 test01.hmatsu47.jp.net
drwxr-xr-x. 2 root root 93  7月  4 04:08 test01.hmatsu47.net
drwxr-xr-x. 2 root root 93  7月  4 04:13 test01.hmatsu47.org
drwxr-xr-x. 2 root root 93  7月  4 02:54 test01.hmatsu47.site
drwxr-xr-x. 2 root root 93  7月  4 04:37 test01.hmatsu47.work
drwxr-xr-x. 2 root root 93  7月  4 04:32 test02.hmatsu47.biz
drwxr-xr-x. 2 root root 93  7月  4 04:30 test02.hmatsu47.cloud
（中略）
drwxr-xr-x. 2 root root 93  7月  4 04:09 test20.hmatsu47.net
drwxr-xr-x. 2 root root 93  7月  4 04:14 test20.hmatsu47.org
drwxr-xr-x. 2 root root 93  7月  4 02:57 test20.hmatsu47.site
drwxr-xr-x. 2 root root 93  7月  4 04:39 test20.hmatsu47.work

# ls -l /etc/letsencrypt/live/test01.hmatsu47.site/
合計 4
-rw-r--r--. 1 root root 682  7月  4 02:40 README
lrwxrwxrwx. 1 root root  44  7月  4 02:54 cert.pem -> ../../archive/test01.hmatsu47.site/cert3.pem
lrwxrwxrwx. 1 root root  45  7月  4 02:54 chain.pem -> ../../archive/test01.hmatsu47.site/chain3.pem
lrwxrwxrwx. 1 root root  49  7月  4 02:54 fullchain.pem -> ../../archive/test01.hmatsu47.site/fullchain3.pem
lrwxrwxrwx. 1 root root  47  7月  4 02:54 privkey.pem -> ../../archive/test01.hmatsu47.site/privkey3.pem
```

※`/etc/letsencrypt/live/`・`/etc/letsencrypt/archive/`の Permission を 755 にしておく（テストのため／本番利用では要検討）。

### nginx（OpenResty）設定

あらかじめ[先の記事](https://qiita.com/hmatsu47/items/54bfcd599546a5988bd2)の通り、`openssl`を使って`certs/ticket.key`ファイルを作成しておく。
`nginx.conf`は入れ替えて動的／静的テストを行う。

 * 動的

```bash:設定／動的
# cat nginx.conf

#user  nobody;
worker_processes auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  5000;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   120;
    types_hash_max_size 2048;

    #gzip  on;

    # HTTPS server

    server {
        listen       443 ssl;
        listen       [::]:443 ssl;
        server_name  _;

        ssl_protocols TLSv1.1 TLSv1.2;
        ssl_certificate     /etc/letsencrypt/live/test01.hmatsu47.site/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test01.hmatsu47.site/privkey.pem;
        ssl_session_tickets on;
        ssl_session_ticket_key certs/ticket.key;
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout 5m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        ssl_certificate_by_lua_block {
            local ssl = require "ngx.ssl"

            local ok, err = ssl.clear_certs()
            if not ok then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local name, err = ssl.server_name()
            if not name then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local ok, err = ngx.re.match(name, "^([A-Za-z0-9][A-Za-z0-9%.%-]{1,251}[A-Za-z])$", "jo")
            if not ok then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local ssl_certificate = string.format("/etc/letsencrypt/live/%s/fullchain.pem", name)
            local ssl_certificate_key = string.format("/etc/letsencrypt/live/%s/privkey.pem", name)

            local file, err = io.open(ssl_certificate, "r")
            if not file then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end
            local pem_cert = file:read("*all")
            io.close(file)

            local cert, err = ssl.cert_pem_to_der(pem_cert)
            if not cert then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local ok, err = ssl.set_der_cert(cert)
            if not ok then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local file, err = io.open(ssl_certificate_key, "r")
            if not file then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end
            local pem_priv_key = file:read("*all")
            io.close(file)

            local priv_key, err = ssl.priv_key_pem_to_der(pem_priv_key)
            if not priv_key then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

            local ok, err = ssl.set_der_priv_key(priv_key)
            if not ok then
                ngx.log(ngx.ERR, err)
                return ngx.exit(ngx.ERROR)
            end

        }

        location /server-status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            deny all;
        }

        location / {

            access_by_lua_block {
                local headers = ngx.req.get_headers()
                local host = headers["Host"]
                local m, err = ngx.re.match(host, "(?<hostname>[^:]+)", "jo")
                if err then
                    ngx.log(ngx.ERR, err)
                    return
                end
            }

            proxy_redirect   off;
            proxy_set_header Host               $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-Host   $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;

            proxy_pass http://localhost:80;
        }
    }
}
```

 * 静的

```bash:設定／静的
# cat nginx.conf

#user  nobody;
worker_processes auto;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  5000;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  logs/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   120;
    types_hash_max_size 2048;

    #gzip  on;


    # HTTPS server

    server {
        listen       443 ssl;
        listen       [::]:443 ssl;
        server_name  test01.hmatsu47.site;

        ssl_protocols TLSv1.1 TLSv1.2;
        ssl_certificate     /etc/letsencrypt/live/test01.hmatsu47.site/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test01.hmatsu47.site/privkey.pem;
        ssl_session_tickets on;
        ssl_session_ticket_key certs/ticket.key;
        ssl_session_cache shared:SSL:1m;
        ssl_session_timeout 5m;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        location /server-status {
            stub_status on;
            access_log off;
            allow 127.0.0.1;
            deny all;
        }

        location / {

            proxy_redirect   off;
            proxy_set_header Host               $host;
            proxy_set_header X-Real-IP          $remote_addr;
            proxy_set_header X-Forwarded-Host   $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;

            proxy_pass http://localhost:80;
        }
    }
}
```

## クライアント（`curl`）側

### `curl`実行用スクリプトを用意

 * 動的

```bash:動的テスト用curl
$ cat curl_https_site.sh
#!/bin/sh
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
curl https://test02.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
curl https://test03.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
curl https://test04.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
curl https://test05.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
（中略）
curl https://test19.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt
curl https://test20.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_site.txt

（後略）
```

 * 静的

```bash:静的テスト用curl
$ cat curl_https_single_01.sh
#!/bin/sh
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
（中略）
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt
curl https://test01.hmatsu47.site/ -o /dev/null -w "%{time_starttransfer}\n" 2> /dev/null >> /home/hmatsu47/https_single_01.txt

（後略）
```

### `crontab`に登録して実行

 * 動的（9 ドメインの場合）

```bash:crontab
$ crontab -l
51 19 * * * /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh && /home/hmatsu47/curl_https_site.sh
51 19 * * * /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh && /home/hmatsu47/curl_https_com.sh
51 19 * * * /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh && /home/hmatsu47/curl_https_net.sh
51 19 * * * /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh && /home/hmatsu47/curl_https_info.sh
51 19 * * * /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh && /home/hmatsu47/curl_https_org.sh
51 19 * * * /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh && /home/hmatsu47/curl_https_cloud.sh
51 19 * * * /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh && /home/hmatsu47/curl_https_biz.sh
51 19 * * * /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh && /home/hmatsu47/curl_https_work.sh
51 19 * * * /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh && /home/hmatsu47/curl_https_jp_net.sh
```

 * 静的（9 ドメインの場合）

```bash:crontab
$ crontab -l
56 19 * * * /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh && /home/hmatsu47/curl_https_single_01.sh
56 19 * * * /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh && /home/hmatsu47/curl_https_single_02.sh
56 19 * * * /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh && /home/hmatsu47/curl_https_single_03.sh
56 19 * * * /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh && /home/hmatsu47/curl_https_single_04.sh
56 19 * * * /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh && /home/hmatsu47/curl_https_single_05.sh
56 19 * * * /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh && /home/hmatsu47/curl_https_single_06.sh
56 19 * * * /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh && /home/hmatsu47/curl_https_single_07.sh
56 19 * * * /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh && /home/hmatsu47/curl_https_single_08.sh
56 19 * * * /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh && /home/hmatsu47/curl_https_single_09.sh
```