GitLab Helm ChartをAzure AKSにインストールする

GitLab v11.0.0からCloud Native GitLab Helm ChartがGitLab Helm Chartに改名され、Beta版になりました。

- https://docs.gitlab.com/ee/install/kubernetes/gitlab_chart.html

AKSへのインストール公式手順はないので頑張って作ってみました。

![Raccoon-4k.jpg](https://qiita-image-store.s3.amazonaws.com/0/260796/44876442-8705-fb91-07d2-403f8dd5f6bc.jpeg)

# 事前準備

Mac OSで作業しています。事前に以下を揃えておく必要があります。

* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-with-homebrew-on-macos)
* [Helm](https://docs.helm.sh/using_helm/#from-homebrew-macos)
* [Azure CLI](https://docs.microsoft.com/ja-jp/cli/azure/install-azure-cli?view=azure-cli-latest)

# AKSクラスタの構築

CLIだけで[成功した試し](https://media.giphy.com/media/1hiVNxD34TpC0/giphy.gif)はないのでAzure Web Portalで作ります。

CLIで作るならこのようなコマンドがきっと役に立ちます：

* `az account list-locations`
* `az group create --name gitlab-chart-group --location eastus`
* `az aks create -g gitlab-chart-group -n bakeneco -l eastus --enable-rbac --node-count 3 --generate-ssh-keys -k 1.9.6`
* `az aks delete -g gitlab-chart-group -n bakeneco`


その後Azure CLIでログインし、クラスタに接続できるようにします。

```shell-session
> az login
Note, we have launched a browser for you to login. For old experience with device code, use "az login --use-device-code"
You have logged in. Now let us find all subscriptions you have access to...
[
  {
    "cloudName": "AzureCloud",
    "id": "nyan",
    "isDefault": true,
    "name": "Microsoft Azure \u30a8\u30f3\u30bf\u30fc\u30d7\u30e9\u30a4\u30ba",
    "state": "Enabled",
    "tenantId": "nyan",
    "user": {
      "name": "nyan@nyan.com",
      "type": "user"
    }
  }
]

> az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
Merged "bakeneco" as current context in /Users/jb/.kube/config

> kubectl get nodes
NAME                       STATUS    ROLES     AGE       VERSION
aks-agentpool-15572265-0   Ready     agent     13m       v1.9.6
aks-agentpool-15572265-1   Ready     agent     13m       v1.9.6

> kubectl get services --all-namespaces
NAMESPACE     NAME                                                  TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                      AGE
default       kubernetes                                            ClusterIP      10.0.0.1       <none>          443/TCP                      13m
kube-system   addon-http-application-routing-default-http-backend   ClusterIP      10.0.169.174   <none>          80/TCP                       12m
kube-system   addon-http-application-routing-nginx-ingress          LoadBalancer   10.0.156.43    168.62.41.195   80:31997/TCP,443:31037/TCP   12m
kube-system   heapster                                              ClusterIP      10.0.76.188    <none>          80/TCP                       12m
kube-system   kube-dns                                              ClusterIP      10.0.0.10      <none>          53/UDP,53/TCP                12m
kube-system   kubernetes-dashboard                                  ClusterIP      10.0.31.110    <none>          80/TCP                       12m

> kubectl get deployments --all-namespaces
NAMESPACE     NAME                                                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   addon-http-application-routing-default-http-backend       1         1         1            1           13m
kube-system   addon-http-application-routing-external-dns               1         1         1            1           13m
kube-system   addon-http-application-routing-nginx-ingress-controller   1         1         1            1           13m
kube-system   azureproxy                                                1         1         1            1           13m
kube-system   heapster                                                  1         1         1            1           13m
kube-system   kube-dns-v20                                              2         2         2            2           13m
kube-system   kubernetes-dashboard                                      1         1         1            1           13m
kube-system   tunnelfront                                               1         1         1            1           13m

> kubectl get pods --all-namespaces
NAMESPACE     NAME                                                              READY     STATUS    RESTARTS   AGE
kube-system   addon-http-application-routing-default-http-backend-66c97fvmg46   1/1       Running   0          13m
kube-system   addon-http-application-routing-external-dns-6c65ddc45d-nlf4k      1/1       Running   0          13m
kube-system   addon-http-application-routing-nginx-ingress-controller-64hzwq8   1/1       Running   0          13m
kube-system   azureproxy-7c677567f6-hlgdl                                       1/1       Running   0          13m
kube-system   heapster-55f855b47-pqf57                                          2/2       Running   0          5m
kube-system   kube-dns-v20-7c556f89c5-mgm9w                                     3/3       Running   0          13m
kube-system   kube-dns-v20-7c556f89c5-xkwrm                                     3/3       Running   0          13m
kube-system   kube-proxy-6pjzz                                                  1/1       Running   0          13m
kube-system   kube-proxy-fzhc6                                                  1/1       Running   0          13m
kube-system   kube-svc-redirect-4r7jm                                           1/1       Running   2          13m
kube-system   kube-svc-redirect-wdcbs                                           1/1       Running   1          13m
kube-system   kubernetes-dashboard-b85c46fc-6sznr                               1/1       Running   5          13m
kube-system   tunnelfront-7b685bd4c5-fkgwh                                      1/1       Running   0          13m
```

# クラスタ環境の設定

### [ネットワーク設定](https://docs.gitlab.com/ee/install/kubernetes/preparation/networking.html)

~~最初からExternal IPがあるので、DNSレコードを登録します。~~


`addon-http-application-routing-nginx-ingress`が付与しているExternal IPとは別のものがインストール時に用意されますのでインストール実行後それを確認しDNS登録する必要があります。

### [ストレージ設定](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md)

AzureではデフォルトのStorageClassものが提供されています。今回それをそのまま使います。

### [Tiller](https://docs.gitlab.com/ee/install/kubernetes/preparation/tiller.html)

> Some clusters require authentication to use kubectl to create the Tiller roles.

Tiller用のClusterRoleとServiceAccountを作成しcluster-adminロールを付与します。

```yaml:cluster-admin-role.yaml 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: cluster-admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'
```

```shell-session
> kubectl create -f cluster-admin-role.yaml 
clusterrole.rbac.authorization.k8s.io "cluster-admin" created
```


```yaml:rbac-config.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
```

```shell-session
> kubectl create -f rbac-config.yaml
serviceaccount "tiller" created
clusterrolebinding.rbac.authorization.k8s.io "tiller" created
```

そしてTillerをインストールし初期化します。

```shell-session
> helm init --service-account tiller
$HELM_HOME has been configured at /Users/jb/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
```

# GitLab Chartインストール

GitLab chart repositoryを追加します。

```shell-session
> helm repo add gitlab https://charts.gitlab.io/
"gitlab" has been added to your repositories
```

HelmでGitLab Chartをインストールします。

```shell-session
> helm upgrade --install gitlab gitlab/gitlab \
                                   --timeout 600 \
                                   --set global.hosts.domain=bakeneco.io \
                                   --set gitlab.migrations.initialRootPassword="nyan" \
                                   --set gitlab.gitaly.persistence.storageClass=default \
                                   --set postgresql.persistence.storageClass=default \
                                   --set gitlab.redis.persistence.storageClass=default \
                                   --set gitlab.minio.persistence.storageClass=default \
                                   --set certmanager-issuer.email=nyan@nyan.com
Release "gitlab" does not exist. Installing it now.
NAME:   gitlab
LAST DEPLOYED: Sat Jul 21 18:38:43 2018
NAMESPACE: default
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/PodDisruptionBudget
NAME                                  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS  AGE
gitlab-gitaly                         N/A            1                0                    8s
gitlab-gitlab-shell                   N/A            1                0                    8s
gitlab-sidekiq                        N/A            1                0                    7s
gitlab-unicorn                        N/A            1                0                    7s
gitlab-minio-v1                       N/A            1                0                    7s
gitlab-nginx-ingress-controller       2              N/A              0                    7s
gitlab-nginx-ingress-default-backend  1              N/A              0                    7s
gitlab-redis-v1                       N/A            1                0                    7s
gitlab-registry-v1                    N/A            1                0                    7s

==> v1/ServiceAccount
NAME                                  SECRETS  AGE
gitlab-certmanager-issuer             1        10s
certmanager-gitlab                    1        10s
gitlab-gitlab-runner                  1        10s
gitlab-nginx-ingress                  1        10s
gitlab-prometheus-alertmanager        1        10s
gitlab-prometheus-kube-state-metrics  1        10s
gitlab-prometheus-node-exporter       1        10s
gitlab-prometheus-server              1        10s

==> v1/Role
NAME                       AGE
gitlab-certmanager-issuer  9s
gitlab-nginx-ingress       9s

==> v1/Job
NAME                           DESIRED  SUCCESSFUL  AGE
gitlab-issuer.1                1        0           8s
gitlab-migrations.1            1        0           8s
gitlab-minio-create-buckets.1  1        0           8s

==> v1/RoleBinding
NAME                       AGE
gitlab-certmanager-issuer  9s
gitlab-nginx-ingress       9s

==> v1beta2/StatefulSet
NAME           DESIRED  CURRENT  AGE
gitlab-gitaly  1        1        8s

==> v1/PersistentVolumeClaim
NAME                      STATUS   VOLUME   CAPACITY  ACCESS MODES  STORAGECLASS  AGE
gitlab-minio              Pending  default  10s
gitlab-postgresql         Pending  default  10s
gitlab-prometheus-server  Pending  default  10s
gitlab-redis              Pending  default  10s

==> v1beta1/CustomResourceDefinition
NAME                               AGE
certificates.certmanager.k8s.io    10s
clusterissuers.certmanager.k8s.io  10s
issuers.certmanager.k8s.io         9s

==> v1beta1/Role
gitlab-gitlab-runner  9s

==> v1beta1/Deployment
NAME                      DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
certmanager-gitlab        1        1        1           0          9s
gitlab-gitlab-runner      1        1        1           0          9s
gitlab-postgresql         1        1        1           0          8s
gitlab-prometheus-server  1        1        1           0          8s

==> v1beta1/Ingress
NAME             HOSTS                 ADDRESS  PORTS  AGE
gitlab-unicorn   gitlab.bakeneco.io    80, 443  8s
gitlab-minio     minio.bakeneco.io     80, 443  8s
gitlab-registry  registry.bakeneco.io  80, 443  8s

==> v2beta1/HorizontalPodAutoscaler
NAME                     REFERENCE                           TARGETS        MINPODS  MAXPODS  REPLICAS  AGE
gitlab-gitlab-shell      Deployment/gitlab-gitlab-shell      <unknown>/75%  2        10       0         8s
gitlab-sidekiq-all-in-1  Deployment/gitlab-sidekiq-all-in-1  <unknown>/75%  1        10       0         8s
gitlab-unicorn           Deployment/gitlab-unicorn           <unknown>/75%  2        10       0         8s
gitlab-registry          Deployment/gitlab-registry          <unknown>/75%  2        10       0         8s

==> v1/Pod(related)
NAME                                                  READY  STATUS             RESTARTS  AGE
certmanager-gitlab-56f5486fdb-f6dpb                   0/2    ContainerCreating  0         9s
gitlab-gitlab-runner-84dff6795b-ffjg6                 0/1    Init:0/1           0         9s
gitlab-gitlab-shell-59c74669b-8fjwg                   0/1    Init:0/1           0         9s
gitlab-sidekiq-all-in-1-f9c56885f-wct2v               0/1    Init:0/2           0         9s
gitlab-task-runner-69f766f6dc-l4tq4                   0/1    Pending            0         9s
gitlab-unicorn-59ccf4d574-m56z4                       0/1    Pending            0         8s
gitlab-minio-567c4bd69d-zrp8m                         0/1    Pending            0         8s
gitlab-nginx-ingress-controller-754587444-7lxn7       0/1    Pending            0         8s
gitlab-nginx-ingress-controller-754587444-hgjwk       0/1    Error              0         8s
gitlab-nginx-ingress-controller-754587444-qx86s       0/1    Pending            0         8s
gitlab-nginx-ingress-default-backend-c955db9c6-plvvw  1/1    Running            0         8s
gitlab-nginx-ingress-default-backend-c955db9c6-qx5jh  0/1    Pending            0         8s
gitlab-postgresql-57c66b8d6b-2sxb6                    0/2    Pending            0         8s
gitlab-prometheus-server-8cf4fdd8-2m4dk               0/2    Pending            0         8s
gitlab-redis-75764d5485-9bcpg                         0/2    Pending            0         8s
gitlab-registry-5d797444d7-fnpb9                      0/1    Pending            0         7s
gitlab-gitaly-0                                       0/1    Pending            0         8s
gitlab-issuer.1-tvnpc                                 0/1    ContainerCreating  0         8s
gitlab-migrations.1-25sv6                             0/1    Pending            0         8s
gitlab-minio-create-buckets.1-5lz7p                   0/1    Pending            0         8s

==> v1/ConfigMap
NAME                                   DATA  AGE
gitlab-certmanager-issuer-certmanager  2     10s
gitlab-gitlab-runner                   3     10s
gitlab-gitaly                          3     10s
gitlab-gitlab-shell                    2     10s
gitlab-nginx-ingress-tcp               1     10s
gitlab-migrations                      4     10s
gitlab-sidekiq-all-in-1                1     10s
gitlab-sidekiq                         5     10s
gitlab-task-runner                     4     10s
gitlab-unicorn                         8     10s
gitlab-unicorn-tests                   1     10s
gitlab-minio-config-cm                 3     10s
gitlab-nginx-ingress-controller        7     10s
gitlab-postgresql                      0     10s
gitlab-prometheus-server               3     10s
gitlab-redis                           2     10s
gitlab-registry                        2     10s

==> v1beta1/ClusterRole
NAME                                  AGE
certmanager-gitlab                    9s
gitlab-prometheus-kube-state-metrics  9s
gitlab-prometheus-server              9s

==> v1beta1/RoleBinding
NAME                  AGE
gitlab-gitlab-runner  9s

==> v1beta1/ClusterRoleBinding
NAME                                  AGE
certmanager-gitlab                    9s
gitlab-prometheus-alertmanager        9s
gitlab-prometheus-kube-state-metrics  9s
gitlab-prometheus-node-exporter       9s
gitlab-prometheus-server              9s

==> v1/Service
NAME                                  TYPE          CLUSTER-IP    EXTERNAL-IP  PORT(S)                                  AGE
gitlab-gitaly                         ClusterIP     None          <none>       8075/TCP,9236/TCP                        9s
gitlab-gitlab-shell                   ClusterIP     10.0.216.145  <none>       22/TCP                                   9s
gitlab-unicorn                        ClusterIP     10.0.226.44   <none>       8080/TCP,8181/TCP                        9s
gitlab-minio-svc                      ClusterIP     10.0.113.89   <none>       9000/TCP                                 9s
gitlab-nginx-ingress-controller       LoadBalancer  10.0.65.171   <pending>    80:32670/TCP,443:31479/TCP,22:31190/TCP  9s
gitlab-nginx-ingress-default-backend  ClusterIP     10.0.104.154  <none>       80/TCP                                   9s
gitlab-postgresql                     ClusterIP     10.0.238.193  <none>       5432/TCP                                 9s
gitlab-prometheus-server              ClusterIP     10.0.222.23   <none>       80/TCP                                   9s
gitlab-redis                          ClusterIP     10.0.148.79   <none>       6379/TCP,9121/TCP                        9s
gitlab-registry                       ClusterIP     10.0.49.168   <none>       5000/TCP                                 9s

==> v1beta2/Deployment
NAME                                  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
gitlab-gitlab-shell                   1        1        1           0          9s
gitlab-sidekiq-all-in-1               1        1        1           0          9s
gitlab-task-runner                    1        1        1           0          9s
gitlab-unicorn                        1        1        1           0          9s
gitlab-minio                          1        1        1           0          8s
gitlab-nginx-ingress-controller       3        3        3           0          8s
gitlab-nginx-ingress-default-backend  2        2        2           1          8s
gitlab-redis                          1        1        1           0          8s
gitlab-registry                       1        1        1           0          8s

```

IngressのExternal IPを確認しDNSレコードに登録します。

```shell-session
> kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
LoadBalancer Ingress:     23.101.134.115
```

しばらくするとすべてのPodsが立ち上がってログイン画面が見れます。

パスワードを取得してログインします。

```shell-session
> kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
nyan
```

<img width="1454" alt="Screen Shot 2018-07-22 at 13.51.31.png" src="https://qiita-image-store.s3.amazonaws.com/0/260796/b69ddf89-d602-4f2f-7d66-9c6c941b3c05.png">


---

# 手順まとめ

```shell-session
az login
az aks get-credentials --resource-group gitlab-chart-group --name bakeneco
kubectl get nodes
kubectl get pods --all-namespaces
kubectl create -f cluster-admin-role.yaml
kubectl create -f rbac-config.yaml
helm init --service-account tiller
helm repo add gitlab https://charts.gitlab.io/
helm upgrade --install gitlab gitlab/gitlab \
                                   --timeout 600 \
                                   --set global.hosts.domain=bakeneco.io \
                                   --set gitlab.migrations.initialRootPassword="nyan" \
                                   --set gitlab.gitaly.persistence.storageClass=default \
                                   --set postgresql.persistence.storageClass=default \
                                   --set gitlab.redis.persistence.storageClass=default \
                                   --set gitlab.minio.persistence.storageClass=default \
                                   --set certmanager-issuer.email=nyan@nyan.com
kubectl describe service gitlab-nginx-ingress-controller | grep Ingress
kubectl get secret gitlab-gitlab-initial-root-password -ojsonpath='{.data.password}' | base64 --decode
```

---

# おまけ：派手に失敗したazure-file StorageClassの採用

あ〜ストレージ、本当に[楽しい](https://media.giphy.com/media/10UHehEC098kAE/giphy.gif)です。
AzureではデフォルトのStorageClassものが提供されていますがAzure-Diskですのであまりうれしくないです。

AzureのStorageClassについて[良い記事](https://pascalnaber.wordpress.com/2018/01/26/persistent-storage-and-volumes-using-kubernetes-on-azure-with-aks-or-azure-container-service/)があったので参考にしました。

まずクラスタのノードと同じリソースグループ（クラスタサービスのグループではない！）にStorage Accountを作ります。

```shell-session
> az storage account create --resource-group MC_gitlab-chart-group_bakeneco_eastus --name gitlabchartstorage  
{
  "accessTier": null,
  "creationTime": "2018-07-21T07:18:12.543726+00:00",
  "customDomain": null,
  "enableHttpsTrafficOnly": false,
  "encryption": {
    "keySource": "Microsoft.Storage",
    "keyVaultProperties": null,
    "services": {
      "blob": {
        "enabled": true,
        "lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
      },
      "file": {
        "enabled": true,
        "lastEnabledTime": "2018-07-21T07:18:12.762473+00:00"
      },
      "queue": null,
      "table": null
    }
  },
  "id": "/subscriptions/cc07cdf5-785f-45be-9718-2283b78ab03a/resourceGroups/gitlab-chart-group/providers/Microsoft.Storage/storageAccounts/gitlabchartstorage",
  "identity": null,
  "kind": "Storage",
  "lastGeoFailoverTime": null,
  "location": "eastus",
  "name": "gitlabchartstorage",
  "networkRuleSet": {
    "bypass": "AzureServices",
    "defaultAction": "Allow",
    "ipRules": [],
    "virtualNetworkRules": []
  },
  "primaryEndpoints": {
    "blob": "https://gitlabchartstorage.blob.core.windows.net/",
    "file": "https://gitlabchartstorage.file.core.windows.net/",
    "queue": "https://gitlabchartstorage.queue.core.windows.net/",
    "table": "https://gitlabchartstorage.table.core.windows.net/"
  },
  "primaryLocation": "eastus",
  "provisioningState": "Succeeded",
  "resourceGroup": "gitlab-chart-group",
  "secondaryEndpoints": {
    "blob": "https://gitlabchartstorage-secondary.blob.core.windows.net/",
    "file": null,
    "queue": "https://gitlabchartstorage-secondary.queue.core.windows.net/",
    "table": "https://gitlabchartstorage-secondary.table.core.windows.net/"
  },
  "secondaryLocation": "westus",
  "sku": {
    "capabilities": null,
    "kind": null,
    "locations": null,
    "name": "Standard_RAGRS",
    "resourceType": null,
    "restrictions": null,
    "tier": "Standard"
  },
  "statusOfPrimary": "available",
  "statusOfSecondary": "available",
  "tags": {},
  "type": "Microsoft.Storage/storageAccounts"
}
```

それからStorageClassを作成してDefaultとします。

```yaml:aks-storage-class.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefilestorage
provisioner: kubernetes.io/azure-file
parameters:
  storageAccount: gitlabchartstorage
reclaimPolicy: Retain
```

```
> kubectl create -f aks-storage-class.yaml 
storageclass.storage.k8s.io "azurefilestorage" created

> kubectl patch storageclass azurefilestorage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io "azurefilestorage" patched

> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted

> kubectl get storageclass
NAME                         PROVISIONER                AGE
azurefilestorage (default)   kubernetes.io/azure-file   43s
managed-premium              kubernetes.io/azure-disk   40m
```

2つのStorageClassが存在すると[厄介なこと](https://giphy.com/gifs/fail-epic-absurd-12WjSnRGVC79EA/fullscreen)が起きるので`default`のStorageClassを削除してましたが、[勝手に復活](https://github.com/kubernetes/kubernetes/issues/53428)してしまいます。


HelmでGitLab Chartをインストールする直前に急いで`default`のStorageClassを削除してしまえば復活しないうちにインストールが始まります。

```shell-session
> kubectl delete sc default
storageclass.storage.k8s.io "default" deleted
```

結果的にはうまくいかず断念しました。Persistent Volume ClaimがちゃんとできましたがminioとpostgresqlがVolumeに対して正しいパーミッションがなかったようです。


```shell-session
> kubectl logs gitlab-minio-567c4bd69d-6cf5s
time="2018-07-21T09:01:18Z" level=error msg="Initializing object layer failed" cause="Unable to initialize '.minio.sys' meta volume, mkdir /export/.minio.sys: permission denied" source="[server-main.go:214:serverMain()]"

> kubectl logs gitlab-postgresql-57c66b8d6b-b7tsh gitlab-postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

initdb: could not change permissions of directory "/var/lib/postgresql/data/pgdata": Operation not permitted
fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... 
```

どうにかできるはずですが、今回諦めました。。