Windows Server 2019でHyper-V上のUbuntu18.04にJupyterHub環境を構築

タイトル通り，Windows Server 2019のHyper-Vで動かすUbuntu18.04にJupyterHub環境を構築した際の備忘録です．
[Windows Server 2019でWSLやHyper-V上のUbuntu18.04にCUDA環境を構築しようとして断念](https://qiita.com/fararrow/items/5f9eb1e4386707c11b3c)
の，Hyper-V部分(Python環境構築)です．

Hyper-VでUbuntu18.04を動かすまでは
[Windows Server 2019でHyper-V上にUbuntu18.04をインストール](https://qiita.com/fararrow/items/0bdc72ac4f80b4cb35c0)
↑こちら

# はじめに

本来はDDA(Discrete Device Assignment)でHyper-V上のLinuxへGPU passthroughしたかったのですが，
[Windows Server 2019でHyper-V上のUbuntu18.04にCUDA環境を構築(断念)](https://qiita.com/fararrow/items/8df089dd17d1699b299c)
にある通り，無理でした．

まぁ何かしらアップデートされればいける気がするので，[WSL](https://qiita.com/fararrow/items/483f0f5a9d5092adead5)同様にPython環境を構築していきます．
(同様にというかほぼ同じ手順です，一応どちらもUbuntu18.04なので当然ですが)


# Anaconda3

[WSL](https://qiita.com/fararrow/items/483f0f5a9d5092adead5)ではせっかくGUI環境を作ったのでGUIで https://www.anaconda.com/ から落としましたが，今回は逆にCUIでいきます．

## Anaconda3のインストール

2019年6月半ば時点では「2019.03」な「Anaconda3-2019.03-Linux-x86_64.sh」が最新なので，rootさんで入れる前提だと

```shell-session
$ cd ~/Downloads/
$ wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh
$ sudo su
# bash Anaconda3-2019.03-Linux-x86_64.sh

Welcome to Anaconda3 2019.03

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>> 
```

Enterを押してLicense確認して

```shell-session
Please answer 'yes' or 'no':'
>>> yes
```

インストール先を聞かれるので

```shell-session
Anaconda3 will now be installed into this location:
/root/anaconda3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/root/anaconda3] >>> /usr/local/lib/anaconda3
```

として，

```shell-session
PREFIX=/usr/local/lib/anaconda3
installing: python-3.7.3-h0371630_0 ...
Python 3.7.3
installing: xxx ...
```

そこそこ待つと

```shell-session
installation finished.
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>>
```

と聞かれる．
[WSL](https://qiita.com/fararrow/items/483f0f5a9d5092adead5)でも書きましたが，公式の[Frequently asked questions](https://docs.anaconda.com/anaconda/user-guide/faq/)に「We do not recommend adding Anaconda to the PATH manually.」とあるのにdefaultはno(笑)
今回もrootで入れているのでnoを選び，
<details><summary>PyCharmをオススメされてインストール完了．</summary><div>

```shell-session
[no] >>> no

You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:

eval "$(/usr/local/lib/anaconda3/bin/conda shell.YOUR_SHELL_NAME hook)" 

To install conda's shell functions for easier access, first activate, then:

conda init

If you'd prefer that conda's base environment not be activated on startup, 
   set the auto_activate_base parameter to false: 

conda config --set auto_activate_base false

Thank you for installing Anaconda3!

===========================================================================

Anaconda and JetBrains are working together to bring you Anaconda-powered
environments tightly integrated in the PyCharm IDE.

PyCharm for Anaconda is available at:
https://www.anaconda.com/pycharm
```
</div></details>

## パスを通す

WSLの時と同様に
<details><summary>``profile.d``へ通す作業…</summary><div>

>```shell-session
# echo 'export PATH="/usr/local/lib/anaconda3/bin:$PATH"' >> /etc/profile.d/anaconda.sh
```
>
とし，さらにrootがprofile.dを読んでくれるよう
>
```shell-session
# sudo visudo
```
>
で
>
```shell
#Defaults       env_reset
Defaults        env_keep += "PATH"
Defaults        mail_badpass
#Defaults       secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
```
</div></details>
…と思ったけど，この手順だとanacondaのパスが複数回追加されることに気付いたし，セキュリティ上アレ．
今回は他に色々入れるわけでもないし，これでは/usr/sbinが読みに行けなくなるし，
<details><summary>``/etc/sudoers.d/``推奨</summary><div>

```shell
Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
```
</div></details>

とあるので，以下のように設定した．

### 通常のパス

↑のechoでも良いけど，複数回追加されるのをどうしても回避できなかったので，Anacondaさんが標準でinitializeする際に追加してくれるスクリプトを書くことにした．

```shell-session
$ sudo vi /etc/profile.d/anaconda.sh
```

```bash:/etc/profile.d/anaconda.sh
#export PATH="/usr/local/lib/anaconda3/bin:$PATH"

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/lib/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/usr/local/lib/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/usr/local/lib/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/usr/local/lib/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
```

最初の1行はただの元のechoのexportの残骸です．

### sudoでのパス
前述の通り``/etc/sudoers``は触らずに``/etc/sudoers.d/``へ

```shell-session
$ sudo visudo -f /etc/sudoers.d/anaconda3
```

などと作成し，``/etc/sudoers``の``secure_path``に``/usr/local/lib/anaconda3/bin``を追加したもので置き換えてあげる．

```bash:/etc/sudoers.d/anaconda3.tmp
Defaults       secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin:/usr/local/lib/anaconda3/bin"
```

とする．
(visudoだと.tmpを編集して構文チェックかけてから本体に保存するんですね．)
ちなみに``/etc/sudoers``のsecure_pathに+=的な事が出来ないかと調べましたが，無理そうです．

### rootでのパス

↑でsudoはできるけど``sudo su``で通らないので.bashrcに追記．

```shell-session
$ sudo su
# cat /etc/profile.d/anaconda.sh >> ~/.bashrc
```

## パスが通ったか確認

ここまで来たらログインし直して

```shell-session
$ conda --version
conda 4.6.11
$ sudo conda --version
conda 4.6.11
$ sudo su
# conda --version
conda 4.6.14
```

こんなノリでcondaが使える事を確認．

- - -
ちなみに一人で使うだけだったりユーザが各自で入れるのであればyesにして
<details><summary>``modified /home/(user_name)/.bashrc``</summary><div>

```shell-session
[no] >>> yes
WARNING: The conda.compat module is deprecated and will be removed in a future release.
no change     /usr/local/lib/anaconda3/condabin/conda
no change     /usr/local/lib/anaconda3/bin/conda
no change     /usr/local/lib/anaconda3/bin/conda-env
no change     /usr/local/lib/anaconda3/bin/activate
no change     /usr/local/lib/anaconda3/bin/deactivate
no change     /usr/local/lib/anaconda3/etc/profile.d/conda.sh
no change     /usr/local/lib/anaconda3/etc/fish/conf.d/conda.fish
no change     /usr/local/lib/anaconda3/shell/condabin/Conda.psm1
no change     /usr/local/lib/anaconda3/shell/condabin/conda-hook.ps1
no change     /usr/local/lib/anaconda3/lib/python3.7/site-packages/xonsh/conda.xsh
no change     /usr/local/lib/anaconda3/etc/profile.d/conda.csh
modified      /home/(user_name)/.bashrc

==> For changes to take effect, close and re-open your current shell. <==

If you'd prefer that conda's base environment not be activated on startup, 
   set the auto_activate_base parameter to false: 

conda config --set auto_activate_base false

Thank you for installing Anaconda3!

===========================================================================

Anaconda and JetBrains are working together to bring you Anaconda-powered
environments tightly integrated in the PyCharm IDE.

PyCharm for Anaconda is available at:
https://www.anaconda.com/pycharm
```
</div></details>
こんな感じで.bashrcをmodifiedしてもらった方が楽．
この場合は``/usr/local/lib/``なので変な事になるけど，defaultの``/home/(user_name)/anaconda3``のようにユーザの配下で．
なお，

```shell
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/usr/local/lib/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/usr/local/lib/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/usr/local/lib/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/usr/local/lib/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
```
こんな感じの文言が追加される．
(はい，以上が寝ぼけてsudo bashした上にyesにしてしまった際のログです，結果的に↑でパス通す時に役立った笑)
そして，パスが2回追加される現象がどうしても回避できなくて調べている間に
[zsh（やshやbash）でパス系環境変数の重複を除去する](https://daily.belltail.jp/?p=1853)
優秀な後輩くんの記事とか発見した笑


# JupyterHub環境の構築

WindowsでJupyterHubを使いたくて始めたことなのでここが本番．

## JupyterHubのインストール

```shell-session
$ sudo conda install jupyterhub
```
途中で``conda 4.6.11-py37_0 --> 4.6.14-py37_0``とcondaのバージョン上げるねって言われたりしつつ，インストール完了後

```shell-session
$ sudo jupyterhub --no-ssl
```

と叩き，Firefoxとかで http://localhost:8000/ にアクセスして
![jupyterhub_login.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/438132/e45ae569-af06-619f-8530-b5e9f274057b.png)
こんなログイン画面が出ればとりあえずOK．
一応ログインもしてみると，``http://localhost:8000/user/(user_name)/tree``というURLになり，ログインしたユーザのhomeが見えるはず．


## JupyterHubの設定

homeからスタートするのは微妙な気がするので，

```shell-session
$ cd /usr/local/lib/anaconda3
$ sudo jupyterhub --generate-config
Writing default config to: jupyterhub_config.py
```

でdefaultのjupyterhub_config.pyを作り，

```shell-session
$ sudo vi jupyterhub_config.py
```

で編集．

```python:jupyterhub_config.py
## Path to the notebook directory for the single-user server.
#
#  The user sees a file listing of this directory when the notebook interface is
#  started. The current interface does not easily allow browsing beyond the
#  subdirectories in this directory's tree.
#
#  `~` will be expanded to the home directory of the user, and {username} will be
#  replaced with the name of the user.
#
#  Note that this does *not* prevent users from accessing files outside of this
#  path! They can do so with many other means.
#c.Spawner.notebook_dir = ''
```

辺りを探して

```python:jupyterhub_config.py
c.Spawner.notebook_dir = '~/notebooks'
```

とする．
当然，普通に``adduser``しただけではnotebooksなんてディレクトリは作られないので，

```shell-session
$ sudo mkdir /etc/skel/notebooks
```

で/etc/skel/にnotebooksディレクトリを用意しておく．
既存ユーザにも無いので，

```shell-session
$ mkdir ~/notebooks
```

既にいっぱいユーザが居るなら…頑張って作って回るか，スクリプト書く．

さらに，いかにもな8000番ポートは嫌なので，

```python:jupyterhub_config.py
## The public facing port of the proxy.
# 
#  This is the port on which the proxy will listen. This is the only port through
#  which JupyterHub should be accessed by users.
# 
#  .. deprecated: 0.9
#      Use JupyterHub.bind_url
#c.JupyterHub.port = 8000
```

辺りを探して

```python:jupyterhub_config.py
c.JupyterHub.port = [お好きなポート番号]
```

とする．ちなみにWSLで同じ設定をしているので，同時に走らせる事はないだろうけれど別のポートにした．
さらに，テストでログイン画面出した際にWarningが出ていた通りhttpsで通信したいので，WSLのときと同様にSSLの設定を

```shell-session
$ cd /usr/local/lib/anaconda3
$ sudo su
# mkdir cert
# chmod 600 cert
# cd cert
# openssl req -x509 -nodes -newkey rsa:2048 -keyout secret.key > jupyter.pem
```

とりあえずCommon NameにIPかドメイン(orマシン)名，後は適当に埋めるか空欄で

```shell-session
Country Name (2 letter code) [AU]:JP
Common Name (e.g. server FQDN or YOUR name) []:xxx.xxx.xxx.xxx
```

jupyter.pemとsecret.keyを作成．

再度``jupyterhub_config.py``を編集して

```python:jupyterhub_config.py
c.JupyterHub.ssl_cert = '/usr/local/lib/anaconda3/cert/jupyter.pem'
c.JupyterHub.ssl_key = '/usr/local/lib/anaconda3/cert/secret.key'
```

と設定．
今度はssl有り起動で

```shell-session
$ (cd /usr/local/lib/anaconda3 && sudo jupyterhub)
```

https://localhost:[お好きなポート番号]/ に繋がればOK．
うまく出来ていれば
![jupyterhub_login_ssl_moz.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/438132/20319237-7726-01d7-ac8f-8281cb50f222.png)
のようにWarning無しのログイン画面になる．


## JupyterHubの自動起動

- [Run jupyterhub as a system service](https://github.com/jupyterhub/jupyterhub/wiki/Run-jupyterhub-as-a-system-service)
- [[メモ] らずぱい: JupyterHubでマルチユーザ対応](https://qiita.com/mt08/items/301f9fb93d01e78bda47)

辺りを参考に，``systemd``でJupyterHubが自動起動するように設定．
すんなりいかなかったけど，とりあえず以降の手順で動いた．


### 必要ライブラリのインストール

``nodejs``と``configurable-http-proxy``を入れる．

```shell-session
$ sudo apt install nodejs
$ sudo npm install -g configurable-http-proxy
```

``configurable-http-proxy``の方は

```shell-session
$ sudo npm install -g configurable-http-proxy
/usr/local/lib/anaconda3/bin/configurable-http-proxy -> /usr/local/lib/anaconda3/lib/node_modules/configurable-http-proxy/bin/configurable-http-proxy
- ms@2.1.1 node_modules/configurable-http-proxy/node_modules/logform/node_modules/ms
/usr/local/lib/anaconda3/lib
└─┬ configurable-http-proxy@4.1.0 
  ├─┬ http-proxy@1.17.0
  │ ├── eventemitter3@3.1.2 
  │ └─┬ follow-redirects@1.7.0 
  │   └── debug@3.2.6 
  └─┬ winston@3.2.1 
    ├── async@2.6.2 
    ├─┬ diagnostics@1.1.1
    │ └── colorspace@1.1.2 
    ├─┬ logform@2.1.2 
    │ ├── colors@1.3.3 
    │ └── ms@2.1.2 
    ├─┬ readable-stream@3.4.0 
    │ └── string_decoder@1.2.0 
    └─┬ winston-transport@4.3.0 
      └─┬ readable-stream@2.3.6 
        └── string_decoder@1.1.1 
```

こんな実行結果が出る．


### jupyterhub_config.pyの設定

続いて，

```shell-session
$ sudo vi /usr/local/lib/anaconda3/jupyterhub_config.py
```

で``jupyterhub_config.py``を再度編集し，
``configurable-http-proxy``と``jupyterhub-singleuser``にパスを通す．

↓Use ConfigurableHTTPProxy.commandって書いてあるから従っておく．

```Python:jupyterhub_config.py
## DEPRECATED since version 0.8. Use ConfigurableHTTPProxy.command
#c.JupyterHub.proxy_cmd = []
c.ConfigurableHTTPProxy.command = '/usr/local/lib/anaconda3/bin/configurable-http-proxy'
```

↓なんで通らないのかわかってないけどとりあえずフルパスを

```Python:jupyterhub_config.py
#  python environment (with virtualenv/conda) than JupyterHub itself.
#
#  Some spawners allow shell-style expansion here, allowing you to use
#  environment variables. Most, including the default, do not. Consult the
#  documentation for your spawner to verify!
#c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.cmd = ['/usr/local/lib/anaconda3/bin/jupyterhub-singleuser']
```


### サービスの登録

適当な名前.serviceを作成し，

```shell-session
$ sudo vi /lib/systemd/system/jupyterhub.service
```

root動作想定で入れたので

```shell-session:/lib/systemd/system/jupyterhub.service
[Unit]
Description=Jupyterhub

[Service]
User=root
ExecStart=/usr/local/lib/anaconda3/bin/jupyterhub
WorkingDirectory=/usr/local/lib/anaconda3

[Install]
WantedBy=multi-user.target
```

こんな感じに．
あとは

```shell-session
$ sudo systemctl enable jupyterhub
$ sudo systemctl start jupyterhub
$ sudo systemctl status jupyterhub
```

この辺りのコマンドで動作確認．
逐一statusを確認していくと

```shell-session
$ sudo systemctl status jupyterhub
● jupyterhub.service - Jupyterhub
   Loaded: loaded (/lib/systemd/system/jupyterhub.service; disabled; vendor pres
   Active: inactive (dead)
$ sudo systemctl enable jupyterhub
Created symlink /etc/systemd/system/multi-user.target.wants/jupyterhub.service → /lib/systemd/system/jupyterhub.service.
$ sudo systemctl status jupyterhub
● jupyterhub.service - Jupyterhub
   Loaded: loaded (/lib/systemd/system/jupyterhub.service; enabled; vendor prese
   Active: inactive (dead)
$ sudo systemctl start jupyterhub
$ sudo systemctl status jupyterhub
● jupyterhub.service - Jupyterhub
   Loaded: loaded (/lib/systemd/system/jupyterhub.service; enabled; vendor prese
   Active: active (running) since xxx ago
 Main PID: 61478 (jupyterhub)
    Tasks: 7 (limit: 19104)
   CGroup: /system.slice/jupyterhub.service
           ├─61478 /usr/local/lib/anaconda3/bin/python /usr/local/lib/anaconda3/
           └─61483 node /usr/local/lib/anaconda3/bin/configurable-http-proxy --i
```

こうなります．