OpenFoam on AWS Batch

2016年末にリリースされたAWSの新サービスBatchを使ってOpenFoamを実行してみました。
![openfoam-batch.png](https://qiita-image-store.s3.amazonaws.com/0/93823/1289b725-0081-8e9b-f37b-5bac1daf6785.png)

# 1.Build OpenFoam Docker Images
ubuntu14.04をベースにして、OpenFoam4.1とParaView5.0をインストールします。
その後AWSCLIをインストールします。OpenFoamで計算した結果をS3に保存するためです。
実際にOpenFoamを実行するShellスクリプトはGithubからダウンロードしてきます。

https://raw.githubusercontent.com/porcaro33/openfoam-docker/master/Dockerfile

```bash:Dockerfile
#OpenFOAM 4.1 and ParaView 5.0.1 on ubuntu 14.04
FROM ubuntu:trusty

MAINTAINER Hiroshi.Kobayashi

WORKDIR /root

RUN set -x && \

    # install openfoam
    sudo apt-get -y update && \
    sudo apt-get -y upgrade && \
    sudo apt-get -y install apt-file && \
    sudo apt-get -y install software-properties-common && \
    sudo add-apt-repository http://dl.openfoam.org/ubuntu && \
    sudo apt-get -y install wget && \
    sudo wget -O - http://dl.openfoam.org/gpg.key | apt-key add - && \
    sudo apt-get -y install apt-transport-https && \
    sudo apt-get -y update && \
    sudo apt-get -y install openfoam4 && \
    sudo echo "source /opt/openfoam4/etc/bashrc" >> /root/.bashrc && \

    # install awscli
    sudo curl -O https://bootstrap.pypa.io/get-pip.py && \
    sudo python3 get-pip.py --user && \
    export PATH=/root/.local/bin:$PATH && \
    /bin/bash -c "source /root/.bashrc" && \
    pip install awscli --upgrade --user

ENV AWS_DEFAULT_REGION=us-east-1 \
    AWS_DEFAULT_OUTPUT=json

ADD https://raw.githubusercontent.com/porcaro33/openfoam-docker/master/openfoam_run.sh /root
RUN sudo chmod +x /root/openfoam_run.sh
CMD ["/root/openfoam_run.sh"]
```

今回はopenfoam-dockerと名前をつけてbuildします。私のMacbook Airで１５分くらいかかりました。
```docker build -t openfoam-docker:latest .```

実際に計算するモデルはチュートリアルから持ってきました。 "incompressible/simpleFoam/pitzDaily"
![simpleFoam_pitzDaily_mesh.png](https://qiita-image-store.s3.amazonaws.com/0/93823/df9feda1-b83f-b798-1965-7c9323a006df.png) ![simpleFoam_pitzDaily_contor.png](https://qiita-image-store.s3.amazonaws.com/0/93823/cf10b50b-2b8a-4ce2-90a0-c08b40c1fec2.png)



# 2.Push Image to AWS ECR
ECRを作成する時に表示される手順にそって、コマンドを実行していくのみです。(ECR get-loginのオプションに --no-include-emailが追加されました)
1) Retrieve the docker login command that you can use to authenticate your Docker client to your registry:
```aws ecr get-login --no-include-email --region us-west-2```
2) Run the docker login command that was returned in the previous step.
3) After the build completes, tag your image so you can push the image to this repository:
```docker tag openfoam-docker:latest <account-id>.dkr.ecr.us-west-2.amazonaws.com/<ecr-name>:latest```
4) Run the following command to push this image to your newly created AWS repository:
```docker push <account-id>.dkr.ecr.us-west-2.amazonaws.com/<ecr-name>:latest```

![docker_push.png.png](https://qiita-image-store.s3.amazonaws.com/0/93823/b2cebafb-ac65-9144-63dc-9c966b0c342f.png)


# 3.Create AWS Batch Environment
ジョブ定義、コンピューテング環境作成、ジョブキュー作成の順番で進めていきます。
1) create job definition
```aws batch register-job-definition --cli-input-json file://<path>/job_definition.json```
https://raw.githubusercontent.com/porcaro33/openfoam-docker/master/job_definition.json

```json
{
    "jobDefinitionName": "openfoam-docker-job",
    "type": "container",
    "parameters": {},
    "containerProperties": {
        "image": "<account-id>.dkr.ecr.us-west-2.amazonaws.com/<ecr-name>",
        "vcpus": 1,
        "memory": 500,
        "command": [],
        "jobRoleArn": "arn:aws:iam::<account-id>:role/openfoam-docker-job-role",
        "volumes": [],
        "environment": [],
        "mountPoints": [],
        "ulimits": []
    },
    "retryStrategy": {
        "attempts": 3
    }
}
```

2) create computing environment
```aws batch create-compute-environment --cli-input-json file://<path>/computing_env.json```
https://raw.githubusercontent.com/porcaro33/openfoam-docker/master/computing_env.json

```json
{
  "computeEnvironmentName": "M3Spot",
  "type": "MANAGED",
  "state": "ENABLED",
  "computeResources": {
    "type": "SPOT",
    "spotIamFleetRole": "arn:aws:iam::<account-id>:role/aws-ec2-spot-fleet-role",
    "minvCpus": 1,
    "maxvCpus": 10,
    "desiredvCpus": 1,
    "instanceTypes": [
      "m3"
    ],
    "bidPercentage": 40,
    "subnets": [
      "subnet-XXXXXXXX",
      "subnet-XXXXXXXX",
      "subnet-XXXXXXXX",
      "subnet-XXXXXXXX"
    ],
    "securityGroupIds": [
      "sg-XXXXXXXX"
    ],
    "ec2KeyPair": "<your-key-pair>",
    "instanceRole": "<your-role>"
  },
  "serviceRole": "arn:aws:iam::<account-id>:role/service-role/AWSBatchServiceRole"
}
```

3) create job queue
```aws batch create-job-queue --cli-input-json file://<path>/job_queue.json```
https://raw.githubusercontent.com/porcaro33/openfoam-docker/master/job_queue.json

```json
{
  "jobQueueName": "openfoam-docker-queue",
  "state": "ENABLED",
  "priority": 1,
  "computeEnvironmentOrder": [
    {
      "order": 1,
      "computeEnvironment": "M3Spot"
    }
  ]
}
```

# 4.Submit Jobs
AWSCLIで同じジョブを回数指定して10秒おきにsubmitします。

```bash:bash
#!/bin/bash
QUEUE="openfoam-docker-queue"
JOBDEF="openfoam-docker-job"
MAXITR=10

for i in `seq 1 $MAXITR`
do
  JOBNAME="JOB_$i"
  aws batch submit-job --job-name $JOBNAME --job-queue $QUEUE --job-definition $JOBDEF
  echo "submitted $JOBNAME"
  sleep 10
done
```

実際にはDocker Imageに埋め込んであるshellスクリプトが実行されます。
流速の結果が格納されているU fileをS3に転送します。

```bash:bash
#!/bin/bash

# geting JOB ID, bucket name
BUCKETNAME="<bucket-name>"

# run OpenFoam4 with tutrial model "pitzDaily"
source /opt/openfoam4/etc/bashrc
source /root/.bashrc
export PATH=/root/.local/bin:$PATH

cd /root
mkdir -p $FOAM_RUN
cd $FOAM_RUN
cp -r $FOAM_TUTORIALS/incompressible/simpleFoam/pitzDaily .
cd pitzDaily
blockMesh
simpleFoam

# upload U to S3
TIMESTAMP=`date +%Y%m%d%H%M%S%N`
aws s3 cp ./298/U s3://$BUCKETNAME/$TIMESTAMP
```

# Check the result on S3
go to S3 and check the U files

# 参考資料
http://qiita.com/pottava/items/d9886b2e8835c5c0d30f
http://qiita.com/pottava/items/452bf80e334bc1fee69a
https://openfoam.org/download/4-1-ubuntu/
http://docs.aws.amazon.com/cli/latest/reference/batch/index.html#cli-aws-batch
http://docs.aws.amazon.com/ja_jp/cli/latest/userguide/installing.html