Project: Apache Hadoop (App Definition and Development / Database)
File: YARN-Services-Examples.md

<! Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> POST URL - http://localhost:8088/app/v1/services ```json { "name": "hello-world", "version": "1.0.0", "description": "hello world example", "components" : [ { "name": "hello", "numberofcontainers": 2, "artifact": { "id": "nginx:latest", "type": "DOCKER" }, "launchcommand": "./startnginx.sh", "resource": { "cpus": 1, "memory": "256" } } ] } ``` GET URL - http://localhost:8088/app/v1/services/hello-world Note, lifetime value of -1 means unlimited lifetime. ```json { "name": "hello-world", "version": "1.0.0", "description": "hello world example", "id": "application15039639855680002", "lifetime": -1, "state": "STABLE", "components": [ { "name": "hello", "state": "STABLE", "resource": { "cpus": 1, "memory": "256" }, "configuration": { "properties": {}, "env": {}, "files": [] }, "quicklinks": [], "containers": [ { "id": "containere031503963985568000201_000002", "ip": "10.22.8.143", "hostname": "ctr-e03-1503963985568-0002-01-000002.example.site", "state": "READY", "launch_time": 1504051512412, "bare_host": "host100.cloud.com", "componentinstancename": "hello-0" }, { "id": "containere031503963985568000201_000003", "ip": "10.22.8.144", "hostname": "ctr-e03-1503963985568-0002-01-000003.example.site", "state": "READY", "launch_time": 1504051536450, "bare_host": "host100.cloud.com", "componentinstancename": "hello-1" } ], "launchcommand": "./startnginx.sh", "numberofcontainers": 1, "runprivilegedcontainer": false } ], "configuration": { "properties": {}, "env": {}, "files": [] }, "quicklinks": {} } ``` PUT URL - http://localhost:8088/app/v1/services/hello-world Note, irrespective of what the current lifetime value is, this update request will set the lifetime of the service to be 3600 seconds (1 hour) from the time the request is submitted. Hence, if a a service has remaining lifetime of 5 mins (say) and would like to extend it to an hour OR if an application has remaining lifetime of 5 hours (say) and would like to reduce it down to an hour, then for both scenarios you need to submit the same request below. ```json { "lifetime": 3600 } ``` PUT URL - http://localhost:8088/app/v1/services/hello-world ```json { "state": "STOPPED" } ``` PUT URL - http://localhost:8088/app/v1/services/hello-world ```json { "state": "STARTED" } ``` PUT URL - http://localhost:8088/app/v1/services/hello-world/components/hello ```json { "numberofcontainers": 3 } ``` Alternatively, you can specify the entire "components" section instead. PUT URL - http://localhost:8088/app/v1/services/hello-world ```json { "state": "FLEX", "components" : [ { "name": "hello", "numberofcontainers": 3 } ] } ``` DELETE URL - http://localhost:8088/app/v1/services/hello-world POST URL - http://localhost:8088:/app/v1/services/hbase-app-1 ```json { "name": "hbase-app-1", "version": "1.0.0", "description": "hbase service", "lifetime": "3600", "components": [ { "name": "hbasemaster", "numberofcontainers": 1, "artifact": { "id": "hbase:latest", "type": "DOCKER" }, "launch_command": "/usr/hdp/current/hbase-master/bin/hbase master start", "resource": { "cpus": 1, "memory": "2048" }, "configuration": { "env": { "HBASELOGDIR": "<LOG_DIR>" }, "files": [ { "type": "XML", "dest_file": "/etc/hadoop/conf/core-site.xml", "properties": { "fs.defaultFS": "${CLUSTERFSURI}" } }, { "type": "XML", "dest_file": "/etc/hbase/conf/hbase-site.xml", "properties": { "hbase.cluster.distributed": "true",

"${CLUSTERZKQUORUM}", "hbase.rootdir": "${SERVICEHDFSDIR}/hbase", "zookeeper.znode.parent": "${SERVICEZKPATH}", "hbase.master.hostname": "hbasemaster.${SERVICE_NAME}.${USER}.${DOMAIN}", "hbase.master.info.port": "16010" } } ] } }, { "name": "regionserver", "numberofcontainers": 3, "artifact": { "id": "hbase:latest", "type": "DOCKER" }, "launch_command": "/usr/hdp/current/hbase-regionserver/bin/hbase regionserver start", "resource": { "cpus": 1, "memory": "2048" }, "configuration": { "env": { "HBASELOGDIR": "<LOG_DIR>" }, "files": [ { "type": "XML", "dest_file": "/etc/hadoop/conf/core-site.xml", "properties": { "fs.defaultFS": "${CLUSTERFSURI}" } }, { "type": "XML", "dest_file": "/etc/hbase/conf/hbase-site.xml", "properties": { "hbase.cluster.distributed": "true", "hbase.zookeeper.quorum": "${CLUSTERZKQUORUM}", "hbase.rootdir": "${SERVICEHDFSDIR}/hbase", "zookeeper.znode.parent": "${SERVICEZKPATH}", "hbase.master.hostname": "hbasemaster.${SERVICE_NAME}.${USER}.${DOMAIN}", "hbase.master.info.port": "16010", "hbase.regionserver.hostname": "${COMPONENTINSTANCENAME}.${SERVICE_NAME}.${USER}.${DOMAIN}" } } ] } } ], "quicklinks": { "HBase Master Status UI": "http://hbasemaster0.${SERVICE_NAME}.${USER}.${DOMAIN}:16010/master-status", "Proxied HBase Master Status UI": "http://app-proxy/${DOMAIN}/${USER}/${SERVICE_NAME}/hbasemaster/16010/" } } ``` POST URL - http://localhost:8088/app/v1/services ```json { "name": "hello-world", "version": "1.0.0", "description": "hello world example with GPUs", "components" : [ { "name": "hello", "numberofcontainers": 2, "artifact": { "id": "nginx:latest", "type": "DOCKER" }, "launchcommand": "./startnginx.sh", "resource": { "cpus": 1, "memory": "256", "additional" : { "yarn.io/gpu" : { "value" : 4, "unit" : "" } } } } ] } ``` POST URL - http://localhost:8088/app/v1/services ```json { "name": "hello-world", "version": "1.0.0", "description": "hello world example with anti-affinity", "components" : [ { "name": "hello", "numberofcontainers": 3, "artifact": { "id": "nginx:latest", "type": "DOCKER" }, "launchcommand": "./startnginx.sh", "resource": { "cpus": 1, "memory": "256" }, "placement_policy": { "constraints": [ { "type": "ANTI_AFFINITY", "scope": "NODE", "node_attributes": { "os": ["linux", "windows"], "fault_domain": ["fd1", "fd2"] }, "node_partitions": [ "gpu", "fast-disk" ], "target_tags": [ "hello" ] } ] } } ] } ``` GET URL - http://localhost:8088/app/v1/services/hello-world Note, for an anti-affinity component no more than 1 container will be allocated in a specific node. In this example, 3 containers have been requested by component "hello". All 3 containers were allocated because the cluster had 3 or more NMs. If the cluster had less than 3 NMs then less than 3 containers would be allocated. In cases when the number of allocated containers are less than the number of requested containers, the component and the service will be in non-STABLE state. ```json { "name": "hello-world", "version": "1.0.0", "description": "hello world example with anti-affinity", "id": "application15039639855680003", "lifetime": -1, "state": "STABLE", "components": [ { "name": "hello", "state": "STABLE", "resource": { "cpus": 1, "memory": "256" }, "placement_policy": { "constraints": [ { "type": "ANTI_AFFINITY", "scope": "NODE", "node_attributes": { "os": ["linux", "windows"], "fault_domain": ["fd1", "fd2"] }, "node_partitions": [ "gpu", "fast-disk" ], "target_tags": [ "hello" ] } ] }, "configuration": { "properties": {}, "env": {}, "files": [] }, "quicklinks": [], "containers": [ { "id": "containere031503963985568000301_000002", "ip": "10.22.8.143", "hostname": "ctr-e03-1503963985568-0003-01-000002.example.site", "state": "READY", "launch_time": 1504051512412, "bare_host": "host100.cloud.com", "componentinstancename": "hello-0" }, { "id": "containere031503963985568000301_000003", "ip": "10.22.8.144", "hostname": "ctr-e03-1503963985568-0003-01-000003.example.site", "state": "READY", "launch_time": 1504051536450, "bare_host": "host101.cloud.com", "componentinstancename": "hello-1" }, { "id": "containere031503963985568000301_000004", "ip": "10.22.8.145", "hostname": "ctr-e03-1503963985568-0003-01-000004.example.site", "state": "READY", "launch_time": 1504051536450, "bare_host": "host102.cloud.com", "componentinstancename": "hello-2" } ], "launchcommand": "./startnginx.sh", "numberofcontainers": 1, "runprivilegedcontainer": false } ], "configuration": { "properties": {}, "env": {}, "files": [] }, "quicklinks": {} } ```