Project: CubeFS (Runtime / Cloud Native Storage)
File: authnode.md

Authnode provides a general authentication & authorization service among CubeFS nodes. Client, Master, Meta and Data node are required to be authenticated and authorized before any resource access in another node. Initially, each node (Auth, Client, Master, Meta or Data node) is launched with a secure key which is distributed by a authenticated person (for instance, cluster admin). With a valid key, a node can be identified in Authnode service and granted a ticket for resource access. The overall work flow is: key creation and distribution > ticket retrieval with key > resource access with ticket. Key: a bit of secret shared data between a node and Authnode that asserts identity of a client. Ticket: a bit of data that cryptographically asserts identity and authorization (through a list of capabilities) for a service for a period of time. Capability: a capability is defined in the format of node:object:action where node refers to a service node (such as auth, master, meta or data), object refers to the resource and action refers to permitted activities (such as read, write and access). See examples below. Capability Example | Capability | Specifications | |--|| | auth:createkey:access | Access permission for createkey in Authnode | | master:\:\ | Any permissions for any objects in Master node | | \:\:\* | Any permissions for any objects in any nodes | cfs-authtool is a client-side utility of Authnode, providing key managements (creation, view and modification) and ticket retrieval in and from Authnode keystore. In particular, Each key is associated with an entity name or id, secret key string, creation time, role and capability specification. Key structure | Key | Type | Description | ||--|| | id | string | Unique key identifier composed of letters and digits | | key | string | Base64 encoded secret key | | role | string | The role of the key (either client or service) | | caps | string | The capabilities of the key | Use the following commands to build client side tool for Authnode: ```bash $ git clone http://github.com/cubefs/cubefs.git $ cd cubefs $ make build ``` If successful, the tool cfs-authtool can be found in build/bin. cfs-authtool ticket -host=AuthNodeAddress [-keyfile=Keyfile] [-output=TicketOutput] [-https=TrueOrFalse -certfile=AuthNodeCertfile] TicketService Service cfs-authtool api -host=AuthNodeAddress -ticketfile=TicketFile [-data=RequestDataFile] [-output=KeyFile] [-https=TrueOrFalse -certfile=AuthNodeCertfile] Service Request cfs-authtool authkey [-keylen=KeyLength] TicketService := [getticket] Service := [AuthService | MasterService | MetaService | DataService] Request := [createkey | deletekey | getkey | addcaps | deletecaps | getcaps | addraftnode | removeraftnode] Authnode use JSON as configuration file

Properties | Key | Type | Description | Mandatory | |-|--||--| | role | string | Role of process and must be set to master | Yes | | ip | string | host ip | Yes | | port | string | Http port which api service listen on | Yes | | prof | string | golang pprof port | Yes | | id | string | identy different master node | Yes | | peers | string | the member information of raft group | Yes | | logDir | string | Path for log file storage | Yes | | logLevel | string | Level operation for logging. Default is error. | No | | retainLogs | string | the number of raft logs will be retain. | Yes | | walDir | string | Path for raft log file storage. | Yes | | storeDir | string | Path for RocksDB file storage,path must be exist | Yes | | clusterName | string | The cluster identifier | Yes | | exporterPort | int | The prometheus exporter port | No | | authServiceKey | string | The secret key used for authentication of AuthNode | Yes | | authRootKey | string | The secret key used for key derivation (session and client secret key) | Yes | | enableHTTPS | bool | Option whether enable HTTPS protocol | No | Example: ```json { "role": "authnode", "ip": "192.168.0.14", "port": "8080", "prof":"10088", "id":"1", "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082", "logDir": "/export/Logs/authnode", "logLevel":"info", "retainLogs":"100", "walDir":"/export/Data/authnode/raft", "storeDir":"/export/Data/authnode/rocksdbstore", "exporterPort": 9510, "consulAddr": "http://consul.prometheus-cfs.local", "clusterName":"test", "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=", "enableHTTPS":false } ``` Run the command: ```bash $ ./cfs-authtool authkey ``` If successful, two key files can be generated `authroot.json` and `authservice.json` under current directory. They represent authServiceKey and authRootKey respectively. example `authservice.json` : ```json { "id": "AuthService", "key": "9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "create_ts": 1573801212, "role": "AuthService", "caps": "{\"*\"}" } ``` Edit `authnode.json` in docker/conf as following: `authRootKey`: use the value of `key` in `authroot.json` `authServiceKey`: use the value of `key` in `authService.json` example `authnode.json` : ```json { "role": "authnode", "ip": "192.168.0.14", "port": "8080", "prof":"10088", "id":"1", "peers": "1:192.168.0.14:8080,2:192.168.0.15:8081,3:192.168.0.16:8082", "retainLogs":"2", "logDir": "/export/Logs/authnode", "logLevel":"info", "walDir":"/export/Data/authnode/raft", "storeDir":"/export/Data/authnode/rocksdbstore", "exporterPort": 9510, "consulAddr": "http://consul.prometheus-cfs.local", "clusterName":"test", "authServiceKey":"9h/sNq4+5CUAyCnAZM927Y/gubgmSixh5hpsYQzZG20=", "authRootKey":"wbpvIcHT/bLxLNZhfo5IhuNtdnw1n8kom+TimS2jpzs=", "enableHTTPS":false } ``` In directory docker/authnode, run the following command to start a Authnode cluster. ```bash $ docker-compose up -d ``` Get Authnode ticket using authServiceKey: ```bash $ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=authservice.json -output=ticket_auth.json getticket AuthService ``` example `ticket_auth.json` : ```json { "id": "AuthService", "session_key": "A9CSOGEN9CFYhnFnGwSMd4WFDBVbGmRNjaqGOhOinJE=", "service_id": "AuthService", "ticket": "RDzEiRLX1xjoUyp2TDFviE/eQzXGlPO83siNJ3QguUrtpwiHIA3PLv4edyKzZdKcEb3wikni8UxBoIJRhKzS00+nB7/9CjRToAJdT9Glhr24RyzoN8psBAk82KEDWJhnl+Y785Av3f8CkNpKv+kvNjYVnNKxs7f3x+Ze7glCPlQjyGSxqARyLisoXoXbiE6gXR1KRT44u7ENKcUjWZ2ZqKEBML9U4h0o58d3IWT+n4atWKtfaIdp6zBIqnInq0iUueRzrRlFEhzyrvi0vErw+iU8w3oPXgTi+um/PpUyto20c1NQ3XbnkWZb/1ccx4U0" } ``` Create admin using Authnode ticket: ```bash $ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticketauth.json -data=dataadmin.json -output=key_admin.json AuthService createkey ``` example `data_admin.json` : ```json { "id": "admin", "role": "service", "caps": "{\"API\":[\"::*\"]}" } ``` Get Authnode ticket using admin key: ```bash $ ./cfs-authtool ticket -host=192.168.0.14:8080 -keyfile=keyadmin.json -output=ticketadmin.json getticket AuthService ``` Create key for Master ```bash $ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticketadmin.json -data=datamaster.json -output=key_master.json AuthService createkey ``` example `data_master.json` : ```json { "id": "MasterService", "role": "service", "caps": "{\"API\":[\"::*\"]}" } ``` Specifications: | Key | Description | ||| | id | will set Client ID | | role | will set the role of id | | caps | will set the capabilities of id | Edit `master.json as` following: `masterServiceKey`: use the value of `key` in `key_master.json` Create key for Client ```bash $ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticketadmin.json -data=dataclient.json -output=key_client.json AuthService createkey ``` example `data_client.json`: ```json { "id": "ltptest", "role": "client", "caps": "{\"API\":[\"::\"], \"Vol\":[\"::\"]}" } ``` Edit `client.json` as following: `clientKey`: use the value of `key` in `key_client.json` example `client.json` ```json { "masterAddr": "192.168.0.11:17010,192.168.0.12:17010,192.168.0.13:17010", "mountPoint": "/cfs/mnt", "volName": "ltptest", "owner": "ltptest", "logDir": "/cfs/log", "logLevel": "info", "consulAddr": "http://192.168.0.101:8500", "exporterPort": 9500, "profPort": "17410", "authenticate": true, "ticketHost":

"clientKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=", "enableHTTPS": "false" } ``` Specifications: | Key | Description | |--|| | authenticate | will enable authentication flow if set true | | ticketHost | will set the IP/URL of Authnode cluster | | clientKey | will set the key generated by Authnode | | enableHTTPS | will enable HTTPS if set true | Run the following to launch CubeFS cluster with AuthNode enabled: ```bash $ docker/run_docker.sh -r -d /data/disk ``` To prevent MITM (Man In The Middle) attacks, HTTPS is required for the communication between client and service. The following steps show the generation of self-sign a certificate with a private (.key) and public key. Generating Key and Self Signed Cert: ```bash $ openssl req \ -x509 \ -nodes \ -newkey rsa:2048 \ -keyout server.key \ -out server.crt \ -days 3650 \ -subj "/C=GB/ST=China/L=Beijing/O=jd.com/OU=Infra/CN=*" ``` Specifications: `server.crt`: `AuthNode` public certificate needed to be sent to `Client` `server.key`: `AuthNode` private key needed to be securely placed in `/app` folder in `Authnode` For easy deployment, current implementation of AuthNode uses TLS option insecureskipverify and tls.RequireAndVerifyClientCert, which would skip secure verification of both client and server. For environment with high security command, these options should be turned off. Master has numerous APIs, such as creating and deleting volumes, so it is necessary to authenticate access to the master API to improve cluster security. With the excellent authentication capability of authnode, we have optimized the existing authentication mechanism to achieve the goal of simplifying the authentication process. example `master.json` ```json { "role": "master", "ip": "127.0.0.1", "listen": "17010", "prof":"17020", "id":"1", "peers": "1:127.0.0.1:17010,2:127.0.0.2:17010,3:127.0.0.3:17010", "retainLogs":"20000", "logDir": "/cfs/master/log", "logLevel":"info", "walDir":"/cfs/master/data/wal", "storeDir":"/cfs/master/data/store", "consulAddr": "http://consul.prometheus-cfs.local", "clusterName":"cubefs01", "metaNodeReservedMem": "1073741824", "masterServiceKey": "jgBGSNQp6mLbu7snU8wKIdEkytzl+pO5/OZOJPpIgH4=", "authenticate": true, "authNodeHost": "192.168.0.14:8080,192.168.0.15:8081,192.168.0.16:8082", "authNodeEnableHTTPS": false } ``` Properties | Key | Description | |--|--| | authenticate | will enable Master API Authentication if set true | | authNodeHost | will set the IP/URL of Authnode cluster | | authNodeEnableHTTPS | will enable HTTPS if set true | When accessing the Master API, the parameter clientIDKey for authentication must be included. When using authtool to create a key, authidkey is generated. This key will be used as the clientIDKey when accessing the master API. Access the master API via HTTP and write the parameter clientIDKey, such as expanding a volume: ```bash curl --location 'http://127.0.0.1:17010/vol/update?name=ltptest&authKey=0e20229116d5a9a4a9e876806b514a85&capacity=100&clientIDKey=eyJpZCI6Imx0cHRlc3QiLCJhdXRoX2tleSI6ImpnQkdTTlFwNm1MYnU3c25VOHdLSWRFa3l0emwrcE81L09aT0pQcElnSDQ9In0=' ``` Access the master API via cfs-cli and write the clientIDKey to configuration file .cfs-cli.json, so that any cluster management command is authenticated for permissions. example `.cfs-cli.json` ```json { "masterAddr": [ "127.0.0.1:17010", "127.0.0.2:17010", "127.0.0.3:17010" ], "timeout": 60, "clientIDKey": "eyJpZCI6Imx0cHRlc3QiLCJhdXRoX2tleSI6ImpnQkdTTlFwNm1MYnU3c25VOHdLSWRFa3l0emwrcE81L09aT0pQcElnSDQ9In0=" } ``` When datanode and metanode are started, they will respectively call the AddDataNode and AddMetaNode APIs, so it is also necessary to prepare serviceIDKey for them. Similarly, use authtool to create keys for datanode and metanode respectively, and write the key as the serviceIDKey to the configuration file. When they start, permission authentication will be performed. ```bash $ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticketadmin.json -data=datadatanode.json -output=key_datanode.json AuthService createkey ``` example `data_datanode` ```json { "id": "DatanodeService", "role": "service", "caps": "{\"API\":[\"::*\"]}" } ``` Edit `datanode.json as` following: `serviceIDKey`: use the value of `authidkey` in `key_datanode.json` ```bash $ ./cfs-authtool api -host=192.168.0.14:8080 -ticketfile=ticketadmin.json -data=datametanode.json -output=key_metanode.json AuthService createkey ``` example `data_metanode` ```json { "id": "MetanodeService", "role": "service", "caps": "{\"API\":[\"::*\"]}" } ``` Edit `metanode.json as` following: `serviceIDKey`: use the value of `authidkey` in `key_metanode.json`