npm package: @ignitial/dlake-service
Description: Ignitial.io Data Lake service
Author: Victor Andritoiu
Latest version: 2.2.0

# Weather service implementation

This service implements basic dlake services to demonstrate both service call
and service UI injection.

## Native process

When using native process execution, take care not to use already in use port.
In order to avoid that, you can start your process using a startup shell script
as following:

```bash
#!/bin/sh

export IIOS_SERVER_PORT=20013
export IIOS_NAMESPACE=ignitialio

node index.js
```

### Redis

Ignitial.io services are based on Redis for service discovery and PUB/SUB RPC
emulation.

You need to start a Redis server to proceed. For example:

```bash
docker run -d --name redis -p 6379:6379 redis
```

## Docker

Configuration file must be based on ENV variables in order to easily configure
Docker container execution.

### Build

```shell
docker build --rm --force-rm -t ignitial/dlake .
```

### Run

```shell
docker run -d -p 20013:20013 --name dlake --link redis:redis -e REDIS_HOST="redis" -e IIOS_SERVER_PORT=20013 ignitial/dlake
```

### Kill

If Docker image does not contain PM2 and node app is not started using pm2-node,
then take care when stopiing Docker container to send TERM signal to the service.
Indeed, TERM signal will allow the service to clean up and unregister from Redis
discovery dictionary.

```shell
docker exec dlake pkill -TERM node
```

## Usage 

__DLake__ sevice aims to provide data services to any other __IIO__ service or application.

### Main concepts  

> Notice:  
> We are mainly targetting NoSQL databases (currently only MongoDB is implemented),
> but same ceoncepts could be implemented for a relational database as well.  

For any MongoDB collection we need to implement a __datum__, which is simply a 
wrapping class providing at least basic CRUD operations (augmented with some sugar). 
These operations are already implemented in the _Item_ class, which is in fact 
using engine specifics wrapping that within a IIO standardized API. For MongoDB, 
implementation can be found in _lib/db/item-mongo.js_ file.  

### Basic operations  

Basic operation available are:  

- _find(args, userId)_: equivalent to _collection.find_. Arguments are passed through an _args_
object that is either a query (in this case a MongoDB query), or an object 
containing a query and an option field:  

```javascript
  mydatum.find({ _id: myID }, userId).then(docs => {
    // do something with docs
  }).catch(err => {})  
  
  // OR
  
  mydatum.find({ 
    query: { _id: myID }, 
    options: { projection: { fieldOne: 1 }}
  }, userId).then(docs => {
    // do something with docs
  }).catch(err => {})  
```

As you can see, one additional parameter to _args_ is _userId_ which is used to 
implement access control.  

- _findAndSort(args, userId)_: similar to previous, excepting that is paginated and sorted. 
_args_ parameter needs then to contain _sort_, _page_, and _pageSize_ fields to 
define corresponding options.  
- _findPaginated(args, userId)_: similar, but only focused on pagination:

```javascript
  mydatum.find({ 
    query: { _id: myID }, 
    page: 1,
    pageSize: 30,
    options: { projection: { fieldOne: 1 }}
  }, userId).then(docs => {
    // do something with docs
  }).catch(err => {})    
```

- _get(args, userId)_: returns one single item from a collection.

```javascript
  mydatum.get({ _id: myID }, userId).then(doc => {
    // do something with doc
  }).catch(err => {})    
  
  // OR
  
  mydatum.get({ 
    query: { _id: myID }, 
    options: { projection: { fieldOne: 1 }}
  }, userId).then(doc => {
    // do something with doc
  }).catch(err => {})  
```

> Notice:  
> __id_ can be a string that will be converted automatically to ObjecID when 
> using Mongo.  

- _put(args, userId)_: inserts or updates an element defined by _args_ object. If
_args_ provides an __id_ field, item is supposed to be updated (if 
corresponding item is found), unless is inserted. Updates are possible only if 
datum options _appendOnly_ is not set to true.  

```javascript
  // update
  mydatum.put({ 
    _id: myID,
    fieldOne: '...',
    // ... 
  }, userId).then(() => {
    console.log('done')
  }).catch(err => {})    
  
  // OR insert
  mydatum.put({ 
    fieldOne: '...',
    // ..
  }, userId).then(result => {
    console.log(result._id)
  }).catch(err => {})  
```

As seen above, in the case of an insert, you can get back the item __id_ field 
for further use.  

- _putsert(args, userId)_: similar, excepting that is using _updateOne_ with 
_insert_ option set (specific to Mongo)  
- _del(args, userId)_: deletes one item defined by _args_ query.  

> Notice:  
> All the methods are returning a _Promise_.  

### Default datums (collections)
  
There are several collections defined by default:
- _users_: users list for user management  
- _roles_: user roles for access control  
- _connections_: logs users connections for reporting  
- _activities_: logs users activity for reporting..
- _notifications_: users application notifications  

### Deploy

In order to deploy __DLake__ service you can either use a derivated class 
creating your own dedicated service, or deploy it as a Docker container providing
to it your additional datum definitions.  

Indeed, configuration file (_config.index.js_) provides a field that allows to 
define the directory where to look for datum definition files:  

```javascript
  {
    /* datum definition pathes */
    datum: {
      paths: [ './lib/datum', '/opt/dlake/datum' ]
    },
    // ...
  }
```

By default there is _/opt/dlake/datum_ which corresponds to service deployment 
directory when build for docker (see _Dockerfile_). Then you can declare for 
example a volume link in order to target your definition directory.  

Observing the configuration file, you can see as well any other environment 
variable that you can use for service tuning. An example of deployment command can be found below:

```bash
#!/bin/sh

docker run -d --name dlake-ottoman \
  -p 20091:20091 \
  -e DLAKE_NAME=dlake-ottoman \
  -e REDIS_HOST=redis \
  -e MONGODB_URI=mongodb://mongo:27017 \
  -e MONGODB_DBNAME=ottoman \
  -e IIOS_NAMESPACE=ottoman \
  -e IIOS_SERVER_PORT=20091 \
  -v ${pwd}/datum:/opt/dlake/datum \
  --link mongo:mongo \
  --link redis:redis \
  ignitial/dlake
```

When deploying in production, you can use Docker volume containers to share 
datum defintions between a source and the dlake service.  

```bash
docker volume create --name datum-defs
```

then 

```bash
#!/bin/sh

docker run -d --name dlake-ottoman \
  -p 20091:20091 \
  -e DLAKE_NAME=dlake-ottoman \
  -e REDIS_HOST=redis \
  -e MONGODB_URI=mongodb://mongo:27017 \
  -e MONGODB_DBNAME=ottoman \
  -e IIOS_NAMESPACE=ottoman \
  -e IIOS_SERVER_PORT=20091 \
  -v datum-defs:/opt/dlake/datum \
  --link mongo:mongo \
  --link redis:redis \
  ignitial/dlake
```