giseldo/neodataset huggingface dataset:
Make a cleanup script more reliable

Goal
===
If the creation of a snapshot is being failed during a few days, the cleanup script can eventually remove all snapshots
```
0 4 * * *   ZFS_POOL="zpool"   PGDATA_SUBDIR="/postgresql/data"   MOUNT_DIR="/var/lib/dblab/clones"   SUDO_CMD="sudo -u postgres"  bash /home/ubuntu/database-lab/scripts/create_zfs_snapshot.sh | logger --stderr --tag "create_zfs_snapshot"
0 6 * * * sudo zfs list -t snapshot -o name | grep -v clone | grep -v NAME | head -n -3 | xargs -n1 --no-run-if-empty sudo zfs destroy -R 2>&1  | logger --stderr --tag "cleanup_zfs_snapshot"
```

TODO / How to implement
===
- Do not remove the last snapshot: 
    - use `head -n -NUM` because the command option with the leading `-` prints all except the last NUM lines of the input
- ~~Do not remove the snapshot from which a clone was created~~ **Update**: clone will not be removed:
   ```
     $ sudo zfs destroy -R dblab_pool_replica/clone_pre_20200611092119
     cannot destroy 'dblab_pool_replica/dblab_clone_6000': dataset is busy
   ```
- Restart sync instance even on failure the snapshot script


Acceptance criteria
===

Story points: 1.0
Improve create_zfs_snapshot script

Goal
===
- [x] Define variable `dockerImage`: `postgres:${pg_ver}-alpine`
- [x] Better comments from `pgdata_subdir=${PGDATA_SUBDIR:-""}`. Like: `export PGDATA_SUBDIR="/<dir name, but not full path, and dont forget '/'>"`
- [x] Set default value for `datastateat` if it's empty
- [x] [Separate issue?] `docker run -it --rm  -v /etc/passwd:/etc/passwd:ro -e POSTGRES_PASSWORD=mysecretpassword -d postgres`

```
# diff create_zfs_snapshot.sh create_zfs_snapshot.sh.base 
32c32
< container_pgdata_dir="/var/lib/postgresql/data"
---
> container_pgdata_dir="/var/lib/postgresql/pgdata"
41c41
< sudo_cmd=${SUDO_CMD:-"sudo -u postgres"} # Use `sudo -u postgres` for default environment
---
> sudo_cmd=${SUDO_CMD:-""} # Use `sudo -u postgres` for default environment
57,59d56
< pg_ver_full=$(${sudo_cmd} cat ${clone_pgdata_dir}/PG_VERSION)
< 
< image_name=${IMAGE_NAME:-"postgres:${pg_ver}-alpine"}
80,81d76
< sed -i 's/^\\(.*stats_temp_directory\\)/# \\1/' ${clone_pgdata_dir}/postgresql.conf
< sed -i 's/^\\(.*include_dir\\)/# \\1/' ${clone_pgdata_dir}/postgresql.conf
130d124
<   --volume /etc/passwd:/etc/passwd:ro \
131a126
>   --user postgres \
133c128
<   ${image_name}
---
>   postgres:${pg_ver}-alpine
182c177
<   sudo docker exec ${container_name} ${sudo_cmd} /usr/lib/postgresql/${pg_ver_full}/bin/pg_ctl -D ${container_pgdata_dir} -W promote
---
>   sudo docker exec ${container_name} pg_ctl -D ${container_pgdata_dir} -W promote
```

TODO / How to implement
===


Acceptance criteria
===

Story points: 3.0
Incorrent clone size in metadata

Goal
===
- CloneSize -> CloneDiffSize in API (console, database lab, update specification)
- Are we sure that data size is correct?

Following the tutorial, when you do `psql ... -c '\l+'`, it shows correct size, as expected (\~1.5 GiB), but when you do `dblab clone list`, it shows weird size something like ~200,000:

```
        "metadata": {
            "cloneSize": 217088,
            "cloningTime": 1.344037454,
            "maxIdleMinutes": 20
        },
```

`cloneSize` seems to be diff size, needs renaming

Current values:
- CLI: `"cloneSize": 218112`, 
- GUI: `Data size: 2.504 GiB`, 
- psql is correct: `1503 MB`

```
$ zfs list -po name,used,mountpoint,compressratio,available,type,origin,creation,referenced,logicalreferenced,logicalused,dblab:datastateat -S dblab:datastateat -S creation -t filesystem
NAME                              USED  MOUNTPOINT                              RATIO        AVAIL  TYPE        ORIGIN             CREATION        REFER      LREFER       LUSED  DBLAB:DATASTATEAT
dblab_pool/dblab_clone_6000     440320  /var/lib/dblab/clones/dblab_clone_6000   2.10  33532292608  filesystem  dblab_pool@initdb  1587380790  532452352  2688620544      592384  -
dblab_pool                   533844480  /var/lib/dblab/data                      5.17  33532292608  filesystem  -                  1587325346  532447232  2688612864  2689524224  -
```


TODO / How to implement
===
Fix clone size in CLI, GUI and Database Lab API response

Data size is correct:
```
/var/lib/dblab/data# du -h --apparent-size | sort -rh | head -10
2.5G	.
1.5G	./base/16384
1.5G	./base
1.1G	./pg_wal
7.8M	./base/13423
7.7M	./base/13422
7.7M	./base/1
623K	./global
17K	./pg_multixact
14K	./pg_stat
```


Acceptance criteria
===

Story points: 4.0
Provide cross-platform CLI artifacts

Goal
===
Curl installation of CLI tool wouldn't work on non-Linux systems. We need MacOS, Windows and FreeBSD versions as well.

TODO / How to implement
===
- Build binaries for: linux, darwin, freebsd, windows in CI
- Improve the cli_install script to automatic downloading CLI binaries

MR: https://gitlab.com/postgres-ai/database-lab/-/merge_requests/98

Acceptance criteria
===

Story points: 4.0
Refactoring API response errors

Goal
===
To add more details, hints for errors and specify `INTERNAL_ERROR` (it should be used only for *unpredicted* errors).
For example, in GUI:
```
Something went wrong. clone with such ID already exists 
Full response: 
{"error": {}, 
"content": {"code": "INTERNAL_ERROR", "hint": "", "detail": "clone with such ID already exists", "message": "Internal server error."}, 
"message": "Internal Server Error", 
"status_code": 500}
```

TODO / How to implement
===
- check the relevance of API responses, adjust if needed
- prepare list of code errors for the docs.



Acceptance criteria
===

Story points: 6.0
Database Lab - Telemetry

Goal
===

As a Database Lab user I want to be able to collect performance metrics about a clone (e.g. for verifying of database migrations).

Implement basic support with following artefacts collection:
- [x] Telemetry collection duration;
- [x] Locks monitoring.
- [x] CI/CD: how can we understand if it fails?

TODO / How to implement
===
Original idea:
New client CLI commands:
- `dblab telemetry start` - starts collection of metrics (deletes previously collected artefacts);
- `dblab telemetry stop` - stops collection of metrics, interval services, executes additional scripts;
- `dblab telemetry status` - gets short summary about migration status;
- `dblab telemetry download` - downloads artefacts from Database Lab machine to a user's machine.

Implement corresponding API handles.

Usage:
```
dblab clone create ...
dblab telemetry start

sqitch  deploy

dblab telemetry stop
dblab telemetry download
```

Suggestions from standup call:
- Use Prometheus format for export of time series data
- `dblab clone observe -f CLONE_ID`
- Use `start`, `stop` or not?

Improved idea:
- Collect telemetry and expose it in Prometheus format or use https://github.com/wrouesnel/postgres_exporter.
- Collect telemetry all the time for all clones.
- Use `dblab clone observe -f CLONE_ID` to retrieve current telemetry for the clone.
- Use `dblab clone start-observe CLONE_ID` and `dblab clone stop-observe CLONE_ID` to get aggregated statistics about duration and locks and the end of a DB migration test.

Fastest implementation:
- Execute monitoring queries from Database Lab CLI itself, do not change server or its API.
- Use `dblab clone observe -f CLONE_ID` to start observing and `dblab clone stop-observe CLONE_ID` to stop observing and show aggregated metrics.


Acceptance criteria
===

Story points: 13.0
Cannot delete a "failed" clone

There is a clone:

```
$ dblab clone list
...json
    {
        "id": "joe-bpfrfqt4h85p27j6426g",
        "snapshot": {
            "id": "zpool/clone_pre_20200304065541@snapshot_20200304065541",
            "createdAt": "2020-03-04 06:58:12 UTC",
            "dataStateAt": "2020-03-04 06:55:33 UTC"
        },
        "protected": false,
        "deleteAt": "",
        "createdAt": "2020-03-04 14:15:07 UTC",
        "status": {
            "code": "FATAL",
            "message": "Cloning failure."
        },
        "db": {
            "connStr": "",
            "host": "",
            "port": "",
            "username": "XXX",
            "password": ""
        },
        "metadata": null,
        "project": ""
    },
...
]
```

If I try to delete it with `clone destroy` I get the following:

```
$ dblab clone destroy joe-bpfrfqt4h85p27j6426g
2020/03/05 03:27:19 [DEBUG]  Database Lab request error: Code "INTERNAL_ERROR". Message: Internal server error. Detail: clone is not started yet Hint: 
url: http://localhost:3000/clone/joe-bpfrfqt4h85p27j6426g
request-dump: DELETE /clone/joe-bpfrfqt4h85p27j6426g HTTP/1.1
Host: localhost:3000
Verification-Token: XXXX
2020/03/05 03:27:19 failed to get response: Code "INTERNAL_ERROR". Message: Internal server error. Detail: clone is not started yet Hint: 
```

Then I see that status is changed to DELETING and the clone eventually is deleted -- however, the initial response looks like a bug.

UPDATE: after deletion attempt, the status indeed changed to DELETING, but it **has not been deleted** after 5 minutes of waiting:

```
    {
        "id": "joe-bpfrfqt4h85p27j6426g",
        "snapshot": {
            "id": "zpool/clone_pre_20200304065541@snapshot_20200304065541",
            "createdAt": "2020-03-04 06:58:12 UTC",
            "dataStateAt": "2020-03-04 06:55:33 UTC"
        },
        "protected": false,
        "deleteAt": "",
        "createdAt": "2020-03-04 14:15:07 UTC",
        "status": {
            "code": "DELETING",
            "message": "Clone is being deleted."
        },
        "db": {
            "connStr": "",
            "host": "",
            "port": "",
            "username": "XXX",
            "password": ""
        },
        "metadata": null,
        "project": ""
    }
```

Story points: 1.0
Extended Docker with Postgres (9.6, 10, 11, 12)

Goal
===
Provide an "extended" Postgres docker image, with additional extensions -- for cases, when we don't want (don't have time) to work on a custom image.

TODO / How to implement
===

- @anatolystansler please provide an example
- CI/CD pipelines have to rebuild it
- it is published to our Docker Hub

Acceptance criteria
===
As a user, I can choose 1 from 3 options:

- use minimalistic (official) image from Docker Hub
- use "extended" image, with various additional extensions
- build a custom image

Story points: 10.0
Determine idle clones by connections and logs

Goal
===
Currently we detect idle clone only by statement logs. It means that we will destroy clones with long running queries, in order to enhance working with clone we should also take into account active connections.

TODO / How to implement
===
```
select * from pg_stat_activity where query not like 'autovacuum: %' and pid <> pg_backend_pid()
```

Acceptance criteria
===

Story points: 4.0
Create a lightweight method for checking of a clones status

Goal
===
If we try to request `GET /clone/cloneID` during reset:
```
	err = ZfsDestroyClone(j.runner, j.config.ModeZfs.ZfsPool, name)
	if err != nil {
		return errors.Wrap(err, "failed to destroy clone")
	}
	err = ZfsCreateClone(j.runner, j.config.ModeZfs.ZfsPool, name, snapshotID,
		j.config.ModeZfs.MountDir, j.config.OSUsername)
	if err != nil {
		return errors.Wrap(err, "failed to create a clone")
	}
```

there is a probability to receive a false-negative result
```
2020/02/12 02:46:38 [DEBUG]  Run(Local): "zfs destroy -R zpool/dblab_clone_6000"
2020/02/12 02:46:39 [INFO]   ->  GET /clone/joe-bp1l0454h85sr82l6fo0
2020/02/12 02:46:39 [DEBUG]  Run(Local): "zfs list -po name,used,mountpoint,compressratio,available,type,origin,creation,referenced,logicalreferenced,logicalused,dblab:datastateat -S dblab:datastateat -S creation -t filesystem -r zpool"
2020/02/12 02:46:39 [DEBUG]  Failed to get clone: failed to get a session state: cannot get session state: specified ZFS pool does not exist
2020/02/12 02:46:39 [DEBUG]  Response: Code "NOT_FOUND". Message: Not found. Detail: Requested object does not exist. Hint: Specify your request.
2020/02/12 02:46:39 [DEBUG]  Not found
2020/02/12 02:46:39 [DEBUG]  Run(Local): "zfs clone -o mountpoint=/var/lib/dblab/clones/dblab_clone_6000 zpool/clone_pre_20200211092040@snapshot_20200211092040 zpool/dblab_clone_6000 && chown -R root /var/lib/dblab/clones/dblab_clone_6000"
2020/02/12 02:46:40 [DEBUG]  Run(Local): output ""
2020/02/12 02:46:40 [DEBUG]  Starting Postgres...
```


TODO / How to implement
===
- Create a new method for retrieving a clone status without a session state.
- Use this method in API for reset and delete methods

OR Alternative: 

- skip the session state error, use default clone size


Acceptance criteria
===

Story points: 1.0
Automated data retrieval (declarative database lab initialization)

Goal
===

```plantuml

frame "Validation" {
 [Config]
}

cloud "Infrastructure\n(optional)\n" {
component Provision [
**Provision machine**

- Create an instance
- Configure environment
- Pre-install software
- Set up extensions
]
}

  component Initialize[
**Initialize**

  - physical copying 
  - logical copying

  - masking ?
  - cleaning data
  - configuration
  
  - WAL shipping (bg)
  - streaming replication (bg)
  - logical replication (bg)
]

frame "Snapshotting" {
  component Snapshot[
**Snapshots**

- masking
- extra data manipulation 
- snapshotting
]


[Config] --> [Provision]: run provisioning
[Provision] -> [Initialize] : start import
[Initialize] --> [Snapshot] : prepare a snapshot
[Snapshot] -> [Snapshot]: schedule/retention \nmanagement

```

TODO / How to implement
===

**Config example**
```yaml
retrieval:
  stages:
    - initialize

  spec:
    initialize:
      jobs:
#        - name: logical-restore
#          options:
#            dumpFile: /tmp/db.dump
#            forceInit: false
#            dbName: test
#            partial:
#              tables:
#                - test
        - name: physical-restore
          options:
            tool: walg
            dockerImage: "postgresai/sync-instance:12"
            envs:
              WALG_GS_PREFIX: "gs://{BUCKET}/{SCOPE}"
            walg:
              storage: gcs
              backupName: LATEST
              credentialsFile: /tmp/sa.json # optional
      
```

-----
---- OLD
Implementation:
- [ ] Stage interface (data retrieval, promotion, mask, etc) 5h
  - Clones/snapshot usage
  - Docker container provision
  - Each stage running in separate container

- [ ] Pipeline:
  - [dump/restore | WAL-G | barman -> PGDATA] -> [PGDATA master/replica -> master -> remove PII -> snapshot]
  - [import] -> [promote] -> [mask]

  - Stage 1: [import] 
    - Configuration 2h
      - snapshot TTL
      - mode
      - dockerImage
      - dump/restore
        - connection params
        - plain-text, directory, custom (-Fc -Fd)
    - Experiments/Preparations 6h
      - Massive diff (!)
        - Delete previous clone snapshots
    - Logic 6h
      - dump/restore
      - set statement timeout to 0
  - Stage 2: [promote]
    - Can be optional. As we want to give SRE an ability to manually promote their clone.
    - Configuration 1h
      - dockerImage
    - Logic 6h
  - Stage 3: [mask] OUT-OF-SCOPE 

- [ ] Pipeline scheduling (run data retrieval on interval) 4h

Documentation:
- [ ] Notify users about autovacuum pause 1h
- [ ] Pipelines docs 6h
  - Configs
  - How it works
  - How to extend


Acceptance criteria
===

Story points: 40.0
Choose the latest snapshot by default on the creation of a new clone

Goal
===
There are 3 snapshots:
```
$ ./bin/dblab snapshot list
[
    {
        "id": "zpool/clone_pre_20200210022148@snapshot_20200210022148",
        "createdAt": "2020-02-10 02:23:26 UTC",
        "dataStateAt": "2020-02-10 02:21:43 UTC"
    },
    {
        "id": "zpool/clone_pre_20200210021941@snapshot_20200210021941",
        "createdAt": "2020-02-10 02:21:34 UTC",
        "dataStateAt": "2020-02-10 02:19:33 UTC"
    },
    {
        "id": "zpool/clone_pre_20200207071546@snapshot_20200207071546",
        "createdAt": "2020-02-07 07:16:57 UTC",
        "dataStateAt": "2020-02-05 07:53:51 UTC"
    }
]
```
The latest is created at `2020-02-10 02:21:43 UTC`
However, we get `"dataStateAt": "2020-02-05 07:53:51 UTC"` for the new clone:
```
$ ./bin/dblab clone list
[
    {
        "id": "joe",
        "snapshot": {
            "id": "zpool/clone_pre_20200207071546@snapshot_20200207071546",
            "createdAt": "2020-02-07 07:16:57 UTC",
            "dataStateAt": "2020-02-05 07:53:51 UTC"
        },
        "protected": false,
        "deleteAt": "",
        "createdAt": "2020-02-10 02:24:53 UTC",
        "status": {
            "code": "OK",
            "message": "Clone is ready to accept Postgres connections."
        },
      ...
    }
]
```

TODO / How to implement
===


Acceptance criteria
===

Story points: 2.0
Rich configuration options

Goal
===
Improve configuration options for Postgres (and other engines in the future). 

Currently we support configuration of two files: `pg_hba.conf` and `postgresql.conf`. `pg_hba.conf` in PGDATA being substituted with the the presented file. And in case of `postgresql.conf` lines being appended to the end of files.

Discuss with @NikolayS 

TODO / How to implement
===
```yaml
files:
  - path: pg_hba.conf
    replaceWithPath: ./configs/postgres/pg_hba.conf 
  - path: postgresql.conf
    append:
      - listen_addresses = '*'
      - log_destination = 'stderr'
    remove:
      - logging_collector
      - data_directory
    comment:
      - ssl
    commentWith: "#"
  - path: standby.signal
    delete: true

# Move to separate issue. Would be nice to have an option to configure such configuration substitute logic and configure custom script that will be launched at different provision stages.
# Requires discussion.
scripts:
  - runner: pg
    script:
      - select 1;
      - select 2;
  - runner: bash
    script:
      - chown user file.txt
  - runner: bash
    fromFile: ./scripts/prepareClone.sh
```

Acceptance criteria
===

Story points: 8.0
Client CLI: "--insecure" in configuration

Goal
===
Currently, if I need to work with a self-signed certificate, I need to use `--insecure` option every time. It's a little bit inconvenient -- I would prefer to configure it once.

TODO / How to implement
===

Allow specifying "insecure" flag in the configuration for a particular Database Lab instance.


Acceptance criteria
===

As a user, I configure the ability to work with a self-signed certificate just once (for each Database Lab instance) and then I don't need to use `--insecure` in CLI at all.

Story points: 2.0
Publish artifacts to external storage

Goal
===
Latest artifacts might be unaccessible on GitLab during pipelines. Also, currently for client CLI we use latest binaries from master which can be unstable. We should upload only stable releases to our server, not GitLab, to improve reliability.

TODO / How to implement
===

Use GCS (Google buckets):
- [x] Use private bucket for temprorary CI build and transfer of artifacts between stage
- [x] Use public bucket for storage of the latest release binaries for CLI download

Previous ideas (Nexus, not actual)
===

We can use Nexus: https://www.sonatype.com/nexus-repository-oss


- OSS

- Self-hosted

- Supports different formats:
  - Raw files (suitable for binary artifacts): https://help.sonatype.com/repomanager3/formats/raw-repositories
    
    ```
    curl -v -u ${NEXUS_USER}:${NEXUS_PWD} --upload-file dblab ${NEXUS_URL}/repository/dblab-cli/${TAG}/dblab
    wget -O dblab ${NEXUS_URL}/repository/dblab-cli/{$VERSION}/${TAG}/dblab

    For example: https://{NEXUS_URL}/repository/dblabcli/ce/0.2.3/dblab
    ```

  - Docker images: https://help.sonatype.com/repomanager3/formats/docker-registry.
    ```
    docker push <nexus-hostname>:<repository-port>/<image>:<tag>
    docker pull <nexus-hostname>:<repository-port>/<image>:<tag>
    ```

- Flexible user access(based on priveledges and roles) with an ability to grant and revoke permissions: https://help.sonatype.com/repomanager3/security

- Supports S3-compartible blob stores: https://help.sonatype.com/repomanager3/high-availability/configuring-blob-stores#ConfiguringBlobStores-AWSSimpleStorageService(S3)

But it could be too difficult and long to implement that. Try to upload artifacts to external storage.

Story points: 5.0
Ask to give a star in README, in the documentation, and in CLI (init, help)

Goal
===
Collect more gitlab stars.

TODO / How to implement
===
Ask to give a star

- [x] in README
- [ ] in the documentation
- [x] in CLI (init, help)

Acceptance criteria
===
Basic places are covered with our ask to give a star, and we are collecting stars constantly.

Story points: 1.0
Use Docker containers in create snapshot script

Goal
===
Remove Postgres dependency.

Additionally: get rid of:
```
2020-01-13 11:53:51.405 UTC [9702]: [1-1] db=,user= (,) FATAL:  could not connect to the primary server: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
```

TODO / How to implement
===

- [x] Use DBLab container 
- [x] Script remains in bash

Acceptance criteria
===

As a user I don't need git clone, and I use only docker exec to create a new snapshot.

Story points: 6.0
Clear CLI output

Goal
===
Clear and clarify CLI output

TODO / How to implement
===
The task might be more complicated than expected. Probably we have to use a custom type of errors and redeclare the default error writer. Need investigation to clarify details.

In addition, it worth to make sure that the standard flags check will not be missed after that.

- [x] add the `ERROR` label before an error message
- [x] change a CLI help message


Acceptance criteria
===

```
$ ~/database-lab/bin/dblab init --url http://localhost:3000 --token some_token --environment_id dev
2020/01/26 23:19:46 ERROR: Environment "dev" is already initialized.
```

Story points: 2.0
CI for API reference deployment

Add API references to the docs.

@akartasov do you remember in which MR you've described CLI commands?

#10 We can use Swagger as initial API reference

Generate mock from Swagger specification? Use plugin to disable "try out" functionality (https://github.com/swagger-api/swagger-ui/issues/3725)?

Deploy from database-lab or load config from gitlab.com?

Story points: 2.0
Database Lab CLI configuration

Goal
===
Single CLI can control multiple Database Lab instances

TODO / How to implement
===
Store CLI configs in the user's home directory.

Acceptance criteria
===
```
// CLI management
dblab config create ENVIRONMENT_ID
dblab config update ENVIRONMENT_ID
dblab config status ENVIRONMENT_ID
dblab config list
dblab config switch ENVIRONMENT_ID
dblab config remove ENVIRONMENT_ID

// Getting started
dblab init
```

Story points: 8.0
Design Database Lab CLI

Goal
===
Create a CLI tool for using Database Lab API.

TODO / How to implement
===
Suggest the next design of the Database Lab CLI:

```
// Clone management
dblab clone list
dblab clone create
dblab clone destroy CLONE_ID
dblab clone reset CLONE_ID
dblab clone get CLONE_ID

// Instance management
dblab instantce status

// Snapshot management
dblab snapshot list

// CLI management
dblab config create ENVIRONMENT_ID
dblab config update ENVIRONMENT_ID
dblab config view [ENVIRONMENT_ID]
dblab config list
dblab config switch ENVIRONMENT_ID
dblab config remove ENVIRONMENT_ID

// Getting started
dblab init

```

Moreover, we are going to change a compiled filename of the Database Lab server from `dblab` to `dblab-server`.
So, the `dblab` name will become free and will be used for the CLI binary.


Acceptance criteria
===

Story points: 12.0