giseldo/neodataset huggingface dataset:
Refactor output of service containers

Goal
===
Since we use CSV logs by default, we cannot rely on Docker logs:
- rewrite health checks for:
  - starting a sync instance on physical restore
- retrieve important output of commands:
   - error logs on fail
   - `InspectCommandResponse` and `ExecCommand` for operations while physical restore: 
      - running Postgres
      - promotion Postgres
      - restore command
      - chown pgdata

TODO / How to implement
===
- use `pg_isready` for health checks
- forward output and errors


Acceptance criteria
===

Story points: 6.0
Show human-readable GiB/MiB/... numbers in API responses

Goal
===
Right now, we show disk usage, DB size, cloneDiffSize, etc in bytes, such as:

```
"cloneDiffSize": 4792320
```

For humans, it is much more convenient to see it in human-readable form.

Especially it makes sense for CLI.

TODO / How to implement
===
In addition to raw numbers, show human-readable string values (with TiB / GiB / MiB / KiB)


Acceptance criteria
===
As an API or CLI user, I can quickly understand the disk usage, object sizes.

Story points: 1.0
Report DLE version in API responses

Goal
===
As an API or CLI user, I want to know the DLE version.

TODO / How to implement
===

- create a new CLI command `dblab instance version`
- make a request `GET /healthz` from CLI

```
dblab instance version
{
    "version": "2.0.3-1-ga9f501c-20201119-0417"
}

```


Acceptance criteria
===
Working with DLE, I can always learn what DLE version it is.

Story points: 1.0
When DLE is restarted, do not lose clones

Goal
===
It's quite inconvenient that we lose all the close when restart is needed.

A restart is needed when:
- we need to change the config
- during upgrades

A restart might also happen unexpectedly (e.g., the machine rebooted).

In all these cases, we risk losing some important work. That's why thin clones need to survive DLE restarts (and machine reboots).

TODO / How to implement
===

- reload configuration https://gitlab.com/postgres-ai/database-lab/-/issues/134
- upgrade:
  - choose serialization format - 2
  - handle an incoming signal - 2
  - back up the state of running instances to file(?) on SIGUSR1 - 6
    - InstanceID
    - clone wrapper list (including clones, sessions)
    - instance status
    - port pool
  - restore the state of backup instances - 12
    - read serialized data
    - initialize the state
  - handle running processes inside DLE - 4
  - add a configuration option to ignore restore file - 1
  - merge existing configuration parameters with new ones - 6

Acceptance criteria
===
As a DLE admin, I can restart DLE not losing clones, so people do not lose their work.

Story points: 33.0
On engine start, if there are some snapshots already, skip creation of the initial snapshot

Goal
===
When we start/restart the engine, if we have some snapshots already, it doesn't make sense to create another one immediately (and wait many minutes till opening the 2345 port). We can simply start working, and create snapshots on schedule.

However, there is a question: what if we **need** to create a new snapshot? There is no API handler yet (related: https://gitlab.com/postgres-ai/database-lab/-/issues/65), and there are no other means for the DLE admin to initiate snapshot creation. -- Let's understand what to do with this before implementing this issue.

TODO / How to implement
===

- disable creation of the initial snapshot by default
- create the initial snapshot only if there are no available snapshots

Acceptance criteria
===
As a DLE admin, I can start DLE much quicker (initial snapshot is not created if there are existing snapshots already).

Story points: 1.0
Review default sync instance (and other Postgres containers) configuration

Goal
===
To make sync instance running is more reliable and observable, including but not limited to:

- all log_*** (e.g., we need `log_checkpoints = on`, autovacuum logging)
- set `hot_standby = 'on'` explicitly (because it's off in 9.6 and older)

For clones, we also need `track_io_timing = 'on'`.

TODO / How to implement
===

- [x] Review and correct default configuration files for a sync instance: https://gitlab.com/postgres-ai/database-lab/-/tree/master/configs/postgres
- [x] Make a configuration plan for each stage (how, which, and where we can configure a PostgreSQL instance):
   - Default adjustment
   - Sync Instance
   - Promote instance
   - Database snapshot
   - User clone
- [x] Provide an ability to configure `recovery.conf`
- [x] Provide an ability to configure sync and promotion instances
- [x] Create a configuration manager to make easy customization for each stage, use include directives
   ```
   include_if_exists postgresql.dblab.postgresql.conf
   include_if_exists postgresql.dblab.pg_control.conf
   include_if_exists postgresql.dblab.sync.conf
   include_if_exists postgresql.dblab.promotion.conf
   include_if_exists postgresql.dblab.snapshot.conf
   include_if_exists postgresql.dblab.user_defined.conf
   ```
- [x] Config management optimization(do not override basic configs if they are already initialized)


Acceptance criteria
===

Story points: 10.0
CI observer: Allow regexp filtering for Postgres logs before sending to the Platform

Goal
===
As a DLE admin, I can set up regexp rules for Postgres logs. These rules are applied *before* sending the logs to the Platform, to ensure that personal data is masked properly.

TODO / How to implement
===
- Extend database Lab configuration
   ```yaml
   observer:
     replacementRules: 
       "regex": "replace"
   ```
- Build a map of processing fields 
   ```
   Processed Fields:
      - message
      - detail
      - hint
      - internal_query
      - query
   ```
- Replace matched strings

   Regex syntax: https://github.com/google/re2/wiki/Syntax
- Add regex tests
- Add docs


Acceptance criteria
===
As a DLE admin, I can set up rules to avoid sending personal log-data (Postgres logs) to the Platform.

Story points: 8.0
Improve preprocessing script experience

Goal
===
For now, it's quite difficult to run SQL-command in a preprocessing script because the PostgreSQL instance is not running

TODO / How to implement
===
Allow users to run a preprocessing script during the promotion stage when PostgreSQL is running and ready to receive commands

- I am able to execute SQL scripts without thinking of where to connect
- I am able to execute Bash scripts too
- 
I am able to use env variables inside SQL scripts

- Add docs
- Parallelization: alphabet order; if scripts are placed in a folder, run them simultaneously. For example,
   ```
   - abc.sql
   - init.sql
   parallel/
     - example.sql
     - new_script.sql
     - run.sql
   - test.sql
   - wrap.sql
   ```


### Example
```yaml
    physicalSnapshot:
      options:
...
        # Promote PGDATA after data fetching.
        promotion:
...
          # It is possible to define pre-precessing SQL queries. For example, "/tmp/scripts/sql".
          # Default: empty string (no pre-processing defined).
          queryPreprocessing:
            # Path to SQL pre-processing queries.
            queryPath: "/sql-scripts"

            # Worker limit for parallel queries.
            maxParallelWorkers: 2
```
Acceptance criteria
===

Story points: 10.0
DLE API for automatic checks of DB migrations

Goal
===
Create an API to provide automatic checks of DB migrations

TODO / How to implement
===
`database-lab`: `observer-start`/`observer-finish` using API, async (16h)
* `observer-start` - starts dblab session and starts observing 3h
  * create session on the platform
  * session ID generated on the side of dblab (\~UUID)
* `observer-finish` - finished dblab session, makes summary, reports to the Platform 10h
  * change clone status (OK -> EXPORTING -> OK)
  * update session on the platform (upload meta)
  * upload logs (single request, what is file size limit :question:; multipart upload/gzip on NGINX)
* do not idle delete clone in EXPORTING status 1h
* CLI, API initiate the processes that is being executed on DLE side, async 2h

https://gitlab.com/postgres-ai/database-lab/-/issues/171#note_424320002

Acceptance criteria
===

Story points: 16.0
Synchronize max_connections parameter on restore

Goal
===
Synchronize `max_connections` parameter to avoid errors on start:
```
2020/09/29 03:59:37 [INFO]   Running container: dblab_phr_btpb29m7graql4bukts0. ID: 62db8684e45f2dce119552f7aba9b123a1229e96a56901a7508a86176e914eda
2020/09/29 03:59:37 [INFO]   Running restore command:  pgbackrest --process-max=4 --type=immediate restore
2020/09/29 04:01:03 [INFO]   Restoring job has been finished
2020/09/29 04:01:03 [DEBUG]  Configuring Postgres...
2020/09/29 04:01:04 [INFO]   Running refresh command
k2020-09-29 11:01:04.661 +07 [32]: [1-1] db=,user= (,) LOG:  listening on IPv4 address "0.0.0.0", port 5432
�2020-09-29 11:01:04.663 +07 [32]: [2-1] db=,user= (,) LOG:  could not create IPv6 socket for address "::": Address family not supported by protocol
y2020-09-29 11:01:04.664 +07 [32]: [3-1] db=,user= (,) LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
�2020-09-29 11:01:04.906 +07 [39]: [1-1] db=,user= (,) LOG:  database system was interrupted; last known up at 2020-09-29 01:21:08 +07
y2020-09-29 11:01:05.037 +07 [39]: [2-1] db=,user= (,) LOG:  starting point-in-time recovery to earliest consistent point
n2020-09-29 11:01:05.052 +07 [39]: [3-1] db=,user= (,) LOG:  restored log file "00000024.history" from archive
v2020-09-29 11:01:05.128 +07 [39]: [4-1] db=,user= (,) LOG:  restored log file "0000002400000149000000DB" from archive
�2020-09-29 11:01:05.149 +07 [39]: [5-1] db=,user= (,) FATAL:  hot standby is not possible because max_connections = 300 is a lower setting than on the master server (its value was 2000)
�2020-09-29 11:01:05.150 +07 [32]: [4-1] db=,user= (,) LOG:  startup process (PID 39) exited with exit code 1
2020-09-29 11:01:05.150 +07 [32]: [5-1] db=,user= (,) LOG:  aborting startup due to startup process failure
Y2020-09-29 11:01:05.175 +07 [32]: [6-1] db=,user= (,) LOG:  database system is shut down
2020/09/29 04:01:05 [INFO]   Removing container ID: 62db8684e45f2dce119552f7aba9b123a1229e96a56901a7508a86176e914eda
2020/09/29 04:02:05 [INFO]   Container "62db8684e45f2dce119552f7aba9b123a1229e96a56901a7508a86176e914eda" has been removed
2020/09/29 04:02:05 [FATAL]  Failed to run the data retrieval service: failed to refresh data: database instance is not running
```

TODO / How to implement
===
Before Postgres starts (on a physical restore step), retrieve the `max_connections` parameter and set it to the configuration file.
Use `pg_controldata`:
```bash
$ sudo -u postgres /usr/lib/postgresql/11/bin/pg_controldata -D /var/lib/postgresql/11/main | grep max_connections
max_connections setting:              100
```


Acceptance criteria
===

Story points: 3.0
Support PostgreSQL 13

Goal
===
Support PostgreSQL which is to be released on September 24.

TODO / How to implement
===
- [ ] Carefully read release notes: https://www.postgresql.org/docs/release/13.0/
- [-] 
Prepare extended Postgres image https://gitlab.com/postgres-ai/custom-images/-/issues/2

- [x] Carefully test
    - [ ] physical mode
    - [ ] logical mode
- [ ] add tests for 13
- [-] 
Pay attention to changes in `pg_stat_***` – there are new columns, and we may benefit from having them

- [-] 
Consider using `pg_stat_progress_basebackup` to show progress of running `pg_basebackup` for physical initialization mode https://www.postgresql.org/docs/13/progress-reporting.html#BASEBACKUP-PROGRESS-REPORTING


Acceptance criteria
===
Database Lab Engine can be used for Postgres 13.

Story points: 5.0
Full refresh without losing clones

Goal
===
Allow performing full refresh in "logical" provisioning mode, without interruptions, not losing the existing clones and snapshots.

TODO / How to implement
===


- [x] reorganize FS configuration
   - create mountpoint configuration structs
- [x] refactor FS pool managers
   - create zfs and lvm filesystem managers
- [x] retrieval pool rotation
   - detect the next pool to update
   - pool switching
   - reload service configuration
- [x] running an actual retrieval chain
   - reproduce retrieval job execution
- [x] running a refresh scheduler
   - create a timetable
   - cancel nested snapshot schedulers
- [x] cloning with different fs pools
   - provide fs managed pools to the cloning service
   - adopt cloning operations
- [x] idle timeouts
   - detect clone idleness for all managed pools

----
The problem is that normally, we refresh the instance fully. To do that, we need to erase PGDATA.

A possible solution would be using a separate volume / ZFS pool (most convenient in clouds: EBS volume on AWS, PD disk on GCP, etc.), and performing initialization of PGDATA there, not interrupting the work of the existing Database Lab Engine.

Next, once new PGDATA is initialized, we have options/ideas: 
- straightforward reload. Cons: disk space overhead. We can try to use ZFS deduplication in order to avoid the negative effect. Steps:
   - remove existing PGDATA
   - load new PGDATA
   - create a new snapshot

- rsync or copy the new data to the "main" disk overwriting the existing data, and create a new snapshot (there are risks that deviation will be so significant in terms "delta" of data blocks, that it will "eat" disk space significantly -- thorough testing is needed); This solution is based on assumption that oids do not change. If the schema has changed, fully refresh of PGDATA might be required

- use "baseline" snapshot to reduce the delta:
    - when performing a fresh setup, create an "initial" ZFS snapshot, when there is no PGDATA yet,
    - always keep this snapshot,
    - before fetching a fresh PGDATA to a new volume, mirror from that snapshot first somehow (using a temporary clone on the "main" zpool, or using `zfs send)
    - once new PGDATA on a temporary zpool is ready, mirror it back to the "main zpool",
    - then create a new snapshot on the main zpool,
    - destroy a temporary zpool / disk / volume.

For clouds (e.g., in the case of AWS / RDS), it is important to implement an automated provision of a temporary disk (AWS: EBS volume), which needs to be auto-deleted at the end of the refresh operation.

Again, first, thorough testing is required to explore the possibilities and resolve possible issues with the approach described.

Acceptance criteria
===
As a Database Lab Engine administrator, I can set up a workflow in which periodical refreshing in "logical" provisioning mode does not lead of temporary downtime and losing the existing clones – I just need an additional disk space (in clouds – temporarily).

Story points: 42.0
logical provisioning: support  pg_dump_anon

Goal
===
We need to be able to use an alternative to pg_dump, for example [pg_dump_anon](https://postgresql-anonymizer.readthedocs.io/en/stable/anonymous_dumps/)

TODO / How to implement
===
We already have postgres_anonymizer in our extended docker image.

But pg_dump is hard-coded. Allow substituting it by pg_dump_anon.

pg_dump_anon doesn't support `-j`, and it supports only `plain` format right now.

Acceptance criteria
===
I'm able to refresh Database Lab from production or another Database Lab with configured masking based on postgres_anonymizer, so no PII reaches my non-production environments.

When testing: ensure that some masking is working: define a masking rule for some column/username pair, see https://postgresql-anonymizer.readthedocs.io/en/stable/dynamic_masking/ -- on the destination, we must see the masked value.

Story points: 4.0
Postgres config management in the case of physical data retrieval

Goal
===
Currently, during physical data retrieval, we rely on postgresql.conf that comes with backup or is brought by pg_basebackup. This is not reliable because such a config may contain various unexpected actions.

We aim to control postgresql.conf fully. (And later, in a separate issue, pg_hba.conf as well).

For that, we want to start from a default postgresql.conf always, and apply transformations needed for Database Lab's work.

TODO / How to implement
===
- [ ] always start with the default postgresql.conf. We need to have them as a starting point. To get it, use `initdb -D /path`.
- [ ] set shared_buffers to some small value that will be enough for maintenance tasks and sync instance activity (should be configurable in dblab config; `100MB` should be enough)
- [ ] apply adjustments that user configures in the main dblab config – we'll ask to provide 3 things:
    - [ ] `work_mem` and all the planner settings such as `effective_cache_size`. We'll ask user to get it using the following query with regexp:
    ```sql
    select format($$%s = '%s'$$, name, setting)
    from pg_settings
    where name ~ '(work_mem$|^enable_|_cost$|scan_size$|effective_cache_size|^jit|^geqo|default_statistics_target|constraint_exclusion|cursor_tuple_fraction|collapse_limit$|parallel|plan_cache_mode)';
    ```
    - [ ] apply `shared_preload_libraries` that user configured in the main dblab config (default is `pg_stat_statements`, its our minimum)
    - [ ] apply extensions-related params, if any
- [ ] take care of `log_***` settings

Acceptance criteria
===
As a user, I know what I need to provide in the main dblab config to configure and use Database Lab with automated "physical" data retrieval. This process doesn't depend on the fact if or if not `postgresql.conf` is prese in PGDATA on the source.

Story points: 3.0
Automatic setup: snapshot schedule and retention policy

Goal
===
Configure snapshot creation schedule and retention policy (autodeletion).

TODO / How to implement
===
- create a new `snapshot` stage
- provide stage configuration. For example,
    ```yaml
    snapshot:
      jobs:
        - name: snapshotting
          options:
            schedule: "0 */6 * * *"
            retention:
              snapshotLimit: 4
    ```
- snapshot creation 
  - parse schedule
  - run a task to create a snapshot
- autodeletion - periodically run a task to remove old snapshots. Check `scripts/cleanup_zfs_snapshot.sh`
  - detect old snapshots
  - remove detected snapshots

Acceptance criteria
===

Story points: 12.0
Sync instance: add a separate stage and configuration

Goal
===
Run sync instance in order to keep data from physical backup up-to-date

TODO / How to implement
===
- options:
    - arbitrary `restore_command` or `primary_conninfo`
    - use WAL-G for the first step
- integrate as an option to existing stages
- prepare stage configuration(docker images, credentials, volumes)

Acceptance criteria
===

Story points: 8.0
Automated setup: always recalculate statistics when restored from a dump

Goal
===
When we use "logical" data provisioning, restoring from a dump, initially, there are no actual stats in the database. As the last step for the restore job, there should be stats recalculation.

Without stats, the optimizer chooses sub-optimal plans.

TODO / How to implement
===
After running pg_restore, always run `vacuum analyze freeze;` in the target database.

Acceptance criteria
===
As us user, using "logical" data retrieval approach (for example, for an RDS database as a source), I get the result with up-to-date stats, so the performance, the SQL execution plans are good.

Story points: 2.0
Automated setup: extend "extended-postgres" – more Postgres extensions

Related: https://gitlab.com/postgres-ai/database-lab/-/issues/138

Goal
===
Add most of all popular Postgres extensions to `extended-postgres`.

TODO / How to implement
===
- [x] define and review the list of the most popular extensions @NikolayS @fomin.list :four: 
    - RDS and others
    - github activity
    - search: any ratings?
    - we expect ~20-30
- [x] extend "extended-postgres" :ten:
    - most are installed via `apt-get install` or `apk add` (if -alpine)
    - we must support 9.6, 10, 11, 12
    - some require compilation (should we take them at all?) -- 80/20 rule
- [x] ensure it's published on Docker Hub https://hub.docker.com/r/postgresai/extended-postgres
- [ ] docs: describe the list of supported Postgres extensions :two: 

Acceptance criteria
===
As a user, I don't do any additional actions to launch my first Database Lab for my real database (extensions I use are already supported by Database Lab, in 99%).

Story points: 16.0
Automated setup: basic Postgres configuration (physical & logical)

Goal
===
Simplify configuration when setting up Database Lab.

TODO / How to implement
===
- [x] first, we take the default `postgresql.conf`
- [x] MAIN: `shared_buffers`: 200M for sync instance and 1/N (or constant; pre-defined in config) of RAM for clones 
- [x] MAIN: override all params required for Database Lab: logging connections, etc
- [x] planner params, `effective_cache_size`, to match production -- we support changing it right in the main Database Lab configuration file (but we don't edit them automatically)
- [x] pg_hba.conf - only changes required for Database Lab – same principle (allow to extend it)
- [x] review what's happening to config during snapshotting


### Config suggestions

from prod:
```
# Use values from the source (production) to make the planner work similarly
effective_cache_size = ..
seq_page_cost = ..
random_page_cost = ..
work_mem = ...
maintenance_work_mem = ...
# ... and the other 'planner' settings, see https://www.postgresql.org/docs/current/runtime-config-query.html
```

this we need to enforce (same as  log_***)
```
min_wal_size = '1GB'
max_wal_size = '16GB'
checkpoint_timeout = '15min'
checkpoint_completion_target = 0.9
```

For the docs
```
select format($$%s = '%s'$$, name, setting)
from pg_settings
where name ~ '(work_mem$|^enable_|_cost$|scan_size$|effective_cache_size|^jit|^geqo|default_statistics_target|constraint_exclusion|cursor_tuple_fraction|collapse_limit$|parallel|plan_cache_mode)';
```

Acceptance criteria
===
As a user, I just edit the main config (paying attention to: `shared_buffers`), launch Database Lab, and it works, without requiring to change any config params.

Story points: 10.0
Automated setup: create the first snapshot (physical case)

Goal
===
Right after retrieval/refresh of PGDATA the very 1st snapshot should be created automatically. :warning: This issue is only about physical retrieval.

TODO / How to implement
===
- [x] adjust pre-snapshot configs
- [x] promote database
- [x] keep metadata to a file
- [x] customize a snapshot script 
- [x] add an ability for custom pre-processing

Acceptance criteria
===
TBD

Related #139

Story points: 14.0
Synchronization or port allocation statuses.

Goal
===
Port may be allocated twice, resulting in an error

TODO / How to implement
===
Don't allocate the same port simultaneously. It's necessary to add a synchronization mechanism.

There is an issue in the StartSession function(pkg/services/provision/mode_local.go):

```
	// TODO(anatoly): Synchronization or port allocation statuses.
	port, err := j.getFreePort()
	if err != nil {
		return nil, errors.New("failed to get a free port")
	}
...
	err = j.setPort(port, true)
	if err != nil {
		return nil, errors.Wrap(err, "failed to set a port")
	}
```
Acceptance criteria
===

Story points: 2.0
API returns the list of clones without specific order (was: "UI: Order list of clones")

Goal
===
Currently, the list of clones on the page of DB Lab instance (https://postgres.ai/console/XXXX/instances/XXXX) is without a specific order. This is inconvenient. We need a predefined order.

TODO / How to implement
===
- This must be implemented at the Database Lab API level. `instance status` and `clone list` – they both have to return the list of clones ordered by creation time, desc (the newest is the 1st).


Acceptance criteria
===
As a user, if I refresh the page, the list of clones doesn't change the order.

Story points: 2.0