Defining the  steps for the PostgreSQL upgrade with a restore of a backup from production using a cluster with a replica ( read-only )

The idea of this task is to create a test scenario for the PostgreSQL upgrade( pg 9.6 and 11.6 ), with a database with a size like production. 

We should have a similar hardware setup as the production boxes in these boxes.

We the idea here is to get estimative of time that we need to execute the task of the upgrade.

The cluster could have 3 nodes.



# Step by step for PG_UPGRADE(master and standby)


@adescoms @sergio.ostapowicz @Finotto 
The following  steps  are for executing `pg_upgrade` for upgrade from PG 9.6 to 11.7, 
this was tested on servers `patroni-migrate-01-db-gstg.c.gitlab-staging-1.internal` and `patroni-migrate-02-db-gstg.c.gitlab-staging-1.internal`. 

***preliminary version***

# Upgrade with pg_upgrade

Before executing the `pg_upgrade` some issue have to be resolved and are related with:

 - Server 9.6 have the extension `pg_repack` installed (solution, install pg_repack for PG 11)
 - Some views, functions, and queries from `postgres exporter` are incompatible with PG 11 (solution, like a workaround, rewrite the views and functions and add some wrapper functions for queries) 
 - Make a copy to all configs file 
 

To execute `pg_upgrade` in gitlab server you must follow the next steps: 

 1. Remove the following view and functions from PG 9.6

	   *  views : "postgres_exporter"."pg_stat_replication", "postgres_exporter"."pg_stat_wal_receiver"
       *  functions : postgres_exporter.f_select_pg_stat_replication(), public.f_pg_stat_wal_receiver()
	  
   1.1 Make sure all connections out , checkpoint in master and check replica status , 

Add to pg_hba.conf from PG 9.6 the line
  ```
host all all 0.0.0.0/0 reject
  ```

Connect to postgres database
    
```
   select pg_terminate_backend(pid) from pg_stat_activity where datname='gitlabhq_production'; --disconnecto all from gitlabhq_production database
   checkpoint ;
   select * from pg_stat_replication ;

```

 2. Stop PG 9.6 if is  running in all server

 3. Install postgresql-11 and stop service,  *ignore if is already installed and stopped*
    * apt-get install postgresql-11
    * pg_ctlcluster 11 main stop
   
 4. Create the path (*data11* or the name that you want) for the new PG cluster a give the correct permission
 ```
     mkdir /var/opt/gitlab/postgresql/data11
     chown gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql/data11
     chmod 700 /var/opt/gitlab/postgresql/data11/
```
 5. Create the new cluster database with user `gitlab-psql` in master
    ```
    su gitlab-psql
    /usr/lib/postgresql/11/bin/./initdb -D /var/opt/gitlab/postgresql/data11 --locale=C.UTF-8 --encoding=UTF8
    ```
 6. Enable to connect locally "trust" on pg_hba(/var/opt/gitlab/postgresql/data/pg_hba.conf) by adding the following line to the beginning of the file
    * local   all         all                              trust
 7. Install the pg_repack extension for PG 11
   ```
apt-get install postgresql-11-repack

--optional, compile from source code
wget http://api.pgxn.org/dist/pg_repack/1.4.5/pg_repack-1.4.5.zip
unzip pg_repack-1.4.5.zip -d ./pg_repack
cd pg_repack/pg_repack-1.4.5/
make  PG_CONFIG=/usr/lib/postgresql/11/bin/pg_config
--like a root 
make install PG_CONFIG=/usr/lib/postgresql/11/bin/pg_config

```


8. Install the postgresql 11 server on standby server and pg_repack extension
```
apt-get install postgresql-11-repack

--optional, compile from source code

wget http://api.pgxn.org/dist/pg_repack/1.4.5/pg_repack-1.4.5.zip
unzip pg_repack-1.4.5.zip -d ./pg_repack
cd pg_repack/pg_repack-1.4.5/
make  PG_CONFIG=/usr/lib/postgresql/11/bin/pg_config
--like a root 
make install PG_CONFIG=/usr/lib/postgresql/11/bin/pg_config
```
  9.	Check if all fine before execute pg_upgrade in master
   ```
   su gitlab-psql
 /usr/lib/postgresql/11/bin/pg_upgrade --old-bindir /usr/lib/postgresql/9.6/bin --new-bindir /usr/lib/postgresql/11/bin --old-datadir /var/opt/gitlab/postgresql/data --new-datadir /var/opt/gitlab/postgresql/data11/ -o "-c config_file=/var/opt/gitlab/postgresql/postgresql.conf" -O "-c config_file=/var/opt/gitlab/postgresql/data11/postgresql.conf"  --check --link
  ``` 
 10.	If the result from above is *Clusters are compatible* then execute the upgrade
```
--takes approximately 10 sec
su gitlab-psql
 /usr/lib/postgresql/11/bin/pg_upgrade --old-bindir /usr/lib/postgresql/9.6/bin --new-bindir /usr/lib/postgresql/11/bin --old-datadir /var/opt/gitlab/postgresql/data --new-datadir /var/opt/gitlab/postgresql/data11/ -o "-c config_file=/var/opt/gitlab/postgresql/postgresql.conf" -O "-c config_file=/var/opt/gitlab/postgresql/data11/postgresql.conf"  --link
  ``` 	
 11. If the result from above is *Upgrade Complete* then test start PG 11 , make sure the config are correct or with the values that you need and test start/stop PG service
 ```
su gitlab-psql
 cd /usr/lib/postgresql/11/bin/ 
./postgres -D /var/opt/gitlab/postgresql/data11 -c "config_file=/var/opt/gitlab/postgresql/data11/postgresql.conf"

stop server
  ``` 	

### *Upgrade standby replicas*
12. Create the structures for data 11 on replicas 
```
mkdir /var/opt/gitlab/postgresql/data11
chown gitlab-psql:gitlab-psql /var/opt/gitlab/postgresql/data11
chmod 700 /var/opt/gitlab/postgresql/data11/
```
13. Copy to standby server the data for PG 11 and delta from PG 9.6, configure the recovery.conf to connecto to master PG 11
```
cd  /var/opt/gitlab/postgresql
rsync --archive --delete --hard-links --size-only --no-inc-recursive  data data11 10.224.42.102:/var/opt/gitlab/postgresql
```  


14. Config the replica setting if is necessary 
15. Confing standby ,  Start master  and standby server
recovery.conf
```
recovery_target_timeline = 'latest'
standby_mode = 'on'
primary_conninfo = 'user=gitlab-replicator password=password host=IP port=5432  application_name=your_name'
primary_slot_name = 'your_slot_name'
```

16. Create the replication slot in master
```
 select pg_create_physical_replication_slot('your_slot_name');
 ``` 
 
 
17. Connect to postgres on Master server, and create the functions and views required by `postgres exporter`
``` 
sudo gitlab-psql 
--and execute the code
CREATE OR REPLACE FUNCTION postgres_exporter.f_select_pg_stat_replication()
  RETURNS SETOF pg_catalog.pg_stat_replication                                         
  LANGUAGE sql                                                              
  SECURITY DEFINER                                                          
 AS $BODY$                                                             
   SELECT * from pg_catalog.pg_stat_replication;                            
 $BODY$;

CREATE VIEW "postgres_exporter"."pg_stat_replication" AS
 SELECT *
   FROM "postgres_exporter"."f_select_pg_stat_replication"() ;
   
 CREATE OR REPLACE FUNCTION public.f_pg_stat_wal_receiver()          
  RETURNS SETOF pg_stat_wal_receiver                                 
  LANGUAGE sql                                                       
  SECURITY DEFINER                                                   
 AS $BODY$
 select * from pg_catalog.pg_stat_wal_receiver
 $BODY$;

CREATE VIEW "postgres_exporter"."pg_stat_wal_receiver" AS
 SELECT *
   FROM "public"."f_pg_stat_wal_receiver"() ;
  
GRANT ALL PRIVILEGES ON postgres_exporter.pg_stat_replication TO postgres_exporter;
GRANT ALL PRIVILEGES ON postgres_exporter.pg_stat_wal_receiver TO postgres_exporter;

CREATE OR REPLACE FUNCTION public.pg_last_xlog_replay_location()
 RETURNS pg_lsn AS
 $BODY$
 SELECT pg_last_wal_replay_lsn();
 $BODY$
 LANGUAGE SQL STABLE;

CREATE OR REPLACE FUNCTION public.pg_current_xlog_insert_location()
 RETURNS pg_lsn AS
 $BODY$
 SELECT  pg_current_wal_insert_lsn();
 $BODY$
 LANGUAGE SQL STABLE;

CREATE OR REPLACE FUNCTION public.pg_current_xlog_location() 
 RETURNS pg_lsn AS $$
 SELECT pg_current_wal_lsn();
 $$ LANGUAGE SQL STABLE;

CREATE OR REPLACE FUNCTION public.pg_xlogfile_name(pg_lsn) 
 RETURNS text AS $$
 SELECT pg_walfile_name($1);
 $$ LANGUAGE SQL STABLE;

``` 
 
 18. To collect statistics 
     *  Add the temporaly line to pg_hba.conf (/var/opt/gitlab/postgresql/data/pg_hba.conf) and reload conf  with `select pg_reload_conf()` : host   gitlabhq_production       gitlab-superuser        127.0.0.1/0        trust
     * Execute the vaccumdb binaries with the following parameters: 
      ```
        --takes approximately 4 min
       /usr/lib/postgresql/11/bin/vacuumdb  --analyze-only --jobs 64 -d gitlabhq_production -U gitlab-superuser -h 127.0.0.1
       ```
    
     * Remove  or comment the lines from pg_hba (/var/opt/gitlab/postgresql/data/pg_hba.conf) and reload conf with `select pg_reload_conf()` on master server:
         * host all all 0.0.0.0/0 reject
         * local   all         all                              trust
         * host   gitlabhq_production       gitlab-superuser        127.0.0.1/0        trust  
 




19. *Integrate PG 11 with HA solution Patroni*

Story points: 8.0