Repository: apache/airavata-custos (language: python)

Issue:
Writing ansible scripts to deploy Custos
As part of the Custos baremetal deployment effort - we need to write ansible scripts to configure and automate deployments.
Writing ansible scripts to deploy Custos
As part of the Custos baremetal deployment effort - we need to write ansible scripts to configure and automate deployments.
Writing ansible scripts to deploy Custos
As part of the Custos baremetal deployment effort - we need to write ansible scripts to configure and automate deployments.


Discussion / hints:

Fix patch:
diff --git a/ansible/README.md b/ansible/README.md
index 58eb0432f..6b653128e 100644
--- a/ansible/README.md
+++ b/ansible/README.md
@@ -49,45 +49,52 @@ You will need 3 Linux VMs (Preferably Ubuntu 20.04) for Custos Deployment. Make
 ### Configure Ansible vars with VM details
 Now, you need to configure Ansible vars with the domain names of these VMs so that ansible knows where to install custos dependencies.
 
-Open the file `test/group_vars/all/vars.yml`. Look for `keycloak_domain`, `hashicorp_domain` and `custos_domain` variables. Set these variables with their respective domain names. For ex:
+Open the file `test/hosts.yml`. Set `ansible_host` for `custos`, `keycloak` and `hashicorp` with their respective domain names. For ex:
+
 ```yml
-⋮
-# Domain names for VMs
-keycloak_domain: my-keycloak-domain.com
-hashicorp_domain: my-hashicorp-domain.com
-custos_domain: my-custos-domain.com
-⋮
+all:
+  hosts:
+    custos:
+      ansible_host: my-custos-domain.com
+      ⋮
+    keycloak:
+      ansible_host: my-keycloak-domain.com
+      ⋮
+    hashicorp:
+      ansible_host: my-hashicorp-domain.com
+      ⋮
 ```
-Ansible connects to these VMs using `ssh`. To allow ansible to do this, you will also need to provide it with ssh passwords for these VMs. To maintain security, these passwords cannot be exposed in plaintext. You can make use of ansible's `ansible-vault` command to encrypt these password strings.
+Ansible connects to these VMs using `ssh`. To allow ansible to do this, you will also need to provide it with ssh passwords for these VMs. To maintain security, these passwords cannot be exposed in plaintext. We use ansible's `ansible-vault` command to encrypt these password strings.
 
-Use the following command to encrypt any string:
+Open the file `test/group_vars/all/vault.yml`. You will need to decrypt the file to access it (it's been encrypted using ansible-vault for security).
+Here's the command:
 ```bash
-ansible-vault encrypt_string "<mysecurepassword>" --ask-vault-pass
+ansible-vault decrypt inventories/test/group_vars/all/vault.yml
 ```
-Ansible will ask you to set up a new vault password everytime you run this command. Make sure to use the same password everytime.
+You will be prompted to provide the `ansible-vault` password. You can reach out to the dev team to get the password for test env.
+
+After you've set the respective passwords, don't forget to encrypot the 
 
-In the file `test/group_vars/all/vars.yml`. Look for `keycloak_ssh_password`, `hashicorp_ssh_password` and `custos_ssh_password` variables. Set these variables with their respective encrypted password strings. For ex:
+In the file `test/group_vars/all/vault.yml`. Look for `keycloak_ssh_password`, `hashicorp_ssh_password` and `custos_ssh_password` variables. Set these variables with their respective password strings. For ex:
 ```yml
 ⋮
-keycloak_ssh_password: !vault |
-  $ANSIBLE_VAULT;1.1;AES256
-  36363039363662616335356566613665346161396138303131616334616666623361633765356434
-  3234326462633763636...43161313064
-hashicorp_ssh_password: !vault |
-  $ANSIBLE_VAULT;1.1;AES256
-  62633531346536643234303565373639626464326135653732323266396335353166346132383230
-  3333316665623861326...462353738656662
-custos_ssh_password: !vault |
-  $ANSIBLE_VAULT;1.1;AES256
-  643562326362326461334303565373639626464326135653732323266396335353166346132383230
-  3333316665623861326...065626463633330
+keycloak_ssh_password: XXX..XXX
+hashicorp_ssh_password: YYY...YYY
+custos_ssh_password: ZZZ...ZZZ
 ⋮
 ```
+
+Anytime you change the `vault.yml` file, don't forget to encrypt it back again.
+Here's the command to encrypt:
+```bash
+ansible-vault encrypt inventories/test/group_vars/all/vault.yml
+```
+
 Now that you've configured the ssh user and password details for your VMs, we can install nginx on each of these servers at once by running the following command:
 ```bash
-ansible-playbook -i inventories/test/ env_setup.yml --ask-vault-pass
+ansible-playbook -i inventories/test/ custos.yml --ask-vault-pass --tags env_setup
 ```
-Enter the Vault Password (that you set up while encrypting your ssh password strings) when prompted by ansible.
+Enter the Vault Password (that you got from airavata dev) when prompted by ansible.
 
 ## Generate SSL certificates<a name="generate-ssl-certificates"/>
 The `env_setup.yml` playbook installed `certbot` along with `nginx` in your VMs. **Certbot** helps manage letsencrypt SSL certificates. You need to generate SSL certs for each of your 3 VMs.
@@ -116,18 +123,12 @@ openssl pkcs12 -export -out certs/keycloak-client-truststore.pkcs12 -inkey /etc/
 ```
 Now, you need to add the pkcs12 passwords in the ansible vars.
 
-Encrypt the passwords using `ansible-vault` just like you did for the ssh passwords. Make sure you still use the same vault password as before. In the file `test/group_vars/all/vars.yml`. Look for `hashicorp_pkcs12_passphrase` and `keycloak_pkcs12_passphrase` variables. Set these variables with their respective encrypted password strings. For ex:
+In the file `test/group_vars/all/vault.yml`. Look for `hashicorp_pkcs12_passphrase` and `keycloak_pkcs12_passphrase` variables. Set these variables with their respective password strings. For ex:
 ```yml
 ⋮
 # pkcs12 passwords
-hashicorp_pkcs12_passphrase: !vault |
-  $ANSIBLE_VAULT;1.1;AES256
-  65646665313232613265363038633662393438376265663363333434303361656565653763313539
-  393735653837326...537
-keycloak_pkcs12_passphrase: !vault |
-  $ANSIBLE_VAULT;1.1;AES256
-  63303962613033353165646639633338306538666233363735393266386432376663656536633663
-  62313163333631...831303961
+hashicorp_pkcs12_passphrase: XXX...XXX
+keycloak_pkcs12_passphrase: YYY...YYY
 ⋮
 ```
 
diff --git a/ansible/custos.yml b/ansible/custos.yml
index 9a2725bd9..ce59e6d49 100644
--- a/ansible/custos.yml
+++ b/ansible/custos.yml
@@ -28,6 +28,9 @@
   roles:
     - nginx
     - keycloak
+- hosts: old_server
+  roles:
+    - migrate_db
 - hosts: custos
   roles:
     - nginx
diff --git a/ansible/roles/database/tasks/create_db.yml b/ansible/roles/database/tasks/import_db.yml
similarity index 73%
rename from ansible/roles/database/tasks/create_db.yml
rename to ansible/roles/database/tasks/import_db.yml
index cd49503bf..3f559b8be 100644
--- a/ansible/roles/database/tasks/create_db.yml
+++ b/ansible/roles/database/tasks/import_db.yml
@@ -25,4 +25,15 @@
     name: "{{ db_name }}"
     state: present
     encoding: "UTF8mb3"
-    collation: "utf8_general_ci"
\ No newline at end of file
+    collation: "utf8_general_ci"
+
+- name: Copy dump file from master node to new server
+  copy:
+    src: tmp/core_services_server_db_arch.sql
+    dest: /home/exouser/backups
+
+- name: import dump file to mysql
+  mysql_db:
+    name: core_services_server_db
+    state: import
+    target: /home/exouser/backups/core_services_server_db_arch.sql
diff --git a/ansible/roles/database/tasks/main.yml b/ansible/roles/database/tasks/main.yml
index 8cc53f76b..62915f80c 100644
--- a/ansible/roles/database/tasks/main.yml
+++ b/ansible/roles/database/tasks/main.yml
@@ -32,43 +32,6 @@
     state: started
     enabled: yes
 
-# 'localhost' needs to be the last item for idempotency, see
-# http://ansible.cc/docs/modules.html#mysql-user
-#- name: update mysql root password for all root accounts
-#  mysql_user: name=root host={{ item }} password={{ custos_core_spring_datasource_password }}
-#  with_items:
-#    - "{{ ansible_hostname }}"
-#    - 127.0.0.1
-#    - ::1
-#    - localhost
 - include: secure_install.yml
-- include: create_db.yml
-#
-#- name: copy .my.cnf file with root password credentials
-##  template: src=templates/root/my.cnf.j2 dest=/root/.my.cnf owner=root mode=0600
-#  copy:
-#    src: templates/my.cnf.j2
-#    dest: /root/.my.cnf
-#    owner: root
-#
-#- name: delete anonymous MySQL server user for $server_hostname
-#  action: mysql_user user="" host="{{ server_hostname }}" state="absent"
-#
-#- name: delete anonymous MySQL server user for localhost
-#  action: mysql_user user="" state="absent"
-#
-#- name: remove the MySQL test database
-#  action: mysql_db db=test state=absent
 
-#- name: Create dynamic MySQL variables.
-#  set_fact:
-#    mysql_users:
-#      - name: mycompany_user
-#        host: "{{groups['database'][0]}}"
-#        password: secret
-#        priv: "*.*:SELECT"
-#      - name: mycompany_user
-#        host: "{{groups['database'][1]}}"
-#        password: secret
-#        priv: "*.*:SELECT"
-#    mysql_replication_master: "{{groups['a4d.lamp.db.1'][0]}}"
+- include: import_db.yml
\ No newline at end of file
diff --git a/ansible/roles/env_setup/tasks/main.yml b/ansible/roles/migrate_db/tasks/main.yml
similarity index 83%
rename from ansible/roles/env_setup/tasks/main.yml
rename to ansible/roles/migrate_db/tasks/main.yml
index 68807666f..5904c2d92 100644
--- a/ansible/roles/env_setup/tasks/main.yml
+++ b/ansible/roles/migrate_db/tasks/main.yml
@@ -19,12 +19,8 @@
 #
 
 ---
-- hosts: hashicorp
-  roles:
-    - nginx
-- hosts: keycloak
-  roles:
-    - nginx
-- hosts: custos
-  roles:
-    - nginx
\ No newline at end of file
+- name: Copy dump file from old server to master node
+  fetch:
+    src: /home/ubuntu/temp_backup/core_services_server_db_arch.sql
+    dest: tmp/
+    flat: yes
\ No newline at end of file
diff --git a/ansible/roles/nginx/tasks/main.yml b/ansible/roles/nginx/tasks/main.yml
index 7c5fa1544..c675b7e08 100644
--- a/ansible/roles/nginx/tasks/main.yml
+++ b/ansible/roles/nginx/tasks/main.yml
@@ -24,12 +24,14 @@
     update_cache: yes
     cache_valid_time: 86400
   become: yes
+  tags: env_setup
 
 - name: install nginx
   apt:
     name: nginx
     state: latest
   become: yes
+  tags: env_setup
 
 - name: allow OpenSSH
   ufw:
@@ -37,6 +39,7 @@
     name: "OpenSSH"
     state: enabled
   become: yes
+  tags: env_setup
 
 # It is recommended to enable the most restrictive profile
 #  that will still allow the traffic you’ve configured.
@@ -48,6 +51,7 @@
 #    delete: yes
     state: enabled # enable the ufw rule
   become: yes
+  tags: env_setup
 
 - name: create www directory at "/var/www/{{ ansible_host }}"
   file:
@@ -57,12 +61,14 @@
     owner: "{{ user }}"
     group: "{{ group }}"
   become: yes
+  tags: env_setup
 
 - name: delete default nginx config
   file:
     path: /etc/nginx/sites-enabled/default
     state: absent
   become: yes
+  tags: env_setup
 
 - name: copy sample index.html from {{ inventory_hostname }} to /var/www/{{ ansible_host }}/html
   template: >
@@ -71,6 +77,7 @@
     owner="{{ user }}"
     group="{{ group }}"
     mode="u=rw,g=r,o=r"
+  tags: env_setup
 
 - name: copy nginx site.conf
   template:
@@ -80,6 +87,7 @@
     group: root
     mode: '0644'
   become: yes
+  tags: env_setup
 
 - name: link sample html to sites-enabled directory
   file:
@@ -87,6 +95,7 @@
     dest: /etc/nginx/sites-enabled/{{ ansible_host }}
     state: link
   become: yes
+  tags: env_setup
 
 - name: adjust server_names_hash_bucket_size
   template: >
@@ -96,6 +105,7 @@
     group="{{ group }}"
     mode="u=rw,g=r,o=r"
   become: yes
+  tags: env_setup
 
 - name: allow HTTPS traffic
   ufw:
@@ -103,6 +113,7 @@
     name: "Nginx Full"
     state: enabled # enable the ufw rule
   become: yes
+  tags: env_setup
 
 - name: delete redundant Nginx HTTP
   ufw:
@@ -111,12 +122,14 @@
     delete: yes
     state: enabled # enable the ufw rule
   become: yes
+  tags: env_setup
 
 - name: Restart nginx
   service:
     name: nginx
     state: restarted
   become: yes
+  tags: env_setup
 
 - name: install certbot
   apt:
@@ -124,4 +137,5 @@
       - certbot
       - python3-certbot-nginx
     state: latest
-  become: yes
\ No newline at end of file
+  become: yes
+  tags: env_setup


Test patch:
diff --git a/ansible/inventories/test/hosts.yml b/ansible/inventories/test/hosts.yml
index 78f07a2e0..36d53bff4 100644
--- a/ansible/inventories/test/hosts.yml
+++ b/ansible/inventories/test/hosts.yml
@@ -33,3 +33,6 @@ all:
       ansible_host: bm-dev-3.usecustos.org
       ansible_user: exouser
       ansible_ssh_pass: "{{ ansible_ssh_password }}"
+    old_server:
+      ansible_host: dev.custos.usecustos.org
+      ansible_user: ubuntu