Repository: zulip/zulip

Issue:
provision: Avoiding spending ~3s on a no-op provision doing apt operations
We spend several seconds on every no-op provision doing apt setup and operations for things that largely don't change. We should be able to optimize this away in the common (not `--force`) case.  

For example, the `apt-get install`, `apt-key add`, etc. operations should probably only run with `--force` unless their dependencies have changed (e.g. the script with the code, the list of files, etc.).  

```
Jun 02 12:53:51 + mkdir -p /home/tabbott/zulip/var
Jun 02 12:53:51 Reading package lists...
Jun 02 12:53:51 Building dependency tree...
Jun 02 12:53:51 Reading state information...
Jun 02 12:53:52 lsb-release is already the newest version (9.20160110ubuntu0.2).
Jun 02 12:53:52 0 upgraded, 0 newly installed, 0 to remove and 58 not upgraded.
Jun 02 12:53:52 + sudo ./scripts/lib/setup-apt-repo
Jun 02 12:53:52 + SOURCES_FILE=/etc/apt/sources.list.d/zulip.list
Jun 02 12:53:52 ++ shasum /etc/apt/sources.list.d/zulip.list
Jun 02 12:53:52 + zulip_source_hash='716d3aaf13728be1b7f04e674415335261d8d627  /etc/apt/sources.list.d/zulip.list'
Jun 02 12:53:52 + apt-get install -y lsb-release
Jun 02 12:53:52 Reading package lists...
Jun 02 12:53:52 Building dependency tree...
Jun 02 12:53:52 Reading state information...
Jun 02 12:53:52 lsb-release is already the newest version (9.20160110ubuntu0.2).
Jun 02 12:53:52 0 upgraded, 0 newly installed, 0 to remove and 58 not upgraded.
Jun 02 12:53:52 +++ dirname ./scripts/lib/setup-apt-repo
Jun 02 12:53:52 ++ dirname ./scripts/lib
Jun 02 12:53:52 + SCRIPTS_PATH=./scripts
Jun 02 12:53:52 + apt-key add ./scripts/setup/zulip-ppa.asc
Jun 02 12:53:52 OK
Jun 02 12:53:52 + apt-key add ./scripts/setup/pgroonga-ppa.asc
Jun 02 12:53:53 OK
Jun 02 12:53:53 ++ lsb_release -sc
Jun 02 12:53:53 + release=xenial
Jun 02 12:53:53 + '[' xenial = trusty ']'
Jun 02 12:53:53 + '[' xenial = xenial ']'
Jun 02 12:53:53 + cat
Jun 02 12:53:53 ++ shasum /etc/apt/sources.list.d/zulip.list
Jun 02 12:53:53 + '[' '716d3aaf13728be1b7f04e674415335261d8d627  /etc/apt/sources.list.d/zulip.list' = '716d3aaf13728be1b7f04e674415335261d8d627  /etc/apt/sources.list.d/zulip.list' ']'
Jun 02 12:53:53 + echo 'zulip.list file did not change; skipping apt-get update'
Jun 02 12:53:53 zulip.list file did not change; skipping apt-get update
Jun 02 12:53:53 + sudo apt-get -y install --no-install-recommends closure-compiler memcached rabbitmq-server redis-server hunspell-en-us supervisor git libssl-dev yui-compressor wget ca-certificates puppet gettext curl netcat build-essential libffi-dev libfreetype6-dev libz-dev libjpeg-dev libldap2-dev libmemcached-dev python3-dev python-dev python-pip python-virtualenv libxml2-dev libxslt1-dev libpq-dev postgresql-9.5 postgresql-9.5-tsearch-extras postgresql-9.5-pgroonga
Jun 02 12:53:54 Reading package lists...
Jun 02 12:53:54 Building dependency tree...
Jun 02 12:53:54 Reading state information...
Jun 02 12:53:54 build-essential is already the newest version (12.1ubuntu2).
Jun 02 12:53:54 ca-certificates is already the newest version (20160104ubuntu1).
Jun 02 12:53:54 gettext is already the newest version (0.19.7-2ubuntu3).
Jun 02 12:53:54 hunspell-en-us is already the newest version (20070829-6ubuntu3).
Jun 02 12:53:54 libffi-dev is already the newest version (3.2.1-4).
Jun 02 12:53:54 libjpeg-dev is already the newest version (8c-2ubuntu8).
Jun 02 12:53:54 libmemcached-dev is already the newest version (1.0.18-4.1).
Jun 02 12:53:54 python-dev is already the newest version (2.7.11-1).
Jun 02 12:53:54 python3-dev is already the newest version (3.5.1-3).
Jun 02 12:53:54 closure-compiler is already the newest version (20130227+dfsg1-8).
Jun 02 12:53:54 netcat is already the newest version (1.10-41).
Jun 02 12:53:54 redis-server is already the newest version (2:3.0.6-1).
Jun 02 12:53:54 yui-compressor is already the newest version (2.4.7-2).
Jun 02 12:53:54 curl is already the newest version (7.47.0-1ubuntu2.2).
Jun 02 12:53:54 git is already the newest version (1:2.7.4-0ubuntu1.1).
Jun 02 12:53:54 libfreetype6-dev is already the newest version (2.6.1-0.1ubuntu2.3).
Jun 02 12:53:54 libldap2-dev is already the newest version (2.4.42+dfsg-2ubuntu3.2).
Jun 02 12:53:54 libpq-dev is already the newest version (9.5.7-0ubuntu0.16.04).
Jun 02 12:53:54 libssl-dev is already the newest version (1.0.2g-1ubuntu4.6).
Jun 02 12:53:54 libxml2-dev is already the newest version (2.9.3+dfsg1-1ubuntu0.2).
Jun 02 12:53:54 libxslt1-dev is already the newest version (1.1.28-2.1ubuntu0.1).
Jun 02 12:53:54 memcached is already the newest version (1.4.25-2ubuntu1.2).
Jun 02 12:53:54 postgresql-9.5 is already the newest version (9.5.7-0ubuntu0.16.04).
Jun 02 12:53:54 rabbitmq-server is already the newest version (3.5.7-1ubuntu0.16.04.1).
Jun 02 12:53:54 wget is already the newest version (1.17.1-1ubuntu1.2).
Jun 02 12:53:54 zlib1g-dev is already the newest version (1:1.2.8.dfsg-2ubuntu4.1).
Jun 02 12:53:54 puppet is already the newest version (3.8.5-2ubuntu0.1).
Jun 02 12:53:54 python-pip is already the newest version (8.1.1-2ubuntu0.4).
Jun 02 12:53:54 python-virtualenv is already the newest version (15.0.1+ds-3ubuntu1).
Jun 02 12:53:54 supervisor is already the newest version (3.2.0-2ubuntu0.1).
Jun 02 12:53:54 postgresql-9.5-tsearch-extras is already the newest version (0.2).
Jun 02 12:53:54 postgresql-9.5-pgroonga is already the newest version (1.2.0-2~xenial1).
Jun 02 12:53:54 0 upgraded, 0 newly installed, 0 to remove and 58 not upgraded.
```

This is extracted from #1461.


Discussion:
Hello @zulip/server-development members, this issue was labeled with the **area: provision** label, so you may want to check it out!
@zulipbot claim.
@timabbott @femnad Should we add the hash same way in #5184 to check the dependencies have been changed or not?
That could work if you figure out some way produce a similar hash for apt dependencies.
So I think we can make a hash for all the dependencies of the `apt` commands (common case) and if it remains same then we'll not be running any of the `apt` command else we'll run all `apt` command because maintaining different hash for all the apt commands can be messy.
Does it sound ok?
Yeah, all or none sounds reasonable for this case.
Agreed all or none seems reasonable.  

I think this is the only block we need to cover to fix this (the rest are mostly negligible):
```
    run(["sudo", "./scripts/lib/setup-apt-repo"])                                                                         
    run(["sudo", "apt-get", "-y", "install", "--no-install-recommends"] + APT_DEPENDENCIES[codename])                     
```

I think hashing the (sorted) list of apt packages from APT_DEPENDENCIES[codename] plus the content of `scripts/lib/setup-apt-repo` seems correct; we should make sure this one respects the `--force` option, however, since one could want to force an update.

Resolving patch:
diff --git a/tools/lib/provision.py b/tools/lib/provision.py
--- a/tools/lib/provision.py
+++ b/tools/lib/provision.py
@@ -6,6 +6,7 @@
 import argparse
 import platform
 import subprocess
+import hashlib
 
 os.environ["PYTHONUNBUFFERED"] = "y"
 
@@ -183,12 +184,35 @@ def main(options):
     # project.
     os.chdir(ZULIP_PATH)
 
+    # setup-apt-repo does an `apt-get update`
+    # hash the apt dependencies
+    sha_sum = hashlib.sha1()
+
+    for apt_depedency in APT_DEPENDENCIES[codename]:
+        sha_sum.update(apt_depedency.encode('utf8'))
+    # hash the content of setup-apt-repo
+    sha_sum.update(open('scripts/lib/setup-apt-repo').read().encode('utf8'))
+
+    new_apt_dependencies_hash = sha_sum.hexdigest()
+    last_apt_dependencies_hash = None
+
     try:
-        install_apt_deps()
-    except subprocess.CalledProcessError:
-        # Might be a failure due to network connection issues. Retrying...
-        print(WARNING + "`apt-get -y install` failed while installing dependencies; retrying..." + ENDC)
-        install_apt_deps()
+        hash_file = open('var/apt_dependenices_hash', 'r+')
+        last_apt_dependencies_hash = hash_file.read()
+    except IOError:
+        run(['touch', 'var/apt_dependenices_hash'])
+        hash_file = open('var/apt_dependenices_hash', 'r+')
+
+    if (new_apt_dependencies_hash != last_apt_dependencies_hash):
+        try:
+            install_apt_deps()
+        except subprocess.CalledProcessError:
+            # Might be a failure due to network connection issues. Retrying...
+            print(WARNING + "`apt-get -y install` failed while installing dependencies; retrying..." + ENDC)
+            install_apt_deps()
+        hash_file.write(new_apt_dependencies_hash)
+    else:
+        print("No need to apt operations.")
 
     if options.is_travis:
         if PY2:


Test patch: