Repository: zulip/zulip

Issue:
upgrade-postgres script fails on Ubuntu 18.04 because of unavailable postgresql-12-pgroonga
The upgrade-postgres script fails on Ubuntu 18.04, because the bionic dist of the Groonga repo has no package  `postgresql-12-pgroonga`, only `postgresql-10-pgroonga`.

It fails with:

```
Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install postgresql-12-pgroonga' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package postgresql-12-pgroonga
Error: /Stage[main]/Zulip::Postgres_appdb_base/Package[postgresql-12-pgroonga]/ensure: change from 'purged' to 'present' failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install postgresql-12-pgroonga' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package postgresql-12-pgroonga
Notice: /Stage[main]/Zulip::Postgres_appdb_base/File[/usr/share/postgresql/12/pgroonga_setup.sql]: Dependency Package[postgresql-12-pgroonga] has failures: true
Warning: /Stage[main]/Zulip::Postgres_appdb_base/File[/usr/share/postgresql/12/pgroonga_setup.sql]: Skipping because of failed dependencies
Notice: /Stage[main]/Zulip::Postgres_appdb_base/Exec[create_pgroonga_extension]: Dependency Package[postgresql-12-pgroonga] has failures: true
Warning: /Stage[main]/Zulip::Postgres_appdb_base/Exec[create_pgroonga_extension]: Skipping because of failed dependencies
Notice: Applied catalog in 10.23 seconds
```

I fixed this by manually downloading the package from the pool and installing it.


Discussion:
@imolein thanks for the report; just to check, are you using pgroonga in your installation?

@kou would it be easy for the Groonga project to provide Pgroonga packages built for the postgres.org package repository (https://www.postgresql.org/download/linux/debian/)?  We've switched to using that in order to support using modern postgres on somewhat older OS releases (and also allow upgrading postgres independently of upgrading the base OS).

Or do you have another recommendation for how we should get Pgroonga for these cases?

@timabbott  Thanks for the fast reply :) Yes, we use pgroonga in our installation.
I think that we can provide them easily. I'll take a look at it.
We need PGroonga packages for Debian 10,  Ubuntu 18.04 and Ubuntu 20.04, right?
@kou `postgresql-12-pgroonga` is available for 20.04 via the Groonga repository.
Ah, it may be difficult to provide PGroonga packages for Ubuntu.
Because PGroonga uses https://launchpad.net/~groonga/+archive/ubuntu/ppa . PPA may require that dependencies exist on PPA but the PostgreSQL packages provided by https://www.postgresql.org/download/linux/ubuntu/ don't exist on PPA.
The current `postgresql-12-pgroonga` package on Ubuntu 20.04 is built with the PostgreSQL package provided by Ubuntu not PostgreSQL.
It may have a problem to use the current `postgresql-12-pgroonga` package with the PostgreSQL package provided by PostgreSQL.
Oh, I already created `postgresql-12-pgroonga` for the PostgreSQL package provided by PostgreSQL on Debian 10: https://pgroonga.github.io/install/debian.html#install-on-buster
I've uploaded `postgresql-12-pgdg-pgroonga` packages for Debian GNU/Linux buster, Ubuntu 18.04 and Ubuntu 20.04. They are for the PostgreSQL packages provided by the PostgreSQL Global Development Group.

Note that we need enable both https://launchpad.net/~groonga/+archive/ubuntu/ppa and packages.groonga.org to use `postgresql-12-pgdg-pgroonga` for Ubuntu.

See also:

  * Install document for Debian GNU/Linux: https://pgroonga.github.io/install/debian.html#install-on-buster
  * Install document for Ubuntu: https://pgroonga.github.io/install/ubuntu.html#install-for-official-postgresql
> The current `postgresql-12-pgroonga` package on Ubuntu 20.04 is built with the PostgreSQL package provided by Ubuntu not PostgreSQL.

Hm, maybe that explains why I don't get any results from the search anymore?

Is it save to deinstall the `postgresql-12-pgroonga` from the PPA and install `postgresql-12-pgdg-pgroonga`? Do I have to run any migration after the installation?
You can just install `postgresql-12-pgdg-pgroonga`. `postgresql-12-pgdg-pgroonga` replaces existing `postgresql-12-pgroonga` automatically.

You just need to disconnect before installing `postgresql-12-pgdg-pgroonga` and connect again after installing `postgresql-12-pgdg-pgroonga`. You don't need to migrate again.

I installed your package, thank you. Unfortunately the search still is kinda broken. For some words, I got 1-2 results, even if there are many more in the least recent messages. But in 99% of the time I got nothing back from the search.
I look into the logfiles, but didn't find any error. Can you or @timabbott give me a hint where I have to look? Thanks :)
Could you try writing "Hello World" message and search by "Hello"?
Does it find the latest "Hello World" message?
@imolein are you actually using PGroonga, or the default search backend?  With the latter, you need to run `manage.py audit_fts_indexes` after the upgrade -- that step is included in our upgrade documentation indirectly via a flag to `upgrade-zulip-stage-2`, but it seems possible because of the error you didn't run that as expected.
@kou yeah, it seems like new messages are found correctly

@timabbott I enabled pgroonga right after the installation of my Zulip server a few months ago. I followed the documentation and edited `zulip.conf` and `settings.py` accordingly and the settings are still there. So I guess it is used as the search backend.
This problem hit me on the Zulip 3.2 update again today. It tries to install `postgresql-12-pgroonga`, but couldn't, because I installed `postgresql-12-pgdg-pgroonga`. I guess the puppet manifest need some kind of check, which package is installed, if possible. (I don't know puppet)

To finish my update, is it safe to edit the puppet manifest in the `deployment/next` folder and run the update again?
Yeah, that should be safe.  Just be sure to run `zulip-puppet-apply` and any other tools from the `/home/zulip/deployments/next` path.

Resolving patch:
diff --git a/tools/lib/provision.py b/tools/lib/provision.py
--- a/tools/lib/provision.py
+++ b/tools/lib/provision.py
@@ -177,7 +177,7 @@
     SYSTEM_DEPENDENCIES = [
         *UBUNTU_COMMON_APT_DEPENDENCIES,
         f"postgresql-{POSTGRESQL_VERSION}",
-        f"postgresql-{POSTGRESQL_VERSION}-pgroonga",
+        f"postgresql-{POSTGRESQL_VERSION}-pgdg-pgroonga",
         *VENV_DEPENDENCIES,
     ]
 elif "rhel" in os_families():
@@ -186,7 +186,7 @@
         f"postgresql{POSTGRESQL_VERSION}-server",
         f"postgresql{POSTGRESQL_VERSION}",
         f"postgresql{POSTGRESQL_VERSION}-devel",
-        f"postgresql{POSTGRESQL_VERSION}-pgroonga",
+        f"postgresql{POSTGRESQL_VERSION}-pgdg-pgroonga",
         *VENV_DEPENDENCIES,
     ]
 elif "fedora" in os_families():


Test patch: