Repository: internetarchive/openlibrary

Issue:
Data Dumps not auto-generating
Despite #5263 being resolved, it looks like the data dumps weren't uploaded on July 1st :/ 

### Relevant URL?
* https://github.com/internetarchive/openlibrary/wiki/Generating-Data-Dumps
* https://archive.org/details/ol_exports?sort=-publicdate

Related issues and pull requests:
* #3989 
* #4621
* #4671
* #4723
* #5546
* #5673 
* #5719 
* #5892 - Worth reading!
* #6158
* #6163

Related files:
* [`docker-compose.production.yml`](../blob/master/docker-compose.production.yml#L90) defines `cron-jobs` Docker container.
* [`docker/ol-cron-start.sh`](../blob/master/docker/ol-cron-start.sh) sets up the cron tasks.
* [olsystem: `/etc/cron.d/openlibrary.ol_home0`](https://github.com/internetarchive/olsystem/blob/master/etc/cron.d/openlibrary.ol_home0#L11) defines the actual job
    * modify and then to reactivate do: `crontab /etc/cron.d/openlibrary.ol_home0`   Also: https://cron.help
    * [x] internetarchive/olsystem#140
* [`scripts/oldump.sh`](../blob/master/scripts/oldump.sh) is the script that gets run.
    * [x] #5860

### Proposal & Constraints
- Run manually for now

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@mekarpeles @jimman2003 



Discussion:
It looks like they were generated, but the file names look off; there's no date. And the `--archive` flag which is used to... you know upload the data dumps, somehow made it into the filename. So I imagine the upload never happened.

On `ol-home0`:
```sh
cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash
```

```sh
cd /1/var/tmp/dumps
openlibrary@ol-home0:/1/var/tmp/dumps$ du -sh *
4.0K    ol_cdump_--archive.txt.gz
4.0K    ol_dump_--archive.txt.gz
```
^ Although the file sizes definitely don't look correct here.
Going to follow the steps here to generate a new dump: https://github.com/internetarchive/openlibrary/issues/4621#issuecomment-818812082

Had to remove the old files manually; complaining about permissions... `sudo rm -rf /1/var/tmp/dumps/ol_*`

Then (from the above comment):

```sh
tmux
# ---
cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash
# ---
cd /1/var/tmp
/olsystem/bin/cron/oldump.sh ol-db1 openlibrary `date -d yesterday '+%Y-%m-%d'` --archive
```


Waiting...
New dump created :+1:
Kicked off manual dump; see https://github.com/internetarchive/openlibrary/issues/3989#issuecomment-896138264
Errored; restarted. See https://github.com/internetarchive/openlibrary/issues/3989#issuecomment-897039442
The dump is still not up :)

Ooops! Auto-closed by a PR
* creating clean directories
* dumping reading log table
real    0m12.712s
user    0m8.430s
sys     0m0.973s

* dumping ratings table
real    0m1.029s
user    0m0.769s
sys     0m0.069s

* dumping the data table
real    110m38.907s
user    106m17.061s
sys     15m48.968s

* generating ol_cdump_2021-11-10.txt.gz
Traceback (most recent call last):
  ...
    import web
ImportError: No module named web

ol-home0% cd /1/var/tmp/dumps
ol-home0% ls -l
```
total 29668
-rw-rw-r-- 1 systemd-coredump systemd-coredump  2222064 Nov 11 17:41 cclauss_ol_dump_ratings_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump 28145076 Nov 11 17:41 cclauss_ol_dump_reading-log_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump       20 Nov 11 19:31 ol_cdump_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump       20 Nov 11 19:31 ol_dump_2021-11-10.txt.gz
```
Two key learnings about `openlibrary/data/dump.py`:
1.  I ~might have~ made bad changes in #4627
2. In Python 3 we need to use `sys.stdin.buffer` to read as ___binary___ a .gz file piped into stdin.

Reopening because the Jan 1 dump did not happen.  Investigating...
~This has large refactors of bash code on ol-home0 which need to be checked over and turned into a PR~ by @cclauss

See: #6253
womp womp
Problems on 01 June 2020:  Generating cdump took 7h28 to process 200M records.
1. Skipping: data.txt.gz -- Because the script deleted the file in the parent directory, not the working directory.
2. Step 4 processed 200M records and wrote its results into ol_cdump_2022-0**5**-31 but the `compgen` logic told Step 5 to process a file that starts with ol_cdump_2022-0**6**.  No such file existed so steps 5 and 6 processed nothing quite quickly. :-(

**ol-home0%** `docker logs -f openlibrary_cron-jobs_1 2>&1 | grep "dump\|Jun"`
```
[openlibrary.dump] * [Wed Jun  1 00:00:01 UTC 2022] /openlibrary/scripts/oldump.sh 2022-05-31 --archive --overwrite
[openlibrary.dump] * <host:ol-home0.us.archive.org> <user:openlibrary> <dir:/1/var/tmp>
[openlibrary.dump] * Cleaning Up: Found --cleanup, removing old files
> === Step 1 ===
[openlibrary.dump] * generating reading log table: ol_dump_reading-log_2022-05-31.txt.gz
> === Step 2 ===
[openlibrary.dump] * generating ratings table: ol_dump_ratings_2022-05-31.txt.gz
> === Step 3 ===
[openlibrary.dump] * Skipping: data.txt.gz   <-- NOT GOOD!!  --overwrite cleared the parent, not working directory.
> === Step 4 ===
[openlibrary.dump] * generating ol_cdump_2022-05-31.txt.gz -- takes approx. 500 minutes for 192,000,000+ records...
/openlibrary/scripts/oldump.py: Python 3.9.4
generate_cdump(data.txt.gz, 2022-05-31) reading
Wed Jun  1 00:00:22 2022 read_data_file(data.txt.gz, max_lines=all)
Wed Jun  1 00:00:22 2022 0
Wed Jun  1 00:02:58 2022 1,000,000
Wed Jun  1 00:04:58 2022 2,000,000
Wed Jun  1 00:07:01 2022 3,000,000
Wed Jun  1 00:09:26 2022 4,000,000
Wed Jun  1 00:12:05 2022 5,000,000
    [ ... ]
Wed Jun  1 07:20:41 2022 196,000,000
Wed Jun  1 07:22:24 2022 197,000,000
Wed Jun  1 07:24:38 2022 198,000,000
Wed Jun  1 07:26:35 2022 199,000,000
Wed Jun  1 07:28:45 2022 200,000,000
[openlibrary.dump] * generated  <-- PROBLEMS START HERE!!  2022-06 vs. 2022-05 issue.
> === Step 5 ===
generating the dump -- takes approx. 485 minutes for 173,000,000+ records...
/openlibrary/scripts/oldump.py: Python 3.9.4
> === Step 6 ===
/openlibrary/scripts/oldump.py: Python 3.9.4
Wed Jun  1 07:29:19 2022 read_tsv() reading <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>
Wed Jun  1 07:29:19 2022 splitting stdin
Wed Jun  1 07:29:19 2022 sorting /1/var/tmp/oldumpsort/00.txt.gz
Wed Jun  1 07:29:19 2022 sorting /1/var/tmp/oldumpsort/01.txt.gz
Wed Jun  1 07:29:19 2022 sorting /1/var/tmp/oldumpsort/02.txt.gz
Wed Jun  1 07:29:19 2022 sorting /1/var/tmp/oldumpsort/03.txt.gz
Wed Jun  1 07:29:19 2022 sorting /1/var/tmp/oldumpsort/04.txt.gz
    [ ... ]
Wed Jun  1 07:29:21 2022 sorting /1/var/tmp/oldumpsort/fb.txt.gz
Wed Jun  1 07:29:21 2022 sorting /1/var/tmp/oldumpsort/fc.txt.gz
Wed Jun  1 07:29:21 2022 sorting /1/var/tmp/oldumpsort/fd.txt.gz
Wed Jun  1 07:29:21 2022 sorting /1/var/tmp/oldumpsort/fe.txt.gz
Wed Jun  1 07:29:21 2022 sorting /1/var/tmp/oldumpsort/ff.txt.gz
splitting the dump: ol_dump_%s_2022-05-31.txt.gz -- takes approx. 85 minutes for 68,000,000+ records...
/openlibrary/scripts/oldump.py: Python 3.9.4
[openlibrary.dump] * dumps are generated at /1/var/tmp/dumps
drwxr-xr-x 2 openlibrary openlibrary 4.0K Jun  1 07:29 ol_cdump_2022-05-31
drwxr-xr-x 2 openlibrary openlibrary 4.0K Jun  1 07:29 ol_dump_2022-05-31
./ol_cdump_2022-05-31:
-rw-r--r-- 1 openlibrary openlibrary 26G Jun  1 07:29 ol_cdump_2022-05-31.txt.gz
./ol_dump_2022-05-31:
-rw-r--r-- 1 openlibrary openlibrary   20 Jun  1 07:29 ol_dump_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary   56 Jun  1 07:29 ol_dump_authors_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary   57 Jun  1 07:29 ol_dump_editions_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary 2.5M Jun  1 00:00 ol_dump_ratings_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary  33M Jun  1 00:00 ol_dump_reading-log_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary   58 Jun  1 07:29 ol_dump_redirects_2022-05-31.txt.gz
-rw-r--r-- 1 openlibrary openlibrary   54 Jun  1 07:29 ol_dump_works_2022-05-31.txt.gz
[openlibrary.dump] * ia version is v2.3.0
ol_dump_2022-05-31:
 uploading /ol_dump_authors_2022-05-31.txt.gz: 100%|██████████| 1/1 [00:00<00:00, 32.94MiB/s]
 uploading /ol_dump_2022-05-31.txt.gz: 100%|██████████| 1/1 [00:00<00:00, 39.22MiB/s]
 uploading /ol_dump_editions_2022-05-31.txt.gz: 100%|██████████| 1/1 [00:00<00:00, 39.23MiB/s]
 uploading /ol_dump_works_2022-05-31.txt.gz: 100%|██████████| 1/1 [00:00<00:00, 34.32MiB/s]
 uploading /ol_dump_ratings_2022-05-31.txt.gz: 100%|██████████| 3/3 [00:00<00:00, 30.41MiB/s]
 uploading /ol_dump_redirects_2022-05-31.txt.gz: 100%|██████████| 1/1 [00:00<00:00, 43.77MiB/s]
 uploading /ol_dump_reading-log_2022-05-31.txt.gz: 100%|██████████| 33/33 [00:04<00:00,  6.80MiB/s]
ol_cdump_2022-05-31:
 uploading /ol_cdump_2022-05-31.txt.gz: 100%|██████████| 25686/25686 [06:53<00:00, 62.12MiB/s] ]
[openlibrary.dump] * Skipping sitemap
Wed Jun 1 07:37:17 UTC 2022: openlibrary has completed /openlibrary/scripts/oldump.sh 2022-05-31 --archive --overwrite in /1/var/tmp on ol-home0.us.archive.org
deleting the data table dump
```
:ballot_box_with_check: 

Resolving patch:
diff --git a/scripts/oldump.py b/scripts/oldump.py
--- a/scripts/oldump.py
+++ b/scripts/oldump.py
@@ -19,6 +19,7 @@ def log(*args) -> None:
 
 
 if __name__ == "__main__":
+    from contextlib import redirect_stdout
     from infogami import config
     from openlibrary.config import load_config
     from openlibrary.data import dump
@@ -29,7 +30,11 @@ def log(*args) -> None:
     ol_config = os.getenv("OL_CONFIG")
     if ol_config:
         logger.info(f"loading config from {ol_config}")
-        load_config(ol_config)
+        # Squelch output from infobase (needed for sentry setup)
+        # So it doesn't end up in our data dumps body
+        with open(os.devnull, 'w') as devnull:
+            with redirect_stdout(devnull):
+                load_config(ol_config)
         sentry = Sentry(getattr(config, "sentry_cron_jobs", {}))
         if sentry.enabled:
             sentry.init()


Test patch: