Repository: internetarchive/openlibrary

Issue:
Data Dumps not auto-generating
Despite #5263 being resolved, it looks like the data dumps weren't uploaded on July 1st :/ 

### Relevant URL?
* https://github.com/internetarchive/openlibrary/wiki/Generating-Data-Dumps
* https://archive.org/details/ol_exports?sort=-publicdate

Related issues and pull requests:
* #3989 
* #4621
* #4671
* #4723
* #5546
* #5673 
* #5719 
* #5892 - Worth reading!
* #6158
* #6163

Related files:
* [`docker-compose.production.yml`](../blob/master/docker-compose.production.yml#L90) defines `cron-jobs` Docker container.
* [`docker/ol-cron-start.sh`](../blob/master/docker/ol-cron-start.sh) sets up the cron tasks.
* [olsystem: `/etc/cron.d/openlibrary.ol_home0`](https://github.com/internetarchive/olsystem/blob/master/etc/cron.d/openlibrary.ol_home0#L11) defines the actual job
    * modify and then to reactivate do: `crontab /etc/cron.d/openlibrary.ol_home0`   Also: https://cron.help
    * [x] internetarchive/olsystem#140
* [`scripts/oldump.sh`](../blob/master/scripts/oldump.sh) is the script that gets run.
    * [x] #5860

### Proposal & Constraints
- Run manually for now

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->

### Stakeholders
@mekarpeles @jimman2003 



Discussion:
It looks like they were generated, but the file names look off; there's no date. And the `--archive` flag which is used to... you know upload the data dumps, somehow made it into the filename. So I imagine the upload never happened.

On `ol-home0`:
```sh
cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash
```

```sh
cd /1/var/tmp/dumps
openlibrary@ol-home0:/1/var/tmp/dumps$ du -sh *
4.0K    ol_cdump_--archive.txt.gz
4.0K    ol_dump_--archive.txt.gz
```
^ Although the file sizes definitely don't look correct here.
Going to follow the steps here to generate a new dump: https://github.com/internetarchive/openlibrary/issues/4621#issuecomment-818812082

Had to remove the old files manually; complaining about permissions... `sudo rm -rf /1/var/tmp/dumps/ol_*`

Then (from the above comment):

```sh
tmux
# ---
cd /opt/openlibrary

COMPOSE_FILE="docker-compose.yml:docker-compose.production.yml" \
HOSTNAME="$HOSTNAME" \
docker-compose run \
    -u openlibrary \
    -e "SCRIPTS=/openlibrary/scripts" \
    cron-jobs \
    bash
# ---
cd /1/var/tmp
/olsystem/bin/cron/oldump.sh ol-db1 openlibrary `date -d yesterday '+%Y-%m-%d'` --archive
```


Waiting...
New dump created :+1:
Kicked off manual dump; see https://github.com/internetarchive/openlibrary/issues/3989#issuecomment-896138264
Errored; restarted. See https://github.com/internetarchive/openlibrary/issues/3989#issuecomment-897039442
The dump is still not up :)

Ooops! Auto-closed by a PR
* creating clean directories
* dumping reading log table
real    0m12.712s
user    0m8.430s
sys     0m0.973s

* dumping ratings table
real    0m1.029s
user    0m0.769s
sys     0m0.069s

* dumping the data table
real    110m38.907s
user    106m17.061s
sys     15m48.968s

* generating ol_cdump_2021-11-10.txt.gz
Traceback (most recent call last):
  ...
    import web
ImportError: No module named web

ol-home0% cd /1/var/tmp/dumps
ol-home0% ls -l
```
total 29668
-rw-rw-r-- 1 systemd-coredump systemd-coredump  2222064 Nov 11 17:41 cclauss_ol_dump_ratings_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump 28145076 Nov 11 17:41 cclauss_ol_dump_reading-log_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump       20 Nov 11 19:31 ol_cdump_2021-11-10.txt.gz
-rw-rw-r-- 1 systemd-coredump systemd-coredump       20 Nov 11 19:31 ol_dump_2021-11-10.txt.gz
```
Two key learnings about `openlibrary/data/dump.py`:
1.  I ~might have~ made bad changes in #4627
2. In Python 3 we need to use `sys.stdin.buffer` to read as ___binary___ a .gz file piped into stdin.

Reopening because the Jan 1 dump did not happen.  Investigating...

Resolving patch:
diff --git a/scripts/oldump.py b/scripts/oldump.py
--- a/scripts/oldump.py
+++ b/scripts/oldump.py
@@ -1,8 +1,13 @@
-#! /usr/bin/env python
+#! /usr/bin/env python3
+
+import sys
+
 import _init_path
-from openlibrary.data import dump
+
 
 if __name__ == "__main__":
-    import sys
+    print("{}: Python {}.{}.{}".format(__file__, *sys.version_info))
+
+    from openlibrary.data import dump
 
     dump.main(sys.argv[1], sys.argv[2:])
diff --git a/scripts/sitemaps/sitemap.py b/scripts/sitemaps/sitemap.py
old mode 100644
new mode 100755
--- a/scripts/sitemaps/sitemap.py
+++ b/scripts/sitemaps/sitemap.py
@@ -1,3 +1,4 @@
+#! /usr/bin/env python3
 """Script to generate XML sitemap of openlibrary.org website.
 
 USAGE:


Test patch: