Repository: internetarchive/openlibrary

Issue:
Create cron jobs docker-compose service
~Fixed in #4387~
Fixed in ~internetarchive/olsystem#118~
Subtask of: #4060

We have a number of cron jobs that are run on `ol-home` ; in order to run this in our docker-based `ol-home0` node, we should place them in a service. Note these cron jobs live in olsystem.

* [x] Transfer ol-home cron jobs to a new cron file.
* [x] Ensure all ol-home cron jobs run on ol-home0
    * [x]  #4622 `/olsystem/bin/cron/oldump.sh ol-db1` --> ~#4627~ + internetarchive/olsystem#118
        * `dumping the data table ` # for a long time... 102 minutes!
        * Fixed in this PR by correcting `$SCRIPTS` in `bin/cron/oldump.sh`
        * Got an error on a job on the IA side; it was trying to rsync from the docker container: `rsync://e1d610933819/var_1/tmp/dumps/ol_cdump_2021-01-30`
        * STATE: Running; waiting to see if it completes
    * [x] `$SCRIPTS/store_counts.py` * 2 -- fixed in internetarchive/infogami#134
        * [x] `if _is_illegal_header_value(values[i]):` TypeError: expected string or bytes-like object 
        * [x] Errored: Unable to access infobase?
    * [x] `$SCRIPTS/2011/08/expire_accounts.py` -- fixed in internetarchive/infogami#134
        * [x] `if _is_illegal_header_value(values[i]):` TypeError: expected string or bytes-like object 
        * [x] Errored: Unable to access infobase?
              * Needed to change infobase link from ol-home0 -> ol-home in openlibrary.yml
    * [x] `openlibrary.solr.process_stats` * 2 (loan stats)
        * `cat /var/log/openlibrary/loanstats.log`
        * [x] `/run/secrets/ia_db_pw_file: not found` should have been fixed by #115
        * From https://github.com/internetarchive/olsystem/blob/master/etc/openlibrary.yml#L31
        * [x] #4542 Python 3 decode error for 1 loan?
    * [x] `python $SCRIPTS/manage-imports.py`
        * [x] `/run/secrets/ia_db_pw_file: not found` should have been fixed by #115
        * No errors now.
* [x] Ensure all ol-www1 cron jobs run in docker + py3
    * [x] (Moved to #4775 and separated from this issue) sitemap rsync should run on ol-home0 [Slack thread](https://internetarchive.slack.com/archives/C075DJ6G1/p1614111981011600) <details>The sitemaps are generated on ol-home0 and copied over to ol-www1 where they are served. We can actually make the `rsync` command run on ol-home0 and target ol-www1 (and/or ol-www0). This is **not** blocked by ol-www0 migration.  `saddr $CLUSTER proto tcp dport rsync ACCEPT;` in `/etc/ferm/ferm.conf` is vital.</details>
    * [x] cron-jobs service stuck in restart loop #4707 fixed in #4717
    * [x] (Moved to #4775 and separated from this issue) MOVE ALL CRON JOBS TO OL-HOME0 ipstats.py. Unclear what ideal course of action is for these, but if we could get them running on ol-home0 with other cron jobs: that would be nice! 


Discussion:

Resolving patch:
diff --git a/openlibrary/solr/process_stats.py b/openlibrary/solr/process_stats.py
--- a/openlibrary/solr/process_stats.py
+++ b/openlibrary/solr/process_stats.py
@@ -149,7 +149,7 @@ def process(data):
 
     if doc.get('t_end'):
         dt = h.parse_datetime(doc.t_end) - h.parse_datetime(doc.t_start)
-        hours = dt.days * 24 + dt.seconds / 3600
+        hours = dt.days * 24 + dt.seconds // 3600
         solrdoc['duration_hours_i'] = hours
 
     #last_updated = h.parse_datetime(doc.get('t_end') or doc.get('t_start'))
diff --git a/openlibrary/solr/solrwriter.py b/openlibrary/solr/solrwriter.py
--- a/openlibrary/solr/solrwriter.py
+++ b/openlibrary/solr/solrwriter.py
@@ -65,7 +65,7 @@ def flush(self):
                 root.append(node)
             logger.info("flushing %d documents", len(self.pending_updates))
             self.pending_updates = []
-            xml = tostring(root).encode('utf-8')
+            xml = tostring(root).decode('utf-8')
             self.request(xml)
 
     def commit(self):
diff --git a/scripts/2011/08/expire_accounts.py b/scripts/expire_accounts.py
similarity index 100%
rename from scripts/2011/08/expire_accounts.py
rename to scripts/expire_accounts.py


Test patch: