Repository: internetarchive/openlibrary

Issue:
scripts/copydocs.py fails when the received content is too large
<!-- What problem are we solving? What does the experience look like today? What are the symptoms? -->
When the copydocs script is used to copy a work that contains data that exceeds a character count of 4094, an error is thrown and the work is not copied.

### Evidence / Screenshot (if possible)

### Relevant url?
<!-- `https://openlibrary.org/works/OL16060982W/Prentice_Hall_Literature_-_Timeless_Voices_Timeless_Themes_-_The_American_Experience` -->

### Steps to Reproduce
<!-- What steps caused you to find the bug? -->
1. Run the copydocs script in your web container (`docker-compose run web bash`, `./scripts/copydocs.py`) using [this work](https://openlibrary.org/works/OL16060982W/Prentice_Hall_Literature_-_Timeless_Voices_Timeless_Themes_-_The_American_Experience)
2. The error indicating the request is too large will be thrown (5495 chars. Limit says it's 4094)

<!-- What actually happened after these steps? What did you expect to happen? -->
* Actual: work is copied to local database
* Expected: error thrown due to data size

### Details

- **Logged in (Y/N)?**  N/A
- **Browser type/version?** N/A 
- **Operating system?** N/A
- **Environment (prod/dev/local)?** dev
<!-- If not sure, put prod -->

### Proposal & Constraints
<!-- What is the proposed solution / implementation? Is there a precedent of this approach succeeding elsewhere? -->

### Related files
<!-- Files related to this issue; this is super useful for new contributors who might want to help! If you're not sure, leave this blank; a maintainer will add them. -->
https://github.com/internetarchive/openlibrary/blob/master/scripts/copydocs.py

### Stakeholders
<!-- @ tag stakeholders of this bug -->



Discussion:
Hey @jimchamp I would like to work on this issue. Can you assign it to me?
@jimchamp can you assign this to me?
@b2pacific what is your plan for fixing this issue?  I suspect that we will have to make a configuration change (either nginx or gunicorn) to address this issue, but I may be wrong.
It's a gunicorn configuration.  @b2pacific, you can update our web GUNICORN_OPTS in `docker-compose.yml` to fix this, here: https://github.com/internetarchive/openlibrary/blob/c713c09e4e09f7521f8e7cd89678a5a2a4ad150c/docker-compose.yml#L8

Here is the setting to use: https://docs.gunicorn.org/en/stable/settings.html#limit-request-line 


Resolving patch:
diff --git a/openlibrary/api.py b/openlibrary/api.py
--- a/openlibrary/api.py
+++ b/openlibrary/api.py
@@ -112,10 +112,11 @@ def get(self, key, v=None):
         response = self._request(key + '.json', params={'v': v} if v else {})
         return unmarshal(response.json())
 
-    def get_many(self, keys):
-        """Get multiple documents in a single request as a dictionary."""
-        if len(keys) > 500:
-            # get in chunks of 500 to avoid crossing the URL length limit.
+    def get_many(self, keys, max_length = 500):
+        """Get multiple documents in a single request as a dictionary.
+        """
+        if len(keys) > max_length:
+            # Process in batches to avoid crossing the URL length limit.
             d = {}
             for chunk in web.group(keys, 100):
                 d.update(self._get_many(chunk))
diff --git a/scripts/copydocs.py b/scripts/copydocs.py
--- a/scripts/copydocs.py
+++ b/scripts/copydocs.py
@@ -231,7 +231,7 @@ def copy(src, dest, keys, comment, recursive=False, saved=None, cache=None):
         cache = {}
 
     def get_many(keys):
-        docs = marshal(src.get_many(keys).values())
+        docs = marshal(src.get_many(keys, max_length = 100).values())
         # work records may contain excerpts, which reference the author of the excerpt.
         # Deleting them to prevent loading the users.
         for doc in docs:


Test patch:
diff --git a/scripts/tests/test_copydocs.py b/scripts/tests/test_copydocs.py
--- a/scripts/tests/test_copydocs.py
+++ b/scripts/tests/test_copydocs.py
@@ -39,7 +39,7 @@ def get(self, key, revision=None):
         else:
             return revisions.get(revision, None)
 
-    def get_many(self, keys):
+    def get_many(self, keys, max_length = 500):
         """
         :param list of str keys:
         :rtype: dict