Repository: dmwm/wmcore
Issue: JobAccountant trying to upload large documents to local CouchDB

**Impact of the bug**
WMAgent

**Describe the bug**
While investigating an ErrorHandler issue in vocms0252, I noticed that JobAccountant log is barely usable, with a dump of a large document that failed to be inserted into local CouchDB for being too large, see part of the error here [1]. And I think this is the equivalent log record in the CouchDB server [2], where one can see that this operation failed with a 
```
413 - Request Entity Too Large
```

We don't override any of the relevant CouchDB document size configuration in:
https://github.com/dmwm/deployment/blob/master/wmagentpy3/local.ini
which means we are using the default values for:
"""
A document exceeds the configured [couchdb/max_document_size](https://docs.couchdb.org/en/3.2.2-docs/config/couchdb.html#couchdb/max_document_size) value or the entire request exceeds the [chttpd/max_http_request_size](https://docs.couchdb.org/en/3.2.2-docs/config/http.html#chttpd/max_http_request_size) value.
"""

From a quick look at the log and the document dump, I think the problem comes from a **very large**  list of LFNs provided under
```
process.mixData.input.
fileNames = cms.untracked.vstring([\\\\\\\\\\\\\\'/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/120000/00456BD1-4A4E-6843-AA18-78B56D65EF38.root, ...
```

Traceback information is provided in [3]

**How to reproduce it**
Create a document in CouchDB larger than what it's configured for.

**Expected behavior**
JobAccountant does not crash when it fails to upload such large documents - and neither are these documents extremely important - so I think we do not need to change anything on this front.

Possible improvements that could be delivered with this issue are:
* whenever this http error `413` is hit, we should not dump the document content in the error message. Ideally, we could actually json serialize it and get its length to see how big the document was.
* update the CouchDB `local.ini` and define those parameters there. Either with the same default value, or with an slightly increase in its size (16MB?)
* (BONUS) maybe stop providing a list of secondary files in the FJR that gets uploaded to CouchDB (and likely ending up in central WMStats, through replication).

**Additional context and error message**
[1]
```
2022-10-12 12:51:36,051:140152399025920:ERROR:ChangeState:Error updating job in couch: NEW ERROR STATUS! UPDATE CMSCOUCH.PY!: 413
CouchError - reason: Request Entity Too Large, data: {'docs': [{'_id': '2490046-0', 'jobid': 2490046, 'jobtype': 'Cleanup', 'jobstate': 'jobfailed', 'retrycount': 0, 'archivestatus': 'skip', 'fwjr': {'WorkerNodeInfo': {'HostName': 'llrgrwn129.in2p3.fr', 'MachineFeatures': {}, 'JobFeatures': {}}, 'task': '/cmsunified_task_HIG-RunIISummer20UL17wmLHEGEN-04807__v1_T_220629_205933_5667/HIG-RunIISummer20UL17wmLHEGEN-04807_0/HIG-RunIISummer20UL17MiniAODv2-06491_0CleanupUnmergedMINIAODSIMoutput', 'steps': {'cleanupUnmergedMINIAODSIMoutput': {'st
...
le "/cvmfs/cms.cern.ch/share/overrides/bin/edm_pset_pickler.py", line 39, in main\\\\\\\\n    code = compile(f.read(), args.input, \\\\\\\\\\\\\\'exec\\\\\\\\\\\\\\')\\\\\\\\n  File "/srv/job/WMTaskSpace/cmsRun3/PSet.py", line 84\\\\\\\\n    process.mixData.input.
fileNames = cms.untracked.vstring([\\\\\\\\\\\\\\'/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/120000/00456BD1-4A4E-6843-AA18-78B56D65EF38.root\\\\\\\\\\\\\\', \\\\\\\\\\\\\\'/store/mc/RunIISummer20ULPrePremix
/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/120000/00C583B9-1C11-D640-8D2C-F27E2DB5D5E3.root\\\\\\\\\\\\\\', \\\\\\\\\\\\\\'/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/120000/00CB222A-
45EA-7C4C-A5B8-DA43C984CBE9.root\\\\\\\\\\\\\\',
...

'status': 0, 'start': 1665240769, 'stop': 1665240778, 'performance': {'storage': {}, 'memory': {}, 'cpu': {}, 'multicore': {}}, 'output': {'logArchive': [{'prep_id': 'TAU-RunIISummer20UL16wmLHEGEN-00014', 'merged': False, 'pfn': 'gsiftp:/
/se.grid.vbc.ac.at:2811//eos/vbc/experiments/cms/store/unmerged/logs/prod/2022/10/8/cmsunified_task_TAU-RunIISummer20UL16wmLHEGEN-00014__v1_T_221007_220847_2422/TAU-RunIISummer20UL16wmLHEGEN-00014_0/0012/2/f515950a-ffc1-4680-a998-ab5951a9
d3fe-287-2-logArchive.tar.gz', 'checksums': {'adler32': '887ea5a0', 'cksum': '488771565'}, 'lfn': '/store/unmerged/logs/prod/2022/10/8/cmsunified_task_TAU-RunIISummer20UL16wmLHEGEN-00014__v1_T_221007_220847_2422/TAU-RunIISummer20UL16wmLHE
GEN-00014_0/0012/2/f515950a-ffc1-4680-a998-ab5951a9d3fe-287-2-logArchive.tar.gz', 'size': 0, 'events': 0, 'runs': {}, 'module_label': 'logArchive', 'location': 'T2_AT_Vienna'}], 'analysis': []}, 'input': {}, 'errors': [], 'cleanup': {}, '
parameters': {}, 'site': 'T2_AT_Vienna', 'analysis': {}, 'logs': {}}}, 'skippedFiles': [], 'fallbackFiles': ['/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/260028/B664263C-EE8E-FF4D-BF9A-E8
5E028BD93B.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/130008/F1CA2195-46DA-1044-9C20-4C0B6D212846.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mc
Run2_asymptotic_v13-v1/130021/D9A62292-3457-4948-8FDA-45FA3FAC3C10.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/120015/D9B18E38-59F5-E743-B12C-759C15EC9DD0.root', '/store/mc/RunIIS
ummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/260008/34DD277B-53F5-FD45-8F6F-35342D95975C.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/130023/E2F95
3B6-89DC-F84E-93AC-D9DDB01E958C.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gun/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/130018/2493A19D-B4A9-294A-80D6-87928C8FAF11.root', '/store/mc/RunIISummer20ULPrePremix/Neutrino_E-10_gu
n/PREMIX/UL16_106X_mcRun2_asymptotic_v13-v1/260026/8C490B56-9D1E-EC4E-889A-4019D5C86820.root'], 'Campaign': 'RunIISummer20UL16wmLHEGEN', 'PrepID': 'TAU-RunIISummer20UL16wmLHEGEN-00014', 'EOSLogURL': ''}, 'type': 'fwjr', 'timestamp': 16652
41967}]} result: b'{"error":"document_too_large","reason":"2446275-2"}\\n'
```

[2]
```
[notice] 2022-10-12T10:58:10.773692Z couchdb@127.0.0.1 <0.20732.2466> f0bbd8867d localhost:5984 127.0.0.1 user POST /wmagent_jobdump%2Ffwjrs/_bulk_docs/ 413 ok 404
```

[3]
```
2023-03-01 18:04:52,178:140522091906816:ERROR:ChangeState:Traceback (most recent call last):
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 131, in makeRequest
    result, status, reason, cached = JSONRequests.makeRequest(
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 185, in makeRequest
    result, response = self.makeRequest_pycurl(uri, data, verb, headers)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 202, in makeRequest_pycurl
    response, result = self.reqmgr.request(uri, data, headers, verb=verb,
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/Utils/PortForward.py", line 69, in portMangle
    return callFunc(callObj, url, *args, **kwargs)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Services/pycurl_manager.py", line 353, in request
    raise exc
http.client.HTTPException: url=http://localhost:5984/wmagent_jobdump%2Ffwjrs/_bulk_docs/, code=413, reason=Request Entity Too Large, headers={'Connection': 'close', 'Date': 'Wed, 01 Mar 2023 17:04:51 GMT', 'Server': 'CouchDB/3.2.2 (Erlang OTP/22)', 'Cache-Control': 'must-revalidate', 'Content-Length': '52', 'Content-Type': 'application/json', 'X-Couch-Request-ID': 'cbf54a0567', 'X-CouchDB-Body-Time': '19'}, result=b'{"error":"document_too_large","reason":"4799572-0"}\\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/JobStateMachine/ChangeState.py", line 178, in propagate
    self.recordInCouch(jobs, newstate, oldstate, updatesummary)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/JobStateMachine/ChangeState.py", line 359, in recordInCouch
    self.fwjrdatabase.queue(fwjrDocument, timestamp=True, callback=discardConflictingDocument)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 240, in queue
    self.commit(viewlist=viewlist, callback=callback)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 306, in commit
    retval = self.post(uri, data)
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Services/Requests.py", line 154, in post
    return self.makeRequest(uri, data, 'POST', incoming_headers,
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 135, in makeRequest
    self.checkForCouchError(getattr(e, "status", None),
  File "/data/srv/wmagent/v2.1.6.1/sw/slc7_amd64_gcc630/cms/wmagentpy3/2.1.6.1/lib/python3.8/site-packages/WMCore/Database/CMSCouch.py", line 178, in checkForCouchError
    raise CouchError(reason, data, result, status)
WMCore.Database.CMSCouch.CouchError: NEW ERROR STATUS! UPDATE CMSCOUCH.PY!: 413
```

Changed files:
src/python/WMCore/Database/CMSCouch.py, etc/WMAgentConfig.py, src/python/WMCore/JobStateMachine/ChangeState.py

diff --git a/etc/WMAgentConfig.py b/etc/WMAgentConfig.py
index a94765ac0..6b77ba072 100644
--- a/etc/WMAgentConfig.py
+++ b/etc/WMAgentConfig.py
@@ -99,6 +99,12 @@ config.JobStateMachine.couchurl = couchURL
 config.JobStateMachine.couchDBName = jobDumpDBName
 config.JobStateMachine.jobSummaryDBName = jobSummaryDBName
 config.JobStateMachine.summaryStatsDBName = summaryStatsDBName
+# Amount of documents allowed in the ChangeState module for bulk commits
+config.JobStateMachine.maxBulkCommitDocs = 250
+# total allowed serialized size for the FJR document that is uploaded to wmagent_jobdump/fwjrs
+# NOTE: this needs to be in sync with CouchDB couchdb.max_document_size parameter
+# see: https://docs.couchdb.org/en/latest/config/couchdb.html#couchdb/max_document_size
+config.JobStateMachine.fwjrLimitSize = 8 * 1000**2  # default: 8 million bytes (not 8MB!!!)
 
 config.section_("ACDC")
 config.ACDC.couchurl = "https://cmsweb.cern.ch/couchdb"
diff --git a/src/python/WMCore/Database/CMSCouch.py b/src/python/WMCore/Database/CMSCouch.py
index 99730b1d1..8f1ac9191 100644
--- a/src/python/WMCore/Database/CMSCouch.py
+++ b/src/python/WMCore/Database/CMSCouch.py
@@ -20,9 +20,11 @@ import urllib.request, urllib.parse, urllib.error
 
 import base64
 import hashlib
+import json
 import logging
 import re
 import time
+import sys
 from datetime import datetime
 from http.client import HTTPException
 
@@ -146,29 +148,31 @@ class CouchDBRequests(JSONRequests):
         Check the HTTP status and raise an appropriate exception.
         """
         if status == 400:
-            raise CouchBadRequestError(reason, data, result)
+            raise CouchBadRequestError(reason, data, result, status)
         elif status == 401:
-            raise CouchUnauthorisedError(reason, data, result)
+            raise CouchUnauthorisedError(reason, data, result, status)
         elif status == 403:
-            raise CouchForbidden(reason, data, result)
+            raise CouchForbidden(reason, data, result, status)
         elif status == 404:
-            raise CouchNotFoundError(reason, data, result)
+            raise CouchNotFoundError(reason, data, result, status)
         elif status == 405:
-            raise CouchNotAllowedError(reason, data, result)
+            raise CouchNotAllowedError(reason, data, result, status)
         elif status == 406:
-            raise CouchNotAcceptableError(reason, data, result)
+            raise CouchNotAcceptableError(reason, data, result, status)
         elif status == 409:
-            raise CouchConflictError(reason, data, result)
+            raise CouchConflictError(reason, data, result, status)
         elif status == 410:
-            raise CouchFeatureGone(reason, data, result)
+            raise CouchFeatureGone(reason, data, result, status)
         elif status == 412:
-            raise CouchPreconditionFailedError(reason, data, result)
+            raise CouchPreconditionFailedError(reason, data, result, status)
+        elif status == 413:
+            raise CouchRequestTooLargeError(reason, data, result, status)
         elif status == 416:
-            raise CouchRequestedRangeNotSatisfiableError(reason, data, result)
+            raise CouchRequestedRangeNotSatisfiableError(reason, data, result, status)
         elif status == 417:
-            raise CouchExpectationFailedError(reason, data, result)
+            raise CouchExpectationFailedError(reason, data, result, status)
         elif status == 500:
-            raise CouchInternalServerError(reason, data, result)
+            raise CouchInternalServerError(reason, data, result, status)
         elif status in [502, 503, 504]:
             # There are HTTP errors that CouchDB doesn't raise but can appear
             # in our environment, e.g. behind a proxy. Reraise the HTTPException
@@ -222,6 +226,13 @@ class Database(CouchDBRequests):
                     doc[label] = int(time.time())
         return data
 
+    def getQueueSize(self):
+        """
+        Return the current size of the queue, i.e., how
+        many documents are already queued up
+        """
+        return len(self._queue)
+
     def queue(self, doc, timestamp=False, viewlist=None, callback=None):
         """
         Queue up a doc for bulk insert. If timestamp = True add a timestamp
@@ -235,8 +246,8 @@ class Database(CouchDBRequests):
         if timestamp:
             self.timestamp(doc, timestamp)
         # TODO: Thread this off so that it's non blocking...
-        if len(self._queue) >= self._queue_size:
-            print('queue larger than %s records, committing' % self._queue_size)
+        if self.getQueueSize() >= self._queue_size:
+            logging.warning('queue larger than %s records, committing', self._queue_size)
             self.commit(viewlist=viewlist, callback=callback)
         self._queue.append(doc)
 
@@ -1117,82 +1128,94 @@ class CouchError(Exception):
         self.status = status
 
     def __str__(self):
-        "Stringify the error"
-        if self.status != None:
-            errorMsg = "NEW ERROR STATUS! UPDATE CMSCOUCH.PY!: %s\\n" % self.status
-        else:
-            errorMsg = ""
-        return errorMsg + "%s - reason: %s, data: %s result: %s" % (self.type,
-                                                                    self.reason,
-                                                                    repr(self.data),
-                                                                    self.result)
+        """Stringify the error"""
+        errorMsg = ""
+        if self.type == "CouchError":
+            errorMsg += "A NEW COUCHDB ERROR TYPE/STATUS HAS BEEN FOUND! "
+            errorMsg += "UPDATE CMSCOUCH.PY IMPLEMENTATION WITH A NEW COUCH ERROR/STATUS! "
+            errorMsg += f"Status: {self.status}\\n"
+        errorMsg += f"Error type: {self.type}, Status code: {self.status}, "
+        errorMsg += f"Reason: {self.reason}, Data: {repr(self.data)}"
+        return errorMsg
 
 
 class CouchBadRequestError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchBadRequestError"
 
 
 class CouchUnauthorisedError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchUnauthorisedError"
 
 
 class CouchNotFoundError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchNotFoundError"
 
 
 class CouchNotAllowedError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchNotAllowedError"
 
 class CouchNotAcceptableError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchNotAcceptableError"
 
 class CouchConflictError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchConflictError"
 
 
 class CouchFeatureGone(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchFeatureGone"
 
 
 class CouchPreconditionFailedError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchPreconditionFailedError"
 
+
+class CouchRequestTooLargeError(CouchError):
+    def __init__(self, reason, data, result, status):
+        # calculate the size of this JSON serialized object
+        docSize = sys.getsizeof(json.dumps(data))
+        errorMsg = f"Document has {docSize} bytes and it's too large to be accepted by CouchDB. "
+        errorMsg += f"Check the CouchDB configuration to see the current value "
+        errorMsg += f"under 'couchdb.max_document_size' (default is 8M bytes)."
+        CouchError.__init__(self, reason, errorMsg, result, status)
+        self.type = "CouchRequestTooLargeError"
+
+
 class CouchExpectationFailedError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchExpectationFailedError"
 
 class CouchRequestedRangeNotSatisfiableError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchRequestedRangeNotSatisfiableError"
 
 
 class CouchInternalServerError(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchInternalServerError"
 
 
 class CouchForbidden(CouchError):
-    def __init__(self, reason, data, result):
-        CouchError.__init__(self, reason, data, result)
+    def __init__(self, reason, data, result, status):
+        CouchError.__init__(self, reason, data, result, status)
         self.type = "CouchForbidden"
 
 
diff --git a/src/python/WMCore/JobStateMachine/ChangeState.py b/src/python/WMCore/JobStateMachine/ChangeState.py
index fea61eaf2..8f0c14fda 100644
--- a/src/python/WMCore/JobStateMachine/ChangeState.py
+++ b/src/python/WMCore/JobStateMachine/ChangeState.py
@@ -4,15 +4,15 @@ _ChangeState_
 
 Propagate a job from one state to another.
 """
-
+import json
+import sys
 from builtins import str
 import logging
 import re
 import time
-import traceback
 
 from WMCore.DataStructs.WMObject import WMObject
-from WMCore.Database.CMSCouch import CouchNotFoundError, CouchError
+from WMCore.Database.CMSCouch import CouchNotFoundError, CouchError, CouchRequestTooLargeError
 from WMCore.Database.CMSCouch import CouchServer
 from WMCore.JobStateMachine.SummaryDB import updateSummaryDB
 from WMCore.JobStateMachine.Transitions import Transitions
@@ -60,6 +60,28 @@ def discardConflictingDocument(couchDbInstance, data, result):
         return result
 
 
+def shrinkLargeFJR(couchDbInstance, sizeLimit):
+    """
+    Look at the CouchDB database queue and empty documents
+    with a 'fwjr' field larger than sizeLimit.
+    :param couchDbInstance: couchdb database instance
+    :param sizeLimit: integer with the limit number of bytes
+    :return: None
+    """
+    for doc in couchDbInstance._queue:
+        if sys.getsizeof(json.dumps(doc)) > sizeLimit:
+            if 'fwjr' in doc:
+                errMsg = f"The 'fwjr' attribute for jobid: {doc.get('jobid')} will be empty "
+                errMsg += f"because it is larger than the configured limit: {sizeLimit} bytes."
+                logging.warning(errMsg)
+                doc['fwjr'] = dict()
+            else:
+                errMsg = f"Found a too large Couch document for job id: {doc.get('jobid')}, which is "
+                errMsg += f"actually missing the 'fwjr' attribute. Current limit: {sizeLimit} bytes."
+                logging.error(errMsg)
+    return
+
+
 def getDataFromSpecFile(specFile):
     workload = WMWorkloadHelper()
     workload.load(specFile)
@@ -89,6 +111,8 @@ class ChangeState(WMObject, WMConnectionBase):
         self.jsumdatabase = None
         self.statsumdatabase = None
 
+        # max total number of documents to be committed in the same Couch operation
+        self.maxBulkCommit = getattr(self.config.JobStateMachine, 'maxBulkCommitDocs', 250)
         self.couchdb = CouchServer(self.config.JobStateMachine.couchurl)
         self._connectDatabases()
 
@@ -101,6 +125,7 @@ class ChangeState(WMObject, WMConnectionBase):
         self.getWorkflowSpecDAO = self.daofactory("Workflow.GetSpecAndNameFromTask")
 
         self.maxUploadedInputFiles = getattr(self.config.JobStateMachine, 'maxFWJRInputFiles', 1000)
+        self.fwjrLimitSize = getattr(self.config.JobStateMachine, 'fwjrLimitSize', 8 * 1000**2)
         self.workloadCache = {}
         return
 
@@ -110,7 +135,8 @@ class ChangeState(WMObject, WMConnectionBase):
         """
         if not hasattr(self, 'jobsdatabase') or self.jobsdatabase is None:
             try:
-                self.jobsdatabase = self.couchdb.connectDatabase("%s/jobs" % self.dbname, size=250)
+                self.jobsdatabase = self.couchdb.connectDatabase("%s/jobs" % self.dbname,
+                                                                 size=self.maxBulkCommit)
             except Exception as ex:
                 logging.error("Error connecting to couch db '%s/jobs': %s", self.dbname, str(ex))
                 self.jobsdatabase = None
@@ -118,7 +144,8 @@ class ChangeState(WMObject, WMConnectionBase):
 
         if not hasattr(self, 'fwjrdatabase') or self.fwjrdatabase is None:
             try:
-                self.fwjrdatabase = self.couchdb.connectDatabase("%s/fwjrs" % self.dbname, size=250)
+                self.fwjrdatabase = self.couchdb.connectDatabase("%s/fwjrs" % self.dbname,
+                                                                 size=self.maxBulkCommit)
             except Exception as ex:
                 logging.error("Error connecting to couch db '%s/fwjrs': %s", self.dbname, str(ex))
                 self.fwjrdatabase = None
@@ -127,7 +154,8 @@ class ChangeState(WMObject, WMConnectionBase):
         if not hasattr(self, 'jsumdatabase') or self.jsumdatabase is None:
             dbname = getattr(self.config.JobStateMachine, 'jobSummaryDBName')
             try:
-                self.jsumdatabase = self.couchdb.connectDatabase(dbname, size=250)
+                self.jsumdatabase = self.couchdb.connectDatabase(dbname,
+                                                                 size=self.maxBulkCommit)
             except Exception as ex:
                 logging.error("Error connecting to couch db '%s': %s", dbname, str(ex))
                 self.jsumdatabase = None
@@ -136,7 +164,8 @@ class ChangeState(WMObject, WMConnectionBase):
         if not hasattr(self, 'statsumdatabase') or self.statsumdatabase is None:
             dbname = getattr(self.config.JobStateMachine, 'summaryStatsDBName')
             try:
-                self.statsumdatabase = self.couchdb.connectDatabase(dbname, size=250)
+                self.statsumdatabase = self.couchdb.connectDatabase(dbname,
+                                                                    size=self.maxBulkCommit)
             except Exception as ex:
                 logging.error("Error connecting to couch db '%s': %s", dbname, str(ex))
                 self.jsumdatabase = None
@@ -181,8 +210,7 @@ class ChangeState(WMObject, WMConnectionBase):
             logging.exception(msg)
             raise
         except Exception as ex:
-            logging.error("Error updating job in couch: %s", str(ex))
-            logging.error(traceback.format_exc())
+            logging.exception("Error updating job in couch: %s", str(ex))
 
         return
 
@@ -214,6 +242,7 @@ class ChangeState(WMObject, WMConnectionBase):
         timestamp = int(time.time())
         couchRecordsToUpdate = []
 
+        countDocs = 0
         for job in jobs:
             couchDocID = job.get("couch_record", None)
 
@@ -275,6 +304,8 @@ class ChangeState(WMObject, WMConnectionBase):
 
                 couchRecordsToUpdate.append({"jobid": job["id"],
                                              "couchid": jobDocument["_id"]})
+                if countDocs >= self.jobsdatabase.getQueueSize():
+                    self.jobsdatabase.commit(callback=discardConflictingDocument)
                 self.jobsdatabase.queue(jobDocument, callback=discardConflictingDocument)
             else:
                 # We send a PUT request to the stateTransition update handler.
@@ -356,6 +387,20 @@ class ChangeState(WMObject, WMConnectionBase):
                                 "archivestatus": archStatus,
                                 "fwjr": jsonFWJR,
                                 "type": "fwjr"}
+                if countDocs >= self.fwjrdatabase.getQueueSize():
+                    try:
+                        # TODO: CouchDB bulk insert may fail at any given document. That means, only by
+                        # parsing the response object we can actually see which documents succeeded or not.
+                        # https://docs.couchdb.org/en/stable/api/database/bulk-api.html#updating-documents-in-bulk
+                        self.fwjrdatabase.commit(callback=discardConflictingDocument)
+                    except CouchRequestTooLargeError as exc:
+                        msg = "Failed to commit bulk of framework job report to CouchDB."
+                        msg += f" Details: {str(exc)}"
+                        logging.warning(msg)
+                        shrinkLargeFJR(self.fwjrdatabase, self.fwjrLimitSize)
+                        # now all the documents should fit in
+                        self.fwjrdatabase.commit(callback=discardConflictingDocument)
+
                 self.fwjrdatabase.queue(fwjrDocument, timestamp=True, callback=discardConflictingDocument)
 
                 updateSummaryDB(self.statsumdatabase, job)
@@ -454,6 +499,8 @@ class ChangeState(WMObject, WMConnectionBase):
                                 jobSummary[prop] = jobSummary[prop] if jobSummary[prop] else currentJobDoc.get(prop, [])
                         except CouchNotFoundError:
                             pass
+                    if countDocs >= self.fwjrdatabase.getQueueSize():
+                        self.jsumdatabase.commit()
                     self.jsumdatabase.queue(jobSummary, timestamp=True)
 
         if len(couchRecordsToUpdate) > 0:
@@ -461,8 +508,16 @@ class ChangeState(WMObject, WMConnectionBase):
                                      conn=self.getDBConn(),
                                      transaction=self.existingTransaction())
 
+        try:
+            self.fwjrdatabase.commit(callback=discardConflictingDocument)
+        except CouchRequestTooLargeError as exc:
+            msg = "Failed to commit bulk of framework job report to CouchDB."
+            msg += f" Error: {str(exc)}"
+            logging.warning(msg)
+            shrinkLargeFJR(self.fwjrdatabase, self.fwjrLimitSize)
+            # now all the documents should fit in
+            self.fwjrdatabase.commit(callback=discardConflictingDocument)
         self.jobsdatabase.commit(callback=discardConflictingDocument)
-        self.fwjrdatabase.commit(callback=discardConflictingDocument)
         self.jsumdatabase.commit()
         return
 
@@ -581,5 +636,4 @@ class ChangeState(WMObject, WMConnectionBase):
                 updateUri += "?location=%s" % (location)
                 self.jobsdatabase.makeRequest(uri=updateUri, type="PUT", decode=False)
         except Exception as ex:
-            logging.error("Error updating job in couch: %s", str(ex))
-            logging.error(traceback.format_exc())
+            logging.exception("Error updating job in couch: %s", str(ex))