Repository: dmwm/wmcore
Issue: Big jsons uploaded to WMArchive 

**Impact of the bug**
WMAgent, WMArchive

**Describe the bug**
While debugging missing monitoring information from ES the Monit team has discovered some quite big json files uploaded to WMarchive and ES [1]. The document's size has been of the order of  2MBs. The documents themselves seem to be sane. Their origin are few workflows which tend to create huge LFNArrays for the input files for the job itself. This  is a behavior on which we need more feedback from other groups like P&R and PDMv in order to estimate the expected future frequency of those. 

In the meantime there was a temporary solution applied by Monit side  so that  we  may continue to use the monitoring properly (all is well explained in the relevant ticket [1]), but we cannot rely on that in the long term. So this ticket is expected to be a  follow up on the decisions taken on our side to revisit the information which we upload to WMArchive and ES.  

[1]
https://cern.service-now.com/service-portal?id=ticket&table=incident&n=INC2961279

**How to reproduce it**
N/A 
In order to reproduce  those huge documents we need some feedback so we can relate to a speciffic type of workflows/campaigns.

**Expected behavior**
Documents uploaded to WMArchive and ES to be in the order of kilobytes rather than megabytes.

**Additional context and error message**
N/A


Changed files:
src/python/WMCore/Services/WMArchive/DataMap.py

diff --git a/src/python/WMCore/Services/WMArchive/DataMap.py b/src/python/WMCore/Services/WMArchive/DataMap.py
index 06ddfbfa1..1523a4cb8 100644
--- a/src/python/WMCore/Services/WMArchive/DataMap.py
+++ b/src/python/WMCore/Services/WMArchive/DataMap.py
@@ -1,6 +1,5 @@
 from __future__ import (division, print_function)
 from builtins import str
-from past.builtins import basestring
 from future.utils import viewitems, listvalues, listitems
 
 import copy
@@ -13,18 +12,17 @@ WMARCHIVE_TOP_LEVEL_REMOVE_OUTSIDE_LAYER = ["steps"]
 # convert data format under stpes["cmsRun1"/"logArch1"/"stageOut1"]["output"]
 WMARCHIVE_REMOVE_OUTSIDE_LAYER = ["checksum", "dataset"]
 # convert to list from str
-WMARCHIVE_CONVERT_TO_LIST = ["OutputPFN", "lfn"]
+WMARCHIVE_CONVERT_TO_LIST = ["lfn"]
 
-WMARCHIVE_DATA_MAP = {"OutputPFN": "outputPFNs", "inputPath": "inputDataset",
-                      "lfn": "outputLFNs", "input": "inputLFNs", "inputpfns": "inputPFNs"}
+WMARCHIVE_DATA_MAP = {"inputPath": "inputDataset",
+                      "lfn": "outputLFNs", "input": "inputLFNs"}
 
-WMARCHIVE_REMOVE_FIELD = ["InputPFN", "pfn", "user_dn", "user_vogroup", "user_vorole"]
+WMARCHIVE_REMOVE_FIELD = ["user_dn", "user_vogroup", "user_vorole"]
 WMARCHIVE_COMBINE_FIELD = {"outputDataset": ["primaryDataset", "processedDataset", "dataTier"]}
 
 WMARCHIVE_LFN_REF_KEY = ["lfn", "files"]
-WMARCHIVE_PFN_REF_KEY = ["pfn"]
-WMARCHIVE_FILE_REF_KEY = {"LFN": WMARCHIVE_LFN_REF_KEY,
-                          "PFN": WMARCHIVE_PFN_REF_KEY}
+WMARCHIVE_PFN_REF_KEY = []
+WMARCHIVE_FILE_REF_KEY = {"LFN": WMARCHIVE_LFN_REF_KEY}
 
 ERROR_TYPE = {'exitCode': int}
 
@@ -94,7 +92,6 @@ STEP_DEFAULT = {  # 'name': '',
         # 'guid': '',
         # 'inputDataset': '',
         'inputLFNs': [],
-        'inputPFNs': [],
         # TODO change to empty string from None
         # 'location': '',
         # 'merged': False,
@@ -102,7 +99,6 @@ STEP_DEFAULT = {  # 'name': '',
         # 'output_module_class': '',
         # 'outputDataset': '',
         'outputLFNs': [],
-        'outputPFNs': [],
         # 'prep_id': '',
         # 'processingStr': '',
         # 'processingVer': -1,
@@ -125,6 +121,23 @@ STEP_DEFAULT = {  # 'name': '',
 }
 
 
+def cleanStep(idict):
+    """
+    perform clean-up of PFNs attributes in place for given dictionary
+
+    :param idict: a FWJR report dictionary
+    :return: a dictionary without PFNs
+    """
+    for step in ['input', 'output']:
+        data = idict.get(step, {})
+        for key, values in data.items():
+            for elem in values:
+                for skip in ['pfn', 'InputPFN', 'OutputPFN', 'inputpfns']:
+                    if skip in elem:
+                        del elem[skip]
+            data[key] = values
+    return idict
+
 def combineDataset(dataset):
     dataset["outputDataset"] = "/%s/%s/%s" % (dataset.pop("primaryDataset"),
                                               dataset.pop("processedDataset"),
@@ -300,12 +313,12 @@ def convertStepValue(stepValue):
 
     return stepValue
 
-
 def convertSteps(steps):
     stepList = []
     for key, value in viewitems(steps):
         stepItem = {}
         stepItem['name'] = key
+        value = cleanStep(value)
         stepItem.update(convertStepValue(value))
         _validateTypeAndSetDefault(stepItem, TOP_LEVEL_STEP_DEFAULT)
         stepList.append(stepItem)