Repository: dmwm/wmcore
Issue: File mismatch investigation

**Impact of the bug**
Identify issue with stuck PnR workflows and allow them to progress.

**Describe the bug**
As was described in [PnR#41](https://gitlab.cern.ch/CMSToolsIntegration/operations/-/issues/41) we have certain level of workflows which have file mismatch between DBS and Rucio and they never get announced for months. The filemismatch usually happens when there are output files in DBS but not recorded by rucio, here is one example:
```
file: /store/data/Run2022D/Muon/RAW-RECO/ZMu-10Dec2022-v1/2550000/025d55a6-8e48-4695-9087-80bf9681229b.root
```
for WF:  `pdmvserv_Run2022D_Muon_10Dec2022_221209_230856_3545`. A recent list of workflows having such issue can be found [here](https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=task_EGM-Run3Winter23wmLHEGS-00001).

**How to reproduce it**
Most likely we need to perform debugging of agent logs and its local databases to identify the issue with stuck workflows.

**Expected behavior**
We should have a file match between DBS and Rucio to allow WF to progress.

**Additional context and error message**


Changed files:
src/python/WMCore/Storage/Backends/GFAL2Impl.py, test/python/WMCore_t/Storage_t/Backends_t/GFAL2Impl_t.py

diff --git a/src/python/WMCore/Storage/Backends/GFAL2Impl.py b/src/python/WMCore/Storage/Backends/GFAL2Impl.py
index 2f8b5fda0..6e1d5f2b1 100644
--- a/src/python/WMCore/Storage/Backends/GFAL2Impl.py
+++ b/src/python/WMCore/Storage/Backends/GFAL2Impl.py
@@ -27,6 +27,8 @@ class GFAL2Impl(StageOutImpl):
         self.setups = "env -i X509_USER_PROXY=$X509_USER_PROXY JOBSTARTDIR=$JOBSTARTDIR bash -c '%s'"
         self.removeCommand = self.setups % '. $JOBSTARTDIR/startup_environment.sh; date; gfal-rm -t 600 %s '
         self.copyCommand = self.setups % '. $JOBSTARTDIR/startup_environment.sh; date; gfal-copy -t 2400 -T 2400 -p %(checksum)s %(options)s %(source)s %(destination)s'
+        self.copyOpts = '-t 2400 -T 2400 -p -v --abort-on-failure %(checksum)s %(options)s %(source)s %(destination)s'
+        self.copyCommand = self.setups % ('. $JOBSTARTDIR/startup_environment.sh; date; gfal-copy ' + self.copyOpts)
 
     def createFinalPFN(self, pfn):
         """
@@ -87,6 +89,8 @@ class GFAL2Impl(StageOutImpl):
           -T   global timeout for the transfer operation
           -p   if the destination directory does not exist, create it
           -K   checksum algorithm to use, or algorithm:value
+          -v   enable the verbose mode (-v for warning level)
+          --abort-on-failure  abort the whole copy as soon as one failure is encountered
         """
         result = "#!/bin/bash\\n"
 
diff --git a/test/python/WMCore_t/Storage_t/Backends_t/GFAL2Impl_t.py b/test/python/WMCore_t/Storage_t/Backends_t/GFAL2Impl_t.py
index 5bb8ba6d8..9c856611f 100644
--- a/test/python/WMCore_t/Storage_t/Backends_t/GFAL2Impl_t.py
+++ b/test/python/WMCore_t/Storage_t/Backends_t/GFAL2Impl_t.py
@@ -17,8 +17,8 @@ class GFAL2ImplTest(unittest.TestCase):
         removeCommand = "env -i X509_USER_PROXY=$X509_USER_PROXY JOBSTARTDIR=$JOBSTARTDIR bash -c " \\
                         "'. $JOBSTARTDIR/startup_environment.sh; date; gfal-rm -t 600 %s '"
         copyCommand = "env -i X509_USER_PROXY=$X509_USER_PROXY JOBSTARTDIR=$JOBSTARTDIR bash -c '" \\
-                      ". $JOBSTARTDIR/startup_environment.sh; date; gfal-copy -t 2400 -T 2400 " \\
-                      "-p %(checksum)s %(options)s %(source)s %(destination)s'"
+                      ". $JOBSTARTDIR/startup_environment.sh; date; gfal-copy -t 2400 -T 2400 -p " \\
+                      "-v --abort-on-failure %(checksum)s %(options)s %(source)s %(destination)s'"
         self.assertEqual(removeCommand, testGFAL2Impl.removeCommand)
         self.assertEqual(copyCommand, testGFAL2Impl.copyCommand)