cms-sw/cmssw — pull request #4505 by makortel (2014-07-03 12:29:51+00:00)
Title: Complete Geant4 MT integration

This PR completes (almost, see below for a known issue) the integration of Geant4 MT into multithreaded CMSSW. Most modifications are again in MT-specific classes, but some classes used by the serial OscarProducer were also modified.

Changes affecting OscarProducer (results should stay the same)
- Use G4Random::setTheEngine() instead of CLHEP::HepRandom::setTheEngine()
- Otherwise incorrect random number engine is used if run in MT G4 build
- Serial G4 build has ```#define G4Random CLHEP::HepRandom```, so should be ok
- Fix CustomUIsession to redirect G4cout/G4cerr to LogInfo/LogWarning
- New class sim::ChordFinderSetter, abstracted out of sim::FieldBuilder, pass it to PhysicsListMakerBase::make()
- The only reason sim::FieldBuilder was passed to physics list was that G4MonopoleTransporation needs to switch between two G4ChordFinder objects. PhysicsListMakerBase::make() must be called in master thread, and sim::FieldBuilder::build() in all worker threads. This separation was the fastest way forward, althought it required changing the interface of PhysicsListMakerBase::make(), and percolating to all physics list classes.

Changes in OscarMTProducer
- Merged RunManagerMTInit to OscarMTMasterThread
- OscarMTMasterThread is now in the GlobalCache
- The master std::thread is started in initializeGlobalCache(), and joined in globalEndRun()
- Begin/end run is signalled to the master thread via enumeration variable and with mutex+condition_variable
- Technically supports multiple runs in a job
- Supported at the same level as in OscarProducer
- E.g. geometry and magnetic field are obtained from EventSetup only during the first Run
- All functionality in OscarProducer+RunManager should now be in OscarMTProducer+OscarMTMasterThread+RunManager+RunManagerMTWorker

I have tested with 100 MinBias events (wf 1.0) (in CMSSW_7_2_GEANT10_X_2014-07-01-0200+#4457+#4476)
- OscarProducer and OscarMTProducer with 1 thread give identical results
- OscarMTProducer with replay mode of RandomNumberGeneratorService checked manually with a couple of quantities
- OscarMTProducer without replay mode checked with runEdmFileComparison.py
- But OscarMTProducer with >= 2 threads gives **different** results for some events (randomly, roughly in 1 event in 10-20 events)
- **Something to be followed up**

In addition I've tested with 200 TTBar events (wf 1302.0) that OscarMTProducer works technically at least up to 20 threads.

To ease testing with cmsDriver.py/runTheMatrix.py, I've included a customise-script to change all OscarProducers in a Process to OscarMTProducers.
```
cmsDriver.py ... --customise SimG4Core/Application/customiseMultithreadedSim.customiseMultithreadedSim --suffix "-n NUMTHREADS"
runTheMatrix.py ... --command="--customise SimG4Core/Application/customiseMultithreadedSim.customiseMultithreadedSim --suffix '-n NUMTHREADS'" --apply 0
```

Following the Geant4 threading model, there are quite a lot of new thread_local statics (especially in RunManagerMTWorker, but also elsewhere). At least the RunManagerMTWorker ones could probably be cleaned up with the "usual tricks" in the future.