ok, this was the hot split.
given that there are more trackRef than gsf the fix proposed may amortize the cost.
I still prefer the c++ solution at
http://cmslxr.fnal.gov/lxr/source/RecoTauTag/RecoTau/src/RecoTauQualityCuts.cc?v=CMSSW_6_2_0#012