Data Quality Punch List

 Item #45
 Date: January 29, 2005
 Description: Jadzia says"
  Since 2nd January Power Outage cft stereo data might be compromised. I am 
  wandering if you noticed something that might point to that. I would be 
  eager to go through the plots myself if you would point me to the data. 
  Idealy I would like to see number of cft axial and cft stereo hits on 
  tracks.
  As you probably know D0 is now reluctant to paricipate in available 
  accesses because of fear of ramping of solenoid. Therefore I need to have 
  evidence of the data being compromised before requesting access to swap 
  bad module." 
  Marj says: "I don't have hits on tracks, just hot or dead fibers. 
  I didn't notice a change, but I can look more closely. Wouldn't the
  vertex examine find a serious problem with the tracking? Marj"

 Action: Point to RecoCert directory. Submitted more. Forwarded to Marj.
 Resolution: 

 Item #44
 Date: December 14, 2004 
 Description: The lumy operations report says run 201585 had some problems.
       Well, it's 223.59 nb-1 "lost in daq". I looked and found 458095
       events written to tape in this run.  
 Action: Email to lumy group.
 Resolution: Lumy group fixed it. Tim Andeen, I think.

 Item #43
 Date: July 19, 2004
 Description: From K. Harder: Last week we had a special run for muon and 
       SMT, and I am trying to figure out whether the track reconstruction 
       looked normal. The captain was unable to start the vertex examine, 
       and it seems that all the GM code was not running either. Are there 
       RecoCert plots or anything else available for this run?  The run 
       number is 195223, a special muon runs with new PDT DSP code.
 Action: Standard run quality evaluation.
 Resolution: SMT Reasonable.

 Item #42
 Date: June 29,2004
 Description: Please recocert 194556 and 194557 TV 12.37 and 13.02 or 3.
 Action: HTD. Done. 
 Resolution: See July 9 ADM on differences in RecoCert plots for 
             TV 12 vs 13.

 Item #41
 Date: April 25, 2004
 Description: Weber: "run 192308 is bad for SMT. Crate x60 was not reading out.
       Crate monitoring was unfortunately not running in crate 60 for that 
       run... No entries in SMT examines, zero occupancy from crate 60 and
       also the 'cft only' track vs phi show a pronounced enhancement (vertex 
       examine).
         It might be useful to look at recocert for 192307 and 192309 (one run 
       before and one run after) to make sure things are fine there.
 Action: HTD ran RecoCert for 192307, 8, 9 on April 30.
 Resolution: SMT reasonable on 192307 and bad on 192308 and 9.

 Item #40
 Date: April 22, 2004
 Description: Tom Nunneman ran RecoCert and noted "Run 192079, which has 
       been selected as the new reference run, shows many significant 
       differences to the old reference.  There is a very prominent problem 
       in the calorimeter at -1<eta<0, -1<phi<0!"
 Action: Email calorimeter experts Petroff and Nelson. They replied 
       "We have noise in crate 9 ADC 6 and 7. The BLS power supply was out 
       of order. We changed the BLS power supply Sunday April 18 between 8 
       and 9 pm..this  explain the hole in the data...  PP".
 Resolved: April 22, 2004.

 Item #39
 Date: April 9, 2004
 Description: "A B-Layer PDT was dead in L2 and maybe the readout". What
       is seen in RecoCert for run 191617?
 Action: HTD ran RecoCert April 16. The plot "N" (hits) shows PDT noise in 
       C-layer Octant 2. But muon examine plots did not indicate
       any dead PDT's in readout. There's no evidence in recocert for a dead
       PDT that I could see. The online examine displays occupancy
       in each PDT, so it's more sensitive anyway.
 Resolved: Maybe a dead PDT in L2 but not dead in Readout. May 26, 2004. HTD.

 Item #38
 Date: April 9, 2004
 Description: "Calorimeter noise occurred". What is seen in RecoCert 
              for run 191616?
 Action: HTD ran RecoCert April 16. Looked at results May 26. Met
       and components are normal, though a bit wider than the comparison 
       run. There's more cells hit and more total energy in the 
       calorimeter but little to no obvious noisy area.
 Resolved: By declaration May 26, 2004.

 
 Item #37
 Date: March 3, 2004
 Description: Emily Nurse shows Z->mumu has poorer width (ratio is 14/9 fwhm)
       and more muon tracks with 0 smt hits. The tracking eff'y is 1 sigma
       lower than preshutdown data. The CFT shows normal number of hits
       for postshudown data (not quite as many as before). The postshutdown 
       fraction is 50% tracks with no SMT hits. Preshutdown was 17%, if I 
       understood clearly.  
 Action: Ginther, Choi, Nomerotski, Diehl, Sanders, Nurse, & Weber met 
       to think. We think the width is worse because of worse momentum 
       resolution with no smt hits.  We don't think Emily has focussed on 
       runs where we know of SMT problems or muon problems, such as 
       SMT off or toroid open. Some postshutdown Recocert plots show number 
       of smt hits looks much more like normal SMT tracking.  
         We don't yet know what the problem is. Emily is to check which 
       runs have problems. We'll focus on that data. - HTD 3/3/4. 
         This turned out to be a code bug. George put us onto it in 
       his rough cross section calculation.
 Resolved: March 4, 2004.

 Item #36
 Date: March 1, 2004
 Description: Run 189818 and runs since 18:30ish Feb 29 through 16:00ish
       March 1 have some kind of noise in NEC and in N bottom PDT's. 
       Are these runs any good. Please run recocert when on, say, 
       189818 when it's available. HTD
 Action: HTD ran RecoCert April 30, 2004.
 Resolved: Reasonable runs. 

 Item #35
 Date: February 13, 2004
 Description: Run 188957. Dean S. writes: "However, in this case I believe 
       that the problem was sever enough that had anyone actually looked at 
       one event on the event display monitor with the standard zero bias 
       trigger one would have known something was wrong. As Robert has 
       recently found, the "bad BLS" is most likely a failed capacitor.  
       This has happened before, but that time the BLS crate tripped, 
       and we thought we had a failed BLS supply.  This time it was not a 
       "good" a short, so only appears to add AC ripple to the power rails.  
       This causes noise  in both the L1 CAl trigger, and in the precision 
       readout system for all BLS cards in that crate (16 trigger towers, 
       64 readout towers).  So the "fix" applied last night to suppress 
       the one really bad BLS card, was not enough.  And the present attempt 
       to kill that one trigger tower in L1 is very unlikely to work.
       If that does not work (which I suspect), the only "reasonable" thing to 
       do, is power down that BLS crate, and wait for an access to 
       replace the BLS card.  Marking all the data taken this store as 
       "BAD CAL".
 Action: Please recocert. 
 Resolution: Done.

 Item #34
 Date: February 2, 2004
 Description: Silke N., on shift, notes that after a pedestal run,
              zero bias run 188917 shows little evidence of a warm 
              region that occurred for 1/2 the previous store. The 
              last run of the previous store was 188911. Now, reco uses
              pedestals taken 1 or 2 years ago. Of course, those are 
              barely relevant. Zero bias, new peds, MET components are 0.8 
              and 0.4 Gev. The question is, what is the value Reco gets
              on previous run? Have we trashed the data?  The 
              GM plot for 188911 shows warm jets in eta 0.3 to 1.0 at 
              phi=6.
                Silke says this is part of 2 ADCs. 
 Action:      Please run Recocert on 188911 when it is reco'd.
              Lima reco'd it Feb 5th. 
              Metx mean is -1.37. Mety is at 2.16. Overall, worse.
              Jet cone NC comparison plots failed. but the warm region
              stands out in eta and phi. Also in good jets. Cal all cells 
              shows the warm region. All cc cells is even more clear.
              The jets are rotten-looking.
                I'd like to see what happens if better peds are used. 
              I asked SN to do so.

              April 30: Silke has made a couple of tries and she is checking 
              thta she has repedestaled the correct cells.
 Resolution: Done. It helped. We didn't write a D0Note, though I wanted 
              to. Silke wrote a talk for the Calgo meeting.

  
 Item #33
 Date: January 30,2004
 Description: Top group throws out runs including 180301-180305. There's
              more MET_x. It's peaked ~-4 GeV. Met_y is similar, but not as 
              far, offset. There's extra jets at ieta = -8 at phi = 6.3 and
              1.5. The logbook for Aug. 20, 2003 indicates there 
              are no hot cells in this area but that there is a "warm region
              in crate 9". It seems to simply go away without solution 
              and have appeared without cause. What happened? (Above
              info is form recocert.
 Call list:   Carried plots to an expert.
 Action:      Silke writes: "The files are not (anymore) on the CAL disk, 
              so I could not actually look to see a hot tower.
              I then started looking at the log book and see:
              (ieta,iphi)=(18,1) Layer FH2 (12) 
                (crate 7 adc 10 bls 6 tower 3 depth 8) 
                E/event: 5.63 RMS: 0.38 occupancy: 1.00, 
              the pT is about 2GeV so I decided not to kill it."
              Details on the pedestals were provided.
              "So there was a hot cell, that didn't quite make the cut to 
              be killed online, but it was suppressed in the next run as 
              the pedestal/sigma was too high.
                So there was a hot tower in addition to the one cell that 
              did not make the "online-suppression"? Laurent also thinks to 
              have seen a warm region, but with these plots, it's hard to tell.
              So this here is a case where you have to decide whether to 
              kill the cell in the run before the pedestals killed it also, 
              or not. But old/new pedestals would not have cured it.
              But then, it was not a hot tower. If there was one and you have 
              the coordinates (any: ieta. iphi is ok), let me know and I'll 
              look.
                About the warm region, I forgot again that there is actually 
              still code missing that would need to be written. I'll see if 
              I find the time to do that today, but this will definitely 
              cause some delay. I'll let you know when I've some further."
 Resolved Date and Comment: April 30, 2004. I think the warm regions we 
              experienced around this time may have been precursors of 
              the "torrid torroid" noise. (Not all agree). -HTD
 
 Item #32 
 Date: January 28, 2004
 Description: It's unclear whether central muon OK in 188032. This applies 
              to all runs Jan 11 to Jan 13. There are NO notes in the 
              GM log.
 Call List:   HTD
 Description: I submitted RecoCert on it 01/28/04.
              207 BC stubs in 100,000 events, 7000 unmatched muons.
              187734 202 in 60,000 events and 4600 unmatched. 
              I don't see how I can declare the runs bad. It turns 
              out to be a shifgter screw up. Linda Stutte showed me how
              to look at the muon rootuples and they are OK.
 Resolved:    01/29/04. HTD

 Item #31 
 Date: January 28, 2004
 Description: It's unclear whether MDT SBWT is OK in 187714
 Call List:   HTD
 Description: I submitted RecoCert on it 01/28/04.
 Resolved:    01/29/04 Seems OK. HTD

 Item #30
 Date: January 24, 2004
 Description: Marco reports checkerboard pattern in calorimeter runs.
              EM, in particular, one has to look at a lot of runs to
              see it.
 Call List:   HTD
 Description: I'd like a run number - something - to go on. I've never seen
              it in the runs I've looked at. MV says one needs a lot of 
              EM data to see it.
 Resolved:    On or about end of winter shutdown this was determined to be 
              due to the cal gain switching errors due to timing problems
              in electronics.

 Item #29
 Date: January 15, 2004
 Description: Sanders reports: "In run 188156, there is a spike around 
              phi=3.2 in the global track distribution, as seen by PhysEx. 
              Do we see this offline?"
 Call List:   HTD or offline shifter.
 Action:      HTD submitted 01/24/04.
              I looked at the recocert plots. There is what I'd call a 
              nubbin of tracks at phi = 3.2 but not a spike. There is a 
              problem with one of the smt layers (4) at that phi.
                I asked M.S. if the plots I sent him showed the same thing as 
              the online.

 Item #28
 Date: January 11, 2004
 Description: HTD reports: "I'm looking at run 187230 using Recocert. The 
              plots are in /rooms/greatlakes/michigan/dataquality/jobs
              /run187230-p140600... I'm comparing to 
              run180951-p140302-DST/run180951-p140302-DST.root
              The tracking plots seem poor in the newer data. Specifically,
              "Tracks All" shows:  many fewer tracks in "ntrks"
              otherwise same?
                "Hits All" seems normal, except seems to be fewer with 16 
              CFT hits. I can't see where they went.
                "Isolated Tracking Eff'y" has lower efficiencies in low 
              trk-multiplicity events.
                Tracking in Jets shows overall lower tracking eff'y. 
                Electrons, matched muons about same, but they have tracking 
              eff'y normalized out.
                There's more unmatched muons.
                Jets are similar except the new run has nothing with F90 > 0.8.
              (I don't think that's related to this problem).
                "Detector Occupancy" says There's way fewer CFT clusters, 
              cal cells, and cps clusters.  This seems to me to be at the 
              heart of the problem.  But "Cal All cells" seems same and 
              SMT cluster plots seems same. 
                There's more dead CFT channels, but I expected that due 
              to post-shutdown problems. CFT occupancy looks about the same.
                There's fewer muon local tracks but I recall normalization 
              problems in this figure of old.
                The Reco timing is significantly faster as if there's fewer 
              clusters.
 Call List: HTD sent email to Melanson and Suyong.
 Action:      HTD ran recocert on more runs. 187800 has similar number of 
            tracks but the eff'y in isolated tracks doesn't look terrible
            at low-pT. Other distributions look similar. Note 187800 is
            p14.06.00. 187230 is same.
              Drew Alton asked about recent RecoCert stuff so I steered him
            at this run. He reports:
            "...page 26.  The number of cps clusters, the number of strips 
            on those clusters, the phi distribution all look very bad.  
            I don't think it leads to the other problem [cft clusters - htd].
            I haven't checked if it's present in other runs, but I figured out
            most of the bug that caused it...it's database constants.  As you 
            know the CFT (and cps) map changed.  When Fred first told us 
            about this he said crate 51 VRB 15 moved to VRB 19, but in fact 
            crate 52 VRB 15 moved to VRB 19.  When I prepared my mip 
            calibration numbers I used the swap for crate 51.  The CPS 
            channels in crate 51 VRB 15 are now coming out without MIP 
            calibrations.  (I don't understand this, because I thought the 
            DB would use the calibration from the previous calibration
            but it smells too much like the problem).  The thing that confuses
            me more is that the actual channels that moved: 
            crate 52 VRB 15 ->19 all seem to be fine.  I guess the DB doesn't 
            work the way I thought it should.  I will get together with 
            Brent and fix this today."

            Drew further replies: "I guess all runs processed since the 
            shutdown are screwed up."

            Brigitte Vachon says (well she didn't actually say it). "Hey, 
            don't try to pin this on the CTT". Actually she asked if the 
            CFT calibration was correct.
            
            HTD forwarded this to Fred, Jadwiga and Stefan. 

            Jadwiga says: "No, there is no run with bad calibration 
            constants. We look online at occupancy plots after calibration 
            is applied for all runs. All runs have expected occupancies. 
            The calibration run number for this run is 187158. I checked that 
            online database has data for this run.  I have log files from 
            transfer to offline databaes, this log file does not show anything
            unusual. May be Taka would be willing to look at offline data 
            base and compare, for example, with the next calibration run 
            187661."

            Taka says: "I find nothing unusual about the pedestals in the 
            offline database for the calibration runs 187158. 96% of channels 
            have values between 50 and 80. For the calibration run 187661, 
            this value is 95%. Do I need to look these in more details?"
            Further: "97.7% of channels in the thresholds table have values 
            between 50 and 100 for run 187158. 97.6% for run 187661.

            HTD: I now believe most of these problems are due to axes 
            differences in p14.06.00 vs p14.03.02 RecoCert. I still don't
            understand why 187230 has low eff'y in the low-pT end of 
            pT vs trk multiplicity on page 4. 

            Jan-29: Start keeping stats on Runs
             187734 trkmulty not ok
             187714 trkmulty not ok
             188032 trkmulty ok

 Item #27
 Date: January 6, 2004
 Description: Horst Wahl notices phi dip in Vertex Examine. It seems to
              be a persistent thing. He writes: "VertexExamine shows dips 
              in the L3 track phi distribution at phi = 1.5, 4.5; (also seen 
              in hit distributions) -- all due to mapping and old algorithm?
              many CFT only tracks, with peak at phi=2, dips at phi=1.5, 4.5."
              We are in runs 187804-812, for instance.
 Call List:   HTD sent email to Sanders.
 Action:      Sanders reports this is fixed up Jan 15th: "Vertex examine 
              will have similar tracking as L3 later today, to "fix" dead 
              cft fibres, as is done in L3 itself.
 Resolved:    Jan 15th, 2004.

 Item #26
 Date: December 19, 2003
 Description: Jadwiga and Mike H. note the post-shutdown CFT dead 
              channels. Not understood. Perhaps associated with 
              warm-ups.
 Call List: Detector people are on it.
 Resolved:    Over the next few days. 1% CFT channels didn't come back
              from the shutdown.

 Item #25
 Date: December 4, 2003
 Description: HTD ran recocert on 185751 and compared it with run 
              180951 and noticed something interesting in the SMT 
              occupancy histograms. It's SMT clusters in barrel 4 plots.
              In "phi layer 4", around phi=1 there's occupancy in the 
              comparison run but not in 185751.  Conversely, in 
              "phi layer 5" it's the other way around at phi = 4.5.
 Call List:   Michael Weber and Sergey Burdin.
 Action:      Michele W. says:"Thanks for looking at the plots.
              If you go through all the plots you will find ca. 35 only black 
              ("appears new") and ca. 9 only red ("disappears").
              This corresponds roughly to number of devices we enabled 
              (repaired, 40)  odd were newly lost (13) since the shutdown.
              I'm in the process of going through our lists and comparing 
              it with the recocert plots.  He reports he doesn't understand
              the shape (tilt) in the occupancy plots as you go across the
              face of the device. He thought it should be flat.

              It was expressed at the 12/15/03 Verification ADM that 
              maybe this is a mechanical problem that causes the HDI's to 
              be flaky. Recall, they seem to recover if you wiggle the 
              connectors. An autopsy might be in order.
 Resolved:    Jan. 19, 2004.  Sergey Burdin: "Drew told us about this about 
              2 weeks ago. I looked offline on run 187910 and it looks like 
              run 187169 in December (the distributions were affected by the 
              CFT map though). The event display does not show anything 
              unusual also. 
                There was some discussion at the beginning of December about 
              the region phi=1 (please see below). Is it the same problem? 
              Can you compare your vertex examine with the run taken at the 
              beginning of December?"
                Sanders: "Rick noticed a spike at phi=1 for CFT only tracks 
              in vertex examine. This means that the L3 tracker didn't pick 
              up any SMT hits in that region. Does this imply that the SMT 
              is "dead" in one small phi-wedge? I think offline (physics 
              examine) things look quite ok.

 Item #24
 Date: December 2, 2003
 Description: HTD noticed out-of-time hits (~-50 ns) in central muon C-layer 
              scintillator plots from all recent data since startup after 
              shutdown.  Previously such hits indicated a bad or flaky 
              SFE.
 Call List:   Al Ito, Steve Doulas, Anatoli Evdokimov.
 Action:      Al Ito and Robert Harrington replaced the SFE on 12/3 after-
              noon. Robert notes: SFE $20 in CES, which is for PDT counters 
              200, 210, 220).  Please let me know if any of you see any more 
              hits < -20 ns.  
 Resolved Date and Comments: Done. See above.

 Item #23
 Date: October 16, 2003
 Description: HTD ran recocert on 175645 for the purpose of studing
              RecoCert and noticed (p13.06.01) that there was terrific 
              calorimeter noise.
 Call List:   Dean and Robert.
 Action:      They identified that the problem was an analog driver (ADC) for
              a specific eta-phi cell set (16 eta-phi units). 1 layer (3rd em)
              all with same energy. It's a little curious becasue only 
              14 cells show hot. But that is not unheard-of. The ADC card 
              is in 8 pairs so if the first pair has a problem the rest
              can get stuck umm, high.  It was noticed by the shift and 
              fixed by Parua the same day.

              It will be interesting to see what p14.03.02 does with this.

 Item #22
 Date: October 3rd, 2003.
 Description: HTD ran recocert on run 180911 for the purpose of studying 
              one thing (cal. noise, see item #19) and noticed something 
              else. The CFT adc distribution, stereo view, has an excess 
              over run 178722 at low adc counts. I guess this is indicative 
              of hot channels in the adc readout but wanted to check with 
              you on that. The cft occpuancies do seem slightly higher and 
              u and v, all layers, but in such a way that doesn't point 
              me to a specifically hot area.  It changed between 180796 
              and 180828.
 Call List: Jadzia W. and Mike Friedman.
 Action: Requested advice as to what this is and how serious.  Jadzia wrote:
              "I looked in CFT eLog for entries for run 178722 and 180911. 
              I do not see anything abnormal there and that is all I know 
              about these runs. It is disturbing that such change occured 
              and we are not aware of that. For the future I will try the 
              get in our cft_examine the plots similar to the ones you 
              sent me."  
              HTD: It changed between 180796 and 180828. 180796 is on 
              9/1/3. There were CFT calib runs ~180801-180805 in between these
              global runs. One of those wasn't cft calib. Well, logbook
              notes problems with the calibration download, but expert 
              thought it worked. The thing is those calibrations occur
              every day. These were for crate 50. The next crate 50 calib
              was 9/5 run 180921. But global runs after that still show
              the problem.
              Mike Hildreth showed plots in the 12/19 ADM and said it was
              OK now. I should compare these with those.
 Resolved Date and Comments: April 24, 2004. This has not re-occurred as of 
              April 2004, so far as we know.

 Item #21
 Date: September 16th, 2003
 Description: Ricardo P. compared run 180881 to 180789. He writes:
              "Normalization when comparing to previous run is screwed 
              up on some plots: all tracks, electrons, muons. Seems to be a 
              software problem on the verifyRecoCert code." 
              Follow-up indicates that comparison with 178722 doesn't
              show a serious normalization problem.
 Call List:   HTD.
 Action:      Follow-up indicates that comparison with 178722 doesn't
              show a serious normalization problem with 180881. It seems
              180789 is the outlier. I sent this off to Harry M. 
              Harry suggests the tracks are the problem in 180789, not
              the normalization.  Maybe. I am sticking with normalization
              for the later muon plots. Also, Harry wonders about the 
              instantaneous luminosity for the later run. It was first
              of store, started at 40e30 and fell over time to 30e30. 
              Run 180881 is 19-15e30. Harry's guess may be borne out.

 Item #20
 Date: September 13th, 2003
 Description: Raimund S. saw run 180748 had no entries in the SMT
              Cluster Bar2 in phi layer 2 in range 0.5-1. (p20 bottom left).
 Call List: TBD
 Action: TBD
 Resolved Date and Comments: 

 Item #19
 Date: September 9th, 2003
 Description: Shift Captain complained that calorimeter jet trigger rates 
              are 50% to 1000% higher than normal in Run 180911. 
              We are curious as to whether something was going on.
              Incidentally, Captain logbook notes: "Just before
              6am there was a glitch in the luminosity and the
              background rates.  About 15 minutes after that, the
              L1Cal trigger lost power due to what appears to be a
              faulty interlock readout box.  It was replaced with
              a 5volt supply, and we began running again
              around 7:20....but with the jet rates returned
              to normal." Curious.
              More logbook notes: "MCR called to say that CDF called them 
              to say that at ~5-6am CDF noticed shift of ~5cm in beam 
              position. Our vertex examine shows a shift of ~2.5cm. MCR is 
              aware of that and working on it. Mike Martins is coming in 
              to work on it in ~20min. They will call before they make 
              any changes."
 Call List:   Any offline shifter, Schamberger and Parua to be informed.
 Action:      Submitted 09/30/03 HTD. VerifyRecoCert against run 178722
              on Oct. 3rd. Vertices distribution close, 180911 has one peak 
              in x and one peak in y, though the y-peak is 1/20th cm higher.
              There is a slight excess on the +z side, maybe centered around
              5 cm, in 180911. Very small, though.
              Electrons(all) identical. Muon matching eff't is a little better
              in 178722. MET is close. There's a little more MET(y) in the 
              downward direction (1.5 GeV?). This caused me to wonder if a net 
              met (or a shift in it) can be a vertex position effect. 
              Well, obviously it can, if you think about it in the limit,
              but how much for a small shift? -1.5 GeV for +1/20 cm shift?
              Jets(coneall) has a softer Et spectrum than 178722. It also 
              shows an excess at eta=1, phi=2. This is not reflected
              in the cal all cells distribution, though. 
                CFT adc summary has excess at low adc counts not seen in the 
              comparison run. I don't usually study this, though and don't
              know what it means. Probably a hot channel. I sent a distribution
              to Mike Freidman (Marge C.) and to Jadzia. This spawned 
              punch list item #22. 
                Time in C-layer FAMUS (scintillator) shows an excess in the
              late peak (~+20 ns) but A and B times are normal. I don't know 
              which end the excess is in. 
 Resolved Date and Comments: I (HTD) was really interested in seeing if there 
              was any effect as bad as ring-of-welding. Nothing like that
              here. 

 Item #18
 Date: September 4th, 2003
 Description: Daria Z. wrote: "I'm looking at recocert plots for run 
              180206 (p14.03.02). /rooms/greatlakes/michigan/dataquality/jobs/
              run180206-p140302-DST/run180206-p140302-DST.ps on clued0.
              The phi distribution of all SMT barrel clusers (p27 in the ps 
              file) has a decreasing trend. In my experience it happens when 
              there is a cutoff on the  max total  number of objects in the 
              reconstruction,  and  the reconstruction goes over sectors 
              increasing in phi. I think the fact that the max is saturated 
              at low phi can lead to an overall inefficiency in cluster 
              finding, and hence in tracking."
                The inst. lum'y fell from 25 to 21 e30 during the run.
 Action:      Forwarded to Melanson and Slava Kulik. 
              Slava replied (9/4): "As far as I know, there is a cut on 
              SVX occupancy (offline), so per chip, and maybe now on HDI 
              (online),  but not on total number of SMT hits. At least not 
              in reco." I replied, asking him what this means.
              10/23/03 (HTD) Inactive item. 

 Item #17
 Date: August 21, 2003
 Description: Please run RecoCert on run 180301. This run was taken
              after a BLS supply was changed, but no pedestal run followed
              it. Runs after 180305 have the pedestals.
 Call List:   Any Offline Shifter.
 Action:      Submitted by HTD 8/21/2003. 
              Aug. 24: No Reco'd data in SAM.
              Submitted by HTD 9/3/2003. Sucessful submit. Comparison
              plots were made against run 178722. MET looks a few 
              GeV higher.  Cal CC cells doesn't look too good. We 
              should compare against a run immediately after 180305.
              I compared against 180329, taken after a new pedestal run
              and download was performed. There is a significant improvement
              in jets, met, and cal cells in the latter run. 
 Resolved Date and Comment: September 9, 2003. I will stress this point
              in an ADM. 
 

 Item #16
 Date: August 18, 2003
 Description: Please run recocert on run 179614 and see if electrons
              look OK. From M.Sanders: "I was paged by the GM shifter earlier 
              today who told me that he didn't see any L2 EM global objects 
              in run 179614. These plots get filled by l2gbl_analyze into 
              the L2gblEM block in the trigsim_analyze roottuple. I checked 
              the roottuple and it did not have anything filled in this block.
              However, the L2gblJet stuff did appear in the roottuple and 
              thus the histograms." 
 Call List:   Any Offline Shifter.
 Action:      Submitted by HTD 8/18/2003.
              Aug. 19: (M. Sanders and L. Duflot). At some luminosity
              L2EM doesn't run (much) and therefore the online plots have 
              no information.  Run 179614 was one such run. 
              Aug. 25: (HTD) Electrons look OK in this run. Jets are also OK, 
              but the are some hot spots in the jet distribution at about 
              eta=-1 phi=4.5
 Resolved Date and Comment: August 25, 2003.
 

 Item #15
 Date: August 18, 2003
 Description: Please run recocert on run 180130 or 180131 when they become 
              available. This has "significant fraction of events with large 
              calorimeter coherent noise, mainly at boundaries of the central 
              and end cryostats. No obvious correlation with any DAB work has 
              been found." - Shift Captain Summary. 
              Please compare to run 179334.
 Call List:   Any Offline Shifter.
 Action:      Submitted by HTD 9/4/3. I looked at the output of RecoCert
              in comparison with run 178722. Run 180130 looked fine. There
              was no notable discrepency between it and the comparison. 
              The muon plots still have the normalization difference because
              we aren't using an update yet. There are tiny little excesses
              in the CC cells that are in the same place as tiny little 
              excess in the much older comparison run. This does not look
              so bad as "Ring-of-Welding". - HTD 9/8/2003.
  Resolved Date: September 8, 2003.

 Item #14
 Date: August 4, 2003
 Description: Thomas Nunneman notes that comparisons for all three "muon 
 local tracks" plots isn't working right. It might be a normalization error.
 See, for instance, run 179236. 
 Call List: Melanson, Daria Z., Mulders.
 Action: 

 Item #13
 Date: July 29, 2003
 Description: Please run recocert on run 179332 when it becomes available. 
              This has "Ring-offire". Please compare to run 179334, maybe.
 Call List:   Any Offline Shifter.
 Action:      Diehl did it.
 Resolved Date and Comment: August 14, 2003.

 Item #12
 Date: July 17, 2003
 Description:  Run 178737. "Jets Cone R=0.7 No Cuts" shows larger horns 
 at ~|eta|=1.5 than the reference. There is some strong phi structure not 
 seen in the reference. Coarse hadronic fraction is wider at the low end. 
 Perhaps all of this reflects the hot areas visible in the Jet phi vs eta plot?
 Question for experts: why does the EMF distribution contain little
 between 0.8 < EMF < 1 in comparison with the reference below?  This was 
 also found by R.I. 7/17/2003 in run 178765. And Sabine in run 178763.
 Call List: 7/17 MC Gao is Cal on-call. Also, sent to Wobisch 7/18. Also sent
 to Bernardi, Groer, Parua on 7/20.
 Action: No, there was no change in the jet selection criteria. 
    Maybe a trigger bias (less electrons triggered) or a higher rate of jets 
    due to the noisier ch, which masks the peak at high EM fraction, or a 
    combination of the two.
       July 23: HTD determined this change may have occurred on Trigger 
    version 11.04 to 12.10 boundary. That occurred 178721 to 178722. 
    Plots exist. Email to Bernardi, Wobisch, Parua, Groer.  
       July 28: The trigger information from Lum'y guys is on a crapped-out
    disk and this has been causing some delay.
 Resolved Date and Comment: August 4, 2003. Attributed to tightening of 
    L3 EM filters at change in trigger version. 

 Item #11
 Date: July 17, 2003
 Description:  Run 178737. "Tracks All" has a minor spike in the Z at DCA 
 plot at Z=0. This looks like it might be axial-only tracks. It's also 
 reflected in the "Vertices Primary" plots. I thought we would not be seeing 
 this kind of thing very frequently any more. Is it still expected to happen 
 from time-to-time?
 Call List: Drew Alton is CFT on-call. -> Gordon Watts.
 Action: Drew commented and went on to summarize:  "In fact I dont think 
    this is axial only tracks.  Since everything you point to is a vertex 
    quantity...I'd ask a vertex expert.  If you want to give me more 
    histograms I'd be glad to look at them." In short, maybe these are 
    zero-bias or minbias or muon events without a reco'd vertex.
    Drew followed up with: "page 2 of your plots shows its not axial-only, 
    because there aren't >100 events with 8 cft hits.  There are only 50."
    July 29, 2003. Gordon W. thinks it might be "hot cal triggers" making
    events without vertices.
 Resolved Date and Comment: 10/23/03 (HTD) Declared obselete.
    

 Item #10
 Date: May 20, 2003
 Description: Several events in zero-bias run 176055 have non-correspondance
    between L1 and L2 (which don't agree) and L3 masks (which agrees with L2). 
    This was identified by Begel.
 Call List: Jon Hayes (script-runner) and Gerry Gugliemo (data-logger).
 Action: Gerry is working to determine if the problem is upstream of the 
    data collector. He found out that there are "pathological" features of
    this run which have not been observed in physics data-taking. It's
    a problem upstream of datalogger/collector router. Some of these are 
    "known" occasional blow-ups or stuck bits.
 Resolved Date and Comment: Never resolved. Obsoleted July 17, 2003.

 Item #: 9
 Date: May 15, 2003
 Description: A dip was seen around phi=1 and phi=5 in the cft track
    distribution from physics examine, at the beginning of run 176929.
    The problem seemed to disappear during the run. Similar dips are
    seen in run 176880. Could be bad CFT or bad SMT, no dips seen in 
    global tracks, nothing seen in vertex examine, L3 tracks or cft examine.
 Call List: Drew Alton
 Action:
 Resolved Date and Comment: Never resolved. Obsoleted July 17, 2003.

 Item #: 8
 Date: April 26, 2003
 Description: A dip is seen around phi=2 in both the L3 tracks plot in
    trigger examine, and in vertex examine. From vertex examine, it is
    clear that a small region is missing in CFT axial layer 2. However, 
    CFT examine doesn't show a problem.
 Call List: Jadzia Warchol, Fred Borcherding, Michiel Sanders
 Action: This was first linked to a "new" CFT calibration, whereas
    vertex examine is using an "old" calibration. However, the real
    problem was in one SVX chip, SVX0 on slot 9A5. It was not seen in
    the examine because: "The confusion was the consequence of the way
    our hit maps are filled: discriminator hits without SVX hits are
    assigned ADC=0 and put into SVX hit maps. This an old issue we
    argue about since somtime but we never reached a consensus.
    Luckily cft_examine delivers pulse hight distribution for chips
    specified on rcp file.
    Pulse height of the bad SVX shows a lot of entries with ADC=0, meaning
    entries to SVX hit map come mainly from events where discriminator
    fired but SVX did not."
 Resolved Date and Comment: April 29, 2003
    Bad AFE board replaced in a controlled access. A similar problem
    occured in July 2002, where CFT examine showed no problem but in
    reality one SVX was bad. 
    This affects runs > 176175 and < 176346.

 Item #: 7
 Date: April 20, 2003
 Description: The GM shifter noticed a spike at eta=1.1, phi=5.7 
    in the tau trigger rate. This corresponds to a hot cell in calorimeter.
    This is run 175890 & 175891. Apparently this hot cell has been 
    around a long time and is getting worse. 
 Call List: Nirmalya Parua
 Action: No response. Obsoleted Jun 9th, 2003.

 Item #: 6
 Date: April 14, 2003
 Description: Run 174212 shows non-standard (fewer) hits in RecoCert tracks
              plots, lower efficiency, and two-peaked vertex distribution.
 Call List: Ron Lipton, Michiel Sanders
 Action: Ron found comments in the logbook. They said: "The vertex examine 
    showed large losses in stereo tracks found look carefully at the 
    fiber fired map and find a region that may be suppressed.
    Reset the cft_examine. Request that the daq shifter give the 
    cft shifter a chance to init_VRBC when a crate goes missing
    rather than continuing via an SCL init."
      Sanders writes: To my knowledge, the beam should be ~x = -0.0193 cm, 
    y = 0.0208 cm. The beam jumped by about 1 mm when the TeVatron 
    magnet was replaced, early March. So, after March 12th, run ~
    174210, I expect a much larger x value. For run 174212, I expect 
    something like x = 0.0760 cm, y = 0.0301 cm. (right now (current 
    store) it is even further away from zero: x = +0.09672, y = +0.03515).
    So, the highest peak in the vertex plots is the correct one. I don't
    expect beam shifts during that run that would explain the second peak.
    Next guess would be a bias in the vertex algorithms for vertices with
    small number of tracks..... But, it is surprising that the second peak
    is exactly at zero. Note that the number of tracks attached to the
    vertex is very small. Also, lots of events with zero tracks. Probably a
    very bad run for tracking.
 Resolved Date and Comment: This seemed to be a one-time effect. 

 Item #: 5
 Date: April 11, 2003
 Description: There is a dip in PHI at DCA on Tracks(All) RecoCert plot
              between 36-180 degrees.
 Call List: Drew Alton
 Action: Drew suggest this will go away with data after CFT timing move 
         to occur April 11th.
 Resolved Date and Comment: It didn't. Now (June) it's thought to be related to
         the vertex position. 

 Item #: 4
 Date: April 11, 2003
 Description: Readout Errors in SMT from end of Jan. (2003) shutdown to 
              ~ Mar. 10th effected the reconstruction. Runs sould be 
              graded as "Not perfect but analyzable".  Can this be seen
              in RecoCert plots?
 Call List: HTD
 Action: Ron Lipton is still worried about this July 17, 2003.
 Resolved Date and Comment: Obsoleted 10/23/03 (HTD)

 Item #: 3
 Date: April 3, 2003
 Description: Muon Examine plots show central scintillator hits out of the 
              expected time-gate. 
 Call List: Al Ito, Steve Doulas, Dmitri Denisov
 Action: DD recommends to check SFE's.
 Resolved Date and Comment: Al Ito replaced SFE $2F in crate x50. Histos look
    good starting in run 175322.

 Item #: 2
 Date: March 12, 2003
 Description: Run 173528 and 173529 have many events with cal readout skipped?
 Call List: Groer, Bernardi, Duflot (calorimeter data quality), Sang-Joon Lee
    (Recocert), Parua (03/25/03 calorimeter data quality)
 Action: According to histograms produced by Reco-cert there are many events 
    with no calorimeter cells. See Sang-Joon's histos at 
    /work/yogsothoth-clued0/sangjoon/Reco_Mon_Plots/*/Cal_All_cells.ps
    where * is 173528_Plots or 173529_Plots. 4/4/2003 - Dean says that unless
    another cut on ET has been applied besides 2.5 sigma in E (not Et units)
    the probability for it to happen even once is vanishingly small. 
 Resolved Date and Comment: April 4. Harry Melanson - "In order to get data 
    from the calorimeter chunk, one needs to supply a vertex (???!!!  
    Well, that's the way the code is written).  I check whether there is 
    a primary vertex found, and use it if so.  I'm guessing the
    spike at zero is for events where no PV is available.  I will fix that 
    in an upcoming release...". A recocert-only problem.

 Item #: 1
 Date: March 12, 2003
 Description: Run 174207-174212 dimuon trigger problem.  Drop in number of 
    N dimuons should be fixed in store with run 174238. 
 Call List: RobMcc (L1 musim), Sang-Joon Lee (recocert), Diehl (muon detector)
 Action: RobMCC discovered it and fixed it.  Muon detector examine plots 
    showed no discrepency. Reco-cert was busy with higher priority item
 Resolved Date and Comment: March 12, 2003. This was discovered ex-post-facto 
    and was resolved on discovery.