<!-- Title: 11:17:57  Wed Sep 14 2005 -->
<!-- Crewchief: EVB / L3 experts -->
<!-- Op1: CDF ACEs -->
<!-- Op2: &nbsp; -->
<!-- Op3: &nbsp; -->
<!-- Op4: &nbsp; -->
<!-- Op5: &nbsp; -->
<!-- Op6: &nbsp; -->
<!-- Notes: -->
<br> <i> We are going to use this Elog to report all pages, problems and features of the Event Builder and Level 3 system. Entries should be made by the SPL, EVB/L3 expert, or ACEs. The Elog is not meant to be a forum for discussions of problems but should show which problems occured and how they have  
been fixed. </i> </br>


<!-- Date: Wed Sep 14 11:32:30 2005 -->
last weeks pages ( 09/07/05 - 09/14/05 ) 
<ol> 
<li>  
Wednesday 09/07/05 - 7.00 pm : Vadim was running a test and used the old ACE control panel 
</li> <li> 
Wednesday 09/07/05 - 7.45 pm : Vadim again, saw corrupted events. It turned out that the SCPUs have been out off memory 
</li> <li> 
Thursday 09/08/05 - 1.20 pm : Corrupted event from COT in shotsetup. HRR helped. 
</li> <li> 
Thursday 09/08/05 - 3.05 pm : Vadim wanted to look at L3 logfiles.  
</li> <li> 
Thursday 09/08/05 - 8.20 pm : CSL problem, L3 did not configure, all nodes green. Asked ACE to stop, cleanup and start CSL. 
</li> <li> 
Friday 09/09/05 - 9.30 am : anti-proton loaded. 
</li> <li> 
Friday 09/09/05 - 12.45 pm: CSL problem, L3 did not configure, all nodes green. Asked ACE to stop, cleanup and start CSL. 
</li> <li> 
Friday 09/09/05 - 4.30 pm : In the middle of L1Cal problems, the EVB had the memory problem again. Rebooted all SCPUs. 
</li> <li> 
Sunday 09/11/05 - 0.35 am : anti-proton loaded. 
</li> <li> 
Monday 09/12/05 - 5:54 am : anti-proton loaded 
</li> <li> 
Tuesday 09/13/05 - 1.30 am :  /cdf on b0l3pcom2 was full. 0.11 pb downtime charged to L3. 0.43am - 2.30am the run was stopped for a scaler mon test and we had a Si trip at 2.00am. The problem was caused by a L3 filter test which crashed on almost all nodes. 
</li></ol> 
  
<!-- Author: markus -->


<!-- Date: Mon Sep 19 15:18:53 2005 -->
Sunday 2am: <p> 
the EVB2 went red trying to take events. Multiple HRRs did not help. They cleaned up but the problem was not fixed.<p> 
There is a problem with the memory in the SCPUs. A reboot is required. Since I did not have a network connection, I asked the ACE to reset the SCPU, which fixed the problem.<p>
<!-- Author: markus -->


<!-- Date: Mon Sep 19 18:36:27 2005 -->
Monday 5pm:<p> 
the L3 proxy failed to partition because node b0l3122 was down. This infomation was displayed in the Error Logger. We excluded subfarm 12, which included b0l3122 to get going.   
<!-- Author: markus -->


<!-- Date: Tue Sep 20 08:23:59 2005 -->
<pre>Tuesday 3.30am (Jeff): 
the L3 proxy died ( again ). The state/transition panel in the L3 display turned red. Run was aborded, reset and L3 farm was cleanup. </pre>
<!-- Author: markus -->


<!-- Date: Wed Sep 21 15:51:35 2005 -->
<pre>Tuesday 4pm: 
Procmon problem. Shifter did not wait for script to exit and finally stopped RTserver. We had to restart, I cleaned up L3 and EVB2 just in case.</pre>
<!-- Author: markus -->


<!-- Date: Wed Sep 21 23:15:31 2005 -->
<pre>Wednesday 5.50pm:</pre>
<!-- Author: markus -->
<!-- Comment: Tue Sep 27 15;08;50 comment by...markus -->
Busy TOs, restarting and cleaning up did not help. The problem was a memory leak in the 2.4.27 Kernel. We implemented a Panic button to reboot all SCPUs and upgraded the Kernel to 2.4.31.
<!-- Date: Wed Sep 21 23:17:00 2005 -->
<pre>Wednesday 11pm (Alberto):</pre>
<!-- Author: markus -->
<!-- Comment: Tue Sep 27 15;09;06 comment by...markus -->
Same a above.
<!-- Date: Thu Sep 22 18:26:06 2005 -->
<pre>Thursday 6.20pm: 
busy TO in a shot setup run - cleared by a HRR. 
The event builder detected a corrupted event and stopped loading further events. The TM issued an auto-HRR. I verified that there was a single corrupted event ( size &gt; 65kB ). Nothing to worry about.</pre>
<!-- Author: markus -->


<!-- Date: Sat Sep 24 02:58:33 2005 -->
<pre>Friday 6.15pm: 
the ACE was on high alert and asked if we should worry about a TM2 pending TO. The answer was no, as long as data taking is not interrupted to often.</pre>
<!-- Author: markus -->




<!-- Date: Thu Sep 29 09:08:59 2005 -->
<pre>Wednesday 10.15pm: 
continuous busy timeouts during shot setup. Even the panic button did not help. 
During the day we removed the old SCPUs and ScramNet cards and moved all DAQ SCPUs to slot 1. The working assumption is that the adapter card for b0evme12 is flaky causing problem for the VME transfer. I moved the card back to its orginal adapter in slot 4, which fixed the problem. </pre>
<!-- Author: markus -->


<!-- Date: Fri Sep 30 22:18:03 2005 -->
<pre>Friday 1.15 am: 
shift crew tried to run a cosmic run during TeV studies. Rates went up and saturated CSL. Ask them to continue with calibrations.</pre>
<!-- Author: markus -->


<!-- Date: Fri Sep 30 22:19:21 2005 -->
<pre>Friday 11.50am (Conor): 
got two corrupted events. Nothing to worry about.</pre>
<!-- Author: markus -->


<!-- Date: Fri Sep 30 22:21:20 2005 -->
<pre>Friday 12.30pm: 
before data taking was started. EVB got stuck (timeout is 30sec) in  Recover transition. The SCPU failed resetting the VRB and aborded after it got a corrupted event. Shifter was asked to reset the partition and cleanup the EVB.</pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 3 12:27:23 2005 -->
<pre>1 Oct 2005 pages: 
 
4pm - DAQ ACE was seeing 'too many' required HRRs. This was in shot-setup. ACE followed the correct procedure, did an evb cleanup after a few HRRs. I was paged, could not find anything specific wrong, so just did a full evb reboot (using the PANIC button) as a precaution 
 
6.45pm - Page was a false alarm. Problem was actually with the software event builder. Shift was doing calibs at the time. Software evb falls under the DAQ responsibilities and has a separate person in the experts list.</pre>
<!-- Author: Conor  -->


<!-- Date: Mon Oct 3 12:29:48 2005 -->
<pre>2 Oct pages: 
 
6pm - While trying to take cosmics, ACE found that EVB proxy was not running. A simple EVB clean-up was all that was needed. 
</pre>
<!-- Author: Conor -->


<!-- Date: Mon Oct 3 12:35:26 2005 -->
<pre>Monday, 1.30 am 
Run 205008 had 1.83% reformatter errors. ACE followed instructions and paged Level3. The subsequent run had already been going for 20 min and showed no errors, so problem looked like it had gone away. Told ACE to continue as normal but monitor carefully and page again immediately if ref errors returned. Sebastian later said that these errors had been caused by an SVX VRB which had been reset. 
 
Monday, 10.15am 
All but 2 Level3 sub-farms had dropped out, and ACE noticed that system started to experience dead-time. This can happen sometimes  for long runs (this run had been ~9 hours). Solution is simple: just reset and start again. Cleanup is not needed.</pre>
<!-- Author: Conor -->


<!-- Date: Wed Oct 5 13:50:04 2005 -->
<pre>Date : 2005/10/05  Wednesday 
---------------------------- 
 
Time : 13:20 
 
They paged us because they saw a white warning saying that b0vrb01 couldn't be contacted. Data taking was not interrupted at all. 
 
B0vrb01 is a board that is in the EVB crate. It makes some tests of the VRB boards. The DAQ people wanted it there. The specific board (b0vrb01) was not plugged in but was disconnected. That's why it complained. It actually had nothing to do with the EVB. 
</pre>
<!-- Author: Georgios -->


<!-- Date: Thu Oct 6 16:56:32 2005 -->
<pre>Date : 2005/10/05  Wednesday 
---------------------------- 
 
Time : 23:50 
 
There was a relay failure with b0l3264. They paged me because they got worried a 
bout it. I told them to start a run without changing anything, as I expect a mis 
sing node not to cause any problems. At that time however an irrelevant problem  
appeared which didn't allow them to start a run. They told me that they would tr 
y to solve that other problem and if they had problems with the Level3 again the 
n they would page me again. 
</pre>
<!-- Author: Georgios -->


<!-- Date: Fri Oct 7 11:26:07 2005 -->
<pre>Thursday, 1.15am 
The Si power supply GUI was showing a red button. Told them to page Si.</pre>
<!-- Author: markus -->


<!-- Date: Fri Oct 7 11:28:22 2005 -->
<pre>Friday 0.30am and 4am: 
The plug sourcing run can not be taken with the default version of the hardware event builder. The reason is the corruption of the very first event, a well known problem likely caused by the init sequence of the taxi or tracer ( not the EVB ). Since the plug sourcing run does not allow a HRR to fix the problem, the run gets stuck.  
 
I have installed a pseudo-fix by removing all event checks in the SCPU and rely on L3 to remove the corrupted events. Since the readout in the SCPU is data driven, this fix is not guaranteed to 
(always) work.  
 
Reset to the default version at 4am.</pre>
<!-- Author: markus -->


<!-- Date: Fri Oct 7 18:20:04 2005 -->
<pre>Friday 6pm: 
htdc problem during calibration run. Recommended to start another calibration or use hardware event builder. Well, we have nothing to do with this.</pre>
<!-- Author: markus -->


<!-- Date: Tue Oct 11 09:18:10 2005 -->
<pre>Date : 2005/10/08  Saturday 
--------------------------- 
 
Time : 07:25 
 
I received a page because there was not data taking. Yuji told me that he was responding to a silicon resonance by pressing HRR when level3 got "stuck" at the transition from ACTIVE to HALTED. The Run Control was also "stuck". He had to "Manually override transition". After some effort he put the RC to START state. 
 
I told him to check L3 proxy from the Ace Control Panel. The AceCP log was: 
 
2005/10/08, 07:32:02 : BEGIN: Stop L3 proxy  
Node b0l3gate2 is alive  
l3proxy process 30394 not found.  
l3proxy-related processes killed.  
 
2005/10/08, 07:32:09 : END: Stop L3 proxy  
  
2005/10/08, 07:32:43 : BEGIN: Check if L3 proxy is running  
Node b0l3gate2 is alive  
l3proxy CHECK: NO PROCESS  
  
2005/10/08, 07:32:44 : END: Check if L3 proxy is running  
 
I told him to restart EVB (to make sure it's OK). 
Then I told him to start L3 Proxy. 
 
2005/10/08, 07:34:06 : BEGIN: Start L3 proxy  
Node b0l3gate2 is alive  
No such process 30394.  
New l3proxy will be started.  
Starting l3proxy after 1 monitoring-check iterations...  
[1] 1604  
Started L3 proxy process, initialization in progress. This might take a few min\ 
utes.  
L3 proxy initialization is completed  
  
2005/10/08, 07:34:44 : END: Start L3 proxy  
 
2005/10/08, 07:34:55 : BEGIN: Check if L3 proxy is running  
Node b0l3gate2 is alive  
l3proxy CHECK: RUNNING  
  
2005/10/08, 07:34:56 : END: Check if L3 proxy is running  
 
I told him then to clean up L3 farm and he did that, but then he pressed Partition before the cleanup was done. So, to make sure I told him to do a cleanup again. Finally, it was done normally. After that, the next run started without problems. 
</pre>
<!-- Author: Georgios -->


<!-- Date: Tue Oct 11 09:18:37 2005 -->
<pre>Date : 2005/10/11  Tuesday 
-------------------------- 
 
Time: 08:30 
----------- 
 
There was a page, but it was not an emergency. They had often EVB timeouts. Markus investigates the deeper reasons. 
</pre>
<!-- Author: Georgios -->


<!-- Date: Fri Oct 14 15:51:01 2005 -->
<pre>Thursday 8.00 pm: 
the ACE was wondering what a red bar behind a processor node means and if we need to take action. This indicates that a L3 filter has crashed. Unless we lose to many nodes ( filter ) nothing has to be done. The L3 filter group and the L3 experts get a notification after the run has ended.</pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 17 15:56:13 2005 -->
<pre>Monday 9.30am: 
one of the subfarms lost monitoring. Cleaned up L3mon.</pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 17 15:57:51 2005 -->
<pre>Saturday ~9.00am ( Jeff ): 
ACE tried to take a cosmic run with beam in TeV. Rates were to high for the CSL. Stopped the run and started L2 torture. 
</pre>
<!-- Author: markus -->


<!-- Date: Wed Oct 19 12:33:51 2005 -->
<pre>Tuesday 9.15am: 
ACE press EVB proxy start button on the ACE control panel. Partition needed to be reset.</pre>
<!-- Author: markus -->


<!-- Date: Thu Oct 20 13:26:03 2005 -->
<pre>Wednesday 21:45 (Alberto, Conor and Markus): 
L3 reformatter errors due to software modification. Fixed by going back to the previous version. </pre>
<!-- Author: markus -->


<!-- Date: Thu Oct 20 13:28:18 2005 -->
<pre>Thursday 9.00am: 
At 8.20am I pointed out to the ACE that 3 subfarms have 0 rate and it's likely due to CSL problem / glitch. Was paged at 9am with exactly this problem. Instructed to stop/cleanup/start the CSL, since the run was stopped anyway.</pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 24 10:51:17 2005 -->
<pre>Monday 3.30am (Alberto): 
Done TOs from FCAL crate. Done TOs are not caused by the EVB/L3 system. The ACE decided to restart the run and cleanup EVB. L3 was not done resetting, when they started the run, which caused EVB proxy to crash on the partition transition. The check of the EVB proxy was performed and showed that is was not running. The right action would have been to reset and cleanup again. Instead they paged Alberto who instructed them to reset the SCPU which delayed the process to &gt;90min. After they were able to start another run, fcal was still not working and needed to be shepherd. </pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 24 10:53:16 2005 -->
<pre>Saturday 11am: 
Problems with L2. 
 
Saturday 11.30am: 
after problems with L2, a few subfarms had problems to connect to the CSL. Told the shift crew that the problem will fix itself after restarting the run.</pre>
<!-- Author: markus -->


<!-- Date: Tue Oct 25 13:30:20 2005 -->
<pre>Tuesday 12.30 ( afternoon ): 
two consecutive BUSY TOs. The ACE asked me to check the system. The BUSY TOs were due to a corrupted event fragment from WCAL00 and channel 8,9 in crate 12. No action required.</pre>
<!-- Author: markus -->


<!-- Date: Fri Oct 28 15:27:51 2005 -->
<pre>Wednesday and Thursday: 
 
2:30pm : XFT problem, rate to high for CSL, paged EVB 
 
11pm: L3 Calib Tags not found by L3 Manager. ACE was not able to contact L3 Filter expert who is in charge. Investigated and asked to page Stephan since the quota for level3 was not set correctly after filesystem work. 
 
3am: pbar loaded - COT problem - TDC expert claimed this is an EVB problem. EVB can not casue DONE TO and is not used in the COT calibration. Later there was a CSL problem causing a relay error for two processor nodes. Needed to cleanup L3 farm and CSL. 
 
5am: high rate of BUSY TOs. WCAL00 crate was sending two event fragments to the EVB. Needed to be shepherded. No more corrupted event from WCAL00. 
</pre>
<!-- Author: markus -->


<!-- Date: Fri Oct 28 15:29:37 2005 -->
<pre>Thursday 3pm (Khaldoun and Markus): 
L3 failed config transition, because the calib tag could not be copied. We needed to remount a disk, due to some work on the file-system Wednesday afternoon.</pre>
<!-- Author: markus -->


<!-- Date: Sat Oct 29 14:26:07 2005 -->
<pre>Saturday 1:30 am (khaldoun and markus): 
 
during level 2 tests, l3/evb was paged because evb2b was red in rc, could not partition.  they already had tried clean up once: it cleaned up ok, but still could not partition.  a second cleanup fixed the problem.   
 
most probably reason (without having looked in detail as log files): l3 had not returned to start state because of slow transition.  waiting and doing a second cleanup is the correct action.</pre>
<!-- Author: khaldoun -->
<!-- Comment: Sat Oct 29 15;29;36 comment by...markus -->
<pre>SCPU 11 had crashed and it took &gt;1 min for it to restart the executable. </pre>


<!-- Date: Mon Oct 31 11:55:51 2005 -->
<pre>Monday 11am: 
BUS errors in Si cartes. We were debugging L2DTOs. Reset the SCPUs.</pre>
<!-- Author: markus -->


<!-- Date: Mon Oct 31 11:57:31 2005 -->
<pre>Sunday 11am: 
CSL problem, excluded output node u06.</pre>
<!-- Author: markus -->


<!-- Date: Wed Nov 2 15:38:46 2005 -->
<pre>1.30pm. Level3 subfarms 4 and 10 'dropped out'. Did not affect data-taking. There were already XFT problems, so they had to restart the run anyway. So, we did a L3 farm cleanup at the same time. All subfarms appeared to run fine after that. Have not yet been able to determine from the logfiles what went wrong. Asked shift crew to page me again if it happens again</pre>
<!-- Author: Conor -->
<!-- Comment: Mon Nov  7 17;21;25 comment by...Conor -->
<pre>This may have been similar to a later page which I describe in more detail below.</pre>
<!-- Date: Sat Nov 5 08:57:02 2005 -->
<pre>Friday, 4:15pm (ie 15 mins after I got off shift) 
I was initially paged because L3 failed the config transition. This was because the ACE ENDed a run, then tried to CONFIG again without a RESET. You always have to RESET after ending a run. So, that was easily fixed. There is no need to clean up the L3 farm, just reset and start again and it usually works. Except this time, because output node 02 still failed config (all others were fine). This was caused by some L3-CSL network adjustments I had made earlier in the afternoon. Although we dont quite understand why yet, it seems that if I do an /sbin/ifdown eth4 then /sbin/ifup eth4 (to check the network configuration for the gbit card) for output node 02, then it would not configure properly when we try to take data. It needs a reboot before behaving normally. No idea why this is, but have checked that it is reproducible. Have not yet checked whether all output nodes have this problem. So, lesson is: be careful with ifdown and ifup! 
Another useful thing to know: if you have to reboot a node, then the RelayService program will not be running on it, so you should clean up the farm to restart this program, otherwise L3 will register it as a relay failure.</pre>
<!-- Author: Conor -->


<!-- Date: Sun Nov 6 11:52:39 2005 -->
<pre>Sat 1.30pm 
 
Not officially a page, but would have been if I were not on ACE shift at the time. 
Multiple clients in control room experienced problems connecting to RTserver. From evb and Level3 side, we saw the same error message:  
WARNING: Lost connection to RTserver: error code = 10. 
      Waiting before reconnecting. 
      ... 
      Attempting to reconnect to RTserver. 
      Connecting to project &lt;cdf_daq&gt; on &lt;tcp:b0dap33&gt; 
      RTserver. 
      Using tcp protocol. 
      connect: Connection refused 
      Could not connect to &lt;tcp:b0dap33&gt; RTserver. 
 
Connection from Level3 was restablished ~15min later and from then on it was fine. The EventBuilder however reported: 
 RTserver denied access 
     Message: Client name &lt;/_b0eproxy01.fnal.gov_3880&gt; is not 
      unique.Cannot reconnect to RTserver. 
 
When it became apparent that all other clients had succesfully reconnected to the RTserver, I cleaned up the evb and then all was well. 
Markus expects that this behaviour from evb side will not happen again when he updates the interface with runControl</pre>
<!-- Author: Conor -->


<!-- Date: Mon Nov 7 17:19:12 2005 -->
<pre>Sat 8.50pm. 
 
Again, not officially a page, because I was reading the shift elog and called them while they were thinking about paging me. 
Three subfarms (5,9,16) went pink and then some minutes later, grey. This did not affect data-taking. For other reasons, they had to restart the run twenty minutes later, and then the subfarms came back fine. I looked into the l3_node logfiles at the time and did not notice any problems. With hindsight, I should also have checked that all the monitoring programs were running - this seems the most likely explanation. In particular, I should have checked if the nodes were still actually processing events but just not displaying this. If such a thing happens again, one should check the monitor logfiles and that all the needed programs are running on the affected nodes. The "clean-up monitoring" button in the AceControlPanel  might be useful for fixing it without starting a new run or doing the full farm clean-up.</pre>
<!-- Author: Conor -->
<!-- Comment: Mon Nov  7 17;23;11 comment by...Conor -->
<pre>This may have been similar to the page from the previous Wed (see above). At that time, I cleaned up the farm, which may not have been necessary.</pre>


<!-- Date: Tue Nov 8 09:04:27 2005 -->
<pre>Mon, 8.15pm 
 
Find from the logbook that the shift crew tried to page Level3, but I never actually got any page! The pager was low on battery for a couple of hours, that might explain it. The problem was not urgent, so the shift crew did not follow it up with any more pages to us. It appears that converter node 11 crashed. This showed up as a relay failure when the ACE tried to config the next run. Fortunately we had a good ACE who read the error message and took the right course of action - mark out the subfarm and start a new run. Things then worked fine. This all happened between stores, so no data was lost. The node was rebooted later (by Markus) before the next store and recovered fine. I find nothing in /var/log/messages or any of the level3 logs that would explain the crash. It is also puzzling that though the standard relay failure message is printed in the error log for the run (206805), no email was sent to us.</pre>
<!-- Author: Conor -->


<!-- Date: Tue Nov 8 23:07:36 2005 -->
<pre>EVB wasn't making it through Partition, because b0evme20 was not accessible. Cleaning up EVB didn't work, so I went downstairs and I manually rebooted the scpu. Problem was resolved with ~5 minutes downtime.</pre>
<!-- Author: Georgios -->


<!-- Date: Wed Nov 9 17:53:19 2005 -->
<pre>L3 was paged due to too many Reformatter errors. The reason was that the L2 expert was tampering L2, so there was a part of the L2 information missing and that's why Reformatter was complaining. It was not a L3 problem.</pre>
<!-- Author: Georgios -->


<!-- Date: Thu Nov 10 10:35:14 2005 -->
<pre>Thursday 9am: 
evb proxy crashed on partition transition. Shifter had cleaned up EVB while L3 was trying to disconnect. EVB failed to reconnect. 
</pre>
<!-- Author: markus -->


<!-- Date: Sat Nov 12 16:34:28 2005 -->
<pre>At 15:36, L3 subfarms 1, 9, 10, 11, and 14 dropped out without affecting datataking.  After trying to discover the cause for this sudden failure, we decided to see if a L3 Cleanup would be able to reinclude them. 
 
We should note that technically, we were able to continue datataking without any penalty in deadtime, but we wanted to see what was happening with L3.  All four subfarms were successfully reincluded, so we do not expect that this will cause a problem in the future. However, we are still investigating the reason for the problem.   
 
We looked at the logfiles and conclude tentatively that the problem seems to come from the converter nodes, i.e. probably not a problem with output nodes.  In particular, we kept seeing an error message mentioning "faucet connection closed" in the processor node logfiles.  Investigation ongoing..........</pre>
<!-- Author: Conor & Khaldoun  -->


<!-- Date: Sun Nov 13 18:47:34 2005 -->
<pre>We added node 109, 117, 122, 217 into the L3 farm.  They were out for various problems, all of which should be fixed now.  We ran L2 torture and saw that these 4 nodes were able to run OK, although we had a very high reformatter error rate for the whole farm, which  Bill Badgett claims is most likely due to svt work he is currently performing.  Our test is complete, and we believe that everything should run fine.</pre>
<!-- Author: Georgios and Khaldoun -->
<!-- Comment: Sun Nov 13 18;51;45 comment by...Khaldoun -->
<pre>Bill just ran L2Torture again and everything looks beautiful.  </pre>


<!-- Date: Sun Nov 13 23:30:43 2005 -->
<pre>I was paged to be informed of the failure of 217. I told them to continue taking data and we would take care of 217 tomorrow.</pre>
<!-- Author: Georgios -->


<!-- Date: Sun Nov 20 17:17:37 2005 -->
<pre>Sat Nov 19 2005 11:45: 
Ace reported relay failure from converter 06. I instructed her to  
remove subfarm 6 from all runs for a few hours until I could  
examine the converter. 
Khaldoun and Georgios were in the control room soon thereafter and 
rebooted it, and included it back in data-taking.  Khaldoun says there 
was a kernel panic, and there is no indication in 
/var/log/messages of anything. 
 
Incidentally, based on Khaldoun's description, there seems to 
have been a miscommunication between Run Control and L3 proxy 
when RC had to be reset with the manual override. L3 became red 
in the Replies and Acknowledgements window, presumably because it 
had not been reset as RC would have been expecting.  I have no 
idea what messages RC is sending in the manual override case.  
Cleaning up L3 brought it back to its normal undefined state, of 
course.</pre>
<!-- Author: Jeff -->


<!-- Date: Fri Dec 2 08:49:35 2005 -->
<pre>Level 3 cannot partition. The l3proxy is trying to use converters that have been marked offline.  from the log file l3proxy.out_200512020615:  
 
 
"ERROR--&gt; src infor: @L3_INVALID_SUBFARM - Subfarm no 7, is marked offline ...  
Failed to define partition 0"  
 
This is an error we feared from the recent change in code.  This needs to be fixed today. 
 
Temp fix is to start new run and check "none" for L3 subfarms and then select the ones that are online.</pre>
<!-- Author: khaldoun -->
<!-- Comment: Tue Dec 13 14;08;04 comment by...khaldoun -->
<pre>If I'm not mistaken, the current version of RC (as of at least one week ago) no longer has this bug.</pre>
<!-- Date: Mon Dec 5 10:39:23 2005 -->
<pre>Sunday Noon: 
 
Alexei was doing plug sourcing and was using an old version of Run Control from his own account.  As a result, the L3ReadoutList message was outdated and sending info about 16 subfarms instead of 20.   
 
--Result: L3 thinks that RunControl is asking to include subfarms that are marked offline (in this case, 17-20) and fails to partition.   
 
--Fix: Use new Run Control!! 
 
</pre>
<!-- Author: Khaldoun & Conor -->
<!-- Comment: Tue Dec 13 14;07;01 comment by...khaldoun -->
<pre>Technically, we now protect against this in at least two ways:

1-- If RC asks for a subfarm that is not online, L3 ignores that request.  

2-- Furthermore, if someone is using an old RC that only knows of 16 subfarms, L3 automatically makes 17-20 offline.</pre>
<!-- Date: Mon Dec 12 11:16:18 2005 -->
<pre>Wed 2005.12.07: 
 
several nodes remained down after the power outage was over.  the following nodes simple needed a hard reboot (turn switch on back to 0 then to 1, and for some nodes, go to the front and press circular power button): 
002 
057 
226 
229 
257 
258 
 
for node 047, i did a reboot but then could not get nfs to work properly.  after stopping and restarting nfs several times, markus suggested we switch the ethernet cable on that node to the other port available.  this fixed the problem.   
 
node 117 is still out: it runs for some time but then crashes every time.   
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
the following three were out from last week: 
 
node 070 will need a hard drive switch; the current one is totally corrupt and it cannot even boot into linux 
  
node 081 received the fsck treatment but still had trouble with the /boot check.  can either go into the intricacies of fsck or just do a little kickstart. 
 
node 083 is back after a serious manual running of fsck.</pre>
<!-- Author: khaldoun -->


<!-- Date: Mon Dec 12 22:30:29 2005 -->
<pre>Tuesday 6 Dec 2005, 9pm: 
 
Level3 was all green, indicating that it was trying to output to CSL. But the output rate was non-zero, and the CSL was working. Reason was that CSL was maxed out, because they were taking cosmics with absurdly high trigger rates, due to some hot muon and/or cal triggers. Under these circumstances, there is nothing that L3 can do, because CSL is the bottleneck. Markus asked the shift crew not to run too long like this, since it is not what the system is designed for. 
 
Friday 9 Dec 2005, 2pm: 
 
Alert ACE recorded in elog that subfarm 11 was grey. It was good that he noticed because our automatic alerts had temporarily failed. This was traced to the fact that a cvs update had removed the executable permission of the alerting script. This annoying cvs feature has been fixed and should never bother us again. Converter 11 had crashed, that was why the subfarm was out. We rebooted the converter and re-included that subfarm, after fixing the script permisssions. </pre>
<!-- Author: Conor -->




<!-- Date: Thu Dec 15 12:13:12 2005 -->
<pre>Tuesday 13 Dec 2005, 2am: 
                                                                                 
Some subfarms were showing much lower activity (factor of 5) than the others. Luminosity was very low, so this was not affecting data-taking. 
Solution: the run had been started from idle. Told the ACE to issue a reset before starting the next run. 
</pre>
<!-- Author: Conor -->


<!-- Date: Fri Dec 16 14:15:10 2005 -->
<pre>Converter node c11 crashed. I restarted it and subfarm 11 was included in the run again. Nothing extraordinary.</pre>
<!-- Author: Georgios -->


<!-- Date: Tue Dec 20 17:39:55 2005 -->
<pre>Took out second hard drive from nodes b0l3122 and b0l3070.</pre>
<!-- Author: Georgios and Khaldoun -->


<!-- Date: Sun Dec 25 15:21:12 2005 -->
<pre>Saturday 11.30am: 
b0evme14 unresponsive. Needed to move the SCPU in a different slot.</pre>
<!-- Author: markus -->


<!-- Date: Thu Dec 29 15:16:16 2005 -->
<pre>Wednesday 8.30pm: 
DAQ problems caused a lot of HHRs and Aborts and finally one SCPU gave up. EVB needed to be cleanup.</pre>
<!-- Author: markus -->


<!-- Date: Tue Jan 3 01:44:09 2006 -->
<pre>Summary of week's pages: 
 
Fri 30 Dec 2005, 8am: 
We were not paged, but 6 min downtime was assigned to Level3. 
Some subfarms were apparently completely dark blue. I havent seen this before, but it was most likely because the run was started from IDLE. Losing 2 subfarms is not a problem for data-taking and if we had been paged, I would have suggested that they simply run like this. But the ACE decided to end the run and clean up Level3 farm. Then there was a relay failure for processor 251. The ACE decided to use the 'Clean-up node' button on the EVB/L3 control panel to try to fix it. The same thing was tried for node 117 as well, presumably because it also shows up as grey in the display, since it is not currently being used. This did not help at all, of course. Then they just continued running. 
Okay, so it was only 6 minutes, but I was a bit unhappy about this incident because: 
(a) We were not paged at all 
(b) The actions undertaken (such as 'clean-up node') are not advocated anywhere in our ACE-help webpages. 
 
Sat 31 Dec 2005, 6am: 
Evb did not respond to a HRR. The ACE handled this very well. He 
found that evb proxy was not running, then cleaned up evb. SCPU 25 did not recover properly. By the time I responded to the page, the ACE was already among the first floor racks, looking for b0evme25 to do a manual reset. I told him to go ahead, the scpu then recovered fine. Then one needs to clean up the evb to get everything back in sync after the reboot, and after that data-taking continued smoothly. Total downtime was ~20mins. 
Not clear from logfiles what the actual problem was. 
 
Sun 1 Jan, 8.30pm: 
Looked similar to above page - evb failed transition, cleanup not 
sufficient, needed ACE to manually reset b0evme23 this time. 
However, this time there was also an RTserver problem at the same time. This may indeed have been the cause of the evb problem in the first place - both Level3 and evb logs show a broken connection to RTserver, although Level3 succesfully reconnected in a few minutes. 
The RTserver problem also complicated the ACEs' task of recovery. 
 An additional feature of this incident was that silicon crate b0svx06 also showed an error at the same time. This is in the same crate as b0evme23. We hypothesize that these two are related, probably due to ACE having to manually override run control state after evbproxy died. The svx06 error recovered after evb was fixed. 
Total downtime was 45mins, assigned to RTserver and evb. 
 
Mon 2 Jan 2006, 10am: 
ACE noticed that converter node 11 was grey and the whole subfarm 
belonging to it was doing nothing, so they paged me. A check of level3 farm revealed that the node was unreachable. This machine crashes the most often of all converters. I contacted Markus to let him know (I was at home at the time, after my owl shift), and he later rebooted the node. I told the ACE to just continue without doing anything, that Level3 would fix itself at the next end of run. But this was a mistake. At the next HRR, the level3 proxy still tried to send commands to this dead converter, then timed out and reported an error to RunControl. The ACE had to manually over-ride the state manager to end the run. It somehow ended up as 20 mins downtime, which seems excessive. We should add this situation to our ACE webpage, to help guide ACEs through the recovery more efficiently. However, I dont remember a converter node crash causing problems like this in the past, so we should really test this under controlled conditions. 
</pre>
<!-- Author: Conor -->


<!-- Date: Thu Jan 5 21:27:38 2006 -->
<pre>Rebooted Level 3 Converter 6 and cleaned up Level 3 so that all subfarms could be included for the high lum store.</pre>
<!-- Author: khaldoun -->
<!-- Comment: Thu Jan  5 21;28;07 comment by...khaldoun -->
<pre>c06 had crashed b/c of kernel panic</pre>


<!-- Date: Fri Jan 6 14:25:26 2006 -->
<pre>Friday 4am: 
ACE had to abort the run a couple of time due to Si problems and did not reset before starting. We need to get this fixed.</pre>
<!-- Author: markus -->


<!-- Date: Mon Jan 9 17:59:38 2006 -->
<pre>Saturday 6pm: 
converter 11 died again, this time at the beginning of the store. SciCo did not page as instructed after pbar were loaded.</pre>
<!-- Author: markus -->
<!-- Comment: Tue Jan 10 14;14;12 comment by...Georgios and Khaldoun  -->
<pre>We replaced the 4-port eth card in c11 and are preparing a spare node in case this again fails to keep c11 stable for more than ~0.5 week.</pre>
<!-- Date: Tue Jan 10 04:16:20 2006 -->
<pre>Tuesday, 4am. 
The shift crew are doing trigger table tests, and spending a lot of time in halted. At one point, the ACE clicked recover, but evb did not respond and turned red. 
We unfortunately do not have a section in our ACE webpage for 'evb fails _any_ transition'. Anyway, I showed Dean how to do the proxy and scpu checks. All checks passed. Which is a bit unusual. I looked at the logfile timestamps to confirm my suspicion that one scpu had died while in halt, then been automatically restarted. This would explain why all the checks passed, but one scpu was in the wrong state to respond to an RC recover 
transition. scpu 11 turned out to be the culprit. I had Dean issue a reset, over-ride as needed, then cleanup evb. And now all is well. 
</pre>
<!-- Author: Conor -->



<!-- Date: Wed Jan 11 22:59:30 2006 -->
<pre>Wednesday 10pm: 
converter 11 died, rebooted, cleaned up L3 and EVB, started again. All this happened while loading pbars.</pre>
<!-- Author: markus -->


<!-- Date: Tue Jan 17 15:49:07 2006 -->
<pre>11am: 
ACE claimed Level3 failed partition. But this was a misunderstanding of the RunControl acknowledgement colors, because Level3 had not in fact been sent the transition request. It was evb which was not responding. Check of evb proxy revealed that proxy for partition 3 was not running. This partition had been in use by the silicon pager-carrier. The DAQ ACE was using partition 0 as usual. The evb cannot be used by different partitions simultaneously. I cleaned up evb and then things were fine.</pre>
<!-- Author: Conor -->