Memorandum:						January 4, 1999
-----------
from J.-F. Grivaz
to   P. Dornan, W.-D. Schlatter, A. Blondel
cc   P. Gay, N. Konstantinidis
The purpose of this memorandum is to express and substantiate my disagreement
with the decisions regarding the Hnn analysis, taken by the spokesman on
the day following the HTF meeting of December 17. I understand that the
advice of the chairman of the steering committee and of the Thursday meeting
organizer has been taken by the spokesman, which is the reason why this
memorandum is also addressed to them. In addition, a copy is sent to the
conveners of the Higgs task force.
This memorandum is intended to be made available to the ALEPH collaboration
soon after being delivered to its addressees.
1. Synopsis:
------------
In a recent ALEPH-note, Marumi Kado and myself gave a detailed account of
a new analysis designed for the search for the standard model Higgs boson
in the Hnn channel. The development of this analysis had been reported by
Marumi in a series of talks to the Higgs task force in October and November.
The essential features of this analysis, its history, and a comparison with
the so-called reference analysis are presented in Sections 2 to 4 of this
memorandum.
During the HTF meeting of December 17, a long discussion took place
regarding the merits and the opportunity of the new analysis. In contrast
to what had been the case for the previous presentations by Marumi, the
spokesman and the chairman of the steering committee were present, but
obviously unaware of the matter under discussion. The Thursday meeting
organizer was not more present that time than in the previous instances.
A summary of the arguments exchanged at the December 17 meeting, and of
its conclusions, is given in Section 5.
In the late afternoon of December 18, and after consulting the chairman of
the steering committee and the Thursday meeting organizer, the spokesman sent
a mail to inform the HTF of his decisions regarding the Hnn analysis. Those
decisions have little to do with what seemed to be the conclusions of the HTF
meeting held on the preceeding day. In addition to being unacceptable from
a purely deontological point of view, it is my belief that the decisions taken
by the spokesman, and the way they are taken, are bad for the Higgs task force
and bad for ALEPH. The spokesman's mail, with my comments to it interleaved,
can be found in Section 6, followed in Section 7 by my conclusions.
2. Features of the revisited Hnn analysis:
------------------------------------------
Although all the necessary material can be found in the note entitled
``The Hnn analysis revisited'' submitted by Marumi and myself on December 9
(ALEPH/98-096 PHYSIC/98-045), a brief account of the specific features of this
analysis is given here.
The initial goal was to perform a full reassessment of the reference analysis
(ALEPH/98-009 PHYSIC/98-009), rather than to try to marginally improve its
performance by added details here and there. The first result was a new
preselection (ALEPH/98-087 PHYSIC/98-037), proposed by Cal and Marumi and
adopted for the LEPC and other future analyses.
The next developments were performed in view of an optimal result for a 95 GeV
signal mass, close to the expected sensitivity for the standard model Higgs
boson at 189 GeV. The issue of the dependence of the analysis performance
on the signal mass was to be addressed in a later stage. For neural network
based analyses, this problem can be solved either through the commonly accepted
method of a signal training sample composed of various signal masses (the
cocktail approach), or through some new technique allowing several neural
networks, each optimal at a given mass, to be interpolated (the sliding neural
network approach). Such techniques have already been proposed by Hongbo
(October 8 HTF) and by Marumi (November 19 HTF).
In the revisited Hnn analysis, the two main backgrounds are first addressed
separately, using kinematic features only.
To discriminate the signal against the qqbar background:
 After an essentially harmless additional preselection, a seven variable
 neural network (NN) was trained. The seven variables are identical to or
 can be reconstructed from variables used in the reference analysis.
 They were chosen for their obvious physical interpretation. The E_12 variable
 was however not considered at this point because it is too sensitive to the
 beam related background.
To discriminate the signal against the WW background:
 Background events with an obvious (W-->l/tau nu) decay are first removed
 by simple cuts on the lepton isolation (if there is such a lepton) and
 on the tau minijet energy. The latter variable was used in the reference
 analysis while the lepton isolation is a new variable, inherited from
 SUSY searches. A three variable NN was then trained. Again, the three
 variables are identical to or can be reconstructed from variables used
 in the reference analysis.
At this point, a four variable NN was trained, using as inputs the outputs of
the above anti-qqbar and anti-WW NN's, and of the b-tagging NN for the two
most energetic jets. For the training, all backgrounds were taken into account
(except for the irreducible ZZ-->bbnn process), in amounts in accordance with
their contributions after all preselections.
The final touch is a cut on E_12, totally harmless in the absence of beam
related background.
The basic philosophy sustaining the revisited analysis is that each problem
should be addressed in turn and understood on its own, unless correlations
prevent from doing so efficiently. The performance can be summarized by an
expected background level of 5.4 events for a 40% efficiency on a 95 GeV mass
Higgs signal.
3. History of the revisited Hnn analysis:
-----------------------------------------
  http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf081098.ps
  http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf211098.ps
  http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf051198.ps
  http://alephwww.cern.ch/LEP2ANALYSIS/HTF/minutes/Marumi_htf191198.ps
The above references direct to transparencies presented by Marumi in HTF
meetings on October 8 and 21, and on November 5 and 19. It is therefore
difficult to argue that the collaboration was faced, at the last minute,
with a completely new analysis, especially in view of the fact that the
essential features were already present in the October 21 presentation.
Indeed, the following mail was sent to the HTF by the conveners on
October 22:
  Dear all,
  At the end of yesterday's meeting, a quick review of the analyses
  status (hZ and hA) was done.
  Some of you had to leave before the end, thus we will give a summary
  here.
  After a review per channel, it was concluded that the extra time needed
  to obtained the finalised versions will be small (not more than four
  weeks). Thus people would have this extra time to finalise the *current*
  developments. Then, It will be nice to have the different persons involved
  in each analysis in touch all together in order to obtain the convergence
  as soon as possible. The main issues after that are the systematics, the
  background subtraction, the problem of the overlap between the
  analyses,...etc.
  Cheers,
  Pascal + Nikos
The deadline of four weeks to finalize the *current* developments was met
with Marumi's presentation on November 19. Following the conveners'
recommendations, people involved should have met at that point to converge
as soon as possible. A meeting was organized for that purpose on December 3
right after the regular HTF. The Wisconsin group decided to boycott that
meeting. A second attempt was made on December 16, which was also cancelled
because Xidong had left for vacation... In the mean time, a comprehensive
note on the revisited Hnn analysis was issued and advertized by a mail to
the HTF, thus allowing any motivated member to ask questions or to raise
objections.
4. Comparison of the LEPC and revisited analyses:
-------------------------------------------------
Except for the new preselection proposed by Cal and Marumi last September
(ALEPH/98-087), the Hnn LEPC analysis is a straightforward extrapolation of
the 183 GeV analysis (ALEPH/98-009). The only modifications to the eleven
variable NN inputs were the removal of the track isolation angle, shown to be
useless, and the introduction of the missing mass, necessitated by the softer
requirement on that variable in the new preselection (Xidong at the Mainz HTF).
The number of input variables happened that way to remain unchanged.
In the absence of any solid documentation, the only uncontroversial anchor for
the Hnn LEPC analysis is the information which had been provided to Patrick in
view of the combination of the various channels. It features a background of
8.2 events for a 40% signal efficiency on a 95 GeV mass Higgs boson.
The reduction from 8.2 to 5.4 background events, for the same 40% signal
efficiency, is quite substantial, and it is interesting to investigate
to which specific features it can be attributed.
The most obvious one is that the revisited analysis was designed to provide
optimal performance at 95 GeV. A LEPC-like analysis, but with a NN trained
on a single signal mass of 95 GeV, rather than on a mass cocktail, provides
a more adequate reference for a comparison, with a background level of
7.2 events.
Training the NN on all relevant backgrounds (rather than on qqbar and WW only
as in the LEPC analysis) brings a clear improvement: the background is reduced
to 5.7 events.
Taking the anti-qq and anti-WW preselections into account improves the
performance only slightly, and for efficiencies in excess of 50%. This
is not too surprising since most of the variables used in the anti-qq and
anti-WW preselections are also input to the eleven variable NN. One
exception is the lepton isolation from which a twelve variable NN could
benefit (Jennifer at the November 19 HTF).
Other combinations were tried: removing E_12 from the NN and adding a post-cut;
using the smaller and larger b-tagging NN outputs rather than two symmetric
combinations; replacing kinematic variables of the reference analysis by
similar ones from the revisited analysis. No further gain was achieved.
The remaining improvement, from 5.7 to 5.4 background events, is therefore
likely to be due to the ``hierarchical'' structure of the revisited analysis
(although this cannot be proved easily).
5. December 17 HTF meeting:
---------------------------
Most of the December 17 HTF meeting was devoted to the fate of the revisited
Hnn analysis. The main objections raised against it were that this analysis
was too fresh to be accepted without further scrutiny, and that it was
unchecked. But it was also felt that the benefits brought in terms of
performance could not be simply ignored. Finally, more emotional aspects
entered the game.
Was this analysis really fresh ?
 As detailed above, all steps in the development of the analysis had been
 regularly reported to the HTF since October 8. The transparencies presented
 were detailed and clear enough to allow a thorough understanding of the
 options taken and of the progress made. The essential features were all
 available on October 21. In accordance with the conveners' request, the end
 of all developments in view of Moriond and of the publication was explicitly
 signified by Marumi on November 19. Finally, a comprehensive ALEPH-note was
 made available on December 9.
 Except for the newcomers, among which the addressees of this memorandum are
 unfortunately to be counted, it is difficult to pretend discovering this new
 analysis on December 17.
Was this analysis unchecked ?
 In addition to the ALEPH-note describing the new analysis, all of the code
 pertaining to it had been made accessible on the web through the HTF page,
 and advertized in the mail calling attention to the ALEPH-note. From this
 point of view, the situation of the revisited analysis is clearer than that
 of the LEPC one.
 The answer to the question ``Was the analysis unchecked ?'' is nevertheless,
 and unfortunately, yes.  This is due to the unavailability of anyone willing
 to perform the necessary checks. With Cal absent from this meeting, Xidong
 away until mid-January, Jennifer kindly accepted to make the checks requested
 in early January. Whether these checks will actually take place or not is
 difficult to predict at the time of writing.
How can the benefits be preserved ?
 For reasons which I am unable to understand, it seems that the anti-qq and
 anti-WW preselections or, more importantly, the improved composition of the
 background training sample, are modifications easy to accept, whereas changes
 to the neural network structure are viewed as exceedingly involved. This may
 have to do with the fact that the addressees of this memorandum have little
 experience with neural networks, which may appear to them more mysterious
 than they really are.
 As detailed above, it is not very difficult to improve the performance of the
 LEPC analysis (from 7.2 to 5.7 background events at 40% efficiency), close to
 that of the revisited analysis (5.4 events). Should the difference between
 5.7 and 5.4 background events be considered as marginal ? In terms of
 efficiency, it corresponds to a relative improvement of 2% for a 40% absolute
 efficiency. Around 30 and 50% efficiencies, the corresponding relative
 improvements are 6 and 3%.
 In many instances, choices have been made in the HTF based on similar or
 smaller differences. Unless there are compelling, and therefore explainable,
 reasons not to do so, this is the only way to preserve, or rather to restore,
 ALEPH's leadership. Asking questions such as ``By how much does it change the
 expected limit ?'' is to be put at the same level as ``Why should we care for
 a 3% degradation of the data taking efficiency ?''.
Emotional aspects:
 Another issue regarding which unbiased arguments are less easy to find is
 the possible ``unfairness'' which would result from a decision to go ahead
 with the revisited analysis. Irrespective of the merits of that analysis,
 such a decision would be unfair to those who had devoted all their strength
 first to the LEPC analysis, then to the assessment of systematic errors in
 view of the December 3 Thursday meeting.
 In fact, in a normal collaboration, it would be a most beneficial sharing of
 efforts that some secure a result while others prepare the following term, and
 all would then be proud of the common achievements. Unfortunately, no result
 was secured at all for the LEPC, and no specific work was reported at the
 December 3 meeting regarding the background subtraction systematics in the Hnn
 channel.
 In practice, the LEPC analysis did not really require any work other than
 keeping up to date with the data collection. The few modifications, already
 mentioned in this memorandum, with respect to the 183 GeV analysis had
 already been made by the time of the Mainz ALEPH week, and the background
 subtraction, in the absence of systematic studies, is a fairly simple
 technical issue. That only little effort was required by the Hnn LEPC
 analysis is demonstrated by the fact that Cal on one side, Jennifer and
 Xidong on the other, found the necessary time before the LEPC to develop
 competing analyses for the ZZ-->qqnn channel.
At the end of the December 17 meeting, no consensus was reached.
 My recollection is that one of the conveners wrote a few lines on a
 transparency stating that the LEPC analysis (defined as what was provided to
 Patrick for the combination) remains as a back-up analysis until January 14,
 that the revisited analysis will be cross-checked by Jennifer in early
 January, that the mass dependence will be handled according to the cocktail
 method, and that, of the two analyses, the one with the best performance in
 terms of confidence level will be retained.
 The spokesman seemingly agreed to this proposal. At least, he did not express
 any disagreement.
6. The day after the December 17 HTF:
-------------------------------------
In the late afternoon of December 18, the following letter was sent by the
spokesman to the HTF conveners, to the head of the Wisconsin group, and to me.
For an easier understanding of the arguments, my comments are interleaved in
the body of the mail, highlighted by *** marks.
*******************************************************************************
Dear Colleagues,
Yesterday's Higgs meeting was one of the most unpleasant ones I have
experienced in ALEPH; it was a bad example for our young physicists. The
contrast with the previous day's WW meeting at Annecy could not have been
greater. When we have differences we should solve them in a manner which
respects all involved. This was not so yesterday.
*** On this, I could not agree more, although I have the feeling that the
*** main responsibility of what happened does not rest on my shoulders.
*** I may elaborate on this later in a private mail.
Following the meeting I checked the minutes of the Dec 3rd Thursday
meeting. It makes a clear statement that the analyses for Moriond should
be based on the ones presented by Nikos once the background systematics
are understood. The Higgs group was thus encouraged to concentrate for the
immediate future on understanding the background systematics for these. At
that time there was no mention of any other Hnunu analysis being ready for
the winter conferences.
*** Some may find it rather surprising that the spokesman has to go back to
*** minutes of meetings to determine what he did or did not approve, and what
*** is good or bad for ALEPH. Anyway, it is hard to draw any conclusions from
*** those minutes (unavailable on the Thursday meeting web page, by the way)
*** other than: 1) the background studies in the Hqq channel give us confidence
*** that we are facing a statistical fluctuation, and we should try to proceed
*** with full background subtraction for Moriond and for the paper (first
*** presentation); 2) the backup analysis ``a la Vancouver'' will be used for
*** the early Winter conferences, including La Thuile (second presentation).
*** And the overall conclusion was:
*** ``The "backup" analyses are approved for the incoming conferences.
***   The finalisation of the "full background subtraction" analyses before
***   Moriond is recommended (supported by the 189 GeV Higgs search paper).''
*** It is very true that the ongoing developments in the Hnn channel were not
*** mentioned at that meeting, the reason being that they had nothing to do
*** with the matters discussed. The new developments had never been intended to
*** be used for the LEPC. The issue of full, partial or no background
*** subtraction is common to all Hnn analyses.
The analysis which was shown to us yesterday has some very attractive
features and is supported by a very good ALEPH note, however the overall
performance is very similar to the one which can be obtained with the
standard 11V neural net.
*** The revisited analysis was not ``shown to us yesterday''. It has been final
*** since November 19. It is too bad that the addressees of this memorandum
*** were not cautious enough to be aware of that point. I appreciate that the
*** quality of the supporting ALEPH-note is acknowledged.
*** The second half of the sentence goes a bit fast. As of today, the only
*** officially approved analysis is the LEPC one, and it is that analysis
*** which provides the reference performance for the 11V-NN. It is the
*** analysis ALEPH would anyway have to live with for Moriond if Marumi and
*** I had not been working on the revisited analysis for more than three
*** months. With respect to the LEPC analysis, the improvement brought by
*** the revisited analysis cannot be questioned.
*** The least we could expect is that our agreement be requested for the
*** use in the 11V NN of some of the improvements which we made. Unless, of
*** course, it is decided to open a new phase of development for Moriond and
*** for the paper, in total contradiction with the policy set by the HTF
*** conveners, as stated in their October 22 mail quoted above.
This morning I heard that we have the Higgs talk at La Thuile which makes
it most important that we concentrate on ensuring we have final results
with systematics by February 18. Pursuing a new analysis at this time for
the winter conferences for what is likely to be a minimal gain in the
final limit would not seem wise. In the past we have missed important
dates because we have been too ambitious.
*** Here too, I find it surprising that the spokesman discovered only on
*** December 18 that we have the Higgs talk in La Thuile, while I was already
*** aware of the fact (maybe only of its very high probability, which does not
*** change a thing).
*** The question of systematic studies in the Hnn channel is essentially
*** independent of whether it is the LEPC or revisited analysis which is used.
*** Nothing prevents these studies from being done, except maybe for the
*** unavailability until mid-January of some of those in charge of the
*** reference analysis.
*** As already explained in this memorandum, pretending that the revisited
*** analysis is too new to be used for Moriond is a statement which cannot be
*** defended in view of its history.
*** How a gain is decided to be minimal is clearly a matter of appreciation.
*** I made the point earlier that such statements clear the way towards
*** decadence. I also do not think we can ever be too ambitious for ALEPH,
*** unless ambition is mistaken for presumption. I rather fear that we now
*** become too shy after being too complacent.
*** At any rate, we had proposed to keep the LEPC analysis as a back up until
*** January 14, thus avoiding any risk that we miss the La Thuile deadline.
*** Had we made a similar decision for the LEPC, we would not have had to face
*** the situation we know. The error at the time was not to try to do our best
*** for the LEPC, it was not to have secured a fall-back solution.
Alain, Dieter and I discussed the situation at length this morning. We
decided that the safest option was to stay with the analyses directly
based on those presented by Nikos at the Dec 3 meeting and to request the
Higgs group to push ahead with all speed to perform these analyses on the
reprocessed data and understand the systematics. In this way we are
confident that, with cooperation from all, we can have final numbers and a
first draft of the paper/conf note by February 18. This is to be the
priority of the group, followed by the preparation of the 1999 analysis
which should run online when data taking resumes.
*** Nothing new here. Any option with a fall-back solution is a safe option.
*** My personnal point of view (hard to prove, I agree), is that progress
*** will be slower in the Hnn channel than it would have been otherwise, if
*** only because, as already stated, some key advocates of the reference
*** analysis are not currently available. Of course, Marumi and I will do the
*** work in the context of the revisited analysis, and a good fraction of it
*** should be useful irrespective of whatever choice is made in the end.
There are a number of features of the Grivaz-Kado analysis which could be
simply implemented in the standard one and we encourage everyone to work
together to make such additions. Additionally, if a sound result from this
analysis is available for the paper, it would be natural to include it as
a supporting analysis as we have so often done in the past. It is our view
that the merits, or otherwise, of the multi-neural net structure require
time to be appreciated but that it is a strong candidate for the 1999
analysis.
*** Making additions now in January to the reference analysis goes well beyond
*** the boundary conditions which had been set within the HTF. At the time of
*** writing, the LEPC analysis is, apart from the revisited analysis, what
*** we have, and I expect that additions to the reference analysis based on
*** Marumi's work would not be made without our consent.
*** It is very true that supporting analyses have often been included in
*** ALEPH papers in the past. As far as I remember, these supporting analyses
*** had weaker performances than the ones used to derive the official result.
*** Here the reverse would happen, by far if the LEPC analysis is to be used,
*** by less, but still by a noticeable amount, if some upgraded reference
*** analysis were to be used instead. I do not see how the choice made could be
*** explained to the outside world. Probably, some kind of understatement
*** such as ``similar performance'' would have to be utilized. The argument
*** which would convince me to accept that is still to be found.
*** ``Multi-neural net structure'' is a phrase, the purpose of which, I fear,
*** is to frighten the reader. In fact, it boils down, in front of a given
*** event, to asking three questions:
*** 1) Does this event look like a qqbar or like a Higgs event ?
*** 2) Does this event look like a WW or like a Higgs event ?
*** 3) Is this event well b-tagged ?
*** Each of these questions is answered by the value of a NN output.
*** This is what we have already done for years with the way the b-tagging
*** information is handled. Finally, these NN ouputs are combined into yet
*** another neural network, which has the virtue of being trained against
*** all relevant backgrounds (i.e., Wen, Zee and ZZ in addition to qq and WW).
*** If anybody can explain to me why it is simpler to put all the information
*** into a single box, mixing up the tau-minijet energy which addresses
*** only the WW background and the acoplanarity (modified or not) which
*** addresses only the qqbar background, I will be most grateful.
*** It is my belief that the addressees of this memorandum did not pay any
*** attention to these questions, and are motivated by considerations which
*** have nothing to do with the supposed complexity of the revisited analysis.
*** Regarding the time needed to appreciate the merits (or otherwise ?) of
*** the approach taken in the revisited analysis, let me remind that it already
*** amounts to more than two months and a half. And as to the vague promise for
*** the 1999 analysis, I take it about as seriously as those which had been
*** made to us in October, and torn in December.
We trust that you will all appreciate that this decision has not been
taken lightly and it is taken, in our view, in the best overall interest
of ALEPH.
        Peter
*** In the end, it is very difficult to understand which decision has really
*** been taken.
*** 1) Should we stay with the analyses reported by Nikos on December 3, and
*** only work on systematics ? From the second paragraph of this mail, the
*** answer seems to be yes. This is repeated at length in the fifth paragraph.
*** 2) It is also stated in the fourth paragraph that pursuing a ``new''
*** analysis for the winter conferences would be unwise.
*** 3) However, mention is made in the third paragraph of a possibly improved
*** performance for the reference analysis. And in the sixth paragraph, people
*** are encouraged to implement features of the revisited analysis into the
*** standard one. This looks in contradiction with the first of the above
*** decisions.
*** 4) Finally, obtaining a ``sound result'' from the revisited analysis is not
*** discouraged, with the possibility of including it somehow in the paper.
*** This also looks contradictory to the second decision.
*** I am therefore sorry to conclude that, in spite of the spokesman's mail,
*** we remain in the dark.
P.S. Could the convenors please circulate this to the Higgs group.
*** It is of course because of this post-scriptum that I took the liberty to
*** reproduce this mail in the present memorandum.
*******************************************************************************
7. Conclusion:
--------------
As detailed in my reply to the spokesman's mail, my impression is that the
situation has not been clarified at all. The same arguments as those used
in that mail can be used to draw different, if not totally opposite,
conclusions. For instance:
1) We should by all means have an essentially final result for La Thuile.
   For that purpose, the safest is to consider the LEPC analysis, as it
   was delivered at the time of the LEPC, as the uncontroversial back-up
   solution.
2) We should not embark now into new developments. Therefore, the only
   contender with respect to the LEPC analysis is the revisited analysis,
   the development of which was declared finished (as far as the 189 GeV
   data analysis is concerned) on November 19. This analysis is fully
   described in an ALEPH-note.
3) We should work on signal and background systematics with the available
   manpower without any further waste of time. Most of the work should be
   usable irrespective of the analysis which is used in the end.
4) Once the necessary checks of the revisited analysis are made, and if its
   improved performance with respect to the back-up solution is confirmed, it
   will be chosen for the publication. If not, no time will have been wasted
   and the LEPC analysis can be used. The decision should be taken no later
   than January 14 (first HTF meeting in 1999).
Indeed, these conclusions turn out to be identical to those which a number
of us had understood to be drawn at the end of the December 17 HTF meeting.