An Evaluation Road Map for Summarization Research

Breck Baldwin,[^1] Robert Donaway,[^2] Eduard Hovy,[^3] Elizabeth
Liddy,[^4]

Inderjeet Mani,[^5] Daniel Marcu,[^6] Kathleen McKeown,[^7] Vibhu
Mittal,[^8]

Marc Moens,[^9] Dragomir Radev,[^10] Karen Sparck Jones,[^11] Beth
Sundheim,[^12]

Simone Teufel,[^13] Ralph Weischedel,[^14] Michael White[^15]

# 1. Background

Recent discussions within the context of TIDES have indicated the need
to develop a coordinated approach to text summarization research. This
document presents the response given by the summarization research
community to the challenge of fleshing out a roadmap for an ambitious
plan that will foster measurable progress in the field. The document
focuses on the evaluation of text summarization systems.

# 

# 

# 2. Introduction

Summarizing is a multi-faceted enterprise that can be understood,
modeled, and evaluated along, at least, the following dimensions
\[Sparck Jones, 1999; Hovy and Lin, 2000\]:

## Input:

-   Source: single-document vs. multi-document.

-   Language: monolingual vs. multilingual.

-   Genre: news vs. technical paper.

-   Specificity: domain-specific vs. general.

-   Length: short (1-2 page docs) vs. long (\> 50 page docs).

-   Media: text, graphics, audio, video.

## Purpose:

-   Use: generic vs. query-oriented.

-   Purpose: wWhat is the summary used for (e.g. alert, preview, inform,
    digest, provide biographical information)?

-   Audience: targeted vs. untargeted.

## Output:

-   Derivation: extract vs. abstract.[^16]

-   Format: running text, tables, geographical displays, timelines,
    charts, etc.

-   Partiality: neutral vs. evaluative.

Thus, a summarizing *task* is defined by a combination of factor
specifications, e.g., summarizing of single English news stories, to
alert technical economic analysts, by neutral abstracts in running text.

The roadmap for summarization that emerged during the discussions
carried out at the TIDES Kickoff Meeting, NAACL/ANLP-2000, and
subsequently over email attempts to address as many of the above facets
as possible. In what follows, we describe our evolving general framework
for evaluating summaries through a series of task specifications and we
propose a five-year plan that uses more and more challenging versions of
the framework in order to keep up with and assess progress in
summarization research. Thus, the detailed form of any particular
evaluation will be firmly grounded in a clear factor characterization of
the task. The evaluation development is focused on summary purpose, but
also covers changes in other factors, such as input source and genre,
summary audience, and output format. These choices will test the
generality and portability of the methods under study.

The roadmap outlined in this document focuses on the evaluation of
summarization systems that take as input text(s) and whose output is
text.

# 

# 

# 

# 3. General framework for evaluating summarization technology

## 3.1 Overview

For the first phases of the evaluation we propose to create and exploit
a relatively straightforward, easily constructed evaluation corpus. This
is intended to promote further testing and development of the current
primarily extractive techniques, and to allow performance comparisons
focused on core system technology that is required for a range of
summarizing situations and that can be evaluated in intrinsic mode
without direct reference to context of use. In the five-year program
that we outline here, we will progress from this corpus towards task
specifications that require deeper source document understanding and
more radical content transformations for summary output. We will also
move from intrinsic evaluations towards evaluating summaries for their
functional purposes.

## 3.2 Development of a summarization corpus

For the first stages of our evaluation program, we will create a
multi-document summarization corpus. This corpus will contain a
collection of summary data sets, such as that shown in Figure 1. Each
summary data set contains *n* documents about one specific topic/event
and *m* multi-document abstracts that summarize at different compression
levels the information in the *n* documents. We propose that each data
set contains multi-document abstracts of approximately 50, 100, 200, and
500 words. Each of the documents in a data set has the important
sentences identified/tagged. This represents the first level of
annotation, a level that is consistent with the capabilities of most of
today's summarization systems, which are only capable of extracting
important sentences from a text. For many text genres though, sentences
may provide a too coarse level to assess textual importance. To
circumvent this, we assume that the important text fragments are also
identified/tagged. These text fragments may be smaller than a sentence.
(In Figure 1, the important text fragments are underlined.) Each
document in a data set also has its own abstract. The documents may be
tagged with source, date, and/or time stamps. If the documents in a data
set do not come with an abstract, such abstracts will be manually
produced by the corpus developers. The identification of important
sentences and sentence fragments will be done manually or automatically
(see Section 4.1).

For example, a data set may contain 20 articles (documents) that were
published by *CNN, AP, ABC News,* and *Fox News*, over a time interval
of one week, on the topic of changes in stock market prices; and four
multi-document abstracts that summarize the information in the articles
using approximately 50, 100, 200, and 500 words, respectively. Another
data set may contain 15 articles (documents) that were published by
these news agencies over a time interval of one day about some topic
such as the Los Alamos fire; and four multi-document abstracts that
summarize the 15 articles and that describe the main events of the day
in 50, 100, 200, and 500 words, respectively. Each article/document in
each data set is paired with a short abstract of itself.

Appendix B contains an example of summary data set, which consists of
eight news articles (documents) published by seven news providers, *New
York Times, CNN, CBS, Fox News, BBC, Reuters,* and *Associated Press,*
on June 3^rd^, 2000, the eve of the meeting between presidents Clinton
and Putin. The data set also contains a collection of four
multi-document summaries, which were written by humans at various
compression rates. To conform with the requirements we enumerated above,
each article in Appendix B will need to be paired with its own abstract.

An initial study of Radev et al. \[2000\] suggests that the creation of
one summarization data set for a multi-document summarization corpus
should not take more than 1 person day. This result is consistent with
the time we spent in creating the data set in Appendix B. The TREC
collection could provide the pool from which such summary data sets can
be selected/constructed.

We assume that as the program develops, it will involve other analogous
data sets, e.g. ones for other types of material than news or for much
longer source documents, where the reduction of source documents to
summaries involves more sophisticated forms of content selection and
generalization. For such types of materials, the resources required for
corpus creation could be more expensive.

![](media/image2.jpeg){width="5.99375in" height="4.49375in"}

> Figure 1: Instance of one summary data set.

A summarization corpus that contains such collections of documents can
provide the basis for carrying out the following evaluations.

## 3.3 Intrinsic Evaluation of Sentence Extraction Summaries 

Since the important sentences of each document are identified, one can
assess the performance of document summarization systems in identifying
the most important passages in document texts. For single documents, the
performance can be assessed using traditional recall and precision
figures, utility figures \[Radev et al., 2000\], and/or content-based
measures \[Donaway et al., 2000\]. (See Appendix A for a description of
these metrics). For multiple documents, the performance can be assessed
using the same metrics over the union of the important sentences in all
documents in a data set and over the longest, human-generated,
multi-document summary. In order to determine an upper bound of the
performance of sentence extraction systems, we will also use the same
metrics to evaluate the human generated abstracts of both single and
multiple documents \[Hirschman and Mani, 2000\]. That is, we will
measure how much of the important information in the underlined
sentences is conveyed in the single- and multi-document, human generated
abstracts. The evaluation of extraction engines will be carried out by
requiring them to produce extracts of single and multiple documents at
pre-specified compression rates.

All these intrinsic evaluations can be carried out completely
automatically. These evaluations will enable us to determine what
methods and systems are best suited to determine the most important
sentences in both single-document and multiple-document summarization
scenarios.

## 3.4 Intrinsic Evaluation of Generated Summaries

The simple concatenation of all important sentences in one or more
documents will often yield incoherent summary. Sentences may be
redundant or may contain conflicting information, referential
expressions may be dangling, etc. The job of the generation component of
a summarization system is to take an underlying representation and
produce a coherent summary, i.e., an abstract. The evaluation discussed
in this subsection will encourage the development of generation
technologies specific to text summarization and will reward
summarization systems capable of generating coherent abstracts.

Since only humans can reliably assess the readability and coherence of
texts, we will use judges to assign readability, coherence, and
grammatical and semantic well-formedness scores to automatically
generated abstracts in a manner similar to that used during the DARPA
evaluation of Machine Translation systems \[White and O'Connell, 1994\].
Human judges will also determine how well an abstract produced by a
single or multi-document summarization system reflects the information
provided in the corresponding set of tagged important sentences
(sentence fragments), human-generated abstract, and/or human-generated
multi-document abstract.

This evaluation will be carried out for both single and multiple
documents. As in the evaluation of sentence extraction summaries, an
upper bound of the performance will be determined by evaluating the
readability, coherence, and semantic content of human generated
abstracts.

Naturally, as in the case of the evaluation method described in Section
3.3, the adequacy of the semantic content of automatically generated
single/multi-document abstracts can be still determined automatically
using the metrics proposed by Donaway et al. \[2000\] and Radev et al.
\[2000\]. The comparison of the evaluation scores of these metrics with
the scores assigned by humans will enable us to better assess the
strengths and weaknesses of these metrics. The adequacy of the semantic
content is, however, only part of this evaluation. A simple extraction
engine will often do poorly with respect to the readability, coherence,
syntactic and semantic well-formedness metrics. To do well with respect
to all metrics, a system will need to worry about generating coherent
abstracts too.

Since the evaluation described here focuses primarily on the ability of
generating coherent abstracts, we envision to implement it in two
parallel tracks:

-   one track uses as input single or multi-documents;

-   the other track uses as input single or multi-documents, with
    important sentences (sentence fragments) identified/marked.

This will give researchers the opportunity to participate in the
evaluation with either a complete summarization system that produces
coherent abstracts starting from single or multiple documents, or with a
generation system that is capable of rendering the content of a set of
important sentences (sentence fragments) taken from one or multiple
documents into a coherent abstract. The generation technologies that
will be developed in this context can be then used in conjunction with
question-answering systems as well.

## 3.5 Extrinsic Evaluation of Single and Multi-Document Summarization Technology by Question-Answering.

Shannon \[1951\] estimated that 75% of English text is redundant.
Shannon's findings suggest that humans should be able to get the
essential information in a document by only reading a summary of it. The
performance of single and multiple document summarization systems can be
estimated using a question-answering experiment similar to that
described below \[Hovy and Lin, 2000\]:

1.  A group of analysts prepares a set of questions that can be answered
    by reading important text fragments in a document.

2.  Summarization systems automatically produce summaries of the
    individual documents.

3.  Human analysts attempt to answer the questions produced by the
    analysts

-   Before reading any text/summary (to factor out their background
    knowledge).

-   After reading the summary produced by a system.

-   After reading the whole document.

4.  The more questions can be answered using an automatically produced
    summary, the better the system that produced that summary.

Obviously, the same methodology can be used in conjunction with multiple
documents.

Question answering is being used here primarily as a method of
evaluating summaries, rather than in its own right, reflecting the fact
that in many situations users will want to exploit summaries as primary
sources of information on more than one point and summarization systems
may be designed so as to produce such concentrated, multi-purpose
information sources. However, while question answering is often treated
as confined to single, specific questions, it is clear that (as in the
parallel document retrieval case) questions may be related to one
another. Thus it may be appropriate in designing question answering
systems to consider their integration with summarization systems, to
support answering multiple related questions. For example, generating
biographical summaries may constitute an excellent vehicle for
integrating these two technologies. As this implies, we are designing
our five year programme so as to encourage some integration of the
summarization and question answering tasks.

This type of evaluation is similar to that used at SUMMAC, 1998, where
the set of questions were developed using as input human-generated TREC
topics. By using as input the human-generated summaries and the
human-identified important sentences, we make this evaluation more
specific to a summarization setting than to an IR setting.

In the first year of the implementation of this evaluation, answers to
questions should be identifiable in single documents. Towards the end of
this evaluation program*,* the answers should not be identifiable in
single documents, but should be identifiable in sets of documents. For
example, the information required to answer a question such as "What was
the price of the CISCO stock at the end of the day on July 2^nd^, 2000?"
may be found in a summary of one document. However, the information
required to answer a question such as "Did the price of the CISCO stock
increase during the week of July 2^nd^, 2000?" may be found only in a
summary of multiple documents. Similarly, to produce a biographical
summary, one will have to integrate information from many documents.

This evaluation differs from the evaluation of question answering
systems because the summarization systems will produce summaries capable
of providing answers to as many questions as possible, questions that
will target the important information in a text or set of texts. In
contrast, question answering system will focus on answering specific
questions. In other words, the summaries produced in this evaluation are
generic (provided without knowledge of the question), while in the
evaluation of question-answering systems the answers provided are
query-specific. In addition, the question-answering evaluation is
focused on extracting answers from a document, as opposed to generating
a summary of what\'s there.

3.6 Extrinsic Evaluation of Multi-Document Summarization Technology in
the Context of Topic Retrieval, Detection, and Tracking

An end-to-end multi-document summarization system can be also evaluated
in conjunction with any form of system that selects a set of documents
from a larger collection, whether by retrieval using a given search
query or standing interest profile, or by a system that identifies new
topics as they arise and follows them, as in current Topic Detection and
Tracking (TDT) systems. A multi-document summarization system that works
as a back-end for a TDT system, for example, identifies the important
information in each cluster of documents that is created by a TDT system
and produces an abstract of that cluster. In such a functional,
context-dependent evaluation, it is possible to focus on the role and
assess the effectiveness of a summarization system component in relation
to others. For instance, one can assess the impact of broad rather than
narrow search queries on the nature of the summary, which might be
forced to become a sequence of document summaries rather than a single
seamless whole.

For evaluation, a human judge can use automatically generated summaries
in order to assess whether a certain cluster of documents is relevant
with respect to a specific purpose. As baselines, for example, one can
present human judges with complete clusters of documents or with the
titles of the documents in a cluster. Ideally, a multi-document
summarization system should enable human judges to determine what
clusters of documents are relevant for a specific purpose faster and
more accurately.

## 

## 

## 

##  {#section-8 .unnumbered}

##  {#section-9 .unnumbered}

##  {#section-10 .unnumbered}

##  {#section-11 .unnumbered}

##  {#section-12 .unnumbered}

##  {#section-13 .unnumbered}

##  {#section-14 .unnumbered}

##  {#section-15 .unnumbered}

##  {#section-16 .unnumbered}

##  {#section-17 .unnumbered}

##  {#section-18 .unnumbered}

##  {#section-19 .unnumbered}

##  {#section-20 .unnumbered}

##  {#section-21 .unnumbered}

##  {#section-22 .unnumbered}

## 

## 

## 

## 

## 

## 

# 4 Strengths of the proposed evaluation framework

## 4.1 Potential to partially automate the construction of the corpus

The framework design with source documents marked up with key sentences
is clearly essential for any extractive summarizing strategy, whether
this simply adopts selected sentences or modifies them for summary
formation. However this information also has an important contribution
to make to evaluation for non-extractive summaries, i.e. abstracts.
Since human abstracters typically scan and mark sources for key
material, abstracts can be evaluated at least for whether they cover
this source content.

Previous research in the automatic creation of corpora for summarization
research \[Marcu, 1999; Jing and McKeown, 1999; Banko et al., 1999\] has
shown that using as input a *Text* and its corresponding *Abstract,* one
can apply text similarity metrics in order to automatically create the
*Extract*, i.e., the set of sentence fragments in the *Text* that were
used to write the *Abstract*, at levels of performance that are close to
those of humans. Assuming that one manually or semi-automatically
identifies sets of documents on specific topics that already have human
generated abstracts (see Figure 1), one can then apply the algorithm
described by Jing and McKeown \[1999\] in order to identify in single
documents the sentence fragments that were used in order to produce the
abstracts; or the algorithms described by Marcu \[1999\] and Banko et
al. \[1999\] in order to automatically identify in single documents the
clauses/sentences that were used in order to produce the abstracts.
Multi-document abstracts at different levels of compression can be then
produced manually; it is unlikely that we will be able to find naturally
occurring corpora of multi-document summaries. If the selected documents
do not have abstracts, the corpus developers will also have to manually
identify the important sentences and sentence fragments in each
document.

4.2 Modularity

The evaluation framework described in this document enables independent
evaluations to be carried out. Single document summarization systems can
be evaluated by focusing only on the single documents in a data set, in
which the important sentences are annotated. Multi-document
summarization systems can be evaluated both with respect to their
ability to rewrite a collection of sentences as a coherent abstract and
as end-to-end systems, which take as input clusters of documents and
produce summaries of them.

4.3 Complexity

The multi-document summarization task in particular, but also
single-document summarization especially for long or technical
documents, is complex enough in order to force researchers tackle a wide
range of complex natural language processing tasks. To identify
important sentences in texts and to produce coherent abstracts,
researchers will need to develop better natural language understanding
components, sophisticated intra-and extra-document coreference
resolution systems, and natural language generation systems that produce
texts that are not only grammatical, but also coherent.

## 4.4 Incrementality

The evaluation framework proposed here enables one to incrementally
develop increasingly sophisticated testbeds. For example, one can start
with modest collections of sets of related documents and very simple
summaries, for instance short individual document summaries that
indicate only the main document theme or finding, or multi-document
summaries that focus primarily on reporting facts and events.
Subsequently, one can focus on other and more sophisticated types of
texts and summaries. As summarization systems mature, one can design
tests for more demanding summary situations, for instance long summaries
of long and complex documents, evaluative summaries, and/or
multi-document summaries of larger source sets or of sets that are
written in several languages.

## 4.5 Specificity

The evaluation framework presented here is specific to the tasks and
problems that need to be solved in order to produce quality abstracts,
not just acceptable extracts: the identification of important
information in texts; and the generation of coherent abstracts that
subsume the important information. The evaluation framework does not put
any constraints on the types of technologies that one may choose to
employ in order to solve these problems. It will be up to each
individual researcher to decide whether they will use symbolic or
statistical techniques in order to solve these problems, whether they
will build intermediate, KR-specific representations in order to
represent the information in the important sentences or will generate
abstracts directly from the sentences identified as important by a given
system.

At the same time, the evaluation framework allows for the possibility of
restricting a subset of the corpus to a constrained domain, with the aim
of facilitating the investigation of using more knowledge-intensive
techniques to achieve higher quality. This move would support two paths
towards developing high quality, unconstrained summarization technology
that trade off quality and task complexity: on one path, unconstrained
input is achieved first and then efforts are pursued to improve quality;
on the other path, high quality in a constrained domain is sought first,
followed by efforts to improve portability. The logic behind pursuing
multiple research paths in this way has recently been recognized in the
speech synthesis community \[cf. Taylor, 2000\]: in this community, the
traditional path has been to achieve unconstrained input first and
improve quality second; the alternative path is to first seek reasonably
high quality in a constrained domain, then to improve flexibility. As
Taylor notes, this alternative path mirrors the one taken in the speech
recognition field, where recognition rates have remained roughly
constant over the years as steadily harder recognition tasks have been
tackled.

4.6 Exploration of summary space

In general, the quality and utility of a summary is affected by its
length, especially when this requires a high degree of source to summary
compression \[Jing et al., 1998\]. While utility depends on summary
purpose (for instance a specific summarization task may require one-line
summaries), for many tasks, different lengths or compression ratios are
in principle equally acceptable. One element of the program is to
investigate summary sensitivity, in quality and utility, under various
degrees of reduction and as dependent on input and purpose factors, with
appropriately calibrated measures. The exploration of summary space can
include evaluations that compare different summary types that serve the
same purpose, for example, the purpose of briefing.

5\. Limitations of the proposed framework

The main limitation of the proposed framework is that the evaluation
criteria, and consequently specific performance metrics, that we propose
to use are not entirely objective or fully automatic. But this is an
unavoidable problem with summarization once past the most primitive
intrinsic evaluation where it is sufficient to compare automatically
extracted sentences with human selections of source document sentences.
However even though assessing the topical content and linguistic
expression of a summary is a subjective enterprise that requires
significant human resources, using key sentence (sentence fragment)
markup and human-generated abstracts for source documents as a
supporting guide for more sophisticated forms of summary evaluation
should help to make the human effort less onerous and more consistent.

We also recognize that the method of summary evaluation by question
answering is more demanding than the others, because of the need to
formulate suitably bounded questions. However as the use of the same
method for educational purposes suggests, it should be viable for our
proposed application, and is certainly worth exploring, particularly
since it will be possible to take advantage of the work that has already
begun on question answering within the TREC program.

Another limitation of this framework is that it focuses primarily on
textual data, downplaying the importance of multimedia and interactive
approaches to summarization.

# 6. Road Map for Evaluating Progress in Summarization Research

This Road Map specifies a series of evaluations in annual cycles,
gradually increasing in challenge through a progression from

-   less to more demanding data;

-   intrinsic system-focused evaluation to extrinsic task-oriented
    evaluation;

-   increasing challenge in task specifications.

It thus includes a number of strands, usually more than one in each
year. These strands represent a mix of regular evaluations and design or
pilot studies, as shown in Figure 2, to ensure program focus and
consolidation of findings and experience. We expect that there would be
a limited choice of strands as compulsory evaluations for all
participants, and that other strands would represent secondary
activities in any one year. We attach particular importance to designing
evaluation resources, e.g. source document corpora as shown in Figure 1,
so that they are suited to use in more than one activity.

Overall, the program is intended both to advance the state of the art in
summarization itself and to move towards the combination of
summarization and question answering technologies, as sketched in the
Vision paper \[Carbonell et al., 2000\].

## Year 1

1.  **Creation of publicly available corpora for summarization
    research**. A few summarization corpora already exist, although not
    in the format described in Section 3.1.

-   

-   

-   Two corpora have been provided as part of the TIPSTER SUMMAC
    evaluation \[Mani et al., 1999\]. The first is a set of 183
    documents including abstracts from the Computation and Language
    (cmp-lg) papers which appeared in Association for Computational
    Linguistics (ACL) sponsored conferences. The markup includes tags
    covering core information such as title, author, date, etc., as well
    as basic structure such as abstract, body, sections, lists, etc. The
    corpus was prepared by the MITRE Corporation based on extensions to
    some fundamental work carried out by Simone Teufel and others at the
    University of Edinburgh. The second is a corpus of 120 texts (4
    topics x 30 relevant docs/topic) from the TREC collection, each of
    which has an associated human-created, topic-focused passage
    extraction summary constructed for the task. Both these corpora are
    available at
    <http://www.itl.nist.gov/iaui/894.02/related_projects/tipster_summac/index.html>.

-   \[Marcu, 1999\] describes a corpus of 7,000 Ziff-Davis *Texts* that
    are paired with human written *Abstracts* and automatically
    generated *Extracts*. The corpus can be used to train and test
    single document summarization systems with respect to their ability
    to identify important clauses/sentences in texts. TREC participants
    can obtain this corpus by sending email to marcu@isi.edu.

-   \[Radev, 2000\] describes a corpus of six clusters of documents,
    each containing between 2 and 10 documents. Important sentences in
    each cluster are identified. The corpus is available at
    <http://perun.si.umich.edu/~radev/share>.

More such corpora will have to be created through a combination of
automatic- and human-based techniques. In the first year, we estimate
that we need to create a corpus of 60 data sets, such as that shown in
Figure 1, each data set having between 5 and 10 documents and 4
multi-document abstracts, each produced at a different compression rate.
The documents in each data set should talk about the same topic. Thirty
data sets can be given to researchers for training; the other thirty can
be used for testing. The initial corpus should contain relatively simple
texts and should focus on event-specific multi-document summaries. To
facilitate more knowledge-intensive approaches to summarization, the
corpus should also include a subset restricted to a constrained domain.

Cover Notes will be provided for each corpus giving the data properties
for their source documents and reference summaries, using the framework
sketched earlier for input, purpose and output feature characterization,
and also indicating the methods used to assemble the corpus, with
background citations where appropriate.

2.  **Development of summarization specific evaluation software.**
    Develop programs that take as input the outputs produced by single-
    and multi-document summarization systems and that

-   automatically assign scores to them, if metrics such as those
    described by Radev et al. \[2000\] and Donaway et al. \[2000\] are
    applicable;

-   support human judges in assigning scores with respect to
    summarization qualities that cannot be assessed reliably through
    automatic means.

3.  **Conduct a** **nation-wide summary evaluation** **conference using
    the same model as TREC/MUC**. This evaluation will focus on

-   an intrinsic evaluation of sentence extraction and

-   an intrinsic evaluation of summary generation

> for both single document and multi-document cases, and for at least
> two degrees of reduction, one relative, i.e., 10% and 20%, and one
> absolute, i.e., 100 words and 200 words. These evaluations are based
> on the assumption that the corpus and its reference summaries are such
> as to allow such neutral or \`multi-purpose\' evaluation schemes.
>
> In order to facilitate both domain dependent and independent
> summarization technologies to be evaluated, participants will have the
> option to participate in only the constrained or unconstrained domain,
> or in both.
>
> We propose that the year 1 cycle begins in July 2000 with the creation
> of the corpus, with the corpus being made available to the community
> by November 2000, with submission of results by March 2000, and with
> the conference being held in June 2000, in conjunction with NAACL. We
> believe that as the evaluation tasks are relatively modest and there
> is an existing interested in the community, this is a viable
> timetable.

## Year 2

The Year 2 work will follow the same intrinsic model as Year 1 with the
aim of consolidating the basic summarization and evaluation technology
used.

-   Increase the difficulty of the source texts in the test collection
    and the size of the multi-document sets, by using a second corpus
    differing in input features from the one distributed in the first
    year.

-   Assess the adequacy of the evaluation metrics used in Year 1, modify
    where necessary. Conduct full-scale evaluation of summarization by
    question answering.

-   Explore a larger range of reduction alternatives.

-   

-   

-   

-   

-   Prepare the design for an evaluation of *evolving* summaries. An
    evolving summary presents a summary of only the new information in a
    set of documents. During the generation of an evolving summary, one
    assumes that the end user has already read earlier documents or a
    summary of them. Such a summary serves as an [update]{.underline} of
    the user's current knowledge. To implement this evaluation, a new
    corpus will need to be developed.

## Year 3

In this year we will introduce at least one overtly task-oriented
evaluation to begin the shift from the less realistic intrinsic
evaluations to more demanding extrinsic evaluations. This is expected
also to require a new corpus, perhaps derived from other evaluations,
and the development of new performance metrics. The current intention is
to conduct a TDT-style evaluation, to assess the utility of a
multi-document summarization system when used in conjunction with a TDT
system.

In this year, we will also begin a second task-oriented evaluation that
explicitly combines summarization and question answering, albeit in a
quite limited form. Thus we will aim to use single search questions (as
opposed to document retrieval topics) that would be designed to retrieve
a body of answer data from more than one document, for assembly into an
overall summary answer. For instance \`\`What happened when Popacatapetl
erupted?\'\' An evaluation specific to biographical summaries could be
also implemented during this year. Systems that participate in this
evaluation should be able not only to produce biographical summaries,
but also to answer questions such as "What new information do we have
about the whereabouts of Osama bin Laden"?

This year would in any case also cover a full-scale evaluation of
evolving summaries, as designed in Year 2.

In addition, on the assumption that as with previous evaluations like
TREC, the program attracts an increasing number of participants, we
expect that intrinsic evaluation will continue to be a subordinate
theme, as a way of proving core technology for varying data types.

## Year 4

We expect that the challenge of the type of task-oriented evaluations
introduced in Year 3 will be such as to call for continuation in Year 4,
for example by a further cycle of the TDT evaluation, along with a full
cycle of the first version of the question answering with

summarization task.

At the same time, to increase the complexity of summarization at least
at the level of intrinsic evaluation, we propose an evaluation of
multi-document summarization where the documents in each set to be
summarized are in different languages and of different genres. This
assumes the construction of a new corpus, which will be specific for
this task. To this end, it may be useful to coordinate with and leverage
activities carried out in the context of text summarization evaluation
in Japan (see <http://galaga.jaist.ac.jp:8000/tsc)>.

## Year 5

Run an integrated question-answering/summarization evaluation. Given a
set of specific and related questions, and the answers to these
identified in one or more source documents, combine the answers together
with any necessary connective background from the source document(s)
into a coherent answer summary. Thus, for example, if we are given a
bunch of explicitly or implicitly related questions such as

-   \`Which countries have embassies in Ruritania?\'

-   \`Has there been a coup in Ruritania?\'

-   \`When did the coup in Ruritania occur?\'

-   \`Are any ambassadors in Ruritania under house arrest?\'

-   \`What do we know about the coup leader?'

generating a situation summary incorporating the answers to these
questions and providing an explanatory context for the answers could
require the identification of other important and pertinent information
from the relevant source documents, so as to obtain a summary such as
that shown below.

\`*The coup which took place in Ruritania on June 1st proved
unsuccessful,* *so the ambassadors from the three countries with
embassies there,* *Frivolia, the Disunity Federation and Cacophonia have
all been* *freed from house arrest. The government of President
Wilbunn,* *against* *whom the coup was directed,* *has returned to
office, and life in Ruritania* *has returned to normal.\'*

This ambitious evaluation would be the focus of the year, along with
further work on partially completed evaluations from year 4.

The overall structure of the program is shown in Figure 2. Unfilled
boxes denote pilot study evaluations.

![](media/image3.png)

> Figure 2 : RoadMap Timeline.

# References

\[Banko M., Vibhu Mittal, Mark Kantrowitz, and Jade Goldstein, 1999\]:
Generating extraction summaries from handwritten summaries by aligning
text spans*. Proceedings of PACLING-99,* Waterloo, Ontario, July 1999.

\[Carbonell Jaime, Donna Harman, Eduard Hovy, Steve Maiorano, John
Prange, and Karen Sparck-Jones, 2000\] Vision Statement to Guide
Research in Question & Answering (Q&A) and Text Summarization, April 20,
2000.

\[Donaway Robert L., Kevin K. Drummey, and Laura A. Mather, 2000\] A
Comparison of Rankings Produced by Summarization Evaluation Measures.
*Proceedings of the ANLP/NAACL-2000 Workshop on Automatic
Summarization,* pages 69-78, Seattle, WA, May 4, 2000.

\[Hirschman Lynette and Inderjeet Mani, 2000\] Evaluation. R. Mitkov,
ed., *Handbook of Computational Linguistics*, Oxford University Press,
to appear.

\[Jing Hongyan, Regina Barzilay, Kathleen McKeown, and Michael Elhadad,
1998\] Summarization Evaluation Methods: Experiments and Analysis.
*Working Notes of the AAAI Spring Symposium on Intelligent Text
Summarization*, pages 60-68, Stanford University, March 23-25, 1998.

\[Jing Hongyan and Kathleen R. McKeown, 1999\] The Decomposition of
Human-Written Summary Sentences. *Proceedings of the 22^nd^
International Conference on Research and Development in Information
Retrieval (SIGIR'99)*, pages 137-144, University of California,
Berkeley, August 1999.

\[Hovy Eduard and Chin-Yew Lin, 2000\] Automated Multilingual Text
Summarization and Its Evaluation. *Forthcoming.*

\[Mani, I., Firmin, T., House, D., Klein, G., Sundheim, B., and
Hirschman, L., 1999\] The TIPSTER SUMMAC Text Summarization Evaluation.
*Proceedings of EACL\'99*, Bergen, Norway, June 1999.

\[Marcu Daniel, 1999\] The automatic construction of large-scale corpora
for summarization research. *Proceedings of the 22^nd^ International
Conference on Research and Development in Information Retrieval
(SIGIR'99)*, pages 137-144, University of California, Berkeley, August
1999.

\[Radev Dragomir R., Hongyan Jing, and Malgorzata Budzikowska, 2000\]
Centroid-based summarization of multiple documents: sentence extraction,
utility-based evaluation, and user studies. *Proceedings of the
ANLP/NAACL-2000 Workshop on Automatic Summarization,* pages 21-30,
Seattle, WA, May 4, 2000.

\[Karen Sparck-Jones, 1999\] Introduction to Text Summarization. In
Inderjeet Mani and Mark Maybury edts., *Advances in Automatic
Summarization*, pages 1-12, The MIT Press, 1999.

\[Taylor P. A., 2000\] Concept-to-speech synthesis by phonological
structure matching. In K.I.B. Sparck Jones, G.J.M. Gazdar and R.M.
Needham, edts., "Computers, language and speech: formal theories and
statistical data," *Philosophical Transactions of the Royal Society:
Mathematical, Physical and Engineering Sciences,* pages 1403-1417,
London, 2000.

\[White J. and T. O'Connell, 1994\] Evaluation in the ARPA
machine-translation program: 1993 methodology. *Proceedings of the ARPA
Human Language Technology Workshop,* pages 135-140, Washington, D.C. See
also <http://ursula.georgetown.edu/>.

See <http://www.si.umich.edu/~radev/summarization/large-bib.doc> for a
comprehensive list of summarization papers.

Appendix A

Evaluation Metrics for Summarization

***Content similarity \[Donaway et al., 2000\]***

Content-based measures can be used to evaluate for semantic content both
extracted and generated summaries. To apply these measures, a suitable
document processor must first perform stemming and create term frequency
vectors for the extracted portions (important sentences) of each single
document, each multiple-document abstract, and each machine-generated
summary. A thesaurus can be used to reduce the number of terms in the
vectors by combining frequencies of synonymous terms. Then to evaluate
the extracted/generated summaries, each machine-generated extract vector
is compared with the key using the cosine measure to produce a score
between 0 and 1. When evaluating single document summarization the score
is calculated by comparing a machine-generated extract to a given
document's important sentences. When evaluating multi-document
summarization the score is the average similarity of the
machine-generated extract with the collection of important sentences
from all of the documents in a data set. To evaluate the abstracts, the
machine-generated summary can be compared with the multiple-document
abstracts or with the union of important sentences in all documents.

The latent semantic indexing approach to summarization evaluation as
described in \[Donaway et al., 2000\] can be used to allow for greater
variation among summaries. In this case it is necessary to generate a
reduced dimension corpus transformation matrix *U~k~* and a reduced
dimension document matrix *~k~V~k~* by calculating the singular value
decomposition of the corpus term-by-document matrix, and setting all but
the largest *k* singular values to 0. At a minimum, this corpus must
include the full text of all documents and all multi-document abstracts
being used for testing. Term frequency vectors of machine-generated
summaries are multiplied by the transformation matrix *U~k~* before
computing the cosine similarity with the appropriate column of
*~k~V~k~*.

## 

## 

## 

## The utility method \[Radev et al., 2000\]

Using metrics such as precision and recall (P&R) or percent agreement
(PA) to evaluate summaries creates the possibility that two equally good
extracts are judged very differently. Suppose that the ideal summary
contains sentences 1 and 2 from a document \[1 2\]. Suppose also that
two systems, A and B produce summaries consisting of \[1 2\] and \[1
3\], respectively. Using P&R or PA, system A will be ranked much higher
than system B. It is quite possible, however that for the purpose of
summarization, sentences 2 and 3 are interchangeable, in which case the
two systems should get the same score. The utility method (UM) \[Radev
et al. 2000\] allows ideal summaries to consist of sentences with
variable-level (fuzzy) membership. In UM, the ideal summary represents
all sentences of the input document(s) with confidence values for their
inclusion in the summary. For example, a document with five sentences
1---5 could be represented as \[1/10 2/9 3/9 4/2 5/4\] which indicates
for each of the sentences the degree to which it should be part of the
summary. In this example, if a system chooses sentences 1 and 2, it will
not get a higher score than a system that chooses sentences 1 and 3 as
both summaries \[1 2\] and \[1 3\] carry the same number of points
(10+9). The UM methods can be further expanded to allow sentences or
paragraphs to exert negative reinforcement on one another, that is,
allow for cases in which the inclusion of a given sentence makes another
redundant and a system that includes both will be penalized more than a
system which only includes one of the two "equivalent" sentences and
another, perhaps less informative sentence.

One important advantage of UM over P&R and PA is that it allows
summaries to be evaluated at various compression rates. Recent studies
\[Jing et al. 98\], \[Goldstein et al. 99\] have shown that optimal
summaries are produced at variable compression rates.

Appendix B

Example of Summary Data Set

**Collection_ID:** 1

**Document_ID:** 1

**Source:** Fox

**Date:** June 3, 2000

**Time:**

CLINTON TAKES STAR WARS PLAN TO RUSSIA

[US president Bill Clinton has arrived in Moscow for his first meeting
with Russia\'s new president Vladmir Putin]{.underline}. The two heads
of state will meet on Saturday night for an informal dinner before
getting down to business on Sunday.

[High on the agenda will be the United State\'s plans to build a missile
shield in Alaska.]{.underline} [Russia opposes the shield]{.underline}
as it contravenes a pact signed by the two countries in 1972 which bans
any anti-missile devices.

Clinton \-- in his last few months of office and keen to make his mark
in American history - will be seeking to secure some sort of concession
from Putin.

The Russian leader has said that [he will suggest an alternative to the
US system]{.underline}.

Kremlin officials said [Putin would propose a system that would shoot
down the missiles with interceptors shortly after they were fired rather
than high in their trajectory]{.underline}.

\"We\'ll talk about it in Russia,\" Clinton told reporters before
leaving Berlin for Moscow. \"It won\'t be long now.\" Accompanying the
President is US Secretary of State Madeline Albright. \"What\'s new is
that Putin is signalling that he is open to discuss it, that he is ready
for talks,\" she said. \"We will discuss it.\"

Arms control will not be the only potentially troublesome issue. US
National Security Adviser Sandy Berger said last week Clinton would
raise human rights and press freedom.

**Collection_ID:** 1

**Document_ID:** 2

**Source:** BBC

**Date:** June 3, 2000

**Time:**

**Clinton set for Moscow summit**

[President Clinton is in Moscow for his first summit meeting with
Russian leader Vladimir Putin]{.underline}.

The potentially thorny question of a US anti-missile defence is likely
to be high on the agenda at Sunday\'s talks.

[The US wants to build an anti-missile system in Alaska to protect
against a potential threat from North Korea.]{.underline}

[But Mr Putin has already warned that if the US goes ahead, Russia might
abandon all arms-control accords with the US.]{.underline}

Soon after his arrival in Moscow, President Clinton joined Mr Putin at
the Kremlin for supper and a jazz concert.

**Nuclear security**

No agreement is expected in these Moscow talks, but discussions will
extend into the rest of Mr Clinton\'s term, with the two presidents
meeting again several times this year.

[Mr Clinton\'s overall message to Russia will be one of
inclusion]{.underline}.

In a speech in Aachen, Germany on Friday, he said that neither the
European Union nor Nato should seal their doors to Russia.

Some practical agreements are likely as both sides seek to ensure the
safety of Russia\'s nuclear materials.

Americans are advising on how to secure plutonium at Russian naval bases
against theft and loss.

It is hoped that a deal can be reached for each side to dispose of 34
tonnes of weapons-grade plutonium, enough for thousands of warheads.

**Resolving tensions**

[Mr Clinton will also be discussing Russia\'s 10-month war in rebel
Chechnya]{.underline}.

He expressed his concern on Friday about Russia\'s direction and dismay
over its military action against rebels in the breakaway republic, but
he praised Russia\'s \"remarkable journey\" since the break-up of the
Soviet Union.

Mr Clinton has been taking part in a conference in Berlin with about a
dozen centre-left world leaders on how to trace a middle way between
capitalism and socialism.

**Collection_ID:** 1

**Document_ID:** 3

**Source:** New York Times

**Date:** June 3, 2000

**Time:**

**Putin Offers Alternative Antimissile Plan**

By MICHAEL R. GORDON

MOSCOW, June 2 \-- In a bid to recast a security debate just before his
summit meeting with President Clinton, [President Vladimir V. Putin has
suggested an alternative to the proposed American defense against
missile attack by nations like North Korea]{.underline}.

Agreeing with the American assessment that so-called rogue states pose a
nuclear threat, [Mr. Putin hinted that the United States and Russia
could collaborate on new ways to shoot down enemy missiles soon after
they were launched, rather than in space]{.underline}. The Russian
proposal is intended to replace the plan for a nationwide shield to
protect the United States against incoming warheads that the Clinton
administration has proposed.

Mr. Putin\'s alternative approach seems to resemble the plan known in
the United States as \"boost phase defense,\" which has been proposed by
a number of arms control advocates. Russian officials have been talking
privately to American negotiators about the idea in recent weeks.

[Advocates of such an approach in the United States say it would provide
the United States, Europe and Russia with the protection that the
administration insists it needs from so-called rogue
states.]{.underline}

[Such a system would be of little use against the Russian nuclear
force]{.underline}. That would make it far more acceptable to the
Russian military, which fears that a solely American missile defense
would be used to gain a strategic edge.

Mr. Putin\'s remarks were made in an interview with \"The NBC Nightly
News.\" Parts of the interview were broadcast Thursday night, and other
parts will be shown over the weekend. A complete transcript was made
available today.

Mr. Clinton is to arrive here on Saturday and stay until Monday.

Administration officials reacted cautiously to Mr. Putin\'s comments,
which were very sketchy and at times hard to interpret.

They said his approach appeared to suggest an unacceptable substitute
for the administration\'s missile defense plan, which involves the
deployment of 100 interceptors and a battle-management radar in Alaska
to protect all 50 American states.

Mr. Clinton says he will make a final decision on whether to develop
such a defense based on a coming test, and new assessments of the threat
and of the effect of a missile system on arms control. But there is
tremendous political momentum behind the program, and Vice President Al
Gore, the presumed Democratic nominee to succeed Mr. Clinton, has
already endorsed such a limited defense.

The remarks by the Russian president were important as an opening move
in what promises to be a difficult negotiation. In the interview, Mr.
Putin acknowledged that the emerging missile threat \-- one that
Washington has associated with North Korea, Iran and other nations \--
was indeed a growing danger for the United States, Russia and Europe. In
the past, Russia dismissed such concerns as unwarranted.

[Rather than flatly rejecting the idea of new antimissile systems, as
many Moscow officials have been doing publicly in recent months, Mr.
Putin\'s remarks indicated that the Kremlin believes that some typeof
political pressures on those nations, as well as antimissile systems,
might be needed]{.underline}.

\"As to new threats that they talk about in the United States, we agree
with that,\" Mr. Putin said. \"We should create new mechanisms that will
protect us from these threats.\"

Without mentioning North Korea or Iran by name, Mr. Putin noted that
certain third world nations lacked long-range bombers or missile-firing
submarines to threaten the United States. But they are, he acknowledged,
developing long-range, land-based missiles.

\"If you are talking about threats that are directed or could be
directed at Russia or at the United States,\" he said, \"there are
countries that have that capability today that can only do that from
their own territory.\"

The Russian leader then went on to describe a possible way to deal with
those threats, apparently alluding to [the possibility of a boost-phase
defense that would prevent emerging third world powers from attacking
Russia and the West with missiles]{.underline}.

\"So we could jointly put up these umbrellas above potential areas of
threat,\" he said. \"With this umbrella, we could jointly protect all of
Europe.\"

[The reference to Europe also seemed to be a way of scoring points with
Western Europeans, who are very skeptical of the American missile
defense plan]{.underline}. The administration plan would protect only
American territory, and senior administration officials have made only
vague comments about Washington\'s willingness to share antimissile
technology with the European allies.

China might not welcome collaboration between Russia and the United
States on missile defense. But the limited approach proposed by Mr.
Putin might be seen by Beijing as far less threatening that the more
extensive American system, which would neutralize the Chinese deterrent.

Mr. Putin\'s remarks developed some themes discussed with American
officials by Foreign Minister Igor S. Ivanov and other Russian
officials, some of whom hinted at the boost-phase approach. Russian
politicians said Mr. Putin\'s comments should be seen more as inviting a
dialogue with Washington on missile defense, albeit on Moscow\'s terms,
than as a detailed proposal.

\"It is an important political gesture toward the United States,\"
Andrei Kokoshin, a senior lawmaker who was national security adviser
when Boris N. Yeltsin was president, said in an interview.

Mr. Kokoshin said the aim of Mr. Putin\'s plan was to explore ways to
meet some of Washington\'s concerns by adapting \"theater missile
defense\" systems. These are land-based interceptors that were
originally intended to counter attacks by short- and medium-range
missiles and which are allowed by the 1972 Antiballistic Missile Treaty.

[The Kremlin\'s goal, he acknowledged, was to persuade the
administration to abandon its more ambitious program, which would
require amending the ABM treaty.]{.underline}

\"Theater missile defense is capable of boost-phase interception,\" Mr.
Kokoshin said in an interview. \"It would be a more perfect solution in
terms of overall strategic stability than the unilateral American
decision to build a national missile defense.\"

Mr. Putin\'s suggestions still leave Washington and Moscow with deep
differences. Administration officials have said they do not expect a
breakthrough at the summit meeting here, but rather are hoping to
establish a dialogue that could lead to an accord on an antimissile
system and nuclear weapons reductions before the end of the Clinton
presidency.

[The administration does not agree that the theater systems Mr. Putin is
referring to would be effective against a third world missile
threat]{.underline}. And the two sides are far apart on the related
issue of slashing strategic arms, with the Russians insisting on deeper
cuts. Still, American officials sought to accent the positive.

\"There are nuggets we can work with in a positive way,\" a senior
administration official said. \"They are recognizing the ballistic
missile threat and talking about cooperating to meet that threat. But we
do not see this as a substitute for our national missile defense
program.\"

Other American experts outside the administration were more positive.
They said Mr. Putin\'s plan might lead to a better system from the point
of view of American as well as Russian interests.

[Richard Garwin, a leading physicist who served on a Congressional panel
on the emerging missile defense, argued that a boost-phase system would
be far more effective than the administration\'s proposal because it
would intercept attacking missiles before they reached their full speed
and dispersed their warheads and decoys.]{.underline}

Mr. Garwin has proposed deploying such a defense on ships near the
Korean peninsula and perhaps on Russian territory. He and Ted Postol, a
professor at the Massachusetts Institute of Technology who also supports
a boost-phase approach, have discussed the plan with Russian officials
and experts and promoted it as an alternative to the administration\'s
approach.

\"It is a lot easier to put a lid over North Korea than an umbrella over
the United States and Eastern Pacific,\" Mr. Garwin said.

The plan promoted by Mr. Garwin, however, would require amending the ABM
treaty.

In his NBC interview, Mr. Putin spoke of the need to preserve the ABM
accord, which sets strict limits on the testing and deployment of
antimissile systems, saying it was a condition for arms cuts.

**Collection_ID:** 1

**Document_ID:** 4

**Source:** CNN

**Date:** June 3, 2000

**Time:** 12:59 p.m. EDT (1659 GMT)

**Clinton meets with Putin at the Kremlin**

**Putin and Clinton meet on Saturday in Moscow**

MOSCOW (CNN) \-- [For the first time since his election in May, Russian
President Vladimir Putin conferred with U.S. President Bill Clinton on
Saturday at the Kremlin, as the two began a summit expected to focus on
Moscow\'s objection to a proposed U.S. missile defense
system.]{.underline}

Before arriving in Moscow, Clinton told reporters he planned to discuss
the National Missile Defense project (NMD), which would be based in
Alaska and designed to shoot down incoming missiles from what the United
States terms \"rogue states,\" such as Iran, Iraq and North Korea.

[White House officials said that Russia is restating an alternative
proposal to combine U.S. and Russian technologies to build a shield that
would target missiles immediately after takeoff \-- rather than high in
their trajectory, as the United States has proposed]{.underline}. The

Russian model would be \"theater-based,\" or near the hot spots that
threaten global security.

U.S. officials said such a \"theater-based\" system would likely have to
be based in Russia, and would raise too many command and control issues
for the United States to count on it.

Kremlin officials have been making it very clear to the United States
that they don\'t want any changes in the 1972 Anti-Ballistic Missile
Treaty between the two nations, which bans deployment of the kind of
missile shield the United States has proposed. Clinton is expected to
suggest amendments to that treaty.

Behind the scenes, U.S. and Russian negotiators have failed to make
major pre-summit progress on arms-control issues, and no headway in
overcoming Moscow\'s opposition to the NMD shield, White House officials
familiar with the discussions have told CNN.

**White House: \'Nothing new\'**

\"There is nothing new in what they are saying,\" a White House official
told CNN as Clinton flew from Berlin to Moscow. \"We\'ll listen closely,
but it appears they are just throwing things out there to confuse the
issue and to knock us off NMD.\"

Still, Secretary of State Madeleine Albright told German media on
Saturday, \"What is important is that President Putin is signaling that
he is open to discussions.\"

[She also said it was \"very interesting\" that the Russian leader seems
to agree that there is some kind of threat from North Korea and that
\"we should look at it together.\"]{.underline}

[This represents a departure from Russia\'s previous statements that it
did not believe North Korea, Iran, Iraq or other nations represented a
threat now or in the near future.]{.underline}

U.S. officials predicted a few modest agreements would be reached during
the summit, including a U.S. commitment to purchase 30 to 36 tons of
Russian plutonium, enough to build thousands of nuclear warheads.

That purchase is part of a continuing U.S. effort to keep Russian
nuclear materials and technology out of the hands of countries
Washington has labeled \"rogue.\"

Clinton and Putin were scheduled to attend a working dinner Saturday
night in Moscow, before beginning the summit on Sunday. On Monday,
Clinton is scheduled to address Russia\'s parliament.

Clinton is expected to reassure Putin that Russia would not be a target
of the proposed anti-missile shield.

**Clinton assures European security**

Since beginning his European tour on Monday[, Clinton has sought to
allay European concerns that the proposed U.S. missile system could
upset the global strategic balance, and leave Europe
unprotected.]{.underline} Clinton has said the United States would be
willing to share its technology with other countries that participated
in international arms control agreements.

During Clinton\'s stop in Berlin, German Chancellor Gerhard Schroeder
warned against upsetting the arms balance and the cohesion of the North
Atlantic Treaty Organization (NATO) with the proposed anti-missile
system.

Schroeder said in a Friday speech preceding a keynote address by Clinton
that the U.S. plans should be debated with its European partners \"as
this issue could have effects well beyond the U.S.A.\"

**European summit**

[Clinton,]{.underline} who spoke Saturday at a summit with 13
politically like-minded European leaders, [appealed to Europe to embrace
Moscow as the former communist capital continues to embrace
democracy.]{.underline}

\"That means no doors can be sealed shut to Russia,\" he said,
specifically mentioning NATO, the Western military alliance, and the
15-nation European Union, which focuses on trade and security issues.
Full normalization of relations with Moscow will bring about \"the end
of our vision of an undivided continent,\" Clinton said.

[But he also expressed concern about alleged human rights violations and
indiscriminate use of force by Russian troops in their eight-month
conflict against rebels fighting for independence in the Russian rebel
republic of Chechnya]{.underline}.

Summit leaders also spoke in broader terms, calling on nations to
combine with market economy forces to create full employment and social
justice the world over.

\"There are things we can do to help each other and help our people,\"
Clinton said.

Clinton said the leaders agreed to set up groups to identify the
problems and move toward a practical way of dealing with them.

Senior White House Correspondent John King, Correspondent Matthew
Chance, The Associated Press and Reuters contributed to this report.

**Collection_ID:** 1

**Document_ID:** 5

**Source:** CBS

**Date:** June 3, 2000

**Time:**

**President Clinton In Moscow**

(CBS) [U.S. President Bill Clinton arrived at the Kremlin on Saturday
for his first meeting with Vladimir Putin since he was elected Russian
president.]{.underline}

Putin was hosting an informal dinner in the Russian leader\'s private
apartment in the Kremlin.

Reporters saw Clinton\'s motorcade drive into the Kremlin, a red-walled
fortress in central Moscow, about two hours after he arrived in Russia
on the penultimate leg of a week-long tour of Europe.

Upon arrival into Moscow, Russian Foreign Minister Igor Ivanov was on
hand to meet Clinton.

The leaders will hold their first formal summit session on Sunday.

But before arriving into Moscow, Mr. Clinton was guarded on Putin\'s
proposed alternative to a U.S. missile defense system.

[The talks are expected to be dominated by U.S. plans to amend a 1972
treaty banning anti-missile defenses in order to allow deployment of a
missile defense shield based in Alaska]{.underline}.

But arms control will not be the only potentially troublesome issue.
[U.S. National Security Adviser Sandy Berger said last week Mr. Clinton
would raise human rights and press freedom.]{.underline}

German Chancellor Gerhard Schroeder warned the president on Friday
against upsetting the arms balance and the cohesion of the Atlantic
alliance with the new anti-missile system.

[Before leaving Moscow on Monday, Mr. Clinton will address the Russian
parliament.]{.underline}

He is scheduled to make a stopover in Kiev before returning to
Washington after a weeklong European tour that also took him to Portugal
and Germany.

[On Friday, President Clinton urged America\'s European allies to ensure
that Russia becomes \"fully part of Europe.\"]{.underline}

\"The job of building a united Europe is certainly not finished and it
is important not to take all of this self-congratulation too far,\" Mr.
Clinton said, as he became the third American ever to receive the
prestigious International Charlemagne Prize for promoting European
unity.

Mr. Clinton spoke in a sun-filled courtyard of the magnificent cathedral
that holds the remains of the eighth-century Holy Roman emperor for whom
the prize is named. Nearby, several thousand Germans watched the
ceremony on an open-air screen set up on the market square in front of
the 14th-century town hall.

Among those in the audience were 10 former winners, including Vaclav
Havel, the dissident who presided over Czechoslovakia\'s return to
democracy. In his remarks, Mr. Clinton noted that Havel has now spent
more time as Czech president than he did as a prisoner.

CBS Worldwide Inc. All Rights Reserved. This material may not be
published, broadcast, rewritten, or redistributed. Reuters Limited
contributed to this report

**Collection_ID:** 1

**Document_ID:** 6

**Source:** Reuters

**Date:** June 3, 2000

**Time:**

**Clinton Declines Comment On New Russia Shield Plan**

WASHINGTON, Jun 3, 2000 \-- (Reuters) U.S. President Bill Clinton would
not comment on Saturday about a new Russian proposal for a system of
shooting down missiles, but [said he and Russian President Vladimir
Putin would talk about it in Moscow.]{.underline}

\"We\'ll talk about it it Russia,\" Clinton told reporters as he entered
into his last meeting in Germany before heading to Russia in the
afternoon. \"It won\'t be long now.\"

Putin, who was inaugurated barely a month ago, has indicated that Russia
would offer a way out of arms control deadlock by proposing an
alternative to a U.S. plan for a Star Wars-style anti-missile defence
shield.

U.S. officials have said the plan was not new and failed to meet
Washington\'s concerns.

In Moscow, [Kremlin officials have said Putin would propose a system of
shooting down the missiles with interceptors, which could be based on
Russian soil, shortly after they are fired rather than high in their
trajectory.]{.underline}

[The proposal]{.underline}, which Putin said he would discuss with
Clinton, [could offer a solution on the sticky subject of the 1972
Anti-Ballistic Missile (ABM) treaty,]{.underline} which Russia regards
as the cornerstone of three decades of arms control negotiations.

[The United States wants to amend the ABM treaty to set up a national
missile defence system.]{.underline}

Under the treaty, the United States and the Soviet Union agreed to limit
defenses so that they could cut their stockpiles of missiles without
worrying about making their deterrent capability ineffective.

Washington says it now needs a missile defence shield to protect it from
what it calls \"rogue states\", such as Iran and North Korea, that may
soon develop long range missiles.

(C)2000 Copyright Reuters Limited. All rights reserved. Republication or
redissemination of the contents of this screen are expressly prohibited
without the prior written consent of Reuters Limited.

**Collection_ID:** 1

**Document_ID:** 7

**Source:** Associated Press

**Date:** June 3, 2000

**Time:**

**Putin May Be Flexible on Missiles**

MOSCOW (AP) - [In their first summit, President Clinton is sounding out
Vladimir Putin on the new Russian president\'s signal that he may be
relaxing his stern opposition to American plans to build a defense
against missile attacks.]{.underline}

[The White House wants to alter the 1972 Anti-Ballistic Missile Treaty
to make room for an American missile defense system,]{.underline}
although Clinton will not decide until fall whether to try to construct
one now being tested.

Clinton, on a weeklong European tour, flew here from a skull session in
Berlin with the leaders of more than a dozen nations trying to forge a
so-called Third Way of governance between liberalism and conservatism.

The leaders talked about economic empowerment, education, closing the
digital divide and \"the importance of reducing income inequality as a
result of globalization,\" Clinton said. He said they also agreed that
there should be a major effort by developed nations to response to
disease in poorer countries, particularly HIV and AIDS, tuberculosis and
malaria.

Here, Clinton and Putin grapple with the threat of nuclear conflict.

Going in, Putin said in an NBC-TV interview that the nearly 30-year-old
ABM treaty must be preserved, calling that a condition for further
nuclear arms reductions.

But in the same interview[, he said he sees the potential missile threat
from states like North Korea. Putin suggested that a different system,
within the ABM treaty, could intercept missiles near liftoff and protect
Europe and Russia as well as the United States.]{.underline}

There are new threats arising that we must address but our proposal is
to answer them together,\" the Russian president said. \"And these
proposals in our view have prospects. \... I think we, on that point,
will have a substantive and useful dialogue with President Clinton.\"

The proposed U.S. system would begin with 100 anti-missile missiles in
Alaska, to knock out attackers in space. [Such a system is barred by the
ABM treaty]{.underline}. It was drafted in the era of East-West mutual
deterrence to assure both sides that neither would lose its power to
retaliate if hit.

[\"What is important is that President Putin is signaling that he is
open to discussions,\" Secretary of State Madeleine Albright
said]{.underline} in Berlin on Saturday.

She told German journalists it was interesting that Putin seemed to see
some risk from North Korea and that he suggested \"we should look at it
together.\"

But ultimately, she said, Clinton will make his decision \"based on
American national interest.\"

Clinton himself said only that \"we\'ll talk about this in Russia.\"

Clinton\'s opening session with Putin was billed as a \"working
dinner.\" It was attended only by the two presidents and two aides
Strobe Talbott, Clinton\'s envoy to Russia, and Sergei Prikhodko, deputy
chief of Putin\'s staff.

Separately, aides to both dined, led by Albright and her Russian
counterpart, Igor Ivanov.

On Sunday, the two meet for more than six hours and deliver statements
on their conclusions. The president also fields questions from Russians
in a call-in radio show in Moscow.

[Also on Clinton\'s agenda was the conflict in the rebellious republic
of Chechnya, where fighting has flare anew]{.underline}. Rebel fighters
have attacked Russian troops across Chechnya and Russian warplanes
bombarded the southern mountains in the latest flareup. The rebels

killed a Russian special forces policeman in the capital Grozny early
Saturday, the Interior Ministry said. The policeman was patrolling near
the city\'s sole surviving Russian Orthodox church. On Monday, Clinton
will become the first American president to address the Duma, the lower
house of the Russian parliament, before he bids Putin

goodbye, pays a courtesy call on ailing former President Boris Yeltsin
and leaves for Kiev, Ukraine, and home. Administration officials said in
advance that they did not expect major accords to emerge at the Moscow
summit, given the U.S.-Russian differences over amending the ABM treaty.
However, they expect an agreement to destroy 34 tons of stockpiled
military-grade plutonium on each side - enough for \"tens of

thousands of nuclear weapons,\" says National Security Adviser Sandy
Berger.

The missile-shield concept, first floated by President Reagan in the
early 1980s, has strong political appeal in the Republican-controlled
U.S. Congress.

Republican candidate George W. Bush, supportive of an even more
ambitious plan, has urged Clinton not to make any arms agreements with
Putin that might bind the next president.

Clinton brushed off such admonitions[. European allies are deeply
skeptical of any missile defense system,]{.underline} even though
Clinton has offered to share some of the technology involved with them
and all other \"civilized\" nations.

As he moved through Europe, Clinton sounded both optimistic and subdued
about prospects of improved relations with Russia. He also voiced
concern about Russia\'s forceful crackdown on rebels in Chechnya.

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

**Collection_ID:** 1

**Document_ID:** 8

**Source:** Reuters

**Date:** June 3, 2000

**Time:** 2:24 PM ET

**Clinton, Putin Dine at Start of First Summit**

By Deborah Charles

MOSCOW (Reuters) - [President Clinton met President Vladimir Putin for
supper and jazz on Saturday at the start of a summit likely to be
dominated by the Russian leader\'s proposed alternative to a U.S.
missile defense system]{.underline}.

Putin met Clinton in a vaulted Kremlin reception room before they headed
off for an informal dinner with just two other guests \-- U.S. Deputy
Secretary of State Strobe Talbott and Sergei Prikhodko, deputy head of
the Kremlin administration.

After dinner, veteran Russian jazzman Oleg Lundstrem and saxophonist
Oleg Butman were to play, Russian news agencies said. Lundstrem told
Itar-Tass news agency he had a spare sax in case Clinton, a jazz fan and
player, wanted to join in.

[It was Clinton\'s fifth visit to Moscow but his first meeting with
Putin since the former KGB spy took office]{.underline}. [On Sunday, the
two hold summit talks covering arms control, the Russian economy,
Chechnya and regional hotspots such as the Balkans]{.underline}.

Clinton\'s motorcade drove into the Kremlin about two hours after he
arrived in Russia in warm evening sunshine on the penultimate leg of a
week-long tour of Europe that has so far taken in Portugal and Germany.
He will also visit Ukraine.

The diminutive Putin strolled across a carpet with his trademark swagger
to greet the towering Clinton with a firm handshake and broad smile.
They briefly paused for photographers before Putin ushered the U.S.
president upstairs for a dinner of chicken consomm, trout and goose with
berry sauce.

COULD BE SERIOUS TALK

Although the supper was informal, the White House did not rule out
serious talks at the long table decked with candelabra. Apart from the
four diners only two interpreters were present.

\"I can\'t imagine that there wouldn\'t be some substantive discussions
but I don\'t know,\" White House spokesman Joe Lockhart told reporters
in the Kremlin.

Putin has met Clinton twice before, but not since he took over from
Boris Yeltsin, with whom the U.S. president had a warm, backslapping
relationship developed over more than a dozen summit meetings. Yeltsin
quit last New Year\'s Eve. Putin was elected in March and inaugurated
last month.

[Arms control is expected to dominate the agenda on Sunday]{.underline}.
They may agree a deal on an early warning center and on destroying some
weapons-grade plutonium from stockpiles. It seems less likely they will
agree on U.S. plans and a Russian alternative proposal for an
anti-missile defense shield.

[The summit talks are expected to be dominated by U.S. plans to amend a
1972 treaty banning anti-missile defenses to allow the deployment of a
missile defense shield based in Alaska.]{.underline}

Putin added spice to the summit run-up by saying in an NBC television
interview on Thursday Moscow would offer an alternative to the U.S.
system, which Russia adamantly opposes.

\"We\'ll talk about it in Russia,\" Clinton told reporters before
leaving Berlin for Moscow. \"It won\'t be long now.\"

CLINTON TO ADDRESS PARLIAMENT

[Clinton leaves Moscow on Monday after addressing parliament, the first
U.S. president to do so]{.underline}. He may visit a new Russian
Orthodox cathedral on Sunday morning, RIA news agency said.

He is scheduled to make a stopover in Kiev before returning to
Washington after a week-long tour during which Clinton heard European
concerns about the U.S. anti-missile plan.

[Kremlin officials said Putin would propose a system that would shoot
down the missiles with interceptors shortly after they were fired rather
than high in their trajectory.]{.underline}

U.S. officials have been cool to the idea, but have said they are
willing to listen to the Russian proposals.

[\"What\'s new is that Putin is signaling that he is open to discuss it,
that he is ready for talks,\" Secretary of State Madeleine Albright told
Germany\'s Phoenix TV.]{.underline}

[The United States says it needs a shield to protect it from what it
calls \"rogue states,\" such as Iran and North Korea]{.underline}, that
may soon develop long-range missiles.

Clinton sought to allay European concerns that the U.S. system could
upset the global strategic balance, and leave Europe unprotected. He
said on Wednesday the United States would be willing to share its
technology with other countries that participated in international arms
control agreements.

But the White House said it was premature to say whether such technology
would be shared with the Russians.

OTHER POTENTIALLY TROUBLESOME ISSUES

Arms control will not be the only potentially troublesome issue. [U.S.
National Security Adviser Sandy Berger said last week Clinton would
raise human rights and press freedom.]{.underline}

Putin, the initiator of Moscow\'s offensive in Chechnya, has drawn a
chorus of criticism from the West for alleged human rights abuses and
indiscriminate use of force by Russian troops during the campaign, which
has lasted more than eight months.

Putin may also come under fire over an armed tax raid on the
headquarters of a key media group, Media-MOST, which is often critical
of the Kremlin. Clinton is scheduled to be interviewed by Ekho Moskvy
radio station, owned by Media-MOST, on Sunday.

# Important sentences in all articles in collection 1

US president Bill Clinton has arrived in Moscow for his first meeting
with Russia\'s new president Vladmir Putin.

High on the agenda will be the United State\'s plans to build a missile
shield in Alaska. Russia opposes the shield

he will suggest an alternative to the US system.

Putin would propose a system that would shoot down the missiles with
interceptors shortly after they were fired rather than high in their
trajectory.

President Clinton is in Moscow for his first summit meeting with Russian
leader Vladimir Putin.

The US wants to build an anti-missile system in Alaska to protect
against a potential threat from North Korea.

But Mr Putin has already warned that if the US goes ahead, Russia might
abandon all arms-control accords with the US.

Mr Clinton\'s overall message to Russia will be one of inclusion.

Mr Clinton will also be discussing Russia\'s 10-month war in rebel
Chechnya.

President Vladimir V. Putin has suggested an alternative to the proposed
American defense against missile attack by nations like North Korea.

Mr. Putin hinted that the United States and Russia could collaborate on
new ways to shoot down enemy missiles soon after they were launched,
rather than in space. Advocates of such an approach in the United States
say it would provide the United States, Europe and Russia with the
protection that the administration insists it needs from so-called rogue
states.

Such a system would be of little use against the Russian nuclear force.

Rather than flatly rejecting the idea of new antimissile systems, as
many Moscow officials have been doing publicly in recent months, Mr.
Putin\'s remarks indicated that the Kremlin believes that some typeof
political pressures on those nations, as well as antimissile systems,
might be needed.

the possibility of a boost-phase defense that would prevent emerging
third world powers from attacking Russia and the West with missiles.

The reference to Europe also seemed to be a way of scoring points with
Western Europeans, who are very skeptical of the American missile
defense plan.

The Kremlin\'s goal, he acknowledged, was to persuade the administration
to abandon its more ambitious program, which would require amending the
ABM treaty.

The administration does not agree that the theater systems Mr. Putin is
referring to would be effective against a third world missile threat.

Richard Garwin, a leading physicist who served on a Congressional panel
on the emerging missile defense, argued that a boost-phase system would
be far more effective than the administration\'s proposal because it
would intercept attacking missiles before they reached their full speed
and dispersed their warheads and decoys.

For the first time since his election in May, Russian President Vladimir
Putin conferred with U.S. President Bill Clinton on Saturday at the
Kremlin, as the two began a summit expected to focus on Moscow\'s
objection to a proposed U.S. missile defense system.

White House officials said that Russia is restating an alternative
proposal to combine U.S. and Russian technologies to build a shield that
would target missiles immediately after takeoff \-- rather than high in
their trajectory, as the United States has proposed.

She also said it was \"very interesting\" that the Russian leader seems
to agree that there is some kind of threat from North Korea and that
\"we should look at it together.\"

This represents a departure from Russia\'s previous statements that it
did not believe North Korea, Iran, Iraq or other nations represented a
threat now or in the near future.

Clinton has sought to allay European concerns that the proposed U.S.
missile system could upset the global strategic balance, and leave
Europe unprotected.

Clinton appealed to Europe to embrace Moscow as the former communist
capital continues to embrace democracy.

But he also expressed concern about alleged human rights violations and
indiscriminate use of force by Russian troops in their eight-month
conflict against rebels fighting for independence in the Russian rebel
republic of Chechnya.

U.S. President Bill Clinton arrived at the Kremlin on Saturday for his
first meeting with Vladimir Putin since he was elected Russian
president.

The talks are expected to be dominated by U.S. plans to amend a 1972
treaty banning anti-missile defenses in order to allow deployment of a
missile defense shield based in Alaska.

U.S. National Security Adviser Sandy Berger said last week Mr. Clinton
would raise human rights and press freedom.

Before leaving Moscow on Monday, Mr. Clinton will address the Russian
parliament.

On Friday, President Clinton urged America\'s European allies to ensure
that Russia becomes \"fully part of Europe.\"

said he and Russian President Vladimir Putin would talk about it in
Moscow.

Kremlin officials have said Putin would propose a system of shooting
down the missiles with interceptors, which could be based on Russian
soil, shortly after they are fired rather than high in their trajectory.

The proposal could offer a solution on the sticky subject of the 1972
Anti-Ballistic Missile (ABM) treaty

The United States wants to amend the ABM treaty to set up a national
missile defence system.

In their first summit, President Clinton is sounding out Vladimir Putin
on the new Russian president\'s signal that he may be relaxing his stern
opposition to American plans to build a defense against missile attacks.

The White House wants to alter the 1972 Anti-Ballistic Missile Treaty to
make room for an American missile defense system

he said he sees the potential missile threat from states like North
Korea. Putin suggested that a different system, within the ABM treaty,
could intercept missiles near liftoff and protect Europe and Russia as
well as the United States.

Such a system is barred by the ABM treaty.

\"What is important is that President Putin is signaling that he is open
to discussions,\" Secretary of State Madeleine Albright said

Also on Clinton\'s agenda was the conflict in the rebellious republic of
Chechnya, where fighting has flare anew.

European allies are deeply skeptical of any missile defense system,

President Clinton met President Vladimir Putin for supper and jazz on
Saturday at the start of a summit likely to be dominated by the Russian
leader\'s proposed alternative to a U.S. missile defense system.

It was Clinton\'s fifth visit to Moscow but his first meeting with Putin
since the former KGB spy took office. On Sunday, the two hold summit
talks covering arms control, the Russian economy, Chechnya and regional
hotspots such as the Balkans.

Arms control is expected to dominate the agenda on Sunday.

The summit talks are expected to be dominated by U.S. plans to amend a
1972 treaty banning anti-missile defenses to allow the deployment of a
missile defense shield based in Alaska.

Clinton leaves Moscow on Monday after addressing parliament, the first
U.S. president to do so.

Kremlin officials said Putin would propose a system that would shoot
down the missiles with interceptors shortly after they were fired rather
than high in their trajectory.

\"What\'s new is that Putin is signaling that he is open to discuss it,
that he is ready for talks,\" Secretary of State Madeleine Albright told
Germany\'s Phoenix TV.

The United States says it needs a shield to protect it from what it
calls \"rogue states,\" such as Iran and North Korea,

U.S. National Security Adviser Sandy Berger said last week Clinton would
raise human rights and press freedom.

# Important sentences grouped by topics

## First meeting Clinton - Putin

US president Bill Clinton has arrived in Moscow for his first meeting
with Russia\'s new president Vladmir Putin.

President Clinton is in Moscow for his first summit meeting with Russian
leader Vladimir Putin.

For the first time since his election in May, Russian President Vladimir
Putin conferred with U.S. President Bill Clinton on Saturday at the
Kremlin, as the two began a summit expected to focus on Moscow\'s
objection to a proposed U.S. missile defense system.

U.S. President Bill Clinton arrived at the Kremlin on Saturday for his
first meeting with Vladimir Putin since he was elected Russian
president.

President Clinton met President Vladimir Putin for supper and jazz on
Saturday at the start of a summit likely to be dominated by the Russian
leader\'s proposed alternative to a U.S. missile defense system.

It was Clinton\'s fifth visit to Moscow but his first meeting with Putin
since the former KGB spy took office.

On Sunday, the two hold summit talks covering arms control, the Russian
economy, Chechnya and regional hotspots such as the Balkans.

## First item on the agenda

High on the agenda will be the United State\'s plans to build a missile
shield in Alaska.

The talks are expected to be dominated by U.S. plans to amend a 1972
treaty banning anti-missile defenses in order to allow deployment of a
missile defense shield based in Alaska.

said he and Russian President Vladimir Putin would talk about it in
Moscow.

Arms control is expected to dominate the agenda on Sunday.

The summit talks are expected to be dominated by U.S. plans to amend a
1972 treaty banning anti-missile defenses to allow the deployment of a
missile defense shield based in Alaska.

##  Russia's position

Russia opposes the shield

he will suggest an alternative to the US system.

Putin would propose a system that would shoot down the missiles with
interceptors shortly after they were fired rather than high in their
trajectory.

But Mr Putin has already warned that if the US goes ahead, Russia might
abandon all arms-control accords with the US.

President Vladimir V. Putin has suggested an alternative to the proposed
American defense against missile attack by nations like North Korea.

Mr. Putin hinted that the United States and Russia could collaborate on
new ways to shoot down enemy missiles soon after they were launched,
rather than in space.

Advocates of such an approach in the United States say it would provide
the United States, Europe and Russia with the protection that the
administration insists it needs from so-called rogue states.

the possibility of a boost-phase defense that would prevent emerging
third world powers from attacking Russia and the West with missiles.

The reference to Europe also seemed to be a way of scoring points with
Western Europeans, who are very skeptical of the American missile
defense plan.

The Kremlin\'s goal, he acknowledged, was to persuade the administration
to abandon its more ambitious program, which would require amending the
ABM treaty.

Richard Garwin, a leading physicist who served on a Congressional panel
on the emerging missile defense, argued that a boost-phase system would
be far more effective than the administration\'s proposal because it
would intercept attacking missiles before they reached their full speed
and dispersed their warheads and decoys.

White House officials said that Russia is restating an alternative
proposal to combine U.S. and Russian technologies to build a shield that
would target missiles immediately after takeoff \-- rather than high in
their trajectory, as the United States has proposed.

Kremlin officials have said Putin would propose a system of shooting
down the missiles with interceptors, which could be based on Russian
soil, shortly after they are fired rather than high in their trajectory.

The proposal could offer a solution on the sticky subject of the 1972
Anti-Ballistic Missile (ABM) treaty

Putin suggested that a different system, within the ABM treaty, could
intercept missiles near liftoff and protect Europe and Russia as well as
the United States.

Kremlin officials said Putin would propose a system that would shoot
down the missiles with interceptors shortly after they were fired rather
than high in their trajectory.

##  US's position

The US wants to build an anti-missile system in Alaska to protect
against a potential threat from North Korea.

Such a system would be of little use against the Russian nuclear force.

The administration does not agree that the theater systems Mr. Putin is
referring to would be effective against a third world missile threat.

Clinton has sought to allay European concerns that the proposed U.S.
missile system could upset the global strategic balance, and leave
Europe unprotected.

The United States wants to amend the ABM treaty to set up a national
missile defence system.

The White House wants to alter the 1972 Anti-Ballistic Missile Treaty to
make room for an American missile defense system

Such a system is barred by the ABM treaty.

European allies are deeply skeptical of any missile defense system,

The United States says it needs a shield to protect it from what it
calls \"rogue states,\" such as Iran and North Korea,

## Russia's shift in perceiving rogue states as a potential threat

Rather than flatly rejecting the idea of new antimissile systems, as
many Moscow officials have been doing publicly in recent months, Mr.
Putin\'s remarks indicated that the Kremlin believes that some typeof
political pressures on those nations, as well as antimissile systems,
might be needed.

She also said it was \"very interesting\" that the Russian leader seems
to agree that there is some kind of threat from North Korea and that
\"we should look at it together.\"

This represents a departure from Russia\'s previous statements that it
did not believe North Korea, Iran, Iraq or other nations represented a
threat now or in the near future.

In their first summit, President Clinton is sounding out Vladimir Putin
on the new Russian president\'s signal that he may be relaxing his stern
opposition to American plans to build a defense against missile attacks.

he said he sees the potential missile threat from states like North
Korea.

\"What is important is that President Putin is signaling that he is open
to discussions,\" Secretary of State Madeleine Albright said

\"What\'s new is that Putin is signaling that he is open to discuss it,
that he is ready for talks,\" Secretary of State Madeleine Albright told
Germany\'s Phoenix TV.

## Other items on the agenda

##  Russia's inclusion in Europe

Mr Clinton\'s overall message to Russia will be one of inclusion.

Clinton appealed to Europe to embrace Moscow as the former communist
capital continues to embrace democracy.

On Friday, President Clinton urged America\'s European allies to ensure
that Russia becomes \"fully part of Europe.\"

##  The war in Chechnya and human rights issues

Mr Clinton will also be discussing Russia\'s 10-month war in rebel
Chechnya.

But he also expressed concern about alleged human rights violations and
indiscriminate use of force by Russian troops in their eight-month
conflict against rebels fighting for independence in the Russian rebel
republic of Chechnya.

U.S. National Security Adviser Sandy Berger said last week Mr. Clinton
would raise human rights and press freedom.

Also on Clinton\'s agenda was the conflict in the rebellious republic of
Chechnya, where fighting has flare anew.

U.S. National Security Adviser Sandy Berger said last week Clinton would
raise human rights and press freedom.

## Other issues

Before leaving Moscow on Monday, Mr. Clinton will address the Russian
parliament.

Clinton leaves Moscow on Monday after addressing parliament, the first
U.S. president to do so.

# Multi-document abstracts

### **Abstract** **1**

US president Bill Clinton will hold talks covering arms control, the
Russian economy, and the situation in Chechnya during his meeting in
Moscow with Russia\'s new president Vladimir Putin.

### **Abstract 2**

US president Bill Clinton will hold talks covering arms control, the
Russian economy, and the situation in Chechnya during his meeting in
Moscow with Russia\'s new president Vladimir Putin. The talks are
expected to be dominated by U.S. plans to amend a 1972 treaty banning
anti-missile defenses in order to allow deployment of a missile defense
shield based in Alaska.

### 

### **Abstract 3**

US president Bill Clinton will hold talks covering arms control, the
Russian economy, and the situation in Chechnya during his meeting in
Moscow with Russia\'s new president Vladmir Putin. The talks are
expected to be dominated by U.S. plans to amend a 1972 treaty banning
anti-missile defenses in order to allow deployment of a missile defense
shield based in Alaska. So far, Russia has opposed the shield. But
recently, Putin has indicated that, after all, some type of political
pressures on rogue nations, such as North Korea and Iran, as well as
antimissile systems, might be needed.

White House officials said that Russia is restating an alternative
'theatre system' proposal to combine U.S. and Russian technologies to
build a shield that would target missiles immediately after takeoff \--
rather than high in their trajectory, as the United States has proposed.
But the administration does not agree that the theater systems Mr. Putin
is referring to would be effective against a third world missile threat.

**Abtract 4**

US president Bill Clinton will hold talks covering arms control, the
Russian economy, and the situation in Chechnya during his meeting in
Moscow with Russia\'s new president Vladmir Putin. The talks are
expected to be dominated by U.S. plans to amend a 1972 treaty banning
anti-missile defenses in order to allow deployment of a missile defense
shield based in Alaska. So far, Russia has opposed the shield. But
recently, Putin has indicated that, after all, some type of political
pressures on rogue nations, such as North Korea and Iran, as well as
antimissile systems, might be needed.

White House officials said that Russia is restating an alternative
'theater system' proposal to combine U.S. and Russian technologies to
build a shield that would target missiles immediately after takeoff \--
rather than high in their trajectory, as the United States has proposed.
But the administration does not agree that the theater systems Mr. Putin
is referring to would be effective against a third world missile threat.

Mr Clinton\'s overall message to Russia will be one of inclusion; he
urged America\'s European allies to ensure that Russia becomes \"fully
part of Europe.\" During his trip to Moscow president Clinton also
intends to express his concern about alleged human rights violations and
indiscriminate use of force by Russian troops in Chechnya.

[^1]: Baldwin Language Technologies, *breck@linc.cis.upenn.edu.*

[^2]: Department of Defense, *rldonaw@super.org.*

[^3]: Information Sciences Institute, University of Southern California,
    *hovy@isi.edu.*

[^4]: School of Information Studies, Syracuse University,
    *LIDDY@syr.edu.*

[^5]: MITRE Corporation, *imani@mitre.org.*

[^6]: Information Sciences Institute, University of Southern California,
    *marcu@isi.edu.*

[^7]: Columbia University, *kathy@cs.columbia.edu.*

[^8]: Xerox Park, *Vibhu@mittal.net.*

[^9]: Edinburgh University, *M.Moens@ed.ac.uk.*

[^10]: University of Michigan, *radev@umich.edu.*

[^11]: Cambridge University, UK, *ksj@cl.cam.ac.uk.*

[^12]: SPAWAR Systems Center, *sundheim@spawar.navy.mil.*

[^13]: Columbia University, *teufel@opus.cs.columbia.edu.*

[^14]: BBN Technologies, *weischedel@bbn.com.*

[^15]: CoGenTex, Inc., *mike@cogentex.com.*

[^16]: An *extract* contains information that is directly copied from
    the original document(s). As a consequence, an extract may contain
    dangling references, redundancies, and may be incoherent. In
    contrast, an *abstract* is coherent and cohesive. It still reflects
    the main information in an original document, but it may express
    this information using different words, phrases, sentence structures
    than those used in the source documents.