**NIHM Project**

I.  **Project Description**

> One of the primary goals of natural language processing and text
> summary in particular, is to categorize documents in terms of subject
> matter. The current primary technique is to compare documents based on
> individual word frequencies and identify common elements. In this case
> we eliminate all words that are relatively common, using TF-IDF, so
> that we may identify more "specialized" words. This technique lacks
> some degree of accuracy, due to the fact that two documents could be
> using many shared, even specialized words and still have completely
> different subject matters.
>
> We propose a new technique of subject matter identification that is
> based on abnormal noun phrase frequencies; we believe that these noun
> phrase frequencies would contain more helpful contextual information
> than word frequencies. We define an abnormal frequency as one that
> deviates significantly from the corpus frequency (CF), or the number
> of times that noun phrase occurs in a large, representative text data
> set.

II. **Project Time Frame**

> June 12, 2007 -- July 30, 2007

III. **Scope of the Project**

> Test the following hypotheses:
>
> If the IDF of a word in a small document set is significantly
> different than the ICF of that word in a large document set, anomaly
> arises. This word can be identified as a "specialized" word - one that
> has some concrete connection to the subject matter.
>
> If the IDF of a simple noun phrase in a small document set is
> significantly different than the ICF of that phrase in a large
> document set, anomaly arises. This phrase can be identified with the
> subject matter, and carries with it more, pertinent information than
> the words identified. We define a simple noun phrase simply as two
> consecutive nouns.
>
> If the IDF of a complex noun phrase in a small document set is
> significantly different than the ICF of that phrase in a large
> document set, anomaly arises. This phrase can be identified with the
> subject matter, and carries with it more, pertinent information than
> the words or simple phrases identified. We define a complex noun
> phrase as one that phrase whose head is a noun or pronoun optionally
> accompanied by a set of modifiers including relative clauses,
> prepositional phrases, determiners, and adjectives.

IV. **Potential Risks & Risk Management**

    -   An open source, complex noun phrase chunker may not be suitably
        modifiable to conform to our needs.

    -   Noun phrase generation is time consuming.

    -   Evaluation of the results may be difficult. For instance, how do
        we determine how "pertinent" information may be to the subject?

    -   Evaluation of subject matter of each document in the document
        set may be difficult. A standard method of subject matter
        identification will have to be established.

    -   A suitably differentiated document set must be chosen. It is not
        clear at this stage of the process whether more concentrated or
        generalized subject matters would be more illustrative.

    -   Interpretation of results may be difficult.

    -   The three hypotheses are proven false.

V.  **Tasks & Tentative Schedule**

  ----------------------------------- -----------------------------------
  **Task**                            **Time**

  Run the tagger on the computer      5/14/2007 -- 06/15/2007
  cluster to generate ICF for noun    
  phrases.                            

  Find a suitable complex noun phrase 6/12/2007 -- 6/22/2007
  chunker and modify it to conform to 
  our needs.                          

  Run the modified chunker on the     6/22/2007 -- 7/13/2007
  computer cluster to generate ICF    
  for complex noun phrases            

  Compare the IDF of test data set    06/22/2007 -- 06/29/2007
  against the ICF based on words and  
  interpret the results. Possibly,    
  adjustments need to be made to the  
  current approach. Generate first    
  graph set.                          

  Compare the IDF of test data set    06/29/2007 -- 07/06/2007
  against the ICF based on noun       
  phrases and interpret the results.  
  Generate second graph set.          

  Determine the evaluation metrics    07/6/2007 -- 07/13/2007
  and analyze data from the first two 
  graphs.                             

  Compare the IDF of test data set    07/13/2007 -- 07/20/2007
  against the ICF based on complex    
  noun phrases and interpret the      
  results. Generate third graph set.  

  Make a final analysis of the graph  07/20/2007 -- 07/27/2007
  sets.                               

  Refine the final paper.             07/27/2007
  ----------------------------------- -----------------------------------