Repository: cltk/cltk

Issue:
Make stopwords list for Old English
To generalize, I observe that there are different approaches to making stopword lists, based either on statistics (most common words, variously calculated) or grammar (definite and indefinite articles, pronouns, etc.) (or some combination).

In doing this ticket, I would like you to do a little research on whether there exist any good lists for OE. If there is one, let's just take it. If not, we can do a little more research about what's right.



Discussion:
Now that the Old-English corpus is finished, I can start the work on this issue

Ok, go for it. Since I opened this issue, I have decided that I only want grammar-based stopwords. That is, no nouns, adjectives, verbs, adverbs, or participles, but only definite and indefinite articles (determiners), pronouns, particles/interjections, prepositions, and conjunctions.

Two steps: to build this list.
1. Look for someone else's Old English stopword list and copy it. We can discuss its pros and cons.
2. Take a Modern English stopword list and adjust for Old English.

The second step will certainly force you to acquaint yourself with the basics of the languages.

@kylepjohnson I can go with the second step for building the list

Please start with step 1., since you and I are not Old English experts. It's always better to use the work of a specialist.

I am not able to find a good list of stopwords for Old English online. I will ask some scholars on this matter

@kylepjohnson I have done a small bit of analysis to obtain a list of most common words from the corpus (I only took Metrical Charms for my testing purposes) - http://paste.ubuntu.com/23082398/
Further checking needs to be done manually for stopwords. I would like know your views on the list.

Ping @kylepjohnson 

Thanks, Sourav. My dilemma is that I don't know what most of the words on the list mean. Let's forget about the corpus approach and just go by grammar. The following I know should be in our stops list:
- wesan: https://lrc.la.utexas.edu/eieol/engol/10#grammar_1060
- personal pronouns: https://lrc.la.utexas.edu/eieol/engol/20#grammar_1062
- ne and its contractions: https://lrc.la.utexas.edu/eieol/engol/20#grammar_1063
- prepositions: https://lrc.la.utexas.edu/eieol/engol/20#grammar_1064
- the verb beon: https://lrc.la.utexas.edu/eieol/engol/40#grammar_1072
- the verb eom: https://lrc.la.utexas.edu/eieol/engol/40#grammar_1073
- all misc. demonstratives and pronouns: https://lrc.la.utexas.edu/eieol/engol/50#grammar_1077
- numerals: https://lrc.la.utexas.edu/eieol/engol/50#grammar_1078
- the verb gan: https://lrc.la.utexas.edu/eieol/engol/60#grammar_1082

A few important notes:
1. Remember to get all declension forms for pronouns, demonstatives, etc
2. There are spelling inconsistencies with the eth ð with thorn þ -- I don't know what to do about these. We'll maybe need a converter like we do with J and V in Latin. 

@kylepjohnson I created a small list of probable stop words by manually searching them from a dictionary- http://paste.ubuntu.com/23085680/

This second, dictionary-based approach is better. But I believe we still need the grammar-based approach I just wrote about. Let's perhaps use your latest list to supplement and improve our results.

I too believe that the grammar-based approach is better for stopword, but I wanted to get started somewhere. So I did a bit of analysis and after obtaining a list of common words, used a dictionary to check the word list generated in the first list.

Oh yes, sorry for continuing off with the approach. I didn't see the links that you gave until after an hour.

@souravsingh What's happening with Old English stop words? 

You have a lot of old issues under your name. I think it's best that you let most of them go.

@kylepjohnson Sorry for being silent on the issue. I will send a PR for the issue ASAP


Resolving patch:
diff --git a/cltk/stop/old_english/__init__.py b/cltk/stop/old_english/__init__.py
new file mode 100644
--- /dev/null
+++ b/cltk/stop/old_english/__init__.py
@@ -0,0 +1 @@
+
diff --git a/cltk/stop/old_english/stops.py b/cltk/stop/old_english/stops.py
new file mode 100644
--- /dev/null
+++ b/cltk/stop/old_english/stops.py
@@ -0,0 +1,232 @@
+"""This stopword list is adapted  from the 
+Introduction to Old English website  at ``https://lrc.la.utexas.edu/eieol/engol``.
+"""
+
+__author__ = 'Sourav Singh <ssouravsingh12@gmail.com>'
+__license__ = 'GPL License.'
+
+STOPS_LIST = ['and',
+              'on',
+              'þonne',
+              'wið', 
+              'to',
+              'þæt',
+              'þe',
+              'ne',
+              'ic',
+              'me',
+              'heo',
+              'him',
+              'he',
+              'swa'
+              'þis'
+              'mid'
+              'þu'
+              'ofer'
+              'his'
+              'þriwa'
+              'seo'
+              'hit'
+              'se'
+              'þas'
+              'cweð'
+              'þæs'
+              'in'
+              'sy'
+              'ða'
+              'ðy'
+              'ær'
+              'ðonne'
+              'næfre'
+              'þone'
+              'ge'
+              'ðas'
+              'þære'
+              'þam'
+              'is' 
+              'of'
+              'gif'
+              'þæm'
+              'nu'
+              'under'
+              'wiþ'
+              'geond'
+              'æfter'
+              'ðis'
+              'do'
+              'hwæt'
+              'her'
+              'þurh'
+              'þus'
+              'lytel'
+              'æt'
+              'ðin'
+              'willian'
+              'cume'
+              'þeos'
+              'þara'
+              'are'
+              'cuman'
+              'com'
+              'ænig'
+              'þon'
+              'for'
+              'us'
+              'ac'
+              'bot'
+              'ende'
+              'wæs',
+              'wǣre',
+              'wes',
+              'wǣron',
+              'wǣren',
+              'wesað',
+              'ic',
+              'wit',
+              'wē',
+              'mīn',
+              'uncer',
+              'ūser',
+              'ūre',
+              'mē',
+              'unc',
+              'ūs',
+              'mec',
+              'uncit',
+              'ūsic',
+              'ðū',
+              'git',
+              'gē',
+              'ðīn',
+              'incer',
+              'ēower', 
+              'ēowre',
+              'ðē',
+              'inc',
+              'ēow',
+              'ðec',
+              'incit',
+              'ēowic',
+              'hē',
+              'hēo', 
+              'hīe',
+              'hit', 
+              'hyt',
+              'hī', 
+              'hȳ',
+              'hire',
+              'hira', 
+              'heora', 
+              'hiera',
+              'heom',
+              'hine',
+              'nǣr',
+              'nǣfre',
+              'nǣnig',
+              'nolde',
+              'noldon',
+              'be',
+              'beforan',
+              'betweox',
+              'for',
+              'from',
+              'fram',
+              'mid',
+              'tō',
+              'geond',
+              'oð',
+              'þurh',
+              'ofer',
+              'under',
+              'bēo',
+              'bist',
+              'biþ',
+              'bēoþ',
+              'bēon',
+              'ēom',
+              'sīe',
+              'eart',
+              'sī',
+              'is',
+              'sēo',
+              'sindon',
+              'sint',
+              'nēom',
+              'neart',
+              'nis',
+              'sīo',
+              'ðæt', 
+              'tæt',
+              'ðæs',
+              'ðǣre',
+              'ðǣm', 
+              'ðām',
+              'ðone',
+              'ðā',
+              'ðȳ', 
+              'ðē', 
+              'ðon',
+              'ðāra', 
+              'ðǣra',
+              'ðes',
+              'ðēos',
+              'ðisse', 
+              'ðeosse',
+              'ðises',
+              'ðisses',
+              'ðisum',
+              'ðissum',
+              'ðisne',
+              'ðās',
+              'ðīs',
+              'ðȳs',
+              'ðissa', 
+              'ðeossa',
+              'ðeosum',
+              'ðeossum',
+              'twēgen',
+              'twā',
+              'tū',
+              'twēgra',
+              'twǣm',
+              'þrīe',
+              'þrēo',
+              'þrēora',
+              'þrīm',
+              'endlefan',
+              'twelf',
+              'twēntig',
+              'þrēotīene',
+              'þrītig',
+              'fēower',
+              'fēowertīene',
+              'fēowertig',
+              'fīf',
+              'fīftīene',
+              'fīftig',
+              'siex',
+              'siextīene',
+              'siextig',
+              'seofon',
+              'seofontīene',
+              'seofontig',
+              'eahta',
+              'eahtatīene',
+              'eahtatig',
+              'nigon',
+              'nigontīene',
+              'nigontig',
+              'tīen',
+              'hund',
+              'gā',
+              'gǣst',
+              'gǣð',
+              'gāð',
+              'gān',
+              'gānde',
+              'gangende',
+              'gegān',
+              'ēode',
+              'ēodest',
+              'ēodon',
+              'ēoden']


Test patch: