ニートの自然言語処理100本ノック:50

諸事情により第6章からする。
正規表現を使ってはいけない問題だと勝手に誤解していたので無理やりnltkモジュールを使って実装した(ただし荒い...)

やり方としてはnltkのtokenizeを使うだけなのだがなんせコロン(:)を判断してくれないので、
無理やり:の横に#をぶち込んでsplitして、２つに文章を分けてから両方ともトークン化して結合した形だが我ながら無理やりすぎる。
:の分離できる数とかも我ながら実装が甘いがもうだるい...

コードに感情移入しすぎて色々見失う時があるのでまじで冷静大事!って学びました　はい

```python

"""
50. 文区切り
(. or ; or : or ? or !) → 空白文字 → 英大文字というパターンを文の区切りと見なし，入力された文書を1行1文の形式で出力せよ．
"""
from nltk import sent_tokenize
from pprint import pprint

def load_text(path):
    '''
    テキストの読み取り
    '''
    
    with open(path,'r') as data:
        sentence = data.read()
    
    return sentence

def find_specialchar_index(sentence,character=":"):
    '''
    １度だけindex返す
    FIXME:indexのリストを返すべき
    '''
    for index,sent in enumerate(sentence):
        if sent == character:
            return int(index+1)

def sent_tokenize_Plus(sentence,index):
    '''
    特定文字がある場所に#を代入する。
    # を基準に分割する
    それぞれのリストをtoken化する
    tokenを結合する。
    '''
    sentence = sentence[:index] +  '#' + sentence[index:]
    sent_splited = sentence.split('#')

    sent_token = []
    for i in range(len(sent_splited)):
        token = sent_tokenize(sent_splited[i])
        sent_token.extend(token)
    
    return sent_token



if __name__ == "__main__":

    from nltk import sent_tokenize
    from pprint import pprint
    import re

    path = "nlp.txt"

    # テキストの読み込み
    sentence = load_text(path)
    # 特殊な文字の位置を取得する
    index = find_specialchar_index(sentence)
    # token化　一文化
    sent_token = sent_tokenize_Plus(sentence,index)
    
    pprint(sent_token)
    
```

# 出力結果
```
 'obvious at all where the effort should be directed.',
 'Automatic learning procedures can make use of statistical inference '
 'algorithms to produce models that are robust to unfamiliar input (e.g.',
 'containing words or structures that have not been seen before) and to '
 'erroneous input (e.g.',
 'with misspelled words or words accidentally omitted).',
 'Generally, handling such input gracefully with hand-written rules -- or more '
 'generally, creating systems of hand-written rules that make soft decisions '
 '-- extremely difficult, error-prone and time-consuming.',
 'Systems based on automatically learning the rules can be made more accurate '
 'simply by supplying more input data.',
 'However, systems based on hand-written rules can only be made more accurate '
 'by increasing the complexity of the rules, which is a much more difficult '
 'task.',
 'In particular, there is a limit to the complexity of systems based on '
 'hand-crafted rules, beyond which the systems become more and more '
 'unmanageable.',
 'However, creating more data to input to machine-learning systems simply '
 'requires a corresponding increase in the number of man-hours worked, '
 'generally without significant increases in the complexity of the annotation '
 'process.',
 'The subfield of NLP devoted to learning approaches is known as Natural '
 'Language Learning (NLL) and its conference CoNLL and peak body SIGNLL are '
 'sponsored by ACL, recognizing also their links with Computational '
 'Linguistics and Language Acquisition.',
 'When the aims of computational language learning research is to understand '
 'more about human language acquisition, or psycholinguistics, NLL overlaps '
 'into the related field of Computational Psycholinguistics.']

```