I am working on the file "NLP_With_Python/Ch5.ipynb".
The first task is:
unigram tagger是統計每個字最常出現的詞性，因此訓練資料越大，就會越準確。但遇到沒看過的字，就會傳回None。因此需要設backoff，當unigram tagger無法判斷時，用另一個tagger來輔助。
Can you write Python code for it?

unigram_tagger = nltk.UnigramTagger(brown.tagged_sents(categories='news')[:500],
                                   backoff = regexp_tagger)

unigram_tagger.evaluate(brown_tagged_sents[500:])

unigram_tagger = nltk.UnigramTagger(brown.tagged_sents(categories='news')[:4000],
                                   backoff = regexp_tagger)

unigram_tagger.evaluate(brown_tagged_sents[4000:])


I am working on the file "NLP_With_Python/Ch5.ipynb".
The first task is:
上面的範例中，CC是對等連接詞、RB是副詞、IN是介系詞、NN是名詞、JJ則是形容詞。如果想知道詳細的tag定義，可以用nltk.help.upenn_tagset('RB')來查詢。
Tagged Corpora
在NLTK的習慣上，tagged token會表示成由單字和tag結合的tuple，但是儲存在corpus中的資料，則是一個字串包含單字和tag，中間以'/'分隔，例如'fly/NN'。利用str2tuple可以從corpus中的字串轉成tuple。
Can you write Python code for it?

nltk.tag.str2tuple('fly/NN')

# tagged_words() 是一個已經表示成tuple形態的資料
nltk.corpus.brown.tagged_words()

# 用參數 tagset='universal' 可以換成簡單的tag
nltk.corpus.brown.tagged_words(tagset='universal')

# 利用 FreqDist 統計詞性的數量
tag_fd = nltk.FreqDist(tag for (word, tag) in nltk.corpus.brown.tagged_words(tagset='universal'))
tag_fd.most_common()

%matplotlib inline
tag_fd.plot()

tag_cd = nltk.ConditionalFreqDist(nltk.corpus.brown.tagged_words(tagset='universal'))

# 查詢某單字的常用POS
tag_cd['yield']


I am working on the file "NLP_With_Python/Ch6.ipynb".
The first task is:
Exploiting Context
Context是指一個單字的前後文，例如單字fly可以作為名詞或動詞，如果前面是a或the，表示是名詞。
Can you write Python code for it?

def pos_features(sentence, i):
    features = {"suf_1": sentence[i][-1:], "suf_2": sentence[i][-2:], "suf_3": sentence[i][-3:]}
    if i == 0:
        features['prev'] = '*'
    else:
        features['prev'] = sentence[i-1]
    return features

pos_features(brown.sents()[0], 8)

brown.sents()[0][7:10]

tagged_sents = brown.tagged_sents(categories='news')
featuresets = []
for tagged_sent in tagged_sents:
    untagged_sent = nltk.tag.untag(tagged_sent)
    for i, (word, tag) in enumerate(tagged_sent):
        featuresets.append( (pos_features(untagged_sent, i), tag) )
size = int(len(featuresets) * 0.1)
train_set, test_set = featuresets[size:], featuresets[:size]
classifier = nltk.NaiveBayesClassifier.train(train_set)
nltk.classify.accuracy(classifier, test_set)

classifier.show_most_informative_features()


I am working on the file "NLP_With_Python/Ch6.ipynb".
The first task is:
Part-of-Speech Tagging
在Chapter 5有介紹過regular expression tagger，當時是用手動設定規則，這裡嘗試用classifier找出規則。
Can you write Python code for it?

import nltk
from nltk.corpus import brown
fdist = nltk.FreqDist()
fdist.update([w[-1:] for w in brown.words()])
fdist.update([w[-2:] for w in brown.words()])
fdist.update([w[-3:] for w in brown.words()])

common_suf = [k for (k, c) in fdist.most_common()[:100]]
common_suf[:5]

def pos_features(word):
    features = {suffix:word.lower().endswith(suffix) for suffix in common_suf}
    return features

featuresets = [(pos_features(n), pos) for (n, pos) in brown.tagged_words(categories='news')]
size = int(len(featuresets) * 0.1)
train, test = featuresets[size:], featuresets[:size]
size

classifier = nltk.DecisionTreeClassifier.train(train)
nltk.classify.accuracy(classifier, test)

# DecisionTree可以列出結構
print classifier.pretty_format(depth=10)


I am working on the file "NLP_With_Python/Ch6.ipynb".
The first task is:
準備一個function，用來產生feature，這裡的feature是用最後的英文字母。
Can you write Python code for it?

def gender_feature(name): return {'last_letter': name[-1]}
featuresets = [(gender_feature(n), g) for (n,g) in name]
train, test = featuresets[500:], featuresets[:500]
classifier = nltk.NaiveBayesClassifier.train(train)

classifier.classify({'last_letter': 'a'})

nltk.classify.accuracy(classifier, test)

classifier.show_most_informative_features()


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
Chinking
另一種作chunking的方法是反向定義，定義那些部分不要放入chunk，這些部分稱為chink。
Can you write Python code for it?

grammar = r"""
    NP: {<.*>+}      # chunk everything
        }<VERB|ADP>+{  # chink sequence of VBD and IN
"""
sentence = preprocess('the little yellow dog barked at the cat')[0]
cp = nltk.RegexpParser(grammar)
cp.parse(sentence)


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
Chunking
要辨識entity最基本的技巧稱為chunking，我們會將多個單字結合起來成為一個chunk，這個過程稱為noun phrase chunking或NP-chunking。
Can you write Python code for it?

sentence = preprocess('the little yellow dog barked at the cat')[0]
sentence

# RegexpParser: 將符合條件的POS組合成一個新的
grammar = "NP: {<DET>?<ADJ>*<NOUN>}"
cp = nltk.RegexpParser(grammar)
cp.parse(sentence)

# 改良版的規則
#   (冠詞|所有格) + 形容詞 + 名詞，如"The black fat cat"
#   連續的專有名詞，如"Mary Brown"
grammar = r"""
    NP: {<DET|PRON>?<ADJ>*<NOUN>}
        {<NOUN>+}
"""
cp = nltk.RegexpParser(grammar)
sentence = preprocess("Rapunzel let down her long golden hair")[0]
cp.parse(sentence)


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
Exploring Text Corpora
一種chunking的應用，是從corpus中，找出特定詞性組合的字。例如want to buy是VERB-PRT-VERB，或連續四個名詞的組合。
Can you write Python code for it?

def find_chunks(user_rule = '{<V.*><TO><V.*>}'):
    cp = nltk.RegexpParser('CHUNK: ' + user_rule)
    for sent in nltk.corpus.brown.tagged_sents():
        tree = cp.parse(sent)
        for subtree in tree.subtrees():
            if subtree.label() == 'CHUNK':
                yield subtree

v_to_v = list(find_chunks())

v_to_v[1]

n4 = list(find_chunks('{<N.*><N.*><N.*><N.*>+}'))

n4[2]


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
Named Entity Recognition
當我們能夠正確辨識出NP chunk後，下一步就是辨識那個NP chunk屬於named entity，這個過程稱為NER(named  entity recognition)。
named entity(NE)有以下類型:
* ORGANIZATION: Georgia-Pacific Corp., WHO
* PERSON: Eddy Bonte, President Obama
* LOCATION: Murray River, Mount Everest
* DATE: June, 2008-06-29
* TIME: two fifty a m, 1:30 p.m.
* MONEY: 175 million Canadian Dollars, GBP 10.40
* PERCENT: twenty pct, 18.75 %
* FACILITY: Washington Monument, Stonehenge
* GPE: South East Asia, Midlothian
NER問題的困難點

人名或組織名更新太快，缺少dictionary可以用查表的方法解決
有些名稱同時屬於不同類型的NE，例如May同時是地名也是日期，也可以是人名
多字組成的名詞，需要辨識起始和結束位置

在conll2002的資料集中，就有包含NER問題的訓練資料。
Can you write Python code for it?

# 使用binary=True，只會標註NE的位置，但不會標註類型
nltk.ne_chunk(nltk.chunk.tree2conlltags(train_sents[99]), binary=True)

# 不使用binary=True，會標註類型 (例如PERSON)
nltk.ne_chunk(nltk.chunk.tree2conlltags(train_sents[99]))


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
Relation Extraction
當所有NE都辨識出來後，就可以試著找它們之間的關係，並輸出為(X, relation, Y)的形式。下面以regular expression為例，示範如何使用relation extration。
Can you write Python code for it?

import re
IN = re.compile(r'.*\bin\b(?!\b.+ing)')
for doc in nltk.corpus.ieer.parsed_docs('NYT_19980315'):
     for rel in nltk.sem.extract_rels('ORG', 'LOC', doc, corpus='ieer', pattern = IN):
            print nltk.sem.rtuple(rel)


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
一般的information extraction分為幾個步驟

輸入raw text，進行sentence segmentation
輸入sentence，進行tokenization
輸入list of words，進行part of speech tagging
輸入list of tuples，進行entity recognition
輸入list of trees，進行relation recognition

前面三個步驟不是本章的重點，因此將三個步驟結合成一個function。接下來要找出proper names(專有名詞)或是definite noun phrases(指示名詞)，作為entity的可能候選。最後，再成對的entity中找出relation的可能pattern。
Can you write Python code for it?

def preprocess(document, tagset="universal"):
    sent = nltk.sent_tokenize(document)
    sent = [nltk.word_tokenize(s) for s in sent]
    sent = [nltk.pos_tag(s, tagset=tagset) for s in sent]
    return sent


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
嘗試第三個版本的feature set，將目前的單字加入feature。
Can you write Python code for it?

def npchunk_features(sentence, i, history):
    """
    sentence -- 需要提取feature的句子，格式為list of (word, pos_tag)
    i -- 目前是句子的第幾個字
    history -- 前面的單字對應的tags
    """
    word, pos = sentence[i]
    if i == 0:
        prevword, prevpos = "<START>", "<START>"
    else:
        prevword, prevpos = sentence[i-1]
    return {"pos": pos, "word": word, "prevpos": prevpos}

chunker = ConsecutiveNPChunker(train_sents)

print chunker.evaluate(test_sents)


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
因為空的parser不會標註任何資料，遇到NP以外的地方就是對的，至少還有43%的正確率。
但因為完全沒有true positive，precision及recall都是0。
Can you write Python code for it?

# 嘗試用最簡單的parser
grammar = r"NP: {<[CDJNP].*>+}"
cp = nltk.RegexpParser(grammar)
print cp.evaluate(test_sents)


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
在上面的例子中，因為加上John thinks導致層數變多，所以thinks就被錯誤處理，沒有合併成VP。這可以用loop=2的參數來解決。
Can you write Python code for it?

cp = nltk.RegexpParser(grammar, loop=2)
cp.parse(preprocess(sentence, "")[0])


I am working on the file "NLP_With_Python/Ch7.ipynb".
The first task is:
最後一個版本，加入後一個字的tag、前後pos的關聯、及冠詞後面的tag。
Can you write Python code for it?

def tags_since_dt(sentence, i):
    tags = set()
    for word, pos in sentence[:i]:
        if pos == 'DT':
            tags = set()
        else:
            tags.add(pos)
    return '+'.join(sorted(tags))

def npchunk_features(sentence, i, history):
    """
    sentence -- 需要提取feature的句子，格式為list of (word, pos_tag)
    i -- 目前是句子的第幾個字
    history -- 前面的單字對應的tags
    """
    word, pos = sentence[i]
    if i == 0:
        prevword, prevpos = "<START>", "<START>"
    else:
        prevword, prevpos = sentence[i-1]
    if i == len(sentence)-1:
        nextword, nextpos = "<END>", "<END>"
    else:
        nextword, nextpos = sentence[i+1]
    return {"pos": pos, "word": word, "prevpos": prevpos, "nextpos": nextpos,
            "prevpos+pos": "%s+%s" % (prevpos, pos),
            "pos+nextpos": "%s+%s" % (pos, nextpos),
            "tags-since-dt": tags_since_dt(sentence, i)}

chunker = ConsecutiveNPChunker(train_sents)

print chunker.evaluate(test_sents)


I am working on the file "NLP_With_Python/Ch8.ipynb".
The first task is:
Parsing with Context-Free Grammar
Can you write Python code for it?

grammar2 = CFG.fromstring("""
    S  -> NP VP
    NP -> Det Nom | PropN
    Nom -> Adj Nom | N
    VP -> V Adj | V NP | V S | V NP PP
    PP -> P NP
    PropN -> 'Buster' | 'Chatterer' | 'Joe'
    Det -> 'the' | 'a'
    N -> 'bear' | 'squirrel' | 'tree' | 'fish' | 'log'
    Adj  -> 'angry' | 'frightened' |  'little' | 'tall'
    V ->  'chased'  | 'saw' | 'said' | 'thought' | 'was' | 'put'
    P -> 'on'
    """)


I am working on the file "NLP_With_Python/Ch8.ipynb".
The first task is:
使用CFG來產生syntax tree，很容易遇到structurally ambiguous的問題，例如"the dog saw a man in the park"，就有兩種不同的tree。在下面的例子中，稱為PP attachment ambiguous，用CFG無法判斷"in the park"是要修飾"a man"還是修飾"saw"。
Can you write Python code for it?

sent = "the dog saw a man in the park".split()
trees = rd_parser.parse(sent)
trees.next()

trees.next()


I am working on the file "NLP_on_non_textual_data/src/main/ipython/clinical2vec.ipynb".
The first task is:
Arthritis
From the Crohn's and Colitis Foundation of America:

Arthritis, or inflammation of the joints, is the most common extraintestinal complication of IBD. It may affect as many as 25% of people with Crohn’s disease or ulcerative colitis. Although arthritis is typically associated with advancing age, in IBD it often strikes the youngest patients.

Dental Abscesses
While not much medical literature exists with a specific link to dental abscesses and Crohn's (there are general oral issues noticed here), you do see lengthy discussions on the Crohn's forums about abscesses being a common occurance with Crohn's.
Yeast Infections
Candidiasis of skin and nails is a form of yeast infection on the skin. From the journal "Critical Review of Microbiology" here.

It is widely accepted that Candidia could result from an inappropriate inflammatory response to intestinal microorganisms in a genetically susceptible host. Most studies to date have concerned the involvement of bacteria in disease progression. In addition to bacteria, there appears to be a possible link between the commensal yeast Candida albicans and disease development.

Drugs associated with HIV/AIDS
The notion of a 'synonym' can also find connections between clinical data types.  Here we look for the drugs most associated with HIV/AIDS
Can you write Python code for it?


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Basic Functions
Can you write Python code for it?

def write_file(x, filepath):
    """Writes a file and generates filename from first 20 characters"""
    bad_file_chars = ['>', '<', ':', '"', '/', '\\', "|", '*', ' ', '\?', '\u2014', '\u2019']
    filename = str(x[:20])
    for char in bad_file_chars:
        filename = filename.replace(char, '_')
    with open(filepath, 'wb') as f:
        f.write(x.encode('utf-8'))

def write_json_file(data, filepath):
    with open(filepath + '.json', 'w') as f:
        json.dump(data, f)

def read_file(filepath, filename):
    """Reads file"""
    return PlaintextCorpusReader(filepath, filename)

def tokenize(text):
    sentences = nltk.sent_tokenize(text.raw())
    words = [nltk.word_tokenize(sentence.lower()) for sentence in sentences]
    return words

def process(text):
    words = tokenize(text)
    tagged_words = [dict(nltk.pos_tag(word)) for word in words]
    return tagged_words


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Basic NLTK Functions
Can you write Python code for it?

def get_words_longer_than(length, text):
    """Returns words from a text that are longer than a length."""
    longer_words = set([w for w in text.words() if len(w) > length])
    print(longer_words)

def get_words_by_char(text, ch):
    """Returns words in a text that contain specified character or string."""
    ch_words = [w.lower() for w in text.words() if ch in w]
    return ch_words


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
CMU Pronouncing Dictionary Functions
The CMU Pronouncing Dictionary is a corpus of words (English) and their pronunciation, available through the Natural Language Toolkit.
Pronunciations are broken into a series of 1- and 2-letter 'phones' that represent the sound of each syllable. Each word is also labeled with information about stressed and unstressed syllables (0: No stress; 1: Primary stress; 2: Secondary stress).
Information about the dictionary is available at http://www.speech.cs.cmu.edu/cgi-bin/cmudict
Can you write Python code for it?

######################################################
#  Helper Functions - CMU Pronunciation Dictionary   #
######################################################

def get_safe_cmudict_list(text):
    """
    Returns a LIST of words in a text that can be
    processed by the CMU Pronuncation Dictionary.
    """
    # get_cmudict_error_words(text)
    cmu_dict_words = sorted([w.lower() for w in text.words()
                             if w.lower() in prondict])
    return cmu_dict_words

def get_safe_cmudict_set(text):
    """
    Returns a SET of words in a text that can be
    processed by the CMU Pronuncation Dictionary.
    """
    return set(get_safe_cmudict_list(text))


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Call functions to create variables and files storing raw and processed Twitter data.
Can you write Python code for it?


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Dictionary Functions
Can you write Python code for it?

def make_pos_dict(text):
    """Returns a dictionary of words sorted by their part of speech."""
    tagged_words = process(text)
    pos = defaultdict(set)
    for i in range(0, len(tagged_words)):
        for k, v in tagged_words[i].items():
            pos[v].add(k)
    return pos

def make_word_pos_dict(text):
    """Returns a flat dict of words and parts of speach in text."""
    tagged_words = process(text)
    word_pos_dict = {}
    for i in range(0, len(tagged_words)):
        for word, pos in tagged_words[i].items():
            word_pos_dict[word] = pos
    return word_pos_dict

def make_phone_pos_dict(phone_list, text):
    """
    Returns a dictionary of words containing the specified phones,
    sorted by their part of speech
    """
    pos = make_pos_dict(text)
    phone_words = get_phone_words(phone_list, text)
    phone_pos_dict = defaultdict(list)
    for k, v in pos.items():
        for word in v:
            if word in phone_words:
                phone_pos_dict[k].append(word)
    return phone_pos_dict


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Examples
Write new text and save it to a file, reading text from file
Can you write Python code for it?

##############################
#  Read text from .txt file  #
##############################

# new_file = read_file('FILEPATH', 'FILENAME.txt')

###  OR  ###

# tweets = PlaintextCorpusReader('FILEPATH', 'FILENAME.txt')

################################################################
# Write sentences based on the file read in the previous step  #
################################################################

# Write new sentences based on words and grammar in a text
# new_text = write_sentences(new_file)

###  OR  ###

# Write new sentences with specified phones
# based on words and grammar in a text
# new_text = write_phone_sentences(['OW', 'OY', 'UW', 'AO'], new_file)

#############################################
#  Create a new file containing the         #
#  sentences written in the previous step   #
#############################################

# write_file(new_text)
# print(new_new_text)


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
NLTK
The Natural Language Toolkit (NLTK) provides users with access to a series of corpora and functions useful for parsing and working with text. Information about the toolkit and documentation is available at http://www.nltk.org/.
NLTK functions require plaintext files (.txt) or variables assigned to values processed by the PlainTextCorpusReader NLTK function.
e.g.
tweets_all = PlaintextCorpusReader(FILEPATH, 'tweets_all.txt')
hashtags_all = PlaintextCorpusReader(FILEPATH, 'hashtags_all.txt')
NLTK Import Statements
Can you write Python code for it?

import nltk
import nltk.data

import random
from collections import defaultdict, OrderedDict

from nltk.corpus import PlaintextCorpusReader
from nltk.corpus import CategorizedPlaintextCorpusReader
from nltk.corpus import cmudict
from nltk.corpus import wordnet

from nltk import load_parser
from nltk.tokenize import *
from nltk.probability import *
from nltk.misc.wordfinder import wordfinder
from nltk.text import Text


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Query Twitter API and Process the Data
The following are very basic functions to query the Twitter API and store the results in variables and .txt files, using plain-text / string and Python-dict formats so the data is easy to work with.
Currently, Tweepy does not support limiting searches by date or the number of results (unless they are paginated).
An alternative to Tweepy is the Twython module, which is more full-featured. However, because of dependencies, (requests-mock) the package can no longer be used with Anaconda / Miniconda.
Can you write Python code for it?

#######################
#  Query Twitter API  #
#######################

def twitter_search(query):
    """Query the Twitter API"""
    tweets = []
    for tweet in api.search(q = query):
        tweets.append(tweet._json)
    return tweets

##################
#  Process Data  #
##################

def process_tweets(data):
    """
    Process the results of a Twitter API query.
    Adds 'text', 'created_at' and 'hashtags' fields
    to Python dictionary.
    """
    # get the text and date fields
    processed_data = []
    for tweet in data:
        entry = {}
        entry['text'], entry['created_at'] = tweet['text'], tweet['created_at']
        if tweet['entities']['hashtags'] != []:
            hashtags = []
            for hashtag in tweet['entities']['hashtags']:
                hashtags.append(hashtag['text'])
            entry['hashtags'] = hashtags
        else:
            entry['hashtags'] = []
        processed_data.append(entry)
    return processed_data

def get_string(tweets):
    """
    Converts unprocessed Twitter data into a string.
    """
    text = ''
    for tweet in process_tweets(tweets):
        text = text + tweet['text'] + '\n'
    return text

##########################
#  Write Data to Files   #
##########################

def write_file_tweets(processed_data, filepath, filename):
    with open(filepath + filename + '.txt', 'wb') as f:
        for tweet in processed_data:
            f.write(bytes(tweet['text'] + '\n', 'utf-8'))

def write_file_hashtags(processed_data, filepath, filename):
    with open(filepath + filename + '.txt', 'wb') as f:
        for tweet in processed_data:
            for hashtag in tweet['hashtags']:
                f.write(bytes(hashtag + '\n', 'utf-8'))


I am working on the file "NLTK TwitterMixer.ipynb".
The first task is:
Twitter
Text remixed by the TwitterMixer is scraped from the Twitter API using the Tweepy (3.5.0) module. Tweepy Documentation: http://docs.tweepy.org/en/v3.5.0/
Twitter Import Statements
Can you write Python code for it?


I am working on the file "NLTK.ipynb".
The first task is:
Given some context surrounding a word, we can discover similar words, e.g. words that that occur frequently in the same context and with a similar distribution: Distributional similarity:
Can you write Python code for it?

print moby.similar("ahab")
austen = nltk.text.Text(nltk.corpus.gutenberg.words('austen-sense.txt'))
print
print austen.similar("monstrous")


I am working on the file "NLTK.ipynb".
The first task is:
Introduction to NLP with NLTK
Natural Language Processing (NLP) is often taught at the academic level from the perspective of computational linguists. However, as data scientists, we have a richer view of the natural language world - unstructured data that by its very nature has latent information that is important to humans. NLP practioners have benefited from machine learning techniques to unlock meaning from large corpora, and in this class we’ll explore how to do that particularly with Python and with the Natural Language Toolkit (NLTK).
NLTK is an excellent library for machine-learning based NLP, written in Python by experts from both academia and industry. Python allows you to create rich data applications rapidly, iterating on hypotheses. The combination of Python + NLTK means that you can easily add language-aware data products to your larger analytical workflows and applications.
Quick Overview of NLTK
NLTK stands for the Natural Language Toolkit and is written by two eminent computational linguists, Steven Bird (Senior Research Associate of the LDC and professor at the University of Melbourne) and Ewan Klein (Professor of Linguistics at Edinburgh University). NTLK provides a combination of natural language corpora, lexical resources, and example grammars with language processing algorithms, methodologies and demonstrations for a very pythonic "batteries included" view of Natural Language Processing.
As such, NLTK is perfect for researh driven (hypothesis driven) workflows for agile data science. Its suite of libraries includes:

tokenization, stemming, and tagging
chunking and parsing
language modeling
classification and clustering
logical semantics

NLTK is a useful pedagogical resource for learning NLP with Python and serves as a starting place for producing production grade code that requires natural language analysis. It is also important to understand what NLTK is not:

Production ready out of the box
Lightweight
Generally applicable
Magic

NLTK provides a variety of tools that can be used...
Can you write Python code for it?


I am working on the file "NLTK.ipynb".
The first task is:
Named Entity Recognition
NLTK has an excellent MaxEnt backed Named Entity Recognizer that is trained on the Penn Treebank. You can also retrain the chunker if you'd like - the code is very readable to extend it with a Gazette or otherwise.
Can you write Python code for it?

print nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize("John Smith is from the United States of America and works at Microsoft Research Labs")))


I am working on the file "NLTK.ipynb".
The first task is:
Preprocessing Text
NLTK is great at the preprocessing of Raw text - it provides the following tools for dividing text into it's constituent parts:

sent_tokenize: a Punkt sentence tokenizer:
This tokenizer divides a text into a list of sentences, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences.  It must be trained on a large collection of plaintext in the target language before it can be used.
However, Punkt is designed to learn parameters (a list of abbreviations, etc.) unsupervised from a corpus similar to the target domain. The pre-packaged models may therefore be unsuitable: use PunktSentenceTokenizer(text) to learn parameters from the given text.

word_tokenize: a Treebank tokenizer
The Treebank tokenizer uses regular expressions to tokenize text as in Penn Treebank. This is the method that is invoked by word_tokenize().  It assumes that the text has already been segmented into sentences, e.g. using sent_tokenize().

pos_tag: a maximum entropy tagger trained on the Penn Treebank
There are several other taggers including (notably) the BrillTagger as well as the BrillTrainer to train your own tagger or tagset.
Can you write Python code for it?

text = u"Medical personnel returning to New York and New Jersey from the Ebola-riddled countries in West Africa will be automatically quarantined if they had direct contact with an infected person, officials announced Friday. New York Gov. Andrew Cuomo (D) and New Jersey Gov. Chris Christie (R) announced the decision at a joint news conference Friday at 7 World Trade Center. “We have to do more,” Cuomo said. “It’s too serious of a situation to leave it to the honor system of compliance.” They said that public-health officials at John F. Kennedy and Newark Liberty international airports, where enhanced screening for Ebola is taking place, would make the determination on who would be quarantined. Anyone who had direct contact with an Ebola patient in Liberia, Sierra Leone or Guinea will be quarantined. In addition, anyone who traveled there but had no such contact would be actively monitored and possibly quarantined, authorities said. This news came a day after a doctor who had treated Ebola patients in Guinea was diagnosed in Manhattan, becoming the fourth person diagnosed with the virus in the United States and the first outside of Dallas. And the decision came not long after a health-care worker who had treated Ebola patients arrived at Newark, one of five airports where people traveling from West Africa to the United States are encountering the stricter screening rules."

for sent in nltk.sent_tokenize(text):
    print sent
    print

for sent in nltk.sent_tokenize(text):
    print list(nltk.wordpunct_tokenize(sent))
    print

for sent in nltk.sent_tokenize(text):
    print list(nltk.pos_tag(nltk.word_tokenize(sent)))
    print


I am working on the file "NLTK.ipynb".
The first task is:
The nltk.text.Text class is a wrapper around a sequence of simple (string) tokens - intended only for the initial exploration of text usually via the Python REPL. It has the following methods:

common_contexts
concordance
collocations
count
plot
findall
index

You shouldn't use this class in production level systems, but it is useful to explore (small) snippets of text in a meaningful fashion.
The corcordance function performs a search for the given token and then also provides the surrounding context:
Can you write Python code for it?


I am working on the file "NLTK.ipynb".
The first task is:
This will open up a window with which you can download the various corpora and models to a specified location. For now, go ahead and download it all as we will be exploring as much of NLTK as we can. Also take note of the download_directory - you're going to want to know where that is so you can get a detailed look at the corpora that's included. I usually export an enviornment variable to track this:
~$ export NLTK_DATA=/path/to/nltk_data

Take a moment to explore what is in this directory
Working with Example Corpora
NLTK ships with a variety of corpora, let's use a few of them to do some work. Get access to the text from Moby Dick as follows:
Can you write Python code for it?


I am working on the file "NLTK.ipynb".
The first task is:
You can also wrap the Stanford NER system, which many of you are also probably used to using.
Can you write Python code for it?

import os
from nltk.tag import StanfordNERTagger

# change the paths below to point to wherever you unzipped the Stanford NER download file
stanford_root = '/Users/benjamin/Development/stanford-ner-2014-01-04'
stanford_data = os.path.join(stanford_root, 'classifiers/english.all.3class.distsim.crf.ser.gz')
stanford_jar  = os.path.join(stanford_root, 'stanford-ner-2014-01-04.jar')

st = StanfordNERTagger(stanford_data, stanford_jar, 'utf-8')
for i in st.tag("John Bengfort is from the United States of America and works at Microsoft Research Labs".split()):
    print '[' + i[1] + '] ' + i[0]


I am working on the file "NLTK.ipynb".
The first task is:
Your turn! Explore some of the text in the available corpora
Frequency Analyses
In statistical machine learning approaches to NLP, the very first thing we need to do is count things - especially the unigrams that appear in the text and their relationships to each other. NLTK provides two very excellent classes to enable these frequency analyses:

FreqDist
ConditionalFreqDist

And these two classes serve as the foundation for most of the probability and statistical analyses that we will conduct.
First we will compute the following:

The count of words
The vocabulary (unique words)
The lexical diversity (the ratio of word count to vocabulary)
Can you write Python code for it?

reuters = nltk.corpus.reuters # Corpus of news articles
counts  = nltk.FreqDist(reuters.words())
vocab   = len(counts.keys())
words   = sum(counts.values())
lexdiv  = float(words) / float(vocab)

print "Corpus has %i types and %i tokens for a lexical diversity of %0.3f" % (vocab, words, lexdiv)

counts.B()

print counts.most_common(40)  # The n most common tokens in the corpus

print counts.max() # The most frequent token in the corpus

print counts.hapaxes()[0:10]  # A list of all hapax legomena

counts.freq('stipulate') * 100 # percentage of the corpus for this token

counts.plot(200, cumulative=False)

from itertools import chain

brown = nltk.corpus.brown
categories = brown.categories()

counts = nltk.ConditionalFreqDist(chain(*[[(cat, word) for word in brown.words(categories=cat)] for cat in categories]))

for category, dist in counts.items():
    vocab  = len(dist.keys())
    tokens = sum(dist.values())
    lexdiv = float(tokens) / float(vocab)
    print "%s: %i types with %i tokens and lexical diveristy of %0.3f" % (category, vocab, tokens, lexdiv)


I am working on the file "NLTK.ipynb".
The first task is:
your turn, go ahead and explore similar words and contexts - what does the common context mean?
NLTK also uses matplotlib and pylab to display graphs and charts that can show dispersions and frequency. This is especially interesting for the corpus of innagural addresses given by U.S. presidents.
Can you write Python code for it?

inaugural = nltk.text.Text(nltk.corpus.inaugural.words())
inaugural.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])


I am working on the file "NLTK_Basics.ipynb".
The first task is:
Another way to match a sentence is to construct a regular grammar:
Regular Grammar
Can you write Python code for it?

from IPython.core.display import display

grammar = nltk.CFG.fromstring("""
S -> 'the' S1
S1 -> 'cat' VP
VP -> 'barked' VP1
VP1 ->  C VP
C  -> 'or' | 'and'
VP -> 'swam'
""")

sent = ['the', 'cat', 'barked', 'or', 'swam']
parser = nltk.ChartParser(grammar)
for tree in parser.parse(sent):
    print tree
    display(tree)


I am working on the file "NLTK_Basics.ipynb".
The first task is:
III. 詞性標註
Can you write Python code for it?

from nltk.tag import StanfordPOSTagger

os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home"
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"

eng_tagger = StanfordPOSTagger(model_filename=r'english-bidirectional-distsim.tagger')
print(eng_tagger.tag('What is the airspeed of an unladen swallow ?'.split()))


I am working on the file "NLTK_Basics.ipynb".
The first task is:
IV. 語法解析
Can you write Python code for it?

from nltk.parse.stanford import StanfordParser
from IPython.core.display import display

os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home"
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"

eng_parser = StanfordParser()
structure=eng_parser.parse("the quick brown fox jumps over the lazy dog".split())

for tree in structure:
    print(tree)
    display(tree)

#print(list(eng_parser.parse("the quick brown fox jumps over the lazy dog".split())))


I am working on the file "NLTK_Basics.ipynb".
The first task is:
Now, here is a sentence which shows ambiguity since it can be parsed into two different trees.
(http://www.nltk.org/book/ch08.html)

A well-known example of ambiguity is shown in (2), from the Groucho Marx movie, Animal Crackers (1930):
(2)        While hunting in Africa, I shot an elephant in my pajamas. How he got into my pajamas, I don't know.
This grammar permits the sentence to be analyzed in two ways, depending on whether the prepositional phrase in my pajamas describes the elephant or the shooting event.
Can you write Python code for it?

from IPython.core.display import display

groucho_grammar = nltk.CFG.fromstring("""
S -> NP VP
PP -> P NP
NP -> Det N | Det N PP | 'I'
VP -> V NP | VP PP
Det -> 'an' | 'my'
N -> 'elephant' | 'pajamas'
V -> 'shot'
P -> 'in'
""")
sent = ['I', 'shot', 'an', 'elephant', 'in', 'my', 'pajamas']
parser = nltk.ChartParser(groucho_grammar)
for tree in parser.parse(sent):
    print tree
    display(tree)


I am working on the file "NLTK_Basics.ipynb".
The first task is:
Stanford NLP + Python NLTK
Stanford NLP
* http://nlp.stanford.edu/software/
Installation guides
* https://github.com/nltk/nltk/wiki/Installing-Third-Party-Software
* http://www.cnblogs.com/baiboy/p/nltk1.html
I. 分詞
Can you write Python code for it?

import nltk
import os

# 提供環境變數
os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home"
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"

from nltk.tokenize import StanfordTokenizer

tokenizer = StanfordTokenizer()
sent = "Good muffins cost $3.88\nin New York.  Please buy me\ntwo of them.\nThanks."
print(tokenizer.tokenize(sent))


I am working on the file "NLTK_Basics.ipynb".
The first task is:
V. 依存語法解析
Can you write Python code for it?

from nltk.parse.stanford import StanfordDependencyParser

os.environ["JAVA_HOME"] = "/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home"
os.environ["CLASSPATH"] = "/Users/chweng/Desktop/StanfordNLP/jars"
os.environ["STANFORD_MODELS"] = "/Users/chweng/Desktop/StanfordNLP/models"

eng_parser = StanfordDependencyParser()
res = list(eng_parser.parse("the quick brown fox jumps over the lazy dog".split()))
for row in res[0].triples():
    print(row)


I am working on the file "NLTK_Basics.ipynb".
The first task is:
the method {str}.startswith()
Can you write Python code for it?

a='123 as2'
print(a.startswith('123 111'))
print(a.startswith('123 a'))

print(inaugural.words("1793-Washington.txt")[:200])


I am working on the file "NLTK_Basics.ipynb".
The first task is:
yet another example of the use of the method nltk.ConditionalFreqDist()
from http://www.nltk.org/_modules/nltk/probability.html
Can you write Python code for it?

import nltk
from nltk.tokenize import word_tokenize

sent = "the the the dog dog some other words that we do not care about"
print(word_tokenize(sent))
cfdist = nltk.ConditionalFreqDist((len(word), word) for word in word_tokenize(sent))
cfdist.plot()
print(cfdist.tabulate())


I am working on the file "NLTK_Basics.ipynb".
The first task is:
先以 gutenberg語料庫中的 Leaves of Grass by Walt Whitman為例。接下來的一些範例可以讓我們了解以下四個方法的意義：

nltk.corpus.gutenberg.fileids()
nltk.corpus.gutenberg.raw()
nltk.corpus.gutenberg.sents()
nltk.corpus.gutenberg.words()
Can you write Python code for it?


I am working on the file "NLTK_Basics.ipynb".
The first task is:
印出文本尺寸，文本總字數，平均詞長，句長等
Can you write Python code for it?

num_char = len(nltk.corpus.gutenberg.raw("whitman-leaves.txt"))    # 原始文本的长度，包括空格、符号等
num_words = len(nltk.corpus.gutenberg.words("whitman-leaves.txt")) # 词的数量
num_sents = len(nltk.corpus.gutenberg.sents("whitman-leaves.txt")) # 句子的数量
num_vocab = len(set([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")])) #文本的尺寸

# 印出文本尺寸，以及文本總字數
print(len(set([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")])),len([w.lower() for w in nltk.corpus.gutenberg.words("whitman-leaves.txt")]))

# 印出平均詞長（包括一个空白符号）、平均句子長度、和文本中每个詞出现的平均次数
print(int(num_char/num_words),int(num_words/num_sents),int(num_words/num_vocab),"whitman-leaves.txt")


I am working on the file "NLTK_Basics.ipynb".
The first task is:
印出該文前兩句
Can you write Python code for it?

for j in range(2):
    print( nltk.corpus.gutenberg.sents("whitman-leaves.txt")[j])


I am working on the file "NLTK_Basics.ipynb".
The first task is:
稍微了解一下Stanford NLP套件後，我們再回到Python NLTK：
http://www.cnblogs.com/baiboy/p/nltk4.html
Can you write Python code for it?


I am working on the file "NN Playground-VariX.ipynb".
The first task is:
Face detection in image
This block performs the actual processing of an image (specified in the code btw), and checks for faces in it
Can you write Python code for it?

orig_scale = 4

def process_chunk(chunk):
    im_input = chunk[np.newaxis, np.newaxis, :, :]
    net.blobs['data'].reshape(*im_input.shape)
    net.blobs['data'].data[...] = imtmp
    return net.forward()['prob'][0][1]

def get_face_prob(chunk):
    return process_chunk(chunk)

def add_outline(img, i, j, scale, confidence):
    color = (255,255,255)
    if confidence > 0.985:
        color = (0,255,0)
    elif confidence >0.96:
        color = (255,255,0)
    else:
        color  =(255,0,0)
    for x in range (int(i*scale), int((i+36)*scale)):
        img[x][int(j*scale)]=color
        img[x][int((j+36)*scale-1)]=color
    for y in range(int(j*scale), int((j+36)*scale)):
        img[int(i*scale)][y]=color
        img[int((i+36)*scale-1)][y]=color
    return img

img =  PIL.Image.open('test_vis4.jpg')
w, h = img.size
img = img.resize((int(w/orig_scale),int(h/orig_scale)), PIL.Image.NEAREST)
imbase = img
img = img.convert('L')
w, h = img.size
scale = 1
print "starting processing"
found_pairs = []
next_scale = 1.3
next_i = 4
while w >=36*2 and h >36*2:
    img = img.resize((int(w/next_scale),int(h/next_scale)), PIL.Image.NEAREST)
    w, h = img.size
    print img.size
    im = np.array(img)
    scale*=next_scale

    i = 0
    j = 0
    last_result = 0
    while i < int(h-36):
        next_i = 4
        while j < int(w-36):
            imtmp = np.array(im [i:i+36, j:j+36]/256.0)
            face_prob = get_face_prob(imtmp)
            if  face_prob > 0.5:
                next_i = 2
                while j < int(w-36) and face_prob > last_result:
                    imtmp = np.array(im [i:i+36, j:j+36]/256.0)
                    last_result = face_prob
                    face_prob = get_face_prob(imtmp)
                    j += 1
                if last_result > 0.92:
                    next_i = 1
                    matched = False
                    """print last_result
                    print "visage trouvé @ %i , %i"%(i, j)
                    showarray(imtmp*255)"""
                    for pair in...


I am working on the file "NN Playground-VariX.ipynb".
The first task is:
Network loading and tests
Can you write Python code for it?

net = caffe.Net('deploy.prototxt', "facenet_iter_200000.caffemodel", caffe.TEST)
#training = caffe.Net('facenet_train_test.prototxt', "facenet_iter_200000.caffemodel", caffe.TRAIN)
#solver = caffe.SGDSolver('facenet_solver.prototxt')
#test_net = solver.testnets[0]

def showarray(a, fmt='jpeg'):
    a = np.uint8(np.clip(a, 0, 255))
    f = StringIO()
    PIL.Image.fromarray(a).save(f, fmt)
    display(Image(data=f.getvalue()))

#im = np.array(PIL.Image.open('train_images/0/137021_102_88_72_72.pgm'))/256.0
im = np.array(PIL.Image.open('train_images/1/image000619.pgm'))/256.0
im_input = im[np.newaxis, np.newaxis, :, :]
net.blobs['data'].reshape(*im_input.shape)
net.blobs['data'].data[...] = im
showarray(im)
print (len(im))
print (len(im[0]))
print (im)
output = net.forward()
print(output)
if output['prob'][0][0] >0.9:
    print "visage"


I am working on the file "NN Playground-VariX.ipynb".
The first task is:
Redraw
Allows you to define a new threshold (provided it's more aggressive than the original), without calculating everything.
Can you write Python code for it?

threshold = 0.96
img =  PIL.Image.open('test_vis4.jpg')
w, h = img.size
img = img.resize((int(w/orig_scale),int(h/orig_scale)), PIL.Image.NEAREST)
imbase = img
print "adding overlay"
for pair in found_pairs:
    if pair[2]>threshold:
        imbase = add_outline(np.array(imbase),pair[0],pair[1],pair[3], pair[2])
showarray(imbase)

img = PIL.Image.open('test_vis4.jpg').convert('LA')
arr = np.array(img)
w, h = img.size
print w

a


I am working on the file "NN Playground-VariX.ipynb".
The first task is:
The cell bellows checks that opencv and its python bindings are properly installed
Can you write Python code for it?


I am working on the file "NN Playground-VariX.ipynb".
The first task is:
Train dataset appender
Can you write Python code for it?

img =  PIL.Image.open('neg1.jpg')
w, h = img.size
#img = img.resize((int(w/4),int(h/4)), PIL.Image.NEAREST)
imbase = img
img = img.convert('L')
w, h = img.size
scale = 1
width = 36
height = 36
import array
findex = 4327
with open('posneg.txt', 'a') as f:
    print "starting processing"
    while w >=36*2 and h >36*2:
        img = img.resize((int(w/1.3),int(h/1.3)), PIL.Image.NEAREST)
        w, h = img.size
        print img.size
        im = np.array(img)
        scale*=1.3
        found_pairs = []
        i = 0
        j = 0
        last_result = 0
        while i < int(h-36):
            while j < int(w-36):
                imtmp = np.array(im [i:i+36, j:j+36]/256.0)
                face_prob = get_face_prob(imtmp)
                if  face_prob > 0.9:
                    showarray(imtmp*255)
                    buff=array.array('B')
                    for k in range(0, 36):
                        for l in range(0, 36):
                            buff.append(int(imtmp[k][l]*255))
                    findex += 1
                    # open file for writing
                    filename = '0/lbe%i.pgm'%findex
                    try:
                        fout=open("train_images/"+filename, 'wb')
                    except IOError, er:
                        print "Cannot open file ", filename, "Exiting … \n", er
                    # define PGM Header
                    pgmHeader = 'P5' + '\n' + str(width) + '  ' + str(height) + '  ' + str(255) + '\n'
                    # write the header to the file
                    fout.write(pgmHeader)
                    # write the data to the file
                    buff.tofile(fout)
                    # close the file
                    fout.close()
                    f.write(filename + " 0\n")
                j+=4
            i+=4
            j = 0
        print "adding overlay"
        for pair in found_pairs:
            imbase = add_outline(np.array(imbase),pair[0],pair[1],scale, pair[2])
showarray(imbase)

print findex


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
An alternative and recommended approach is to use StandardScaler in a Pipeline
* Finding a reasonable regularization parameter \alpha is best done using GridSearchCV, usually in the range 10.0 ** -np.arange(1,7).
* Empirically, we observed that L-BFGS converges faster and with better solutions on small datasets. For relatively large datasets, however, Adam is very robust. It usually converges quickly and gives pretty good performance. SGD with momentum or nesterov’s momentum, on the other hand, can perform better than those two algorithms if learning rate is correctly tuned.
More control with warm_start
If you want more control over stopping criteria or learning rate in SGD, or want to do additional monitoring, using warm_start=True and max_iter=1 and iterating yourself can be helpful:
Can you write Python code for it?

X = [[0., 0.], [1., 1.]]
y = [0, 1]
clf = MLPClassifier(hidden_layer_sizes=(15,), random_state=1, max_iter=1, warm_start=True)
for i in range(10):
     clf.fit(X, y)
    # additional monitoring / inspection


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
Classification
Class MLPClassifier implements a multi-layer perceptron (MLP) algorithm that trains using Backpropagation.
MLP trains on two arrays: array X of size (n_samples, n_features), which holds the training samples represented as floating point feature vectors; and array y of size (n_samples,), which holds the target values (class labels) for the training samples:
Can you write Python code for it?

## Input
X = [[0., 0.], [1., 1.]]

## Labels
y = [0, 1]

## Create Model
clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                    hidden_layer_sizes=(5, 2), random_state=1)
## Fit
clf.fit(X, y)

## Make Predictions
clf.predict([[2., 2.], [-1., -2.]])


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
Compare Stochastic learning strategies for MLPClassifier
This example visualizes some training loss curves for different stochastic learning strategies, including SGD and Adam. Because of time-constraints, we use several small datasets, for which L-BFGS might be more suitable. The general trend shown in these examples seems to carry over to larger datasets, however.
Note that those results can be highly dependent on the value of learning_rate_init.
Can you write Python code for it?

import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier
from sklearn.preprocessing import MinMaxScaler
from sklearn import datasets

# different learning rate schedules and momentum parameters
params = [{'solver': 'sgd', 'learning_rate': 'constant', 'momentum': 0,
           'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'constant', 'momentum': .9,
           'nesterovs_momentum': False, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'constant', 'momentum': .9,
           'nesterovs_momentum': True, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': 0,
           'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': .9,
           'nesterovs_momentum': True, 'learning_rate_init': 0.2},
          {'solver': 'sgd', 'learning_rate': 'invscaling', 'momentum': .9,
           'nesterovs_momentum': False, 'learning_rate_init': 0.2},
          {'solver': 'adam', 'learning_rate_init': 0.01}]

labels = ["constant learning-rate", "constant with momentum",
          "constant with Nesterov's momentum",
          "inv-scaling learning-rate", "inv-scaling with momentum",
          "inv-scaling with Nesterov's momentum", "adam"]

plot_args = [{'c': 'red', 'linestyle': '-'},
             {'c': 'green', 'linestyle': '-'},
             {'c': 'blue', 'linestyle': '-'},
             {'c': 'red', 'linestyle': '--'},
             {'c': 'green', 'linestyle': '--'},
             {'c': 'blue', 'linestyle': '--'},
             {'c': 'black', 'linestyle': '-'}]

def plot_on_dataset(X, y, ax, name):
    # for each dataset, plot learning for each learning strategy
    print("\nlearning on dataset %s" % name)
    ax.set_title(name)
    X = MinMaxScaler().fit_transform(X)
    mlps = []
    if name == "digits":
        # digits is larger but converges fairly quickly
        max_iter = 15
    else:
        max_iter = 400

    for label, param in zip(labels, params):
       ...


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
Currently, MLPClassifier supports only the Cross-Entropy loss function, which allows probability estimates by running the predict_proba method.
MLP trains using Backpropagation. More precisely, it trains using some form of gradient descent and the gradients are calculated using Backpropagation. For classification, it minimizes the Cross-Entropy loss function, giving a vector of probability estimates P(y|x) per sample x:
Can you write Python code for it?


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
MLP can fit a non-linear model to the training data. clf.coefs_ contains the weight matrices that constitute the model parameters:
Can you write Python code for it?

## Weight matrices/ model parameters
[coef.shape for coef in clf.coefs_]

## Coefficents of classifier
clf.coefs_


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
MLPClassifier supports multi-class classification by applying Softmax as the output function.
Further, the model supports multi-label classification in which a sample can belong to more than one class. For each class, the raw output passes through the logistic function. Values larger or equal to 0.5 are rounded to 1, otherwise to 0. For a predicted output of a sample, the indices where the value is 1 represents the assigned classes of that sample:
Can you write Python code for it?

X = [[0., 0.], [1., 1.]]
y = [[0, 1], [1, 1]]
clf = MLPClassifier(solver='lbfgs', alpha=1e-5,
                     hidden_layer_sizes=(15,), random_state=1)

 clf.fit(X, y)

clf.predict([1., 2.])

clf.predict([0., 0.])


I am working on the file "NN in Scikit Learn.ipynb".
The first task is:
Regularization
Both MLPRegressor and class:MLPClassifier use parameter alpha for regularization (L2 regularization) term which helps in avoiding overfitting by penalizing weights with large magnitudes. Following plot displays varying decision function with value of alpha.
Varying regularization in Multi-layer Perceptron¶
A comparison of different values for regularization parameter ‘alpha’ on synthetic datasets. The plot shows that different alphas yield different decision functions.
Alpha is a parameter for regularization term, aka penalty term, that combats overfitting by constraining the size of the weights. Increasing alpha may fix high variance (a sign of overfitting) by encouraging smaller weights, resulting in a decision boundary plot that appears with lesser curvatures. Similarly, decreasing alpha may fix high bias (a sign of underfitting) by encouraging larger weights, potentially resulting in a more complicated decision boundary.
Can you write Python code for it?

import numpy as np
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn.neural_network import MLPClassifier

h = .02  # step size in the mesh

alphas = np.logspace(-5, 3, 5)
names = []
for i in alphas:
    names.append('alpha ' + str(i))

classifiers = []
for i in alphas:
    classifiers.append(MLPClassifier(alpha=i, random_state=1))

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2,
                           random_state=0, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(noise=0.3, random_state=0),
            make_circles(noise=0.2, factor=0.5, random_state=1),
            linearly_separable]

figure = plt.figure(figsize=(17, 9))
i = 1
# iterate over datasets
for X, y in datasets:
    # preprocess dataset, split into training and test part
    X = StandardScaler().fit_transform(X)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.4)

    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
    # and testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], c=y_test, cmap=cm_bright, alpha=0.6)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax =...


I am working on the file "NN playground.ipynb".
The first task is:
Cost (or error) functions
Suppose that the expected output for an input vector ${\bf x} \equiv {\bf a^1}$ is ${\bf y} = {\bf y_x}^ = (0,1,0)$, we can now compute the error vector ${\bf e}= {\bf e_x}= {\bf a_x}^L-{\bf y_x}^$. With this error, we can now compute a cost $C=C_x$ assotiated with the output $\bf{y_x}$ of the input vector ${\bf x}$ (also called loss) function. For convinience of notations we will frequently omitt the subscript $x$.
Popular loss functions are:
* Absolute cost $C = C({\bf a}^L)=\sum_i |e_i|$
* Square cost $C= C({\bf a}^L) = \sum_i e_i^2$
* Cross entropy loss $C=C({\bf a}^L) = -\sum_i y_i^\log{a^L_i} \equiv -\sum_i y_i^\log{y_i}$. The rationale here is that the output of the softmax function is a probability distribution and we can also view the real label vector $y$ as a probability distribution (1 for the corerct label and 0 for all other labels). The cross entropy function is a common way to measure difference between distributions.
The total error from all $N$ data vectors is computed as the average of the individual error terms associated with each input vector ${\bf x}$, that is:$\frac{1}{N} \sum_x C_x$
Can you write Python code for it?

def abs_loss(e):
    return np.sum(np.abs(e))

def sqr_loss(e):
    return np.sum(e**2)

def cross_entropy_loss(y_estimated,y_real):
    return -np.sum(y_real*np.log(y_estimated))

y_real = np.array([0.,1.,0])
err    = a2 - ystar

print ' Error                                          ',err
print ' Absolute loss                                  ',abs_loss(err)
print ' Square loss                                    ',sqr_loss(err)
print ' Cross entropy loss                             ',cross_entropy_loss(a2,y_real)


I am working on the file "NN playground.ipynb".
The first task is:
General terminology and notations
The notation in here follow the notations in Chapter 2 of the deep learning online book. We note that there are different notations in different places, all refer differently to the same mathematical construct.

We consider $L$ layers marked by $l=1,\ldots,L$ where $l=1$ denotes the input layer
Layer $l$ has $s_l$ units referred to by  $a^{l}_j, j = 1,\ldots,s_l$.

The matrix $W^l : s_{l} \times s_{l-1} , l=2,\ldots, L$ controls the mapping from layer $l-1$ to layer $l$. The vector $\mathbf{b}^l$ of size $s_{l}$ corresponds to the bias term and in layer $l$. The weight $w^l_{ij}$ is the weight associated with the connection of neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$

Forward propagation: $\mathbf{a}^l = \sigma(\mathbf{z}^l)$ where $\mathbf{z}^l=W^l \mathbf{a}^{(l-1)}+ \mathbf{b}^{(l)} , l =2,\ldots,L$ where the activation function $\sigma \equiv \sigma_l$ is applied to each component of its argument vector. For simplicity of notations we often write $\sigma$ instead of $\sigma_l$

Synonims

Neuron - inspired from biology analogy
Unit -  It’s one component of a large network
Feature - It implements a feature detector that’s looking at the input and will turn on iff the sought feature is present in the input

A note about dot product
The activation function works on the outcome of the done way to think about this is in terms of correlation which are normalized dot products. Thus what we really measure is the degree of correlation, or dependence, between the input vector and the coefficient vector. We can view at the dot product as:
* A correlation filter - fires if a correlation between input and weights exceeds a threshold
* A feature detector - Detect if a specific pattern occur in the input
Output unit
The output values are computed by similarly multplying the values oh $h$ by another weight matrix,
$\def \mathbf \mathbf {}$

$\mathbf{a}^L = \sigma_L(\mathbf{W^L} \cdot \mathbf{a}^{(L-1)} + \mathbf{b}^L) = \sigma_L(\mathbf{z^L}) $

Linear regression...
Can you write Python code for it?

def sigmoid(x):
    return 1./(1.+np.exp(-x))

def rectifier(x):
    return np.array([max(xv,0.0) for xv in x])

def softplus(x):
    return np.log(1.0 + np.exp(x))

x     = np.array([1.0,0,0])
w     = np.array([0.2,-0.03,0.14])
print ' Scalar product between unit and weights ',x.dot(w)
print ' Values of Sigmoid activation function   ',sigmoid(x.dot(w))
print ' Values of ta      activation function   ',np.tanh(x.dot(w))
print ' Values of sofplus activation function   ',softplus(x.dot(w))

import pylab

z  = np.linspace(-2,2,100) # 100 linearly spaced numbers
s  = sigmoid(z)    # computing the values of
th = np.tanh(z)    # computing the values of
re = rectifier(z)  # computing the values of rectifier
sp = softplus(z)  # computing the values of rectifier

# compose plot
pylab.plot(z,s)
pylab.plot(z,s,'co',label='Sigmoid')   # Sigmoid
pylab.plot(z,th,label='tanh')          # tanh
pylab.plot(z,re,label='rectifier')     # rectifier
pylab.plot(z,sp,label='softplut')     # rectifier
pylab.legend()
pylab.show() # show the plot


I am working on the file "NN_demo.ipynb".
The first task is:
Build Neural Network
Functions to build NN
Can you write Python code for it?

def store_stats(i, avg_train_error, best_train_error, avg_valid_error,
                best_valid_error,**_):
    if i==1:
        global errors_stored
        errors_stored = []
    errors_stored.append((avg_train_error, best_train_error, avg_valid_error,
                          best_valid_error))

def build_nn(method, actv_fnc, hid_neur, learning_rule, pp_str,
             batch_size=100, n_iter=None, n_stable=None,
             learning_rate=0.01, learning_momentum=0.9,
             regularize='L2', weight_decay=0.0, valid_size=0.5,
             f_stable=.001):
    """Builds a multi-layer perceptron via the scikit neural network interface"""
    # First build layers
    layers = [sknn.mlp.Layer(f,units=h) for f,h in zip(actv_fnc,hid_neur)]
    # Append a linear output layer
    layers.append(sknn.mlp.Layer("Linear"))
    mlp = sknn.mlp.Regressor(layers, n_iter=n_iter, batch_size=batch_size,
        learning_rule=learning_rule, learning_rate=learning_rate,
        learning_momentum=learning_momentum, regularize=regularize,
        weight_decay=weight_decay, n_stable=n_stable, valid_size=valid_size,
        f_stable=f_stable, callback={'on_epoch_finish': store_stats})
    # Create a name for the neural network
    # First build names of each layer
    layerstr = '_'.join([str(h) + f[0] for h, f in zip(hid_neur, actv_fnc)])
    # Get str of appropriate learning rate
    if learning_rule == 'momentum':
        lrn_str = str(learning_momentum)
    else:
        lrn_str = str(learning_rate)
    # Construct name
    mlp_str = pp_str + method[0] + "_" +  layerstr + "_" + \
              learning_rule[0:3] + lrn_str
    # If using regularization, add that to the name too
    if weight_decay > 0.0:
        mlp_str = mlp_str + 'reg' + str(weight_decay)
    # Add the number of iterations too
    mlp_str = mlp_str + '_Niter' + str(n_iter)
    return mlp, mlp_str


I am working on the file "NN_demo.ipynb".
The first task is:
Check the distribution of output values
Can you write Python code for it?

# Plot histogram showing how well true and predicted values match
def check_output_distribution(yt_unscl, yt_scl, yp_unscl, yp_scl, lat, lev):
    # For unscaled variables
    fig, ax = plt.subplots(2, 2)
    x1, x2, bins = _plot_distribution(unpack(ypred_unscl,'T'), lat, lev, fig, ax[0,1],
                                      r'Pred. $\Delta$Temp [K/day]')
    _ , _ , _    = _plot_distribution(unpack(ytrue_unscl,'T'), lat, lev, fig, ax[0,0],
                                      r'True $\Delta$Temp [K/day]', x1, x2, bins)
    x1, x2, bins = _plot_distribution(unpack(ypred_unscl,'q'), lat, lev, fig, ax[1,1],
                                      r'Pred. $\Delta$Humidity [g/kg/day]')
    _ , _ , _    = _plot_distribution(unpack(ytrue_unscl,'q'), lat, lev, fig, ax[1,0],
                                      r'True $\Delta$Humidity [g/kg/day]', x1, x2, bins)

check_output_distribution(ytrue_unscl, ytrue_scl, ypred_unscl, ypred_scl,
                      lat, lev)


I am working on the file "NN_demo.ipynb".
The first task is:
Evaluate the NN
Load a more fully trained neural network as well as new (different) data for testing purposes
Also load validation data
Can you write Python code for it?

# Define a stored neural net to load
r_str = 'X-StandardScaler-qTindi_Y-SimpleY-qTindi_r_60R_60R_mom0.9reg1e-05'
# Load the NN
r_mlp_eval, _, errors, x_ppi, y_ppi, x_pp, y_pp, lat, lev, dlev = \
       pickle.load(open('./data/regressors/' + r_str + '.pkl', 'rb'))
# Open the ***VALIDATION*** data set
datasource='./data/convection_50day_validation.pkl'
x_unscl, ytrue_unscl, _,_,_,_,_,_ = nnload.loaddata(datasource, minlev=min(lev))
# Scale data using input scalers
x_scl     = nnload.transform_data(x_ppi, x_pp, x_unscl)
ytrue_scl = nnload.transform_data(y_ppi, y_pp, ytrue_unscl)
# Apply neural network to get predicted output
ypred_scl   = r_mlp_eval.predict(x_scl)
ypred_unscl = nnload.inverse_transform_data(y_ppi, y_pp, ypred_scl)


I am working on the file "NN_demo.ipynb".
The first task is:
Plot Mean Statistics
Can you write Python code for it?

def do_mean_or_std(method, vari, true, pred, lev, ind):
    out_str_dict = {'T':'K/day','q':'g/kg/day'}
    methods = {'mean':np.mean,'std':np.std}
    plt.subplot(2,2,ind)
    m = lambda x: methods[method](unpack(x,vari), axis=0).T
    plt.plot(m(true), lev, label='true')
    plt.plot(m(pred), lev, label='pred')
    plt.ylim(np.amax(lev),np.amin(lev))
    plt.ylabel('$\sigma$')
    plt.xlabel(out_str_dict[vari])
    plt.title(vari + " " + method)
    plt.legend()

def plot_means_stds(y3_true, y3_pred, lev):
    fig = plt.figure()
    do_mean_or_std('mean','T',y3_true,y3_pred, lev, 1)
    do_mean_or_std('mean','q',y3_true,y3_pred, lev, 2)
    do_mean_or_std('std','T',y3_true,y3_pred, lev, 3)
    do_mean_or_std('std','q',y3_true,y3_pred, lev, 4)

def plot_pearsonr(y_true, y_pred, vari, lev, label=None):
    r = np.empty(y_true.shape[1])
    prob = np.empty(y_true.shape[1])
    for i in range(y_true.shape[1]):
        r[i], prob[i] = scipy.stats.pearsonr(y_true[:,i],y_pred[:,i])
    plt.plot(unpack(r,vari, axis=0), lev, label=label)
    plt.ylim([np.amax(lev), np.amin(lev)])
    plt.ylabel('$\sigma$')
    plt.title('Correlation Coefficient')

# Plot means and standard deviations
plot_means_stds(ytrue_unscl, ypred_unscl, lev)

# Plot correlation coefficient versus height
fig = plt.figure()
plot_pearsonr(ytrue_unscl, ypred_unscl, 'T', lev, label=r'$\Delta$Temperature')
plot_pearsonr(ytrue_unscl, ypred_unscl, 'q', lev, label=r'$\Delta$Humidity')
plt.legend(loc="upper left")


I am working on the file "NN_demo.ipynb".
The first task is:
Preprocess Data
Can you write Python code for it?

def unpack(data,vari,axis=1):
    """Reverse pack operation to turn ouput matrix into T & q"""
    N = int(data.shape[axis]/2)
    varipos = {'T':np.arange(N),'q':np.arange(N,2*N)}
    out = np.take(data,varipos[vari],axis=axis)
    return out

def pack(d1,d2,axis=1):
    """Combines T & q profiles as an input matrix to NN"""
    return np.concatenate((d1,d2), axis=axis)

# Initialize & fit scaler
def init_pp(ppi, raw_data):
    """Initialize list of scaler objects"""
    if ppi['name'] == 'MinMax':
        pp =[preprocessing.MinMaxScaler(feature_range=(-1.0,1.0)), # for temperature
                 preprocessing.MinMaxScaler(feature_range=(-1.0,1.0))] # and humidity
    elif ppi['name'] == 'MaxAbs':
        pp =[preprocessing.MaxAbsScaler(), # for temperature
                 preprocessing.MaxAbsScaler()] # and humidity
    elif ppi['name'] == 'StandardScaler':
        pp =[preprocessing.StandardScaler(), # for temperature
                 preprocessing.StandardScaler()] # and humidity
    elif ppi['name'] == 'RobustScaler':
        pp =[preprocessing.RobustScaler(), # for temperature
                 preprocessing.RobustScaler()] # and humidity
    elif ppi['name'] == 'SimpleY':
        pp =[15.,10.] # for temperature
    else:
        ValueError('Incorrect scaler name')
    #Initialize scalers with data
    if ppi['method'] == 'individually':
        pp[0].fit(unpack(raw_data,'T'))
        pp[1].fit(unpack(raw_data,'q'))
    elif ppi['method'] == 'alltogether':
        pp[0].fit(np.reshape(unpack(raw_data,'T'), (-1,1)))
        pp[1].fit(np.reshape(unpack(raw_data,'q'), (-1,1)))
    elif ppi['method'] == 'qTindividually':
        if ppi['name'] != 'SimpleY':
            pp = pp[0]
            pp.fit(raw_data)
    else:
        raise ValueError('Incorrect scaler method')
    return pp

# Transform data using initialized scaler
def transform_data(ppi, pp, raw_data):
    """Scales data before ML algorithm"""
    if ppi['method'] == 'individually':
        T_data = pp[0].transform(unpack(raw_data,'T'))
       ...


I am working on the file "NN_demo.ipynb".
The first task is:
Preprocessing output values
Can you write Python code for it?

fig, ax = plt.subplots(2, 2)
_, _, _ = _plot_distribution(unpack(y_orig, 'T'), lat, lev, fig, ax[0,0],
                             'Temperature (unscaled) [K]')
_, _, _ = _plot_distribution(unpack(y,      'T'), lat, lev, fig, ax[0,1],
                             'Temperature (scaled) []')
_, _, _ = _plot_distribution(unpack(y_orig, 'q'), lat, lev, fig, ax[1,0],
                             'Humidity (unscaled) [g/kg]')
_, _, _ = _plot_distribution(unpack(y,      'q'), lat, lev, fig, ax[1,1],
                             'Humidity (scaled) []')
fig.suptitle('Distributions of Raw and Preprocessed Outputs', fontsize=20)


I am working on the file "NN_demo.ipynb".
The first task is:
Some examples of the NN in action
Can you write Python code for it?

def plot_sample_profile(y_true, y_pred, lev):
    """Plots the vertical profiles of input T & q and
       predicted and true output tendencies"""
    f, (ax1, ax3) = plt.subplots(1,2)
    T = nnload.unpack(x, 'T', axis=0)
    q = nnload.unpack(x, 'q', axis=0)
    # Plot temperature tendencies
    ax1.plot(unpack(y_true, 'T', axis=0), lev, color='red' ,
             ls='-' , label=r'$\Delta$T true')
    ax1.plot(unpack(y_pred, 'T', axis=0), lev, color='red' ,
             ls='--', label=r'$\Delta$T pred')
    ax1.set_xlabel(r'$\Delta$Temperature [K/day]')
    # Plot humidity tendencies
    ax3.plot(unpack(y_true, 'q', axis=0), lev, color='blue',
             ls='-' , label=r'$\Delta$q true')
    ax3.plot(unpack(y_pred, 'q', axis=0), lev, color='blue',
             ls='--', label=r'$\Delta$q pred')
    ax3.set_xlabel(r'$\Delta$Humidity [g/kg/day]')
    # Set axis properties
    for ax in [ax1, ax3]:
        ax.set_ylim(1, 0.25)
        ax.legend()
        ax.grid(True)
    fig.suptitle('Sample True and Predicted Outputs', fontsize=20)
samp = np.random.randint(0, x.shape[0])
print('Random sample index value: ' + str(samp))
plot_sample_profile(ytrue_unscl[samp,:], ypred_unscl[samp,:], lev)


I am working on the file "NN_demo.ipynb".
The first task is:
Train the Neural Network
Can you write Python code for it?

def train_nn(mlp,mlp_str,x,y, w=None):
    """Train each item in a list of multi-layer perceptrons and then score
    on test data. Expects that mlp is a list of MLP objects"""
    # Initialize
    start = time.time()
    # Train the model using training data
    mlp.fit(x, y, w)
    train_score = mlp.score(x, y)
    end = time.time()
    print("Training Score: {:.4f} for Model {:s} ({:.1f} seconds)".format(
                                              train_score, mlp_str, end-start))
    # This is an N_iter x 4 array...see score_stats
    errors = np.asarray(errors_stored)
    # Return the fitted models and the scores
    return mlp, errors

# Train neural network
r_mlp, r_errors = train_nn(r_mlp, r_str, x, y)

# Save neural network
pickle.dump([r_mlp, r_str, r_errors, x_ppi, y_ppi, x_pp, y_pp, lat, lev, dlev],
            open('data/regressors/' + r_str + '.pkl', 'wb'))


I am working on the file "NN_example_new/nn-from-scratch.ipynb".
The first task is:
Finally, here comes the function to train our Neural Network. It implements batch gradient descent using the backpropagation derivates we found above.
Can you write Python code for it?

# This function learns parameters for the neural network and returns the model.
# - nn_hdim: Number of nodes in the hidden layer
# - num_passes: Number of passes through the training data for gradient descent
# - print_loss: If True, print the loss every 1000 iterations
def build_model(nn_hdim, num_passes=20000, print_loss=False):

    # Initialize the parameters to random values. We need to learn these.
    np.random.seed(0)
    W1 = np.random.randn(nn_input_dim, nn_hdim) / np.sqrt(nn_input_dim)
    b1 = np.zeros((1, nn_hdim))
    W2 = np.random.randn(nn_hdim, nn_output_dim) / np.sqrt(nn_hdim)
    b2 = np.zeros((1, nn_output_dim))

    # This is what we return at the end
    model = {}

    # Gradient descent. For each batch...
    for i in xrange(0, num_passes):

        # Forward propagation
        z1 = X.dot(W1) + b1
        a1 = np.tanh(z1)
        z2 = a1.dot(W2) + b2
        exp_scores = np.exp(z2)
        probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

        # Backpropagation
        delta3 = probs
        delta3[range(num_examples), y] -= 1
        dW2 = (a1.T).dot(delta3)
        db2 = np.sum(delta3, axis=0, keepdims=True)
        delta2 = delta3.dot(W2.T) * (1 - np.power(a1, 2))
        dW1 = np.dot(X.T, delta2)
        db1 = np.sum(delta2, axis=0)

        # Add regularization terms (b1 and b2 don't have regularization terms)
        dW2 += reg_lambda * W2
        dW1 += reg_lambda * W1

        # Gradient descent parameter update
        W1 += -epsilon * dW1
        b1 += -epsilon * db1
        W2 += -epsilon * dW2
        b2 += -epsilon * db2

        # Assign new parameters to the model
        model = { 'W1': W1, 'b1': b1, 'W2': W2, 'b2': b2}

        # Optionally print the loss.
        # This is expensive because it uses the whole dataset, so we don't want to do it too often.
        if print_loss and i % 1000 == 0:
          print "Loss after iteration %i: %f" %(i, calculate_loss(model))

    return model


I am working on the file "NN_example_new/nn-from-scratch.ipynb".
The first task is:
Generating a dataset
Let's start by generating a dataset we can play with. Fortunately, scikit-learn has some useful dataset generators, so we don't need to write the code ourselves. We will go with the make_moons function.
Can you write Python code for it?

# Generate a dataset and plot it
np.random.seed(0)
X, y = sklearn.datasets.make_moons(200, noise=0.20)
plt.scatter(X[:,0], X[:,1], s=40, c=y, cmap=plt.cm.Spectral)


I am working on the file "NN_example_new/nn-from-scratch.ipynb".
The first task is:
The graph shows the decision boundary learned by our Logistic Regression classifier. It separates the data as good as it can using a straight line, but it's unable to capture the "moon shape" of our data.
Training a Neural Network
Let's now build a 3-layer neural network with one input layer, one hidden layer, and one output layer. The number of nodes in the input layer is determined by the dimensionality of our data, 2. Similarly, the number of nodes in the output layer is determined by the number of classes we have, also 2. (Because we only have 2 classes we could actually get away with only one output node predicting 0 or 1, but having 2 makes it easier to extend the network to more classes later on). The input to the network will be x- and y- coordinates and its output will be two probabilities, one for class 0 ("female") and one for class 1 ("male"). It looks something like this:
<img src='./nn-3-layer-network.png' style='width: 50%'/>
We can choose the dimensionality (the number of nodes) of the hidden layer. The more nodes we put into the hidden layer the more complex functions we will be able fit. But higher dimensionality comes at a cost. First, more computation is required to make predictions and learn the network parameters. A bigger number of parameters also means we become more prone to overfitting our data.
How to choose the size of the hidden layer? While there are some general guidelines and recommendations, it always depends on your specific problem and is more of an art than a science. We will play with the number of nodes in the hidden later later on and see how it affects our output.
We also need to pick an activation function for our hidden layer. The activation function transforms the inputs of the layer into its outputs. A nonlinear activation function is what allows us to fit nonlinear hypotheses. Common chocies for activation functions are tanh, the sigmoid function, or [ReLUs](https://en.wikipedia.org/wiki/Rectifier_(neural_networks). We will use tanh, which performs quite well in...
Can you write Python code for it?

num_examples = len(X) # training set size
nn_input_dim = 2 # input layer dimensionality
nn_output_dim = 2 # output layer dimensionality

# Gradient descent parameters (I picked these by hand)
epsilon = 0.01 # learning rate for gradient descent
reg_lambda = 0.01 # regularization strength


I am working on the file "NN_example_new/nn-from-scratch.ipynb".
The first task is:
We also implement a helper function to calculate the output of the network. It does forward propagation as defined above and returns the class with the highest probability.
Can you write Python code for it?

# Helper function to predict an output (0 or 1)
def predict(model, x):
    W1, b1, W2, b2 = model['W1'], model['b1'], model['W2'], model['b2']
    # Forward propagation
    z1 = x.dot(W1) + b1
    a1 = np.tanh(z1)
    z2 = a1.dot(W2) + b2
    exp_scores = np.exp(z2)
    probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)
    return np.argmax(probs, axis=1)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Binary Threshold Neuron Model
Decision unit of output $y$ from weighted sum $z$.$^{[2,3]}$
$$
y = \left{
\begin{array}{ll}
1 & \mbox{if $z \geq 0$}\
0 & \mbox{otherwise}
\end{array}
\right.
$$
Output clustering $y$ using binary threshold neuron model.
Can you write Python code for it?


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Combined Examples and Labels
Combined both negative and positive examples into $x$.
Can you write Python code for it?

X = np.vstack([neg_examples, pos_examples])

X


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Construct negative examples with bias.
Can you write Python code for it?

neg_examples = np.hstack([neg_examples_nobias, np.ones((ttl_neg_examples, 1))])

neg_examples


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Construct negative labels.
Can you write Python code for it?

neg_labels = np.zeros(ttl_neg_examples, dtype=bool)

neg_labels


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Construct positive examples with bias.
Can you write Python code for it?

pos_examples = np.hstack([pos_examples_nobias, np.ones((ttl_pos_examples, 1))])

pos_examples


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Construct positive labels.
Can you write Python code for it?

pos_labels = np.ones(ttl_pos_examples, dtype=bool)

pos_labels


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Evaluate cost/error/mistake $J$. True means the clustering result agrees with given label. Whereas, False means the opposite.
Can you write Python code for it?


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Evaluate perceptron using feasible weight.
Can you write Python code for it?

J = evaluate_perceptron(X, w_gen_feas, labels)
plot_classification_boundary(X, w_gen_feas, J, labels)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Evaluate perceptron using initial weight.
Can you write Python code for it?

J = evaluate_perceptron(X, w, labels)
plot_classification_boundary(X, w, J, labels)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
First iteration.
Can you write Python code for it?

w = update_weights(X, w, labels)
J = evaluate_perceptron(X, w, labels)
plot_classification_boundary(X, w, J, labels)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Generously Feasible Weight with Bias
Provided or expected feasible wight with bias for comparison purposes.
Can you write Python code for it?

w_gen_feas = dataset.get('w_gen_feas', [])
if len(w_gen_feas):
    w_gen_feas = np.transpose(w_gen_feas)[0]

w_gen_feas


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Get weight from initial weight or random generator.
Can you write Python code for it?

if w_init is not None:
    w = np.copy(np.transpose(w_init))[0]
else:
    w = np.random.random_sample(ttl_features + 1)

w


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Negative Examples and Labels
Multiple examples of two input variables (without bias) under negative class (target output is 0).
Can you write Python code for it?

neg_examples_nobias = dataset['neg_examples_nobias']

neg_examples_nobias

ttl_neg_examples = neg_examples_nobias.shape[0]
print("Total negative examples: ", ttl_neg_examples)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Overall perceptron evaluation function.
Can you write Python code for it?

def evaluate_perceptron(X, w, labels):
    Z = np.dot(X, w)  # weighted sum
    Y = Z >= 0        # binary threshold clustering
    J = Y == labels   # cost function

    return J


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Plot training history.
Can you write Python code for it?

def plot_training_history(ttl_errors_history, w_dist_history):
    iteration_vector = np.arange(1, len(ttl_errors_history) + 1)

    fig, ax1 = plt.subplots()
    ax1.set_xlabel('Iteration')

    if len(w_dist_history):
        ax1.plot(iteration_vector, w_dist_history, marker='o', color='blue')
        ax1.set_ylabel('Distance', color='blue')

    ax2 = ax1.twinx()
    ax2.plot(iteration_vector, ttl_errors_history, marker='*', color='red')
    ax2.set_ylabel('Number of errors', color='red')

    plt.show()


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Positive Examples and Labels
Multiple examples of two input variables (without bias) under positive class (target output is 1).
Can you write Python code for it?

pos_examples_nobias = dataset['pos_examples_nobias']

pos_examples_nobias

ttl_pos_examples = pos_examples_nobias.shape[0]
print("Total positive examples: ", ttl_pos_examples)


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Read Dataset
Can you write Python code for it?

def read_dataset(filename):

    dataset = sio.loadmat(filename)

    ttl_features = dataset['neg_examples_nobias'].shape[1]

    neg_examples_nobias = dataset['neg_examples_nobias']
    ttl_neg_examples = neg_examples_nobias.shape[0]
    neg_examples = np.hstack([neg_examples_nobias, np.ones((ttl_neg_examples, 1))])
    neg_labels = np.zeros(ttl_neg_examples, dtype=bool)

    pos_examples_nobias = dataset['pos_examples_nobias']
    ttl_pos_examples = pos_examples_nobias.shape[0]
    pos_examples = np.hstack([pos_examples_nobias, np.ones((ttl_pos_examples, 1))])
    pos_labels = np.ones(ttl_pos_examples, dtype=bool)

    X = np.vstack([neg_examples, pos_examples])
    labels = np.hstack([neg_labels, pos_labels])

    w_init = dataset.get('w_init', None)
    if w_init is not None:
        w = np.copy(np.transpose(w_init))[0]
    else:
        w = np.random.random_sample(ttl_features + 1)

    w_gen_feas = dataset.get('w_gen_feas', [])
    if len(w_gen_feas):
        w_gen_feas = np.transpose(w_gen_feas)[0]

    return ttl_features, X, labels, w, w_gen_feas


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Second iteration.
Can you write Python code for it?


I am working on the file "NNfML-W3/Perceptron.ipynb".
The first task is:
Training
Can you write Python code for it?

def perform_perceptron_training(dataset_filename, max_iteration):
    ttl_features, X, labels, w, w_gen_feas = read_dataset(dataset_filename)
    ttl_labels = len(labels)

    J = evaluate_perceptron(X, w, labels)
    ttl_errors = ttl_labels - np.count_nonzero(J)
    print("Number of errors in iteration %d: %2d" % (0, ttl_errors))
    plot_classification_boundary(X, w, J, labels)

    ttl_errors_history = []
    w_dist_history = []
    for i in range(1, max_iteration):
        w = update_weights(X, w, labels)
        J = evaluate_perceptron(X, w, labels)
        ttl_errors = ttl_labels - np.count_nonzero(J)
        print("Number of errors in iteration %d: %2d" % (i, ttl_errors))
        ttl_errors_history.append(ttl_errors)
        if len(w_gen_feas):
            w_dist_history.append(np.linalg.norm(w - w_gen_feas))
        plot_classification_boundary(X, w, J, labels)
        if ttl_errors <= 0:
            break

    return ttl_errors_history, w_dist_history

ttl_errors_history, w_dist_history = perform_perceptron_training('dataset1.mat', 21)

plot_training_history(ttl_errors_history, w_dist_history)


I am working on the file "NOAA_sandbox.ipynb".
The first task is:
CSV Generator
Can you write Python code for it?

def gen_csv(df, query_dict):
    """
    Arguments: PANDAS DataFrame, a query parameters dictionary
    Returns: A CSV of the df with dropped index and named by dict params
    """

    # extract params
    station= query_dict['stationid']
    start= query_dict['startdate']
    end= query_dict['enddate']

    # using os.path in case of future expansion to other directories
    path= os.path.join(station + '_' + start + '_' + end + '.' + 'csv')

    # remove problem characters (will add more in future)
    exclude_chars= ':'
    path= path.replace(exclude_chars, "_")

    # export to csv

    my_csv= df.to_csv(path, index= False)

    return my_csv, path

stuff, path= gen_csv(test, query_dict)

path

ls *csv

#!/usr/bin/env python
# coding: utf-8

"""Python code for querying NOAA daily summary weather and returnig a CSV per year
for a specfic station.  Code is intended to be executed from CLI."""

import sys

# set path to tools library and import
sys.path.append(r'noaa_weather_tools')
import noaa_weather_tools

NOAA_Token_Here= 'enter token as string'

print("Check dt format('DD-MM-YYYY', and whether dates span <= 1 year from a current or past date")
print("If dates exceed one year, NCDC query returns a null object")
print("Need a token take a token, have a token, keep it to yourself @ https://www.ncdc.noaa.gov/cdo-web/token")
print('start_dt: {}\n end_dt: {}'.format(sys.argv[1], sys.argv[2]))

def noaa_dailysum_weather_processor(start_dt, end_dt, station):

    """Function accepts a station ID, and beginning/end datetime as strings with date format as
    'MM-DD-YYYY' which span <= 1 year from a current or past date, passing them to the query_builder function.
    Function creates a .csv file of NOAA (NCDC) Daily Summary data for a specific station."""

    print(15 * '.' + "reticulating splines" + 5* '.' + "getting records")
    df, record_count, query_parameters= noaa_weather_tools.get_ncdc(start_dt, end_dt, station)

    print(15* '.' + "exporting to...


I am working on the file "NOAA_sandbox.ipynb".
The first task is:
Query Generator
TODO

refactor with a decorator
make key an attribute that can be hidden
Can you write Python code for it?

def execute_query(query):

    """
    Function accepts an NOAA query for daily summaries for a specfic location
    and executes the query.
    Function returns a response (JSON)
    """
    url = query
    # replace token with token provided by NOAA.  Enter token as string
    headers = {'token': NOAA_Token_Here} # https://www.ncdc.noaa.gov/cdo-web/token
    response = requests.get(url, headers = headers)
    response = response.json()

    return response

working_1= execute_query(query_1)['results']
working_2 = execute_query(query_2)['results']


I am working on the file "NOTEBOOKS/.ipynb_checkpoints/Ej2_Estatica-checkpoint.ipynb".
The first task is:
Lo que queda ahora es resolver el sistema de ecuaciones de equilibrio. Para ello lo escribimos de la forma:
$$AX = B$$
Can you write Python code for it?


I am working on the file "NOTEBOOKS/.ipynb_checkpoints/Ej2_Estatica-checkpoint.ipynb".
The first task is:
Nota. Problema tomado de Mecánica Vectorial para Ingenieros. Estatica. Sexta Edición. Beer Ferninand;  Johnston, Russell.
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open('./custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "NOTEBOOKS/.ipynb_checkpoints/Ej2_Estatica-checkpoint.ipynb".
The first task is:
Por medio de un par de cables se quiere sostener un bloque de peso $W = 200\; kgf$. Determine la tensión en cada cuerda si las coordenadas de posición de los puntos A, B, C son A(0, -20 cm, 40 cm), B(-40 cm, 50 cm, 0cm)  y C(45 cm, 40 cm, 0cm), respectivamente. Suponga que no hay fricción entre la rampa y el contrapeso.
Can you write Python code for it?


I am working on the file "NOTEBOOKS/.ipynb_checkpoints/Ej2_Estatica-checkpoint.ipynb".
The first task is:
Resolviendo el sistema
Can you write Python code for it?

X = dot(inv(A),B)
print '[TB, TC, N]=', X.round(2)


I am working on the file "NOTEBOOKS/Ej1_EcPlano.ipynb".
The first task is:
De esta forma, la ecuación de plano es:
Can you write Python code for it?

N = str(N[0]) + 'x + ' + str(N[1]) + 'y + ' + str(N[2]) + 'z = ' + str(-D)

print N

from IPython.core.display import HTML
def css_styling():
    styles = open('./custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "NOTEBOOKS/Ej1_EcPlano.ipynb".
The first task is:
Primero determinemos el vector posición $\vec{r_{1}}$, $\vec{r_{2}}$ y $\vec{r_{3}}$ de cada punto:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej1_EcPlano.ipynb".
The first task is:
Via producto cruz se determina el vector normal al plano
$$ \vec{N} = \vec{A} \times \vec{B} $$
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej2_Estatica.ipynb".
The first task is:
Lo que queda ahora es resolver el sistema de ecuaciones de equilibrio. Para ello lo escribimos de la forma:
$$AX = B$$
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej2_Estatica.ipynb".
The first task is:
Nota. Problema tomado de Mecánica Vectorial para Ingenieros. Estatica. Sexta Edición. Beer Ferninand;  Johnston, Russell.
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej2_Estatica.ipynb".
The first task is:
Por analogía, la fuerza normal prodría N podría ser escrita como:
$\vec{N} = N \; \hat{n_p} $,
donde las componentes de $\hat{n_p} $ son determinadas por la inclinación de la rampa. De esta forma.
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej2_Estatica.ipynb".
The first task is:
Por medio de un par de cables se quiere sostener un bloque de peso $W = 200\; kgf$. Determine la tensión en cada cuerda si las coordenadas de posición de los puntos A, B, C son A(0, -20 cm, 40 cm), B(-40 cm, 50 cm, 0cm)  y C(45 cm, 40 cm, 0cm), respectivamente. Suponga que no hay fricción entre la rampa y el contrapeso.
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej2_Estatica.ipynb".
The first task is:
Si llamamos $\vec{T_{B}}$ y $\vec{T_{C}}$ la tensión en el cable AB y AC (incógnitas) respectivamente, $\vec{W}$ el peso del elemento y $\vec{N}$ el vector normal al plano de la rampa, y hacemos equilibro sobre el bloque, se tiene:
$$ \begin{align}
&\sum \; F_{X} = T_{BX} + T_{CX} = 0\
&\sum \; F_{Y} = - W + T_{BY} + T_{CY} + N_{Y} = 0\
&\sum \; F_{Z} =  T_{BZ} + T_{CZ} + N_{Z} = 0
\end{align}$$
Por otro lado, los vectores de tensión pueden ser escritos como:
$\vec{T} = T \; \hat{n} $ , donde $\hat{n}$ es un vector director unitario.
De esta forma para la tensión en el cable AB:
$\vec{T_B} = T_B \; \hat{n_B} = T_B \;\frac{\vec{R_{AB}}}{MR_{AB}} $
Can you write Python code for it?

from numpy import array, sqrt, cross, dot, arctan, cos, sin
from scipy.linalg import inv

R_AB = [-40., 70., -40.]
MR_AB = sqrt(dot (R_AB,R_AB))
nB = R_AB / MR_AB
print 'Magnitud =', MR_AB, ',', 'nB =', nB.round(2)


I am working on the file "NOTEBOOKS/Ej3_Cauchy.ipynb".
The first task is:
2.Calcular el esfuerzo normal al que está sometida la cara del punto $P$, que es paralela al plano que contiene los puntos $A$ , $B$ y $C \\$.
Solución:
El esfuerzo normal no es más que la proyección escalar de $\vec{t}$ sobre el vector normal unitario $\hat{n}$. Es otras palabras $\sigma_{nn} = {\vec{t} \cdot \hat{n}}$
Can you write Python code for it?

Sigmann = dot(t,n)
print Sigmann.round(2)


I am working on the file "NOTEBOOKS/Ej3_Cauchy.ipynb".
The first task is:
3.Calcular la magnitud del esfuerzo tangencial al que está sometida la cara del punto $P$, que es paralela al plano que contiene los puntos $A$ , $B$ y $C \\$
Solución
Teniendo el vector $\vec{t}$ y su componente normal la componente normal $\sigma_{nn}$, la magnitud de la compenente tangencial $\tau$, se calcularía a partir de:
$\|\vec{t}\|^2 =  \|\sigma_{nn}\|^2 + \|\tau\|^2$
Despejando $\tau$, se obtiene:
Can you write Python code for it?

tau = sqrt(dot(t,t) - Sigmann**2)
print tau.round(2)

from IPython.core.display import HTML
def css_styling():
    styles = open('./custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "NOTEBOOKS/Ej3_Cauchy.ipynb".
The first task is:
Con esto, el vector de tracciones $\vec{t}$ sería entonces:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej3_Cauchy.ipynb".
The first task is:
El vector unitarion $\hat{n}$ es entonces obtenido como $$ \vec{n} = \frac{\vec{AB} \times \vec{AC}}{\|\vec{AB} \times \vec{AC}\|} $$
Can you write Python code for it?

Nor = cross(AB,AC)
MNor = sqrt(dot(Nor,Nor))
n = Nor / MNor
print 'Normal=',Nor, ',' 'Magnitud',MNor


I am working on the file "NOTEBOOKS/Ej3_Cauchy.ipynb".
The first task is:
Por otro lado, el tensor $[\sigma]$ sería
Can you write Python code for it?


I am working on the file "NOTEBOOKS/Ej4_Eingen.ipynb".
The first task is:
De esta manera escribamos en tensor asociado a las direcciones principales:
Can you write Python code for it?

print
Sp = np.array([
    [la[0],0,0],
    [0,la[1],0],
    [0,0,la[2]]])
print 'Sp =',np.round(Sp,1)
print
Image(filename='FIGURES/Sprinc.png',width=400)


I am working on the file "NOTEBOOKS/Ej4_Eingen.ipynb".
The first task is:
Los vectores $i'$, $j'$ y $k'$ están dados por:
Can you write Python code for it?

print "i'=", np.round(n[:,0],2)
print "j'=", np.round(n[:,1],2)
print "k'=", np.round(n[:,2],2)
print


I am working on the file "NOTEBOOKS/Ej4_Eingen.ipynb".
The first task is:
Resolviendo vía polinomio característico:
Can you write Python code for it?

coeff=[1.0,-IS,IIS,-IIIS]
ps=np.roots(coeff)

print
print "Esfuerzos principales:", np.sort(np.round(ps,1))
print


I am working on the file "NOTEBOOKS/Ej4_Eingen.ipynb".
The first task is:
Verifiquemos que se cumplen los invariantes en el tensor asociado a direcciones principales:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Ahora determinemos la componente normal de $\vec{t}$ sobre el plano inclinado usando:
$${\sigma _n} = \vec t \cdot \hat n$$
Can you write Python code for it?

magt = sqrt(t.dot(t))
signn = dot(t,n)
print signn


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Ahora, para examinar el bloque este fue cortado como se muestra en la figura 3:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Con el cual es posible calcular el vector de tracciones sobre el plano con vector normal $n$ usando la formula de Cauchy:
$$\vec t = \left[ \sigma  \right] \cdot \hat n$$
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Determine cual es el tiempo mínimo que se debe esperar tras aplicar el pegamento para poder colocar el bloque nuevamente en el muro sin que falle.
Solución:
Se determina inicialmente el vector normal al plano de corte para lo cual es conveniente seleccionar 3 puntos contenidos en el plano. En este caso es también conveniente fijar el origen en la esquina inferior izquierda (ver figura 3), y tomar desde allí vectores posición a 2 puntos sobre el plano. Marcando estos puntos como $A$ y $B$ se tiene:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Notando que la tensión normal es a compresión se concluye que la falla se presenta a corte. Usando la ecuación:
$$\tau = 7t$$
se tiene que el tiempo para montar la pieza sin que se produzca la falla es:
Can you write Python code for it?

t = tau/7
print t

from IPython.core.display import HTML
def css_styling():
    styles = open('./custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "NOTEBOOKS/PROBLEMA_3_12.ipynb".
The first task is:
Y finalmente para determinar la tensión a cortante se usa:
$${\tau ^2} = \vec t \cdot \vec t - \sigma _n^2$$
Can you write Python code for it?

tausq = dot(t,t)-signn*signn
tau = sqrt(tausq)
print tau


I am working on the file "NOTEBOOKS/PROBLEMA_3_7.ipynb".
The first task is:
Se calcula ahora la componente normal y tangencial de acuerdo con:
$${\sigma} = \vec t \cdot \hat n$$
$${\tau ^2} = \vec t \cdot \vec t - {\sigma ^2}$$
Can you write Python code for it?

magt = dot(t,t)
signn = dot(t,n)
taus =sqrt(magt-signn*signn)
print signn , sqrt(magt) , taus

from IPython.core.display import HTML
def css_styling():
    styles = open('./custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "NOTEBOOKS/PROBLEMA_3_7.ipynb".
The first task is:
Solución:
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_3_7.ipynb".
The first task is:
con lo cual es posible calcular el vector de tracciones usando la formula de Cauchy:
$$\vec t = \left[ \sigma  \right] \cdot \hat n$$
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
Con lo que se concluye que la tubería B falla primero
 f) Tarea
* ¿Como sería la matriz de transformación que llevaría el tensor desde el sistema de referencia de la carga $P$ al sistema de la carga $W$?
* Hacer el ejercicio de transformación sin considerar el hecho que el estado de tensiones debido a la carga $W$ es invariante a rotación
Can you write Python code for it?


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
Determine cuál tubería falla primero y en que tiempo se produce la falla si se sabe que el material de la tubería A es infinitamente resistente a esfuerzos cortantes pero no soporta esfuerzos axiales mayores o iguales a $210,5\dfrac{Ton}{m^2}$  y el material de la tubería B es infinitamente resistente a esfuerzos axiales pero no sorporta esfuerzos cortantes mayores o iguales a $100,25\dfrac{Ton}{m^2}$.
Definicion de variables:

Solución:
Definicion de variables:
Can you write Python code for it?

P = 100
Ha = 1.0 / np.pi
Hb = 0.75 * Ha
Wo = 100
print 'Ha=', np.round(Ha,2), 'Hb=', np.round(Hb,2)


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
Para la tubería B
Can you write Python code for it?

sigmaWPb = sigmaW + sigmaPb

print 'sigmaWPb='
print
print sigmaWPb


tetab = np.arctan(Ha / Hb)
Rb = Ha / np.sin(tetab)
sigmaPb = FacP / Rb * np.array([[np.cos(tetab),0,0],[0,0,0],[0,0,0]])

print
print 'sigmaPb ='
print
print sigmaPb


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
a) Definición de estado de esfuerzo (tensión) debidos a la carga $P$

Para la tubería A
Can you write Python code for it?

FacP = -2.0 * P / (np.pi)
Ra = np.sqrt(2.0) * Ha         # Distancia al punto
tetaa = np.pi / 4.0            # Angulo np.

sigmaPa = FacP / Ra * np.array([[np.cos(tetaa),0,0],[0,0,0],[0,0,0]])
print
print 'sigmaPa ='
print sigmaPa


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
b) Definición del estado de tension debido la carga $w$

Para la tubería A y B

Si se revisan las funciones de esfuerzo se puede ver que como en ángulo $\theta$ es el mismo para ambas tuberías la solución también es la misma. En este caso $\theta=0^{\circ}$
Can you write Python code for it?

W = Symbol('W')

tetaW = 0                       # Angulo
sigmaW = -W / np.pi * np.array([[(np.pi) + 2.0*tetaW - np.sin(2.0*tetaW), 1.0 - np.cos(2.0 * tetaW),0],[1.0 - np.cos(2.0 * tetaW),np.pi + 2.0*tetaW + np.sin(2.0*tetaW),0],[0,0,0]])
print
print 'sigmaW='
print
print (sigmaW)


I am working on the file "NOTEBOOKS/PROBLEMA_4_16.ipynb".
The first task is:
c) Superposición de cargas
Para hacer la superposición de las cargas es necesario que ambas estén asociadas a las mismas caras perpendiculares. En otras palabras es necesario que estén en el mismo sistema de referencia. Para hacer esto se debe considerar transformar por rotación un estado de esfuerzos para llevarlo a la dirección del otro. No obstante, si se observa el tensor de la carga $w$ se puede notar que en el plano $xy$ (plano a considerar la rotación) es invariante ante rotación. Esto debido a que en ese plano ambos sigmas son iguales lo que origina que el círculo de Mohr es un punto. Con esta aclaración se tiene que la superposición en este caso se puede obtener sumando ambos tensores de forma directa.

Para la tubería A
Can you write Python code for it?

sigmaWPa = sigmaW + sigmaPa

print 'sigmaWPa='
print
print sigmaWPa