I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
less than
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
less than or equal to
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
top
Checking if an item is in a list
You can check if an item is in a list using the in keyword.
Can you write Python code for it?

vowels = ['a', 'e', 'i', 'o', 'u']
'a' in vowels

vowels = ['a', 'e', 'i', 'o', 'u']
'b' in vowels


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
top
Inequality
Two items are inequal if they do not have the same value. In Python, we test for inequality using the exclamation point and one equals sign.
Sometimes you want to test for equality and if that fails, assume inequality. Sometimes it makes more sense to test for inequality directly.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/if_statements-checkpoint.ipynb".
The first task is:
top
Other Inequalities
greater than
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Add Library to Python path
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Export environment variable
export INDIC_RESOURCES_PATH=<path>

 OR

set it programmatically
We will use that method for this demo
Can you write Python code for it?

from indicnlp import common
common.set_resources_path(INDIC_NLP_RESOURCES)


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Get Phonetic Similarity
Using the phonetic feature vectors, we can define phonetic similarity between the characters (and underlying phonemes). The library implements some  measures for phonetic similarity between the characters (and underlying phonemes). These can be defined using the phonetic feature vectors discussed earlier, so users can implement additional similarity measures.
The implemented similarity measures are:

cosine
dice
jaccard
dot_product
sim1 (Kunchukuttan et al., 2016)
softmax

 References
Anoop Kunchukuttan, Pushpak Bhattacharyya, Mitesh Khapra. Substring-based unsupervised transliteration with phonetic and contextual knowledge. SIGNLL Conference on Computational Natural Language Learning  (CoNLL 2016) . 2016.
Can you write Python code for it?

from indicnlp.script import  indic_scripts as isc
from indicnlp.script import  phonetic_sim as psim

c1=u'क'
c2=u'ख'
c3=u'भ'
lang='hi'

print u'Similarity between {} and {}'.format(c1,c2)
print psim.cosine(
    isc.get_phonetic_feature_vector(c1,lang),
    isc.get_phonetic_feature_vector(c2,lang)
    )

print

print u'Similarity between {} and {}'.format(c1,c3)
print psim.cosine(
    isc.get_phonetic_feature_vector(c1,lang),
    isc.get_phonetic_feature_vector(c3,lang)
    )


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Initialize the Indic NLP library
Can you write Python code for it?

from indicnlp import loader
loader.load()


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Let's actually try out some of the API methods in the Indic NLP library
Many of the API functions require a language code. We use 2-letter ISO 639-1 codes. Some languages do not have assigned 2-letter codes. We use the following two-letter codes for such languages:

Konkani: kK
Manipuri: mP
Bodo: bD

Text Normalization
Text written in Indic scripts display a lot of quirky behaviour on account of varying input methods, multiple representations for the same character, etc.
There is a need to canonicalize the representation of text so that NLP applications can handle the data in a consistent manner. The canonicalization primarily handles the following issues:
- Non-spacing characters like ZWJ/ZWNL
- Multiple representations of Nukta based characters
- Multiple representations of two part dependent vowel signs
- Typing inconsistencies: e.g. use of pipe (|) for poorna virama
Can you write Python code for it?

from indicnlp.normalize.indic_normalize import IndicNormalizerFactory

input_text=u"\u0958 \u0915\u093c"
remove_nuktas=False
factory=IndicNormalizerFactory()
normalizer=factory.get_normalizer("hi",remove_nuktas)
output_text=normalizer.normalize(input_text)

print output_text
print 'Length before normalization: {}'.format(len(input_text))
print 'Length after normalization: {}'.format(len(output_text))


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Romanization
Convert script text to Roman text in the ITRANS notation
Can you write Python code for it?

from indicnlp.transliterate.unicode_transliterate import ItransTransliterator

input_text=u'राजस्थान'
lang='hi'

print ItransTransliterator.to_itrans(input_text,lang)


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Script Conversion
Convert from one Indic script to another. This is a simple script which exploits the fact that Unicode points of various Indic scripts are at corresponding offsets from the base codepoint for that script. The following scripts are supported:
Devanagari (Hindi,Marathi,Sanskrit,Konkani,Sindhi,Nepali), Assamese, Bengali, Oriya, Gujarati, Gurumukhi (Punjabi), Sindhi, Tamil, Telugu, Kannada, Malayalam
Can you write Python code for it?

from indicnlp.transliterate.unicode_transliterate import UnicodeIndicTransliterator
input_text=u'राजस्थान'
print UnicodeIndicTransliterator.transliterate(input_text,"hi","pa")


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Script Information
Indic scripts have been designed keeping phonetic principles in nature and the design and organization of the scripts makes it easy to obtain phonetic information about the characters.
Get Phonetic Feature Vector
With each script character, a phontic feature vector is associated, which encodes the phontic properties of the character. This is a bit vector which is can be obtained as shown below:
Can you write Python code for it?

from indicnlp.script import  indic_scripts as isc

c=u'क'
lang='hi'

isc.get_phonetic_feature_vector(c,lang)


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Some similarity functions like sim do not generate values in the range [0,1] and it may be more convenient to have the similarity values in the range [0,1]. This can be achieved by setting the normalize paramter to True
Can you write Python code for it?

slang='hi'
tlang='ml'
sim_mat=psim.create_similarity_matrix(psim.sim1,slang,tlang,normalize=True)

c1=u'क'
c2=u'ഖ'
print u'Similarity between {} and {}'.format(c1,c2)
print sim_mat[isc.get_offset(c1,slang),isc.get_offset(c2,tlang)]


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
This fields in this bit vector are (from left to right):
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Tokenization
A trivial tokenizer which just tokenizes on the punctuation boundaries. This also includes punctuations for the Indian language scripts (the purna virama and the deergha virama). It returns a list of tokens.
Can you write Python code for it?

from indicnlp.tokenize import indic_tokenize

indic_string=u'अनूप,अनूप?।फोन'

print u'Input String: {}'.format(indic_string)
print u'Tokens: '
for t in indic_tokenize.trivial_tokenize(indic_string):
    print t


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Transliteration
We use the BrahmiNet REST API for transliteration.
Can you write Python code for it?

import urllib2
from django.utils.encoding import *
from django.utils.http import *

text=iri_to_uri(urlquote('anoop, ratish kal fone par baat karenge'))
url=u'http://www.cfilt.iitb.ac.in/indicnlpweb/indicnlpws/transliterate_bulk/en/hi/{}/statistical'.format(text)

response=urllib2.urlopen(url).read()
print response


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
Word Segmentation
Unsupervised morphological analysers for various Indian language. Given a word, the analyzer returns the componenent morphemes.
The analyzer can recognize inflectional and derivational morphemes.
The following languages are supported:
Hindi, Punjabi, Marathi, Konkani, Gujarati, Bengali, Kannada, Tamil, Telugu, Malayalam
Support for more languages will be added soon.
Can you write Python code for it?

from indicnlp.morph import unsupervised_morph
from indicnlp import common

analyzer=unsupervised_morph.UnsupervisedMorphAnalyzer('mr')

indic_string=u'आपल्या हिरड्यांच्या आणि दातांच्यामध्ये जीवाणू असतात .'

analyzes_tokens=analyzer.morph_analyze_document(indic_string.split(' '))

for w in analyzes_tokens:
    print w


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
You can check the phonetic information database files in Indic NLP resources to know the definition of each of the bits.

For Tamil Script: database
For other Indic Scripts: database

Query Phonetic Properties
Note: This interface below will be deprecated soon and a new interface will be available soon.
Can you write Python code for it?

from indicnlp.langinfo import *

c=u'क'
lang='hi'

print 'Is vowel?:  {}'.format(is_vowel(c,lang))
print 'Is consonant?:  {}'.format(is_consonant(c,lang))
print 'Is velar?:  {}'.format(is_velar(c,lang))
print 'Is palatal?:  {}'.format(is_palatal(c,lang))
print 'Is aspirated?:  {}'.format(is_aspirated(c,lang))
print 'Is unvoiced?:  {}'.format(is_unvoiced(c,lang))
print 'Is nasal?:  {}'.format(is_nasal(c,lang))


I am working on the file ".ipynb_checkpoints/indic_nlp_examples-checkpoint.ipynb".
The first task is:
You may have figured out that you can also compute similarities of characters belonging to different scripts.
You can also get a similarity matrix which contains the similarities between all pairs of characters (within the same script or across scripts).
Let's see how we can compare the characters across Devanagari and Malayalam scripts
Can you write Python code for it?

from indicnlp.script import  indic_scripts as isc
from indicnlp.script import  phonetic_sim as psim

slang='hi'
tlang='ml'
sim_mat=psim.create_similarity_matrix(psim.cosine,slang,tlang,normalize=False)

c1=u'क'
c2=u'ഖ'
print u'Similarity between {} and {}'.format(c1,c2)
print sim_mat[isc.get_offset(c1,slang),isc.get_offset(c2,tlang)]


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Add / Delete a note   ↑
Can you write Python code for it?

#add or modify a note
def add_note(wit, rowID, token, note):
    try:
        #select token
        t = collation[rowID][witnesses.index(wit)][token]
        if note is '':
            print('Your note is empty.')
        elif 'note' in t:
            #add comment to an already existing note
            t['note'] += ' '+note
        else:
            #or create a new note
            t['note'] = note
    except:
        print('This token is not valid.')

#delete completely a token's note
def del_note(wit, rowID, token):
    try:
        #select token
        t = collation[rowID][witnesses.index(wit)][token]
        if 'note' in t:
            #delete note
            t.pop('note')
        else:
            #or print error message
            print('There is no note to delete')
    except:
        print('This token is not valid.')


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Add / delete a row
Can you write Python code for it?

def add_row_after(rowID):
    #rowID must be within collation table
    if rowID < 0 or rowID > len(collation)-1:
        print('Row '+str(rowID)+' does not exist.')
    else:
        #create an empty row
        new_row = []
        #for each witness in the collation
        for wit in witnesses:
            #add an empty list of tokens to the row
            new_row.append([])
        #insert new row in the collation, after the row passed in argument (+1)
        collation.insert(rowID+1, new_row)
        print('Row added!')

def delete_row(rowID):
    #rowID must be within collation table
    if rowID < 0 or rowID > len(collation)-1:
        print('Row '+str(rowID)+' does not exist.')
    elif
    else:
        collation.pop(rowID)
        print('Row deleted!')


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Clarify one reading  ↑
Can you write Python code for it?

#Examples: 459/C1, 932/M1, 9/LH
#uncomment the next line to see the widget
#interact(print_info, rowID=w_rowID, wit=w_wit)


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Compare cells
Can you write Python code for it?

#compare two cells, original text
def compare_cell(c1,c2):
    return cell_to_string(c1) == cell_to_string(c2)

#compare two cells, normalized text
def compare_cell_norm(c1,c2):
    return cell_to_string_norm(c1) == cell_to_string_norm(c2)

#compare a list of cells, original text
#return true if all the cells are equivalent (they contain the same string of tokens)
def compare_multiple_cell(cell_list):
    #compare each cell to the next
    for c1,c2 in zip(cell_list, cell_list[1:]):
        if compare_cell(c1,c2) is False:
            comparison = False
            break
    else:
        comparison = True
    return comparison

#compare a list of cells, normalized text
#return true if all the cells are equivalent (they contain the same string of tokens)
def compare_multiple_cell_norm(cell_list):
    #compare each cell to the next
    for c1,c2 in zip(cell_list, cell_list[1:]):
        if compare_cell_norm(c1,c2) is False:
            comparison = False
            break
    else:
        comparison = True
    return comparison


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
FUNCTIONS  ↑
Transform a cell c (list of tokens) into a string of text
Can you write Python code for it?

#original text
def cell_to_string(c):
    #tokens t are joined together, separated by a space
    string = ' '.join(token['t'] for token in c)
    return string

#text with normalized tokens
def cell_to_string_norm(c):
    string = ''
    #word division is not taken into account when comparing the normalized text
    #for this reason we do not add a space in between tokens
    for token in c:
        if 'n' in token:
            string += token['n']
        elif 't' in token:
            string += token['t']
    return string


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Find all variants in the collation table
Can you write Python code for it?

def view_variants(table):
    result_table = []
    #go through the collation table, row by row
    for row in table:
        #if there is a variant in the row (i.e. at least one cell is different from another cell, normalized form)
        if compare_multiple_cell_norm(row) is False:
            #add row to the result
            result_table.append(row)
    return result_table


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Get the ID of a row
Can you write Python code for it?

def get_pos(row):
    return collation.index(row)


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
INTERACTIVE EXPLORATION OF THE COLLATION TABLE  ↑
Update the collation results  ↑
In this section, we will see how to view a selection of the collation table and update it.
Possible updates:
  1. Add/delete a row
  3. Move one token up/down
  4. Add notes
  3. Save
Note : it is only possible to move the last token (or word) in a cell down to the first place in the next cell; or vice-versa to move the first token of a cell up to the last place in the previous cell. This is to prevent any change in the word order of a witness, and to keep the correct text sequence.
Can you write Python code for it?

#select an extract of the collation table with interactive widgets

#widget for HTML display
w1_html = widgets.HTML(value="")

#define the beginning of extract
w_from = widgets.BoundedIntText(
    value=6,
    min=0,
    max=len(collation)-1,
    description='From:',
    continuous_update=True,
)

#define the end of extract
#because of python list slicing, the last number is not included in the result.
#to make it more intuitive, the "to" number is added +1 in collation_extract function
w_to = widgets.BoundedIntText(
    value=11,
    min=0,
    max=len(collation)-1,
    description='To:',
    continuous_update=True,
)

#binding widgets with table_to_html function
def collation_extract(a, b):
    x = a
    y = b+1
    if y <= x:
        print("The table you have requested does not exist.")
    w1_html.value = table_to_html(collation,collation[x:y])

#uncomment the next lines to see the widgets
##interactive selection of a collation table extract
#interact(collation_extract, a=w_from, b=w_to)
##display HTML widget (rows 6-11)
#display(w1_html)

#Widgets for:
#move tokens up/down
#add/delete rows
#add/delete notes to a specific token

#widget to select a witness
w_wit = widgets.Dropdown(
    options = witnesses,
    description = 'Witness:',
)
#widget to select a row
w_rowID = widgets.BoundedIntText(
    min=0,
    max=len(collation)-1,
    description='ID:',
)
#widget to select a specific token
w_token = widgets.Text(
    min0,
    description = 'Token position:',
)
#widget to enter text note
w_note = widgets.Text(
    description = 'Note:',
)

out = widgets.Output()

#link buttons and functions
@out.capture(clear_output=True)#wait=True, clear_output=True
def modif_on_click(b):
    if b.description == 'add row after':
        #add row
        add_row_after(rowID=w_rowID.value)
    if b.description == 'delete row':
        #delete
        delete_row(rowID=w_rowID.value)
    if b.description == 'move token down':
        move_token_down(rowID=w_rowID.value, wit=w_wit.value)
    if b.description == 'move...


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Move a token to the previous (up) or next (down) row   ↑
Can you write Python code for it?

def move_token_up(rowID, wit):
    try:
        #the token cannot be in the first row
        rowID > 0
        #select the first token
        token = collation[rowID][witnesses.index(wit)].pop(0)
        #append it at the end of the cell in the previous row
        collation[rowID-1][witnesses.index(wit)].append(token)
        print("Token '"+token['t']+"' moved up!")
    except:
        print("There is no token to move.")

def move_token_down(rowID, wit):
    try:
        #the token cannot be in the last row
        rowID < len(collation)-1
        #select the last token
        token = collation[rowID][witnesses.index(wit)].pop()
        #add it at the beginning of the cell in the next row
        collation[rowID+1][witnesses.index(wit)].insert(0, token)
        print("Token '"+token['t']+"' moved down!")
    except:
        print("There is no token to move.")


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Print collation in a text orientation
To read a short passage quickly. The collation table is reversed so that each witness is displayed on a row instead of a column.
Can you write Python code for it?

def print_witnesses_text(table):
    reverse_table = [[row[i] for row in table] for i in range(len(witnesses))]
    for index,row in enumerate(reverse_table):
        text = ''
        for cell in row:
            #the row starts and ends with a token, not a space
            if row.index(cell) == 0 or text == '' or not cell:
                text += cell_to_string(cell)
            #if it is not the start of the string or an empty cell, add a space to separate tokens
            else:
                text += ' '+cell_to_string(cell)
        text += ', '+str(witnesses[index])
        print(text)
    #return reverse_table


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Print information about a reading
Can you write Python code for it?

def print_info(rowID, wit):
    #select cell
    cell = collation[rowID][witnesses.index(wit)]
    #if cell is empty, there is no token
    if len(cell) == 0:
            print('-')
    else:
        for token in cell:
            #position of token in cell + content
            print(cell.index(token), ':', ', '.join(token[feature] for feature in token))


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Save results   ↑
Can you write Python code for it?

#save the json file with update in the collation table
def save_json(path, table):
    #combine new collation table with witnesses, so as to have one data variable
    data = {'witnesses':witnesses, 'table':table}
    #open a file according to path
    with open(path, 'w') as outfile:
        #write the data in json format
        json.dump(data, outfile)

#save a subset of the collation table into fancy HTML version, with a small text description
def save_table(descr, table, path):
    #path to template
    template_path = 'alignment-tables/template.html'

    #load the text of the template into a variable html
    with open(template_path, 'r', encoding='utf-8') as infile:
        html = infile.read()

    #add base text to description
    if base_text:
        descr += '<br>Agreement with the base text '+base_text+' is marked in green.'
        descr += ' Variation from '+base_text+' is marked in red.'

    #modify template: replace the comment with descr paragraph
    html = re.sub(r'<!--descr-->',descr,html)

    #modify template: replace the comment with table
    html = re.sub(r'<!--table-->',table,html)

    #save
    with open(path, 'w', encoding='utf-8') as outfile:
        outfile.write(html)


I am working on the file ".ipynb_checkpoints/interactive-collation-checkpoint.ipynb".
The first task is:
Search   ↑
Can you write Python code for it?

def search(table,text):
    #result table to build
    result_table = []

    #go through each row of the collation table
    for row in table:
        #go through each cell
        for cell in row:
            #if the search text matches the cell text (original or normalized form)
            if text in cell_to_string_norm(cell) or text in cell_to_string(cell):
                #add row to the result table
                result_table.append(row)
                #go to next row
                break

    #if the result table is empty, the text was not found in the collation
    if result_table == []:
        print(text+" was not found!")

    return result_table


I am working on the file ".ipynb_checkpoints/ipython-intro-checkpoint.ipynb".
The first task is:
Interactive widgets: Example
Can you write Python code for it?

from IPython.html.widgets import interact
import matplotlib.pyplot as plt
import networkx as nx
%matplotlib inline

# wrap a few graph generation functions so they have the same signature

def random_lobster(n, m, k, p):
    return nx.random_lobster(n, p, p / m)

def powerlaw_cluster(n, m, k, p):
    return nx.powerlaw_cluster_graph(n, m, p)

def erdos_renyi(n, m, k, p):
    return nx.erdos_renyi_graph(n, p)

def newman_watts_strogatz(n, m, k, p):
    return nx.newman_watts_strogatz_graph(n, k, p)

def plot_random_graph(n, m, k, p, generator):
    g = generator(n, m, k, p)
    nx.draw(g)
    plt.show()

interact(plot_random_graph, n=(2,30), m=(1,10), k=(1,10), p=(0.0, 1.0, 0.001),
        generator={'lobster': random_lobster,
                   'power law': powerlaw_cluster,
                   'Newman-Watts-Strogatz': newman_watts_strogatz,
                   u'Erdős-Rényi': erdos_renyi,
                   });


I am working on the file ".ipynb_checkpoints/ipython-intro-checkpoint.ipynb".
The first task is:
New paragraph
This is rich text with links,
equations:
$$\hat{f}(\xi) = \int_{-\infty}^{+\infty} f(x)\,
                 \mathrm{e}^{-i \xi x}$$
code with syntax highlighting:
python
print("Hello world!")
and images:
Can you write Python code for it?

%lsmagic

from IPython.display import HTML, SVG
HTML('''

         ''' +
         ''.join(['
' +
                  ''.join(['{row},{col}	'.format(
                                 row=row, col=col
                                 ) for col in range(5)]) +
                  '
' for row in range(5)]) +
         '''

         ''')


I am working on the file ".ipynb_checkpoints/ipython-intro-checkpoint.ipynb".
The first task is:
Remark: Generation of presentations from notebooks:
(Thats how this presentation was generated ....)
Can you write Python code for it?

!ipython nbconvert --to slides ipython-intro.ipynb --post serve


I am working on the file ".ipynb_checkpoints/itoledoc_coffee-checkpoint.ipynb".
The first task is:
A DataFrame can be converted into a numpy array by using the method .values:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/itoledoc_coffee-checkpoint.ipynb".
The first task is:
But also you can use the column names and index keys, to substract, for example, the name of the first antenna in a baseline pair from row 3:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/itoledoc_coffee-checkpoint.ipynb".
The first task is:
DataFrame are objects containgin tabular data, that can be grouped by columns and then used to aggreate data. Let's say you want to obtaing the mean frequency for the baselines and the number of channels used:
Can you write Python code for it?

data_group = df.groupby(['ant1name', 'ant2name'])
df2 = data_group.agg({'freq': np.mean, 'chan': np.count_nonzero}).reset_index()
df2.head()

data_raw = df.groupby(['ant1name', 'ant2name', 'chan']).y.mean()
data_raw.head(30)

data_raw.unstack().head(20)

pd.options.display.max_columns = 200
data_raw.unstack().head(20)

data_raw = data_raw.unstack().reset_index()
data_raw.head()

data_raw.to_excel('test.xls', index=False)

todegclean = np.degrees(np.arcsin(np.sin(np.radians(data_raw.iloc[:,2:]))))

todegclean.head()

todegclean['mean'] = todegclean.mean(axis=1)

todegclean.head()

data_clean = todegclean.iloc[:,:-1].apply(lambda x: x - todegclean.iloc[:,-1])
data_clean.head(20)

data_ready = pd.merge(data_raw[['ant1name', 'ant2name']], todegclean, left_index=True, right_index=True)
data_ready.head()


I am working on the file ".ipynb_checkpoints/itoledoc_coffee-checkpoint.ipynb".
The first task is:
For numpy expert, you have also methods to access the data using the numpy standards. If you want to extract the data at the coordinate (0,1) you can do:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/itoledoc_coffee-checkpoint.ipynb".
The first task is:
Plot.ly
Can you write Python code for it?

data_clean2 = data_clean.unstack().reset_index().copy()

data_clean2.query('100 < level_1 < 200')

data_clean2.query('100 < level_1 < 200').iplot(kind='scatter3d', x='chan', y='level_1', mode='markers', z=0, size=6,
                  title='Phase BL', filename='phase_test', width=1, opacity=0.8, colors='blue', symbol='circle',
                  layout={'scene': {'aspectratio': {'x': 1, 'y': 3, 'z': 0.7}}})

ploting = data_clean2.query('100 < level_1 < 200').figure(kind='scatter3d', x='chan', y='level_1', mode='markers', z=0, size=6,
                  title='Phase BL', filename='phase_test', width=1, opacity=0.8, colors='blue', symbol='circle',
                  layout={'scene': {'aspectratio': {'x': 1, 'y': 3, 'z': 0.7}}})

# ploting

ploting.data[0]['marker']['color'] = 'blue'
ploting.data[0]['marker']['line'] = {'color': 'blue', 'width': 0.5}
ploting.data[0]['marker']['opacity'] = 0.5

plotly.offline.iplot(ploting)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Affinity Propagation

Software Engineer:
Can you write Python code for it?

se_sims = calSimScores('Software Engineer', df=df, out_fmt='matrix_topic_sim') # 'Financial and Insurance Activities',

se_posts = df[df.title == 'Software Engineer']
se_cluster = AF_clustering(se_posts, job_title='se', sim_mat=se_sims)

se_cluster.groupby('af_label').size().sort_values(ascending=False)

fig = plotCluster(0, job_title='SE',cluster_res=se_cluster)
# plt.show(); plt.close()

for i in range(9):
    fig = plotCluster(i, job_title='SE',cluster_res=se_cluster)

fig = plotCluster(10, job_title='SE',cluster_res=se_cluster)
plt.show(); plt.close()

fig = plotCluster(22, job_title='SE', cluster_res=se_cluster, figsize=(12, 6))
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
After the merging, we need to re-comp the statistics for job titles as the stats change.
Can you write Python code for it?

stats = getTitleStats(df)
stats.to_csv(SKILL_RES + 'stats.csv', index=False)

hk_titles = set([t for t in df.title if found(hk_kw, t)])
hk_titles

_ = df[df.title.isin(hk_titles)]
print('# posts: %d' %_.shape[0])


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Analyze cases where cscore is 1 (posts are 100% consistent):
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
As we already cover a large number of relevant titles. We may not need to retrieve more titles based on key words.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Based on this, we guessed the following titles for PST.
Can you write Python code for it?

cc_teachers = ['Pre-School Teacher', 'Kindergarten Teacher', 'Child Care Teacher', 'Pre-Primary Education Teacher',
            'Teacher, Kindergarten', 'Teacher, Child Care', 'Day Care Teacher']

teacher_stat.describe().round(1)

teacher_stat[teacher_stat.title.isin(cc_teachers)]


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
But first let us look at basic stats of the titles.
Can you write Python code for it?

rel_title_stats = stats[stats.title.isin(rel_titles)]
rel_title_stats.describe().round(1)

rel_title_stats.to_csv(HK_DIR + 'rel_title_stats.csv', index=False)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Comparing Different Job Titles
We compare job titles to see if their job posts are consistent (thru the topic sim distribution among the job posts).
Can you write Python code for it?

se_sims_df = calSimScores('Software Engineer')

se_sims_df.sort_values('topic_sim', ascending=False, inplace=True)
se_sims_df.head()

se_sims_df.query('skill_sim < 1 and employer1 != employer2').head()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Distribution of c-scores
Can you write Python code for it?

def vizCScores(res):
    fig = plt.figure(figsize=(6,5))
    plt.hist(res.cscore)
    avg, std = round(res.cscore.mean(), 3), round(res.cscore.std(), 3)
    xl = 'Consistency score' + r'$(\mu = {}, \sigma = {})$'.format(avg, std)
    plt.xlabel(xl, fontsize=16);
    plt.ylabel('Count', fontsize=16)
    plt.grid(True)
    return fig

res = agg_df.query('2 <= n_post < 100')
res.reset_index(inplace=True)

_ = agg_df.query('2 <= n_post')
print('# pairs with at least 2 posts: %d' % _.shape[0])

del res['index']
res.head()

res['cscore'] = res.apply(cScoreAtRow, axis=1)

res_55 = pd.read_csv(JOB_PROF + 'cscore_jobs_100posts.csv')
res = pd.concat([res, res_55])
res.to_csv(JOB_PROF + 'cscore_all.csv', index=False)

fig = vizCScores(res)
fig.savefig(JOB_PROF + 'cscore_dist.pdf')
plt.show(); plt.close()

res = res.sort_values('cscore', ascending=False)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Early Childhood Care and Education (ECCE)
Pre-School Teachers (PST)
The list of titles for PST may be formed by looking at the titles from certain pre-schools in SG. We tried with the top pre-schools obtained from https://skoolopedia.com/preschool-singapore-2015-infographic/. First try return empty results! Checking with employer name in data shows that we need to append 'PTE. LTD.' to school names. We then added schools found in data.
Can you write Python code for it?

keys = map(str.upper, ['PreSchool', 'Skool', 'CHILDCARE', 'Kid', 'toddler', 'infant'])

guessed_preschools = set([s for s in df.employer_name if found(keys, s)])
print('# guessed preschools: %d' %len(guessed_preschools))

guessed_preschools

# ['Shaws CDLC', 'childfirst',  'kiddiWinkie', 'little footprints', 'brighton montessori', 'posso', 'little skool-house',
# 'little village', 'mulberry', 'learning vision', 'Star Learners', 'global eduhub', 'sunflower childcare', 'frobel']

preschool_keys = ['E-BRIDGE', 'ETONHOUSE PRESCHOOL', 'MINDCHAMPS', 'LECLARE', "Pat's Schoolhouse",
                  'SKOOL4KIDZ', 'VIVA KIDS', 'JUST KIDS LEARNING PLACE', 'BIBINOGS KIDS ACADEMY',
                  'CREATIVE LAND CHILDCARE', 'Lorna Whiston',
                  'Carpe Diem', 'Crestar', 'nurture edu', 'safari house']
preschool_keys = map(str.upper, preschool_keys)

preschools = [s for s in df.employer_name if found(preschool_keys, s)]
preschool_posts = df[df.employer_name.isin(preschools)]
print('# posts from preschool employers: %d, distributed as follows:' %preschool_posts.shape[0])
sortDown(preschool_posts.groupby('employer_name').size())

titles = set(preschool_posts['title'])
titles


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Financial and Insurance Activities
Can you write Python code for it?

fin_df = agg_df.query('industry == "Financial and Insurance Activities"')

fin_top50 = fin_df.iloc[range(50)]
fin_top50['cscore'] = fin_top50['title'].apply(consistency, industry='Financial and Insurance Activities')
fin_top50.to_csv(JOB_PROF + 'consistency/fin50_cscore.csv', index=False)

fin_top50.cscore.describe()

fin_top50 = fin_top50.sort_values('cscore', ascending=False)

fin_top50.head()

fin_top50.tail()

fin_res = fin_df.query('n_post >= 2').copy()
print('# cscores to be computed: %d' %fin_res.shape[0])
fin_res['cscore'] = fin_res['title'].apply(consistency, industry='Financial and Insurance Activities')
fin_res.to_csv(JOB_PROF + 'consistency/fin_cscore.csv', index=False)

fin_res.cscore.describe()

# Plot dist of cscore


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Find "Synonym" Job Titles

Software Engineer vs. Software Developer
Can you write Python code for it?

se_and_sd = simScore('Software Engineer', 'Software Developer')

se_and_sd = se_and_sd.sort_values('topic_sim', ascending=False)
se_and_sd.reset_index(inplace=True); del se_and_sd['index']
se_and_sd.head()

fig = plotSimDists(se_and_sd, 'SE_and_SD')
fig.savefig(SKILL_RES + 'fig/se_and_sd_sims.pdf')
plt.show(); plt.close()

plt.close('all')

vizPair(0, se_and_sd, abbv_title='se_vs_sd')

last = se_and_sd.shape[0] - 1
vizPair(last, se_and_sd, abbv_title='se_vs_sd')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
First, analyze tuples (job title, industry) with $\ge 100 $ posts.
Can you write Python code for it?

res = agg_df.query('n_post >= 100').copy()
print('# pairs to analyze: %d' %res.shape[0])

cScoreAtRow(res.iloc[0])

res['cscore'] = res.apply(cScoreAtRow, axis=1)

res.cscore.describe().round(3)

res = res.sort_values('cscore', ascending=False)

res.head()

res.tail()

res.to_csv(JOB_PROF + 'cscore_jobs_100posts.csv', index=False)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
From the summary, we see that:

25% of titles have <= 8 post
50% of titles have <= 22 posts.
Can you write Python code for it?

hk_title_stats = stats[stats.title.isin(hk_titles)]
hk_title_stats


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Gotcha:
All of them are re-posts, some with a bit editing (cases 5, 7 and 17).
Can you write Python code for it?

res.query('cscore < 1').head()

checkRepost(res.query('cscore < 1').head())


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Manager in Finance
Can you write Python code for it?

fin_man = calSimScores('Manager', 'Financial and Insurance Activities', df)

print fin_man.topic_sim.describe().round(2)
print fin_man.skill_sim.describe().round(2)

fin_man = fin_man.sort_values('topic_sim', ascending=False); del fin_man['index']

fin_man.head().to_csv(SKILL_RES + 'fin_man_top5.csv', index=False)
fin_man.tail().to_csv(SKILL_RES + 'fin_man_bottom5.csv', index=False)

posts = getPostsInPairs(fin_man.tail(), df)

top5 = fin_man.query('employer1 != employer2 and skill_sim <= 0.8').head()

fig = vizTopicDists(posts, doc_topic_distr, figsize=(12, 6))
plt.savefig(SKILL_RES + 'fig/fin_man_bottom5.pdf')
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Manager in Finance:
Can you write Python code for it?

spectral.fit(fin_man_sims)

fin_man_posts = df[(df.title == 'Manager') & (df.industry == 'Financial and Insurance Activities')]

fin_man_posts['cluster'] = spectral.labels_
fin_man_posts = fin_man_posts.sort_values('cluster')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Measure the similarity among the retrieved posts:
Can you write Python code for it?

tech_sims = pairwiseSim(tech_posts, doc_topic_distr)

# plot dists of the sims
fig = plotSimDists(tech_sims, 'Technician jobs')
fig.savefig(SKILL_RES + 'fig/tech_jobs_sim.pdf')
plt.show(); plt.close()

tech_sims.skill_sim.describe().round(2)

tech_sims.query('skill_sim == 1')

tech_sims.topic_sim.describe().round(2)

tech_sims = tech_sims.sort_values('topic_sim', ascending=False)

n_pair = tech_sims.shape[0]; last = n_pair - 1
vizPair(last, tech_sims)


cc_sims = pairwiseSim(cc_df, doc_topic_distr)

print('# pairs: %d' %cc_sims.shape[0])

# del cc_sims['index']
cc_sims.topic_sim.describe().round(3)

fig = plotSimDists(cc_sims, 'Pre-School Teacher')
fig.savefig(SKILL_RES + 'fig/pst_sims.pdf')
plt.show(); plt.close()

cc_sims = cc_sims.sort_values('skill_sim', ascending=False)

cc_sims.query('skill_sim >= 0.8').to_csv(SKILL_RES + 'job_prof/pst_variants.csv', index=False)

cc_sims.query('(employer1 != employer2) and (topic_sim >= 0.9) and (skill_sim < 0.8)')

niwa_df = cc_sims.query('employer1 == "NIWA SCHOOLHOUSE"')
print(niwa_df.shape[0])

niwa_df.head()

tmp = niwa_df.head()
plt.close('all')

vizPair(0, niwa_df)

vizPair(1, niwa_df)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Niche vs. General Job Titles
InfoCom industry
Can you write Python code for it?

info_df = agg_df.query('industry == "Information and Communications"')

info_top50 = info_df.iloc[range(50)]
info_top50['cscore'] = info_top50['title'].apply(consistency, industry='Information and Communications')
info_top50.to_csv(JOB_PROF + 'consistency/infocom_cscore.csv', index=False)

info_top50.cscore.describe()

info_top50.sort_values('cscore', ascending=False, inplace=True)

info_top50.head()

info_top50.tail()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Pinnacle Leader
Hotel and Accomodation Services (HAS)

Obtain an exhaustive list of job titles available for hotels, hostels in SG:
Can you write Python code for it?

# query employers in HAS in ds
hotel_kws = map(str.upper, ['hotel', 'hostel', 'motel', 'lodging', 'resort'])
names = [s for s in df.employer_name if found(hotel_kws, s)]
hotels = pd.DataFrame({'name': names}).drop_duplicates()

print('# employers in HAS: %d' %hotels.shape[0])

hotels.to_csv(SKILL_RES + 'hotels.csv', index=False)

# query all posts of the employers
has_posts = df[df.employer_name.isin(hotels)]
print('# posts in HAS: %d, distributed as follows:' %has_posts.shape[0])
# sortDown(has_posts.groupby('employer_name').size())

# query possible job titles for the employers in HAS
has_title = set(has_posts.title)
print('# titles in the HAS employers: %d' %len(has_title))

has_title_stats = stats[stats.title.isin(has_title)]
has_title_stats.to_csv(SKILL_RES + 'has_title_stats.csv', index=False)

has_title_stats.describe().round(1)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Precision Engineering - Technician Track

Find an exhaustive list of job titles for Technician:
Can you write Python code for it?

# Machinist/Technician are suggested by SF
tech_kw = ['Machinist', 'Technician']
tech_titles = [t for t in df.title if found(tech_kw, t)]
c = Counter(tech_titles)

tech_titles = pd.DataFrame({'title': c.keys(), 'n_post': c.values()}).sort_values('n_post', ascending=False)

tech_titles.n_post.describe().round(1)

tech_titles = tech_titles.query('n_post > 10')
print('# titles: %d' %tech_titles.shape[0])

tech_titles.tail()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Range of Job Post Similarity Scores
We will look at distribution of topic similarity scores of certain samples in data. We investigate how the distribution varies with n_topic and sample size.
Can you write Python code for it?

with(open(SKILL_RES + 'doc_topic_distr.mtx', 'r')) as f:
    doc_15topic_distr = mmread(f)

with(open(SKILL_RES + 'doc_20topic_distr.mtx', 'r')) as f:
    doc_20topic_distr = mmread(f)

with(open(SKILL_RES + 'doc_30topic_distr.mtx', 'r')) as f:
    doc_30topic_distr = mmread(f)

print(doc_15topic_distr.shape)
print(doc_20topic_distr.shape)
print(doc_30topic_distr.shape)

# Allow us to loop over doc-topic dists wrt diff no. of topics
topic_distr = {15: doc_15topic_distr, 20: doc_20topic_distr, 30: doc_30topic_distr}

rd.seed(1234567) # time()
size = 500; posts = good_df.sample(size)

sims_15 = pairwiseSim(posts, doc_15topic_distr)
sims_20 = pairwiseSim(posts, doc_20topic_distr)
sims_30 = pairwiseSim(posts, doc_30topic_distr)

sims_15.topic_sim.describe().round(3)
sims_20.topic_sim.describe().round(3)
sims_30.topic_sim.describe().round(3)

sims_15.sort_values('topic_sim', inplace=True)
vizPair(0, sims_15)
vizPair(1, sims_15)

medium = sims_15.query('0.7 < topic_sim and topic_sim <= 0.8')
medium.sort_values('topic_sim', inplace=True)

vizPair(0, medium, abbv_title='medium')

last = medium.shape[0]-1
vizPair(last, medium, abbv_title='medium')

large = sims_15.query('topic_sim > 0.8')
vizPair(0, large, abbv_title='large')

n_pair = sims_15.shape[0]; last = n_pair - 1
vizPair(last, sims_15)

sims = {15: sims_15, 20: sims_20, 30: sims_30}
ks = [15, 20, 30]

for k in ks:
    fig = plotSimDists(sims[k])
    fig.suptitle('Sample size: {} posts'.format(size), fontsize=20)
    fname = SKILL_RES + 'fig/sim_dists_{}topics.pdf'.format(k)
#     fig.set_tight_layout(True)
    fig.savefig(fname)
    plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Relevant titles for HK

From employers of HK positions:
Can you write Python code for it?

hk_employers = hk_posts.employer_name.drop_duplicates()
hk_employers.to_csv(HK_DIR + 'employers.csv', index=False)

print('# employers having HK positions: %d' %len(hk_employers))

# posts from HK employers
posts_of_hk_employers = df[df.employer_name.isin(hk_employers)]
print('# posts from HK employers: %d' %posts_of_hk_employers.shape[0])

titles_from_hk_employers = set(posts_of_hk_employers['title'])
print('# titles from HK employers: %d' %len(titles_from_hk_employers))

rel_titles = titles_from_hk_employers.union(hk_titles)
print('# relevant titles: %d' %len(rel_titles))

hk_titles.difference(titles_from_hk_employers)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Relevant titles for PST
Can you write Python code for it?

# employers having PST positions
pst_employers = np.unique(pst_posts.employer_name)
print('# PST employers: %d' %len(pst_employers))
# pst_employers


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Research Fellow
Can you write Python code for it?

rf_sims = calSimScores(job_title='Research Fellow', industry='Education', df=df)

print rf_sims.topic_sim.describe().round(3)
print rf_sims.skill_sim.describe().round(2)

rf_sims = rf_sims.sort_values('topic_sim', ascending=False)
rf_sims.head()


rf_sim_mat = calSimScores(job_title='Research Fellow', industry='Education', df=df, out_fmt='matrix_topic_sim')

rf_posts = df[(df.title=='Research Fellow') & (df.industry == 'Education')]
print('# posts of Research Fellow: %d' %rf_posts.shape[0])

rf_cluster = AF_clustering(rf_posts, job_title='rf', sim_mat=rf_sim_mat)

rf_cluster.groupby('af_label').size().sort_values(ascending=False)

rf_c0 = plotCluster(0, job_title='RF', cluster_res=rf_cluster)
rf_c1 = plotCluster(1, job_title='RF', cluster_res=rf_cluster)
plt.close('all')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Result: The topic similarity scores are very high with a mean value of 0.99!

Categorize the skills in these posts using the SF framework for PST:

The SF framework for PST has 10 categories of skills (listed below). When we manually labeled the top-100 popular skills using the categories, we detected another group of skills, namely language skills e.g. chinese, mandarin. We labeled this group as Language Skill. We also found several skills which cannot go into any categories, we thus create Others group for these skills. In short, we have 12 following categories of skills.
Can you write Python code for it?

categs = {'c-dev': 'Child Development', 'ped': 'Curriculum & Pedagogy',
          'env': 'Learning environment', 'rel' : 'Interaction & Relationships',
          'safety' : 'Health, Safety & Nutrition', 'family & com' : 'Family & Community partnerships',
          'prof dev' : 'Professional Mastery', 'prof val' : 'Professional Values & Ethics',
          'teamwork' : 'Teamwork & Collaboration', 'plan' : 'Visioning & Planning',
          'lang': 'Language Skill', 'others' : 'Others'}

skill_sets = map(lambda s: set(s.split(',')), cc_df.occur_skills)
cc_skills = unionAll(skill_sets)
print('# skills in child care posts: %d' %len(cc_skills))

freqs = [freq(sk, skill_sets) for sk in cc_skills]
cc_skill_df = pd.DataFrame({'skill': list(cc_skills), 'freq': freqs})

# cc_skill_df.sort_values('freq', ascending=False).to_csv(SKILL_RES + 'cc_skills.csv', index=False)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
SE in Finance:
Can you write Python code for it?

spectral.fit(fin_se_sims)

fin_se_posts = df[(df.title == 'Software Engineer') & (df.industry == 'Financial and Insurance Activities')]
fin_se_posts['cluster'] = spectral.labels_
fin_se_posts = fin_se_posts.sort_values('cluster')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Senior Software Engineer vs. Marketing Executive (Wholesale and Retail Trade)
Denote Senior Software Engineer as SSE, Administrative Assistant as AA, and Marketing Executive as ME.
Can you write Python code for it?

sse_cscore = consistency('Senior Software Engineer', 'Wholesale and Retail Trade', abbv_job='sse', abbv_industry='wholesale')
aa_cscore = consistency('Administrative Assistant', 'Wholesale and Retail Trade', abbv_job='aa', abbv_industry='wholesale')
me_sims = consistency('Marketing Executive', 'Wholesale and Retail Trade', abbv_job='me', abbv_industry='wholesale')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Similarity with relevant titles
Can you write Python code for it?

def sims2RelTitles(min_post):
    print('Picked titles with at least {} posts'.format(min_post))
    res = rel_title_stats.query('n_post >= {}'.format(min_post))
    res['topic_sim_with_hk'] = res['title'].apply(simScore, t2='Housekeeping Attendant')

    res = res.round(2)
    res = res.sort_values('topic_sim_with_hk', ascending=False)

    res.reset_index(inplace=True); del res['index']
    return res

res_38 = sims2RelTitles(min_post=38)

res_38.head(10).to_csv(HK_DIR + 'top_sim_titles.csv', index=False)

res_38.head(10)

res_38.describe().round(2)

# limit to titles with > 22 posts.
_ = rel_title_stats.query('n_post > 22')
print('# titles with > 22 posts: %d' %_.shape[0])

res = rel_title_stats.query('n_post > 22')
res['topic_sim_with_hk'] = res['title'].apply(simScore, t2='Housekeeping Attendant')

res = res.round(2)
res = res.sort_values('topic_sim_with_hk', ascending=False)
res.head(10)

res.to_csv(HK_DIR + 'sims_to_hk.csv', index=False)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Software Engineer vs. Manager
Can you write Python code for it?

se_and_man = simScore('Software Engineer', 'Manager')

se_and_man.topic_sim.describe().round(3)

fig = plotSimDists(se_and_man)
fig.savefig(SKILL_RES + 'fig/se_and_man_sims.pdf')
plt.show(); plt.close()


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
The description shows us the following:
+ totally $128$ job titles are compared against PST
+
Can you write Python code for it?

res.query('n_post >= 6').describe().round(3)

res.query('n_post >= 6').head(10)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
The final result is follows.
Can you write Python code for it?

cc_skill_df = pd.read_csv(SKILL_RES + 'cc_skills.csv')

# top 100 skills
top_100 = pd.DataFrame(cc_skill_df.head(100))
top_100['Category'] = top_100.label.apply(lambda x: categs[x])
print('Distribution of categories among top-100 skills')
sortDown(top_100.groupby('Category').size())

top_100['skill_freq'] = top_100.skill + ' (' + map(str, top_100.freq) + ')'
top_100.head(1)

tmp = top_100.groupby('Category').agg({'skill_freq': joinEntries, 'skill': len})
tmp = tmp.reset_index().rename(columns = {'skill_freq': 'skills (freq)', 'skill': 'n_skill'})
tmp

tmp.to_csv(JOB_PROF + 'cc_skill_categ.csv', index=False)

print [sk for sk in cc_skills if ('child' in sk)]
print [sk for sk in cc_skills if ('curriculum' in sk)]


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
The plots show that topic similarity distribution $sim_{topic}$ changes gradually with the number of topics $k$ and with sample size. For a sample of size 300, we have
+ $k=15$: $sim_{topic}$ is distributed in the range $ [0.63, 0.93] $
+ $k=20$: $sim_{topic}$ is distributed in the range $ [0.62, 0.92] $
+ $k=30$: $sim_{topic}$ is distributed in the range $ [0.6, 0.9] $
Similarity of Job Posts and Consistency Score
We will look at consistency score of a given job title in a given industry. First, let's see how many pairs (job title, industry) we have and get stats for the pairs.
Can you write Python code for it?

df.head(1)

by_job_and_industry = df.groupby(['title', 'industry'])

agg_df = by_job_and_industry.agg({'job_id': len, 'employer_id': 'nunique', 'n_skill': 'mean'})

agg_df = agg_df.rename(columns={'job_id': 'n_post', 'employer_id': 'n_employer',
                               'n_skill': 'avg_n_skill'})
agg_df = agg_df.reset_index()
agg_df.describe().round(1)

agg_df.sort_values('n_post', inplace=True)

agg_df.head(10)


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
The two posts above are from hotels of the same group: Park Hotel Group.
Can you write Python code for it?

vizPair(1, hk_sims, labels, abbv_title='hk')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
This case a job agency reposted the job for the employer.
Can you write Python code for it?

vizPair(2, hk_sims, labels, abbv_title='hk')

last = hk_sims.shape[0]-1
vizPair(last, hk_sims, labels, abbv_title='hk')


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
Using the titles to retrieve posts for Technician:
Can you write Python code for it?

tech_posts = df[df.title.isin(tech_titles.title)]
print('# posts for Technician: %d' %tech_posts.shape[0])

getTitleStats(tech_posts).to_csv(SKILL_RES + 'job_prof/tech_titles.csv', index=False)

sortDown(tech_posts.groupby('title').size())


I am working on the file ".ipynb_checkpoints/job_post_cluster-checkpoint.ipynb".
The first task is:
We need to get basic stats of job titles to understand more about them. Given a title, we need to know:

How many posts for the title in whole ds
avg n_skill in the posts
n_employer having the title
Can you write Python code for it?

stats = getTitleStats(df)
stats.describe().round(1)
stats.to_csv(SKILL_RES + 'stats.csv', index=False)

k = 20;  fname = SKILL_RES + 'lda/{}_topics.csv'.format(k)
doc_topic_distr = topic_distr[k]
topic_df = pd.read_csv(fname)
labels = map(str.upper, topic_df['label'])


I am working on the file ".ipynb_checkpoints/jobtitle_skill-checkpoint.ipynb".
The first task is:
Collapse all posts of the same job title into a single document
Can you write Python code for it?

by_job_title = jobs.groupby('title')
job_title_df = by_job_title.agg({'job_id': lambda x: ','.join(x), 'doc': lambda x: 'next_doc'.join(x)})

job_title_df = job_title_df.add_prefix('agg_').job_title_dfet_index()
job_title_df.head()

n_job_title = by_job_title.ngroups
print('# job titles: %d' %n_job_title)

reload(cluster_skill_helpers)
from cluster_skill_helpers import *

jd_docs = job_title_df['agg_doc']

# This version of skills still contain stopwords
doc_skill = buildDocSkillMat(jd_docs, skill_df)


I am working on the file ".ipynb_checkpoints/jobtitle_skill-checkpoint.ipynb".
The first task is:
Most popular skills by job title
Can you write Python code for it?

job_title_df.head(1)

idx_of_top_skill = np.apply_along_axis(np.argmax, 1, jobtitle_skill)

# skill_df = skills
skills = skill_df['skill']
top_skill_by_job_title = pd.DataFrame({'job_title': job_titles, 'idx_of_top_skill': idx_of_top_skill})
top_skill_by_job_title['top_skill'] = top_skill_by_job_title['idx_of_top_skill'].apply(lambda i: skills[i])

top_skill_by_job_title.head(30)

with(open(SKILL_DIR + 'jobtitle_skill.mtx', 'r')) as f:
    jobtitle_skill = mmread(f)

jobtitle_skill = jobtitle_skill.tocsr()
jobtitle_skill.shape

job_titles = job_title_df['title']

# for each row (corresponding to a jobtitle) in matrix jobtitle_skill, get non-zero freqs
global k
k = 3

def getTopK_Skills(idx):
    title = job_titles[idx]
    print('Finding top-{} skills of job title {}...'.format(k, title))

    skill_occur = jobtitle_skill.getrow(idx)
    tmp = find(skill_occur)
    nz_indices = tmp[1]
    values = tmp[2]
    res = pd.DataFrame({'job_title': title, 'skill_found_in_jd': skills[nz_indices], 'occur_freq': values})

    res.sort_values('occur_freq', ascending=False, inplace=True)
    return res.head(k)

# getTopK_Skills(0)

frames = map(getTopK_Skills, range(n_job_title))
res = pd.concat(frames) # concat() is great as it can concat as many df as possible
res.head(30)

res.to_csv(RES_DIR + 'top3_skill_by_jobtitle.csv', index=False)


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Create Dataset
Create a data set that can be retrieved for each Code.
First, convert Nan in stock_price_df to 0, bool to int, and 'Date' to datetime.
Can you write Python code for it?

# stock_price_df['ExpectedDividend'] = stock_price_df['ExpectedDividend'].fillna(0)
# stock_price_df['SupervisionFlag'] = stock_price_df['SupervisionFlag'].map({True: 1, False: 0})
# stock_price_df['Date'] = pd.to_datetime(stock_price_df['Date'])
# stock_price_df.info()


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Explanation of data
Loading Modules
First, load the required modules.
In this case, we will use pandas to load the data.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Prediction
The trained model will be used to make predictions on the submitted data.
DataFrame → Ndarray → tensor and transform the data to make predictions.
Can you write Python code for it?

from datetime import datetime

columns = ['Open', 'High', 'Low', 'Close', 'Volume', 'AdjustmentFactor', 'ExpectedDividend', 'SupervisionFlag']

def predict(model, X_df, sequence=30):
    pred_df = X_df[['Date', 'SecuritiesCode']]
    # Grouping by `groupby` and retrieving one by one
    code_group = X_df.groupby('SecuritiesCode')
    X_all = np.array([])
    for sc, group in code_group:
        # Standardize target data
        group_std = stdsc.transform(group[columns])
        # Calling up past data of the target data
        X = dataset_dict[str(sc)][-1*(sequence-1):, :-1]
        # concat
        group_std_add = np.zeros((group_std.shape[0], group_std.shape[1]+1))
        group_std_add[:, :-1] = group_std
        dataset_dict[str(sc)] = np.vstack((dataset_dict[str(sc)], group_std_add))

        X = np.vstack((X[:, :input_size], group_std[:, :input_size]))
        X_all = np.append(X_all, X)
    X_all = X_all.reshape(-1, sequence, X.shape[1])
    y_pred = np.array([])
    for it in range(X_all.shape[0]//512+1):
        data = X_all[it*512:(it+1)*512]
        data = torch.from_numpy(data.astype(np.float32)).clone()
        data = data.to(torch.float32)
        data = data.to(device)
        print('input size', data.size())
        # print(data)
        output = model.forward(data)
#         print(output)
        # output = output.view(1, -1)
        output = output.to('cpu').detach().numpy().copy()
        y_pred = np.append(y_pred, output)
    pred_df['target'] = y_pred
    # print(y_pred, y_pred.shape)
    pred_df['Rank'] = pred_df["target"].rank(ascending=False, method="first") - 1
    pred_df['Rank'] = pred_df['Rank'].astype(int)
    pred_df = pred_df.drop('target', axis=1)
    return pred_df


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Some of them contained missing values, so they were removed.
Can you write Python code for it?

# stock_price_df = stock_price_df.dropna(how='any')
# # Confirmation of missing information
# stock_price_df_na = (stock_price_df.isnull().sum() / len(stock_price_df)) * 100
# stock_price_df_na = stock_price_df_na.drop(stock_price_df_na[stock_price_df_na == 0].index).sort_values(ascending=False)[:30]
# missing_data = pd.DataFrame({'Missing Ratio' :stock_price_df_na})
# missing_data.head(22)


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Store data for each issue in dictionary form and store it in such a way that it can be recalled for each issue.
Can you write Python code for it?

dataset_dict = {}
for sc in train_df['SecuritiesCode'].unique():
    dataset_dict[str(sc)] = train_df[train_df['SecuritiesCode'] == sc].values[:, 3:].astype(np.float32)
print(dataset_dict['1301'].shape)


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Trainig
For each stock, LSTM training is conducted by repeatedly creating a data set and training the model.
To see the learning status, check the phenomenon of the loss function.
　→Can be learned.
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.plot(log_train[0][1:], log_train[1][1:])
plt.plot(log_eval[0][1:], log_eval[1][1:])
plt.xlabel('epoch')
plt.ylabel('loss')
plt.show()

from tqdm import tqdm
import time
import os
import copy

output_dir = "output_lstm"
if not os.path.exists(output_dir):
    os.makedirs(output_dir)

epochs = 20
batch_size = 512
seq_len = 14
num_layers = 2
input_size = 5
lstm_dim = 64
dropout = 0.

# Check wheter GPU is available
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

# Model Instantiation
model = LSTM(d_feat=input_size, hidden_size=lstm_dim, num_layers=num_layers, dropout=dropout)
model.to(device)
model.train()
# setting optimizer
lr = 0.0001
weight_decay = 1.0e-05
optimizer = torch.optim.Adagrad(model.parameters(), lr=lr, weight_decay=weight_decay)
# setting criterion
criterion = nn.MSELoss()

def train_epoch(train_df, model, seq_len=30, batch_size=512):
    groups = train_df.groupby(['SecuritiesCode'])
    total_loss = 0.
    iteration = 0
    model.train()

    def collect_batch_index():  # index for a stock with seq_len continuous days
        batch_index = []
        for sc, group in groups:
            indices = np.arange(len(group))
            for i in range(len(indices))[:: seq_len]:
                if len(indices) - i < seq_len:
                    break
                batch_index.append(group.index[i: i + seq_len])
        return batch_index

    batch_index = collect_batch_index()
    indices = np.arange(len(batch_index))
    np.random.shuffle(indices)

    for i in range(len(indices))[:: batch_size]:

        # if len(indices) - i < batch_size:
        #     break

        x_train = []
        y_train = []
        for index in indices[i: i + batch_size]:
            values = train_df.loc[batch_index[index]].values
            x_train.append(values[:, 3: 3 + input_size].astype(np.float32))
            y_train.append(values[:, -1][-1])

        # print(y_train)

        feature =...


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
Use Pytorch dataloader to recall data for each mini-batch.
Can you write Python code for it?

from torch.utils.data.sampler import SubsetRandomSampler

class MyDataset(torch.utils.data.Dataset):
    def __init__(self, X, sequence_num=31, y=None, mode='train'):
        self.data = X
        self.teacher = y
        self.sequence_num = sequence_num
        self.mode = mode

    def __len__(self):
        return len(self.teacher)

    def __getitem__(self, idx):
        out_data = self.data[idx]
        if self.mode == 'train':
            out_label = self.teacher[idx[-1]]
            return out_data, out_label
        else:
            return out_data

def create_dataloader(dataset, dataset_num, sequence_num=31, input_size=8, batch_size=32, shuffle=False):
    sampler = np.array([list(range(i, i+sequence_num)) for i in range(dataset_num-sequence_num+1)])
    if shuffle is True:
        np.random.shuffle(sampler)
    dataloader = torch.utils.data.DataLoader(dataset, batch_size, sampler=sampler)
    return dataloader

test_df.loc[test_df['Date'] == "2021-12-06"]


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
define metric: sharp rate and compute for the training data with known Target
Can you write Python code for it?

def calc_spread_return_sharpe(df: pd.DataFrame, portfolio_size: int = 200, toprank_weight_ratio: float = 2, rank='Rank') -> float:
    """
    Args:
        df (pd.DataFrame): predicted results
        portfolio_size (int): # of equities to buy/sell
        toprank_weight_ratio (float): the relative weight of the most highly ranked stock compared to the least.
    Returns:
        (float): sharpe ratio
    """
    def _calc_spread_return_per_day(df, portfolio_size, toprank_weight_ratio, rank='Rank'):
        """
        Args:
            df (pd.DataFrame): predicted results
            portfolio_size (int): # of equities to buy/sell
            toprank_weight_ratio (float): the relative weight of the most highly ranked stock compared to the least.
        Returns:
            (float): spread return
        """
        assert df[rank].min() == 0
        assert df[rank].max() == len(df[rank]) - 1
        weights = np.linspace(start=toprank_weight_ratio, stop=1, num=portfolio_size)
        purchase = (df.sort_values(by=rank)['Target'][:portfolio_size] * weights).sum() / weights.mean()
        short = (df.sort_values(by=rank, ascending=False)['Target'][:portfolio_size] * weights).sum() / weights.mean()
        return purchase - short

    buf = df.groupby('Date').apply(_calc_spread_return_per_day, portfolio_size, toprank_weight_ratio, rank)
    sharpe_ratio = buf.mean() / buf.std()
    return sharpe_ratio

# add rank according to Target for train

train_df['Rank'] = train_df.groupby("Date")['Target'].transform('rank', ascending=False, method="first") - 1
train_df['Rank'] = train_df['Rank'].astype(int)
# print(train_df['Rank'].min())
df_astock = train_df[train_df['Date'] == '2021-12-03']

# make sure it's correct
print(df_astock['Rank'].min(), df_astock['Rank'].max())
df_astock.sort_values(by=['Target'], ascending=False)
# sharp = calc_spread_return_sharpe(train_df)

tmpdf = test_df.copy()
tmpdf["Close_shift1"] = tmpdf["Close"].shift(-1)
tmpdf["Close_shift2"] = tmpdf["Close"].shift(-2)

tmpdf["rate"] =...


I am working on the file ".ipynb_checkpoints/jpx-tokyo-simple-lstm-network-scuec-checkpoint.ipynb".
The first task is:
数据预处理是否应该把所有股票放在一起？

stdscStandardize the features (other than RowId, Date, and SecuritiesCode) to be used in this project using sklearn's StandardScaler.
Can you write Python code for it?

# from sklearn.preprocessing import StandardScaler
# stdsc = StandardScaler()
# columns = ['Open', 'High', 'Low', 'Close', 'Volume', 'AdjustmentFactor', 'ExpectedDividend', 'SupervisionFlag']
# stock_price_df[columns] = stdsc.fit_transform(stock_price_df[columns])
# stock_price_df.head()


I am working on the file ".ipynb_checkpoints/knn-checkpoint.ipynb".
The first task is:
Cross-validation
We have implemented the k-Nearest Neighbor classifier but we set the value k = 5 arbitrarily. We will now determine the best value of this hyperparameter with cross-validation.
Can you write Python code for it?

num_folds = 5
k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]

X_train_folds = []
y_train_folds = []
################################################################################
# TODO:                                                                        #
# Split up the training data into folds. After splitting, X_train_folds and    #
# y_train_folds should each be lists of length num_folds, where                #
# y_train_folds[i] is the label vector for the points in X_train_folds[i].     #
# Hint: Look up the numpy array_split function.                                #
################################################################################
pass
################################################################################
#                                 END OF YOUR CODE                             #
################################################################################

# A dictionary holding the accuracies for different values of k that we find
# when running cross-validation. After running cross-validation,
# k_to_accuracies[k] should be a list of length num_folds giving the different
# accuracy values that we found when using that value of k.
k_to_accuracies = {}

################################################################################
# TODO:                                                                        #
# Perform k-fold cross validation to find the best value of k. For each        #
# possible value of k, run the k-nearest-neighbor algorithm num_folds times,   #
# where in each case you use all but one of the folds as training data and the #
# last fold as a validation set. Store the accuracies for all fold and all     #
# values of k in the k_to_accuracies dictionary.                               #
################################################################################
pass
################################################################################
#                                 END OF YOUR CODE                            ...


I am working on the file ".ipynb_checkpoints/knn-checkpoint.ipynb".
The first task is:
We would now like to classify the test data with the kNN classifier. Recall that we can break down this process into two steps:

First we must compute the distances between all test examples and all train examples.
Given these distances, for each test example we find the k nearest examples and have them vote for the label

Lets begin with computing the distance matrix between all training and test examples. For example, if there are Ntr training examples and Nte test examples, this stage should result in a Nte x Ntr matrix where each element (i,j) is the distance between the i-th test and j-th train example.
First, open cs231n/classifiers/k_nearest_neighbor.py and implement the function compute_distances_two_loops that uses a (very inefficient) double loop over all pairs of (test, train) examples and computes the distance matrix one element at a time.
Can you write Python code for it?

# Open cs231n/classifiers/k_nearest_neighbor.py and implement
# compute_distances_two_loops.

# Test your implementation:
dists = classifier.compute_distances_two_loops(X_test)
print dists.shape

# We can visualize the distance matrix: each row is a single test example and
# its distances to training examples
plt.imshow(dists, interpolation='none')
plt.show()


I am working on the file ".ipynb_checkpoints/knn-checkpoint.ipynb".
The first task is:
You should expect to see approximately 27% accuracy. Now lets try out a larger k, say k = 5:
Can you write Python code for it?

y_test_pred = classifier.predict_labels(dists, k=5)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct) / num_test
print 'Got %d / %d correct => accuracy: %f' % (num_correct, num_test, accuracy)


I am working on the file ".ipynb_checkpoints/lcacoffee-checkpoint.ipynb".
The first task is:
How come it is only adding the wednesday data in the results. It needs to have all the datas.
Needs to take the number in brackets away from the number not in brackets.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/lcacoffee-checkpoint.ipynb".
The first task is:
I need to filter the - - from the results. I really only need the values that have numbers.
Take number in brackets away from number not in brackets.
The number in brackets is total amount of coffees sold. The number not in brackets is amount of volchers used.
The number that I get when i take away is the coffee sold without volchers.
New dict that shows only the times that coffee were sold and the amount of coffgfges that were solf. Maybe that would works.
Can you write Python code for it?

betra = []

for azl in azlis:
    betra.append(azl)

anoe = []
anez = []

for betr in betra:
    betr.append(anoe)

for deta in betr:
    #print deta
    if '- -' in deta:
        print deta
    else:
        anez.append(deta)

fdic = []

for resut in anez:
    print resut
    fdic.append(resut)


I am working on the file ".ipynb_checkpoints/lcacoffee-checkpoint.ipynb".
The first task is:
ok if i cycle through jdunp between 0 and 23 i get the results.
cycle through ints but as a string. must add ' '
Can you write Python code for it?

for numtwn in range(0,24):
    print "'" + str(numtwn) + "'"

for jdr in jdunp:
    print jdr['0']

for numtwn in range(0,24):
        print "'" + str(numtwn) + "'"

for dej in jdunp:
    print dej.values()
    valia.append(dej.values())

dezrand = len(valia)

azlis = []

for vals in valia:
    print vals
    azlis.append(vals)


I am working on the file ".ipynb_checkpoints/logistic-regression-checkpoint.ipynb".
The first task is:
Logistic regression model by Rabst
This is a sample logistic regression model found in the internet.
Several functions exists, I recommend to use a more robust function for real problems, such as in sklearn.
Can you write Python code for it?

class LogisticRegression(object):
    """LogisticRegression classifier from the the Rasbt machine learning book.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    cost_ : list
        Cost in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=50):
        self.eta = eta
        self.n_iter = n_iter

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []
        for i in range(self.n_iter):
            y_val = self.activation(X)
            errors = (y - y_val)
            neg_grad = X.T.dot(errors)
            self.w_[1:] += self.eta * neg_grad
            self.w_[0] += self.eta * errors.sum()
            self.cost_.append(self._logit_cost(y, self.activation(X)))
        return self

    def _logit_cost(self, y, y_val):
        logit = -y.dot(np.log(y_val)) - ((1 - y).dot(np.log(1 - y_val)))
        return logit

    def _sigmoid(self, z):
        return 1.0 / (1.0 + np.exp(-z))

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        """ Activate the logistic neuron"""
        z = self.net_input(X)
        return self._sigmoid(z)

    def predict_proba(self, X):
        """
        Predict class probabilities for X.

        Parameters
        ----------
        X : {array-like, sparse matrix}, shape = [n_samples, n_features]
            Training vectors, where n_samples is...


I am working on the file ".ipynb_checkpoints/logistic-regression-checkpoint.ipynb".
The first task is:
Pandas
Use pandas to read a database.
In this example, I use the popular seed's database at the UCL site.
In the UCL web site (https://archive.ics.uci.edu/ml/datasets.html), one can find many useful academic and real databases.
Can you write Python code for it?

#Import data from files
import pandas as pd

#I use this dataset because this has clearly separated cathegories,
#Read the database using pandas,
#Note that bad lines are omitted with error_bad_lines=False
df = pd.read_csv('https://archive.ics.uci.edu/ml/'
        'machine-learning-databases/00236/seeds_dataset.txt', header=None, sep="\t", error_bad_lines=False)

#The headers are not given in the dataset, so we give them afterwords:
#1. area A,
#2. perimeter P,
#3. compactness C = 4*pi*A/P^2,
#4. length of kernel,
#5. width of kernel,
#6. asymmetry coefficient
#7. length of kernel groove.
#8. Class: 1=Kama, 2=Rosa, 3=Canadian

df.columns = ["area","perimeter","compactness","kernel-length","kernel-width",
              "asymetry","kernel-groove-length","class"]

#This shows the header of the database:
df.head()


I am working on the file ".ipynb_checkpoints/logistic-regression-checkpoint.ipynb".
The first task is:
We take only two classes from the dataset and we standarize features.
Standarization is a common practice in machine learning algorithms to give the same weight to all features.
To standarize the values of a given feature, just use:
X_i = (X_i - M) / D
Where X_i is a given entry, M is the statistical mean and D is the standard deviation (https://en.wikipedia.org/wiki/Standard_deviation).
These functions are provided in numpy: see mean() and std().
Can you write Python code for it?

#In the database there are 3 classes of seeds:
#We will just focus on two classes: 2 and 3:
df=df.loc[(df["class"]==2) | (df["class"]==3)]

import numpy as np

#This sets class=2 to 0 and 3 to 1:
y = df.loc[:,'class']
y = np.where(y == 2 , 0, 1)

#Extract some cathegories:
X=df.loc[:,["area","perimeter"]]
#This is to convert the csv dictionary into a numpy matrix to later standarize:
X=X.as_matrix()

# standardize features
X_std = np.copy(X)
X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()
X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()

from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, classifier, resolution=0.02):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                         np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    # plot class samples
    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                    alpha=0.8, c=cmap(idx),
                    marker=markers[idx], label=cl)

%matplotlib inline
import matplotlib.pyplot as plt

lr = LogisticRegression(n_iter=5000, eta=0.1).fit(X_std, y)
plt.plot(range(1, len(lr.cost_) + 1), np.log10(lr.cost_))
plt.xlabel('Samples')
plt.ylabel('Cost')
plt.title('Logistic Regression - Learning rate 0.01')

plt.tight_layout()
plt.show()
#You can see that the logistic regression algorithm converges nicely:

#Finally we plot the decision boundary:
plot_decision_regions(X_std, y,...


I am working on the file ".ipynb_checkpoints/logistic-regression-sciki-checkpoint.ipynb".
The first task is:
Plot different SVM classifiers in the seeds dataset
This example is a small modification of the sciki-learn tutorial test.
Comparison of different linear SVM classifiers on a 2D projection of the iris
dataset. Here I consider only two features of the dataset:

Seed area
Seed asymmetry

This example shows how to plot the decision surface for four SVM classifiers
with different kernels.
The linear models LinearSVC() and SVC(kernel='linear') yield slightly
different decision boundaries. This can be a consequence of the following
differences:

LinearSVC minimizes the squared hinge loss while SVC minimizes the
  regular hinge loss.

LinearSVC uses the One-vs-All (also known as One-vs-Rest) multiclass
  reduction while SVC uses the One-vs-One multiclass reduction.

Pandas
Use pandas to read a database.
In this example, I use the popular seed's database at the UCL site.
In the UCL web site (https://archive.ics.uci.edu/ml/datasets.html), one can find many useful academic and real databases.
Can you write Python code for it?

print(__doc__)

import numpy as np
import matplotlib.pyplot as plt
from sklearn import svm, datasets
import pandas as pd

#I use this dataset because this has clearly separated cathegories,
#Read the database using pandas,
#Note that bad lines are omitted with error_bad_lines=False
df = pd.read_csv('https://archive.ics.uci.edu/ml/'
        'machine-learning-databases/00236/seeds_dataset.txt', header=None, sep="\t", error_bad_lines=False)

#The headers are not given in the dataset, so we give them afterwords:
#1. area A,
#2. perimeter P,
#3. compactness C = 4*pi*A/P^2,
#4. length of kernel,
#5. width of kernel,
#6. asymmetry coefficient
#7. length of kernel groove.
#8. Class: 1=Kama, 2=Rosa, 3=Canadian

df.columns = ["area","perimeter","compactness","kernel-length","kernel-width",
              "asymmetry","kernel-groove-length","class"]

#This shows the header of the database:
df.head()


I am working on the file ".ipynb_checkpoints/logistic-regression-sciki-checkpoint.ipynb".
The first task is:
We take only two classes from the dataset and we standarize features.
Standarization is a common practice in machine learning algorithms to give the same weight to all features.
To standarize the values of a given feature, just use:
X_i = (X_i - M) / D
Where X_i is a given entry, M is the statistical mean and D is the standard deviation (https://en.wikipedia.org/wiki/Standard_deviation).
These functions are provided in numpy: see mean() and std().
In this particular example, there are 3 classes of seeds.
Can you write Python code for it?

import numpy as np

#This sets class=2 to 0 and 3 to 1:
y = df.loc[:,'class']

#Extract some cathegories:
X=df.loc[:,["area","asymmetry"]]
#This is to convert the csv dictionary into a numpy matrix to later standarize:
X=X.as_matrix()

# standardize features
X_std = np.copy(X)
X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()
X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
Bland's Rule
This seemingly arbitrary rule will seem less arbitrary in just a while.
Multiple Optimal Solutions
So, given the graphical intuition we now have for how the simplex method works, do we know if there ever a time when we would encounter more than 1 optimal solution for a given problem?
\begin{align}
\text{maximise}   \quad &  5x_1 +  7x_2          \
\text{subject to} \quad &  2x_1 +   x_2 \leq 4 \
                        & 10x_1 + 14x_2 \leq 30 \
\text{and}        \quad & x_1, x_2 \geq 0
\end{align}
Can you write Python code for it?

fig = plt.figure()
axes = fig.add_subplot(1,1,1)

# define view
x_1_min = 0.0
x_1_max = 3.0
x_2_min = 0.0
x_2_max = 5.0
res = 50

# plot axes
axes.axhline(0, color='k')
axes.axvline(0, color='k')

# plot constraints
x_1 = numpy.linspace(x_1_min, x_1_max, res)
c_1 = lambda x: 4.0 - 2.0*x
c_2 = lambda x: (30.0 - 10.0*x)/14.0
c_1_line = axes.plot( x_1, c_1(x_1), label='Constraint 1' ) # 2x_1 +   x_2 \leq 4
c_2_line = axes.plot( x_1, c_2(x_1), label='Constraint 2' ) # 10x_1 + 14x_2 \leq 30

# plot objective
x_2 = numpy.linspace(x_2_min, x_2_max, res)
c = numpy.empty([x_1.size, x_2.size])
for i, x_1_i in enumerate(x_1):
    c[:,i] = 5 * x_1_i + 7 * x_2

axes.contourf(x_1, x_2, c, res, cmap='Oranges', alpha=0.5)
# shade feasible region
c_1_bottom = numpy.linspace(0.0, 2.0, res)
c_2_bottom = numpy.linspace(0.0, 3.0, res)
axes.fill_between(c_1_bottom, c_1(c_1_bottom), color=plt.rcParams['axes.color_cycle'][0], alpha=0.5)
axes.fill_between(c_2_bottom, c_2(c_2_bottom), color=plt.rcParams['axes.color_cycle'][1], alpha=0.5)

# label graph
axes.set_title('How many solutions?')
axes.set_xlabel(r'x_1')
axes.set_ylabel(r'x_2')
axes.legend(loc=1)

plt.show()


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
Choosing $x_2$ to pivot back to seems like a good idea, right? Nope.
Can you write Python code for it?

pivot('x_2', 's_2', tableaux_degenerate[1])
tableaux_degenerate[1]


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
In dual form, this is
\begin{align}
\text{minimise}   \quad & - 4y_1 - 3y_2 - 6y_3       \
\text{subject to} \quad &   2y_1 + 3y_2 + 3y_3 = 5 \
                        &    y_1 + 3y_2 + 4y_3 = 7 \
\text{and}        \quad & { y_i \geq 0 }_{i=1}^3 \
\end{align}
Which can be seen as minimising the objective function on the line segment  formed by intersecting the 2 constraint planes. We can also interpret this as wanting to maximise the nutritional value of our meals, given that trying to increase the quantity of one nutrient will necessarily mean that we have to give up some amount of another nutrient.
The Simplex Method
Standard Form (for the Simplex Method)
\begin{align}
\text{maximise}   \quad & c_1 x_1 + c_2 x_2 + \ldots + c_m x_m \
\text{subject to} \quad & a_{11} x_1 + a_{12} x_2 + \ldots + a_{1m} x_m \leq b_1 \
                        & a_{21} x_2 + a_{22} x_2 + \ldots + a_{2m} x_m \leq b_2 \
                        & \vdots \
                        & a_{n1} x_n + a_{n2} x_n + \ldots + a_{nm} x_m \leq b_n \
\text{and}        \quad & { x_i \geq 0 }{i=1}^m \text{ and } { b_j \geq 0 }{j=1}^n
\end{align}

If you are currently trying to minimise the objective function, turn it into a maximisation problem by taking the negative of the expression
Turn all the inequality constraints into equality constraints by adding slack variables
If these transformation still don't allow your system of equations to fit the form, solve the dual form of the problem!

System of Constraint Equations
\begin{align}
\text{maximise}   \quad & c_1 x_1 + c_2 x_2 + \ldots + c_m x_m = z \
\text{subject to} \quad & a_11 x_1 + a_12 x_2 + \ldots + a_1m x_m + s_1 = b_1 \
                        & a_21 x_2 + a_22 x_2 + \ldots + a_2m x_m + s_2 =  b_2 \
                        & \vdots \
                        & a_n1 x_n + a_n2 x_n + \ldots + a_nm x_m + s_n =  b_n \
\text{and}        \quad & { x_i \geq 0 }{i=1}^{m}, ~ { s_i \geq 0 }{j=1}^{n}, ~ \text{ and } { b_j \geq 0 }_{j=1}^n
\end{align}
Taking another look at our diet problem, we can...
Can you write Python code for it?

import pandas as pd
pd.set_option('display.notebook_repr_html', True)

def pivot(departing, entering, tab):
    dpi = tab[tab['basic_variable']==departing].index[0] # index of the departing row

    # update basic variable
    tab['basic_variable'][dpi] = entering

    # normalise departing_row
    tab.ix[dpi,0:-1] = tab.ix[dpi,0:-1] / tab[entering][dpi]

    departing_row = tab.ix[dpi,0:-1]

    # do gauss-jordan on entering variable column
    for row in tab.index[tab.index!=dpi]:
        tab.ix[row, 0:-1] = tab.ix[row, 0:-1] - tab[entering][row] * departing_row

# Bland's rule
def calculate_ratio(entering, tab):
    ratios = tab.ix[0:-1, 'value'] * 0 - 1

    for index, is_valid in enumerate(tab.ix[0:-1, entering] > 0):
        if is_valid==True:
            ratios[index] = tab.ix[index, 'value']/tab.ix[index, entering]
    return ratios

def find_entering(tab):
    return tab.ix['z',0:-2].idxmin()

def find_departing(ratios, tab):
    return tab.ix[ratios[ratios>=0].idxmin(),'basic_variable']

def update_stats(tab):

    print "Basic variables: "
    basic_variables = tab.ix[0:-1, 'basic_variable'].values
    print basic_variables

    print "Non-basic variables: "
    non_basic_variables = numpy.setdiff1d(tab.columns[0:-2], basic_variables)
    print non_basic_variables

    print "Entering variable: "
    entering_variable = find_entering(tab)
    print entering_variable

    print "Ratios: "
    ratios = calculate_ratio(entering_variable, tab)
    print ratios

    print "Departing variable: "
    departing_variable = find_departing(ratios, tab)
    print departing_variable

    return departing_variable, entering_variable

def is_optimum(tab):
    return (tab.ix['z',0:-2] >= 0).all()

def run_simplex(tableau_dict, tableau_orig, max_iterations=10, force_iterations=0):
    if force_iterations == 0:
        for i in xrange(max_iterations):
            tableau_dict[i] = tableau_orig.copy()
            display(tableau_orig)
            if is_optimum(tableau_orig):
      ...


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
The (College Student) Diet Problem
Consider the canonical college student. After a hard afternoon's work of solving way too many partial differential equations, she emerges from her room to obtain sustenance for the day.
She has a choice between getting chicken over rice (\$5) from the halal cart on her street ($r$), or subs (\$7) from the deli ($s$). She's a poor college student, so she will obviously want to get her money's worth. This is obviously an optimisation problem: she wants to find the amount of chicken over rice and subs she has to buy in order to minimise the total cost she spends on food.
$$
\text{minimise} \quad 5r + 7s
$$
In optimisation, we like to call this expression the objective function.
Well, it's not as simple as that. A girl's got to get her fill of daily nutrients. Fibre, protein, and carbohydrates are all important, and however far away food pyramids are from the quotidien thoughts of college students, a girl can still dream of a pseudo-healthy diet with at least 4 servings of fibre, 3 servings of protein, and 6 servings of carbohydrates.
A chicken over rice has 2 servings of fibre, 3 servings of protein, and 3 servings of carbohydrates, while a sub has 1 serving of fibre, 3 servings of protein, and 4 servings of carbohydrates. To find the combination of meals that satisfies the daily nutritional requirements, we impose the following constraints:
\begin{align}
\text{Fibre: } &2r +  s \geq 4 \
\text{Protein: } &3r + 3s \geq 3 \
\text{Carbohydrates: } &3r + 4s \geq 6
\end{align}
Visualising the Problem
Can you write Python code for it?

fig = plt.figure()
axes = fig.add_subplot(1,1,1)

# define view
r_min = 0.0
r_max = 3.0
s_min = 0.0
s_max = 5.0
res = 50

r = numpy.linspace(r_min, r_max, res)

# plot axes
axes.axhline(0, color='#B3B3B3', linewidth=5)
axes.axvline(0, color='#B3B3B3', linewidth=5)

# plot constraints
c_1 = lambda x: 4 - 2*x
c_2 = lambda x: 1 - x
c_3 = lambda x: 0.25 * ( 6 - 3*x )
c_1_line = axes.plot( r, c_1(r), label='Fibre' ) # 2r +  s \geq 4
c_2_line = axes.plot( r, c_2(r), label='Protein' ) # 3r + 3s \geq 3
c_3_line = axes.plot( r, c_3(r), label='Carbohydrate' ) # 3r + 4s \geq 6

# plot objective
s = numpy.linspace(s_min, s_max, res)
c = numpy.empty([r.size, s.size])
for i, r_i in enumerate(r):
    c[:,i] = 5 * r_i + 12 * s

axes.contourf(r, s, c, res, cmap='Oranges', alpha=0.5)
r_cut = numpy.linspace(0.0, 2.0, 100)
axes.fill_between(r_cut, c_1(r_cut), color='w')

# plot cost minimising point
axes.plot(2.0, 0, 'o')

# label graph
axes.set_title('Visualising the Diet Problem')
axes.set_xlabel('Chicken Over Rice')
axes.set_ylabel('Sub')
axes.legend()

plt.show()


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
We got an error!
ValueError: attempt to get argmin of an empty sequence
Usually, errors are bad things, but in this case, the error is trying to tell us something
In the code:
return tab.ix[ratios[ratios>=0].idxmin(),'basic_variable']
Which is telling us that no non-negative ratio was found! Why is this a problem for us? Let's take a look at the equations our tableau, and our equations at this point in time:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
Without Bland's Rule, one could potentially choose to pivot on $s_2$, which will give us
Can you write Python code for it?

pivot('s_2', 'x_2', tableaux_degenerate[1])
tableaux_degenerate[1]


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
You think you're moving, but you get nowhere. — Stop and Stare, OneRepublic

As its name suggests, degeneracy is when you get a basic variable (that's supposed to have a non-zero value) with a value of 0, and you are able to modify the value of the objective function without moving on the simplex.
In general, predicting when degeneracy will occur is non-trivial; one source claims that it is NP-complete. You can read more about it here.
Bland's Rule

Choose the non-basic variable with the most negative coefficient as the entering variable
Choose the basic variable producing the smallest value/pivot ratio as the departing variable

Using Bland's Rule, the Simplex Method will never cycle if it encounters degeneracy (i.e. it halts on all inputs)
So what is cycling?
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
\begin{align}
\text{maximise}   \quad &  5x_1 +  7x_2          \
\text{subject to} \quad &  -x_1 +   x_2 + s_1 = 5 \
                        & -\frac{1}{2}x_1 + x_2 + s_2 = 7 \
\text{and}        \quad & x_1, x_2, s_1, s_2 \geq 0
\end{align}
Can you write Python code for it?

c_1 = numpy.array([[  -1,  1, 1, 0,  5, 's_1']])
c_2 = numpy.array([[-0.5,  1, 0, 1,  7, 's_2']])
z   = numpy.array([[  -5, -7, 0, 0,  0,    '']])
rows= numpy.concatenate((c_1, c_2, z), axis=0)

tableau_unbounded = pd.DataFrame(rows, columns=['x_1','x_2','s_1','s_2','value', 'basic_variable'], index=['c_1','c_2','z'])
tableau_unbounded.ix[:,0:-1] = tableau_unbounded.ix[:,0:-1].astype('float')

tableaux_unbounded = dict()
run_simplex(tableaux_unbounded, tableau_unbounded)


I am working on the file ".ipynb_checkpoints/lpsm-checkpoint.ipynb".
The first task is:
\begin{align}
\text{maximise}   \quad &  5x_1 +  7x_2          \
\text{subject to} \quad &  2x_1 +   x_2 + s_1 = 4 \
                        & 10x_1 + 14x_2 + s_2 = 30 \
\text{and}        \quad & x_1, x_2, s_1, s_2 \geq 0
\end{align}
Can you write Python code for it?

c_1 = numpy.array([[ 2,  1, 1, 0,  4, 's_1']])
c_2 = numpy.array([[10, 14, 0, 1, 30, 's_2']])
z   = numpy.array([[-5, -7, 0, 0,  0,    '']])
rows= numpy.concatenate((c_1, c_2, z), axis=0)

tableau_multiple = pd.DataFrame(rows, columns=['x_1','x_2','s_1','s_2','value', 'basic_variable'], index=['c_1','c_2','z'])
tableau_multiple.ix[:,0:-1] = tableau_multiple.ix[:,0:-1].astype('float')

tableaux_multiple = dict()
run_simplex(tableaux_multiple, tableau_multiple, force_iterations=3)

step_coords = numpy.array([[0.0, 0.0], [0.0, 2.14286], [tableaux_multiple[2].ix['c_1','value'], tableaux_multiple[2].ix['c_2','value']]])
step_value = numpy.array([tableaux_multiple[0].ix['z','value'], tableaux_multiple[1].ix['z','value'], tableaux_multiple[2].ix['z','value']])

def multiple_problem(step):
    fig = plt.figure()
    axes = fig.add_subplot(1,1,1)

    # define view
    x_1_min = 0.0
    x_1_max = 3.0
    x_2_min = 0.0
    x_2_max = 5.0

    # plot axes
    axes.axhline(0, color='k')
    axes.axvline(0, color='k')

    # plot constraints
    x_1 = numpy.linspace(x_1_min, x_1_max, res)
    c_1 = lambda x: 4.0 - 2.0*x
    c_2 = lambda x: (30.0 - 10.0*x)/14.0
    c_1_line = axes.plot( r, c_1(r), label='Constraint 1' ) # 2x_1 +   x_2 \leq 4
    c_2_line = axes.plot( r, c_2(r), label='Constraint 2' ) # 10x_1 + 14x_2 \leq 30

    # plot objective
    x_2 = numpy.linspace(x_2_min, x_2_max, res)
    c = numpy.empty([x_1.size, x_2.size])
    for i, x_1_i in enumerate(x_1):
        c[:,i] = 5 * x_1_i + 7 * x_2

    # color map of objective function values
    axes.contourf(x_1, x_2, c, res, cmap='Oranges', alpha=0.5)

    # shade feasible region
    c_1_bottom = numpy.linspace(0.0, 2.0, res)
    c_2_bottom = numpy.linspace(0.0, 3.0, res)
    axes.fill_between(c_1_bottom, c_1(c_1_bottom), color=plt.rcParams['axes.color_cycle'][0], alpha=0.5)
    axes.fill_between(c_2_bottom, c_2(c_2_bottom), color=plt.rcParams['axes.color_cycle'][1], alpha=0.5)

    # plot point
    axes.plot(step_coords[step][0],...


I am working on the file ".ipynb_checkpoints/lv_notebook-checkpoint.ipynb".
The first task is:
Calculate Quality elements
Create an instance of NP Qualityfactor class
Can you write Python code for it?

    qf_NP = core.QualityFactorNP()
    # use set_data_handler to load the selected data to the QualityFactor
    qf_NP.set_data_handler(data_handler = loaded_filtered_data)


I am working on the file ".ipynb_checkpoints/lv_notebook-checkpoint.ipynb".
The first task is:
Filter parameters in QualityFactorNP
THIS SHOULD BE DEFAULT
Can you write Python code for it?

    print('{}\nApply season filters to parameters in QualityFactor\n'.format('*'*nr_marks))
    # First general filter
    qf_NP.filter_data(data_filter_object = first_filter)
    # winter filter
    qf_NP.filter_data(data_filter_object = winter_filter, indicator = 'TOTN_winter')
    qf_NP.filter_data(data_filter_object = winter_filter, indicator = 'DIN_winter')
    # summer filter
    qf_NP.filter_data(data_filter_object = summer_filter, indicator = 'TOTN_summer')
    print('done\n{}.'.format('*'*nr_marks))


I am working on the file ".ipynb_checkpoints/lv_notebook-checkpoint.ipynb".
The first task is:
Load reference values
Can you write Python code for it?

    print('{}\nLoading reference values'.format('*'*nr_marks))
    core.RefValues()
    core.RefValues().add_ref_parameter_from_file('DIN_winter', resources_directory + '/classboundaries/nutrients/classboundaries_din_vinter.txt')
    core.RefValues().add_ref_parameter_from_file('TOTN_winter', resources_directory + '/classboundaries/nutrients/classboundaries_totn_vinter.txt')
    core.RefValues().add_ref_parameter_from_file('TOTN_summer', resources_directory + '/classboundaries/nutrients/classboundaries_totn_summer.txt')
    print('done\n{}.'.format('*'*nr_marks))


I am working on the file ".ipynb_checkpoints/lv_notebook-checkpoint.ipynb".
The first task is:
Select
Directories and file paths
Can you write Python code for it?

start_year = interactive(return_input,
        value = widgets.Dropdown(
        options=[2009, 2010, 2011, 2012, 2013],
        value=2009,
        description='Select start year:',
        disabled=False)
        )
end_year = interactive(return_input,
        value = widgets.Dropdown(
        options=[2011, 2012, 2013, 2014, 2015, 2016],
        value=2015,
        description='Select start year:',
        disabled=False)
        )

from IPython.display import display
display(start_year, end_year)

print(start_year.result, end_year.result)

test_widget = core.jupyter_eventhandlers.MultiCheckboxWidget(['Bottenfauna', 'Växtplankton','Siktdjup','Näringsämnen'])

test_widget # Display the widget

if __name__ == '__main__':
    nr_marks = 60
    print('='*nr_marks)
    print('Running module "lv_test_file.py"')
    print('-'*nr_marks)
    print('')

    #root_directory = os.path.dirname(os.path.abspath(__file__)) # works in
    root_directory = os.getcwd() # works in notebook
    resources_directory = root_directory + '/resources'
    filter_directory = root_directory + '/workspaces/default/filters'
    data_directory = root_directory + '/workspaces/default/data'
#    est_core.StationList(root_directory + '/test_data/Stations_inside_med_typ_attribute_table_med_delar_av_utsjö.txt')
    core.ParameterList()

    #--------------------------------------------------------------------------

    print('{}\nSet directories and file paths'.format('*'*nr_marks))
    raw_data_file_path = data_directory + '/raw_data/data_BAS_2000-2009.txt'
    first_filter_data_directory = data_directory + '/filtered_data'

    first_data_filter_file_path = filter_directory + '/selection_filters/first_data_filter.txt'
    winter_data_filter_file_path = filter_directory + '/selection_filters/winter_data_filter.txt'
    summer_data_filter_file_path = filter_directory + '/selection_filters/summer_data_filter.txt'

    tolerance_filter_file_path = filter_directory + '/tolerance_filters/tolerance_filter_template.txt'