I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Token Definitions
Next, we need to provide the definition of the tokens.  One way to define tokens is via python functions.
In this notebook we are only going to use these functional token definitions.
The document string of these functions is a raw string that contains the regular expression defining the semantics of the token.  The regular expression can be followed by code that is needed to further process the token.  The name of the function defining a token has to have the form t_name, where name is the name of the token as declared in the list tokens.
The HEADER Token
The token HEADER matches any string that is made up of upper and lower case characters followed by a colon.  This colon may be followed by arbitrary characters.
The token extends to the end of the line and includes the terminating newline.
When the function t_HEADER is called it is provided with a token t.  This is an object that has five
attributes:
- t.lexer is an object of class Lexer that contains the scanner that was used to extract the token t.
  We are free to attach additional attributes to this Lexer object.
- t.type is  a string containing the type of the token.  For tokens processed in the function
  t_HEADER this type is always the string HEADER.
- t.value is the actual string matched by the token.
- t.lexpos is the position of the token in the input string that is scanned.
Furthermore, the lexer object has one important attribute:
- t.lexer.lineno is the line number.  However, it is our responsibility to update this variable
  by incrementing t.lexer.lineno every time we read a newline.
In the case of the token HEADER we need to increment the attribute t.lineno, as the regular expression contain a newline.
Can you write Python code for it?

def t_HEADER(t):
    r'[A-Za-z]+:.*\n'
    t.lexer.lineno += 1


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Token Definitions
Next, we need to provide the definition of the tokens.  One way to define tokens is via python functions.
In this notebook we are only going to use these functional token definitions.
The document string of these functions is a raw string that contains the regular expression defining the semantics of the token.  The regular expression can be followed by code that is needed to further process the token.  The name of the function defining a token has to have the form t_name, where name is the name of the token as declared in the list tokens.
The HEADER Token
The token HEADER matches any string that is made up of upper and lower case characters followed by a colon.  This colon may be followed by arbitrary characters.
The token extends to the end of the line and includes the terminating newline.
When the function t_HEADER is called it is provided with a token t.  This is an object that has five
attributes:
- t.lexer is an object of class Lexer that contains the scanner that was used to extract the token t.
  We are free to attach additional attributes to this Lexer object.
- t.type is  a string containing the type of the token.  For tokens processed in the function
  t_HEADER this type is always the string HEADER.
- t.value is the actual string matched by the token.
- t.lineno is the line number.  However, it is our responsibility to update this variable
  by incrementing t.lexer.lineno every time we read a newline.
- t.lexpos is the position of the token in the input string that is scanned.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Tricking Ply
The line below is necessary to trick ply.lex into assuming this program is part of an ordinary python file instead of being a Jupyter notebook.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Converting Html to Text
This notebook shows how we can use the package ply
to extract the text that is embedded in an Html file.
In order to be concise, it only supports a small subset of
Html.  Below is the content of my old
web page that I had used when I still worked at the DHBW Stuttgart.  The goal of this notebook is to write
a scanner that is able to extract the text from this web page.
Can you write Python code for it?

data = \
'''

    <meta charset="utf-8">
    <title>Homepage of Prof. Dr. Karl Stroetmann</title>
    <link type="text/css" rel="stylesheet" href="style.css" />
    <link href="http://fonts.googleapis.com/css?family=Rochester&subset=latin,latin-ext"
          rel="stylesheet" type="text/css">
    <link href="http://fonts.googleapis.com/css?family=Pacifico&subset=latin,latin-ext"
          rel="stylesheet" type="text/css">
    <link href="http://fonts.googleapis.com/css?family=Cabin+Sketch&subset=latin,latin-ext" rel="stylesheet" type="text/css">
    <link href="http://fonts.googleapis.com/css?family=Sacramento" rel="stylesheet" type="text/css">

Prof. Dr. Karl Stroetmann

          Duale Hochschule Baden-Württemberg

Coblitzallee 1-9

68163 Mannheim

Germany

Office:     Raum 344B

Phone:    +49 621 4105-1376

Fax:        +49 621 4105-1194

Skype:     karlstroetmann

          Some links:

	      My lecture notes,
              as well as the programs presented in class, can be found
              at

              https://github.com/karlstroetmann.

Most of my papers can be found at researchgate.net.

The programming language SetlX can be downloaded at...


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Definition of the States
Once we are inside an Html header or inside of some
JavaScript code the rules of the scanning game change.  Therefore, we declare two new exclusive scanner states:
- header is the state the scanner is in while it is scanning an
  Html header.
- script is the state of the scanner while scanning JavaScript code.
These states are exclusive states and hence the other token definitions do not apply in these
states.
Can you write Python code for it?

states = [ ('header', 'exclusive'),
           ('script', 'exclusive')
         ]


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Error Handling
The function t_error is called when a substring at the beginning of the input can not be matched by any of the regular expressions defined in the various tokens.  In our implementation we print the first character that could not be matched, discard this character and continue.
Can you write Python code for it?

def t_error(t):
    print(f"Illegal character: '{t.value[0]}'")
    t.lexer.skip(1)


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Imports
We will use the package ply to remove the
Html tags and extract the text that
is embedded in the Html shown above.
In this example, we will only use the scanner that is provided by the module ply.lex.
Hence we import the module ply.lex that contains the scanner generator from ply.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
In order to scan the data that we provided in the last line, we iterate over all tokens generated by our scanner.
Can you write Python code for it?

def scan(lexer):
    for t in lexer:
        pass

scan(lexer)


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Next, we feed our input string into the generated scanner.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Running the Scanner
The line below is necessary to trick ply.lex into assuming this program is written in an ordinary python file instead of a Jupyter notebook.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token ANY
If the scanner is either in the state header or the state script, the function
t_header_script_ANY eats up all characters without echoing them.
Can you write Python code for it?

def t_header_script_ANY(t):
    r'.|\n'
    pass


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token ANY
The regular expression . matches any character that is different from a newline character.  These characters are printed unmodified.  Note that the scanner tries the regular expressions for a given state in the order that they are defined in this notebook.  Therefore, it is crucial that the function t_ANY is defined after all other token definitions for the INITIAL state are given.  The INITIAL state is the default state of the scanner and therefore the state the scanner is in when it starts scanning.
Can you write Python code for it?

def t_ANY(t):
    r'.'
    print(t.value, end='')


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token HEAD_END
The regular expression </head> matches the closing head tag.  Note that is regular expression is only
active in state header as the name of this function starts with t_header.  Once the closing tag has been found, the function lexer.begin switches the lexer back into the state INITIAL, which is the
start state of the scanner.  In the state INITIAL, all token definitions are active, that do not start with either t_header or t_script.
Can you write Python code for it?

def t_header_HEAD_END(t):
    r'</head>'
    t.lexer.begin('INITIAL')


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token NAMED_ENTITY
In order to support named Html entities we need to import
the dictionary html5 from the module html.entities.  For every named
Html entity e, html[e] is the unicode symbol that is specified by e.
Can you write Python code for it?

from html.entities import html5

html5['auml']


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token SCRIPT_END
If the scanner is either in the state  script, the function
t_header_script_END recognizes the matching closing tag and switches back to the state
INITIAL.
The regular expression </script> matches the closing script tag.  Note that this regular expression is only
active in state script.  Once the closing tag has been found, the function lexer.begin switches the lexer back into the state INITIAL, which is the start state of the scanner.
Can you write Python code for it?

def t_script_SCRIPT_END(t):
    r'</script>'
    t.lexer.begin('INITIAL')


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token SCRIPT_END
If the scanner is in the state script, the function t_script_SCRIPT_END recognizes the matching closing tag and switches back to the state
INITIAL.
The regular expression </script> matches the closing script tag.  Note that this regular expression is only
active in state script.  Once the closing tag has been found, the function lexer.begin switches the lexer back into the state INITIAL, which is the start state of the scanner.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token SCRIPT_START
Once the scanner reads the opening tag <script> it switches into the state script.  In this state it will continue to read and discard characters until it sees the closing tag </script>.
Can you write Python code for it?

def t_SCRIPT_START(t):
    r'<script>'
    t.lexer.begin('script')


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token UNICODE
The regular expression &\#[0-9]+;? searches for Html entities that specify a unicode character numerically.  The corresponding strings start with the character &
followed by the character # followed by digits and are optionally ended by the character ;.
Note that we had to escape the character # with a  backslash because otherwise this character would signal the begin of a comment.
Note further that the function chr takes a number and returns the corresponding unicode character.
For example, chr(128034) returns the character '🐢'.
Can you write Python code for it?

def t_UNICODE(t):
    r'&\#[0-9]+;?'
    if t.value[-1] == ';':
        number = t.value[2:-1]
    else:
        number = t.value[2:]
    print(chr(int(number)), end='')

chr(8555)

chr(128034)


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The Definition of the Token `LINEBREAK``
Groups of newline characters are condensed into a single newline character.
As we are not interested in the variable t.lexer.lineno in this example, we don't have to count the newlines.
This token is active in any state.
Can you write Python code for it?

def t_LINEBREAK(t):
    r'\n+'
    print()


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The function t_header_error is called when a substring at the beginning of the input can not be matched by any of the regular expressions defined in the various tokens and the scanner is in state header.  Actually, this function can never be called.
Can you write Python code for it?

def t_header_error(t):
    print(f"Illegal character in state 'header': '{t.value[0]}'")
    t.lexer.skip(1)


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The function t_script_error is called when a substring at the beginning of the input can not be matched by any of the regular expressions defined in the various tokens and the scanner is in state script.
Can you write Python code for it?

def t_script_error(t):
    print(f"Illegal character in state 'script': '{t.value[0]}'")
    t.lexer.skip(1)


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The function t_script_error is called when a substring at the beginning of the input can not be matched by any of the regular expressions defined in the various tokens and the scanner is in state script.  Actually, this function can never be called.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The line below generates the scanner.  Because the option debug=True is set, we can see the regular expression that is generated for scanning.
Can you write Python code for it?


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
The regular expression &[a-zA-Z]+;? searches for Html
entity names.  These are strings that start with the character & followed by the name of the entity, optionally followed by the character ;.  If a Unicode entity name is found, the corresponding character is printed.
Can you write Python code for it?

def t_NAMED_ENTITY(t):
    r'&[a-zA-Z]+;?'
    if t.value[-1] == ';':          # ';' is not part of the entity name
        entity_name = t.value[1:-1] # so chop it off
    else:
        entity_name = t.value[1:]
    unicode_char = html5[entity_name]
    print(unicode_char, end='')


def t_NAMED_ENTITY(t):
    r'&[a-zA-Z]+;?'
    if t.value[-1] == ';':          # ';' is not part of the entity name
        entity_name = t.value[1:-1] # so chop it off
    else:
        entity_name = t.value[1:]
    unicode_char = html5[entity_name]
    print(unicode_char, end='')     # don't print a line break


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Token Declarations
We begin by declaring the tokens.  Note that the variable tokens is a keyword of ply to define the names of the token classes.  In this case, we have declared nine different tokens.
- HEAD_START will match the tag <head> that starts the definition of the
  Html header.
- HEAD_END will match the tag </head> that ends the definition of the
  Html header.
- SCRIPT_START will match the tag <script> that starts embedded JavaScript code.
- SCRIPT_END will match the tag </script> that ends embedded JavaScript code.
- TAG is a token that represents arbitrary Html tags.
- LINEBREAK is a token that will match the newline character \n at the end of a line.
- NAMED_ENTITY is a token that represents named
  Html5 entities.
- UNICODE is a token that represents a unicode entity.
- ANY is a token that matches any character.
Can you write Python code for it?

tokens = [ 'HEAD_START',
           'HEAD_END'
           'SCRIPT_START',
           'SCRIPT_END',
           'TAG',
           'LINEBREAK',
           'NAMED_ENTITY',
           'UNICODE',
           'ANY'
         ]


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Token Definitions
We proceed to give the definition of the tokens.  Note that none of the function defined below
returns a token.  Rather all of these function print the transformation of the
Html that they have matched.
The Definition of the Token HEAD_START
Once the scanner reads the opening tag <head> it switches into the state header.  The function begin of the lexer can be used to switch into a different scanner state.  In the state header, the scanner continues to read and discard characters until the closing tag </head> is encountered.  Note that this token is only recognized in the state INITIAL.  The state INITIAL is the initial state of the scanner, i.e. the scanner always starts in this state.
Can you write Python code for it?

def t_HEAD_START(t):
    r'<head>'
    t.lexer.begin('header')


I am working on the file "Ply/Html2Text.ipynb".
The first task is:
Token Definitions
We proceed to give the definition of the tokens.  Note that none of the function defined below
returns a token.  Rather all of these function print the transformation of the
Html that they have matched.
The Definition of the Token HEAD_START
Once the scanner reads the opening tag <head> it switches into the state header.  The function begin of the lexer can be used to switch into a different scanner state.  In the state header, the scanner continues to read and discard characters until the closing tag </head> is encountered.  Note that this token is only recognized in the state START.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
... lots of token definitions here ...
Can you write Python code for it?

def t_formula_error(t):
    print(f"Illegal character in state 'formula': '{t.value[0]}'")
    t.lexer.skip(1)


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:

Below are some predefined functions that you can use to create the Html file.

The function start_html writes the header of the Html file
and the opening <body> tag to the file opened above.
Can you write Python code for it?

def start_html():
    outfile.write('<!doctype html>\n')
    outfile.write('<html>\n')
    outfile.write('<head>\n')
    outfile.write('<script type="text/javascript" ')
    outfile.write('src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML">\n')
    outfile.write('</script>\n')
    outfile.write('<body>\n\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Arguments of functions like the square root or exponents have to be enclosed in pairs of <mrow> and </mrow> tags.
Can you write Python code for it?

def start_row():
    outfile.write('<mrow>\n')

def end_row():
    outfile.write('</mrow>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Converting $\LaTeX$ to Html
The purpose of the following exercise is to implement a translator from $\LaTeX$ to
MathML.  $\LaTeX$ is a document markup language
that is especially well suited to present text that contains mathematical formulas.  MathML is the part of Html that deals with the representation of mathematical formulas.  As $\LaTeX$ provides a very rich
document markup language and we can only afford to spend a few hours on this exercise, we confine
ourselves to a small subset of $\LaTeX$.  The file example.tex contains some $\LaTeX$.  The goal of this exercise is to implement a translator that is able to transform this file into MathML.
We start with reading the file.
Can you write Python code for it?

with open('example.tex') as f:
    data = f.read()


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
In order to scan the data that we provided in the last line, we iterate over all tokens generated by our scanner.
Can you write Python code for it?

def scan(lexer):
    for t in lexer:
        pass

scan(lexer)

outfile.close()

!open 'example.html'


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
In order to write exponents we have to use the tag <msup>.  For example, the expression $a^2$
is equivalent to the following markup:
<msup>
<mi>a</mi>
<mrow>
<mn>2</mn>
</mrow>
</msup>
Note that the exponent is enclosed in <mrow> </mrow> tags.
Can you write Python code for it?

def start_super():
    outfile.write('<msup>\n')

def end_super():
    outfile.write('</msup>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
In order to write fractions we have to use the tag <mfrac>.  For example, the expression $\frac{1}{6}$
is equivalent to the following markup:
<mfrac>
<mrow>
<mn>1</mn>
</mrow>
<mrow>
<mn>6</mn>
</mrow>
</mfrac>
Note that both nominator and denominator are enclosed in <mrow> </mrow> tags.
Can you write Python code for it?

def start_fraction():
    outfile.write('<mfrac>\n')

def end_fraction():
    outfile.write('</mfrac>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Let us look at the output file example.pdf that is produced if we run $\LaTeX$ on this file.
If you are nor running MacOS you have to replace the command
open with the name of an executable that can open a .pdf-file.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Mathematical operators should be enclosed in pairs of <mo> and </mo> tags.  For example, the operator $+$ is displayed by the following MathML:
<mo>+</mo>
Can you write Python code for it?

def write_operator(op):
    outfile.write('<mo>' + op + '</mo>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Next, we feed our input string into the generated scanner.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Next, we open the file example.html.  The scanner we are going to implement has to write its output into this file.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Now data contains the text that is stored in this file.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The function start_math_block ends a math block.
Can you write Python code for it?

def end_math():
    outfile.write('</math>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The function start_math_inline starts an inline formula, i.e. a formula enclosed in $.  Formulas of this type are part of the surrounding text.
Can you write Python code for it?

def start_math_inline():
    outfile.write('<math xmlns="http://www.w3.org/1998/Math/MathML" display="inline">\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The function write_any writes a single character unadorned to the output file.
Can you write Python code for it?

def write_any(char):
    outfile.write(char)


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The functions start_sqrt and end_sqrt write code to display formulas involving square roots.  For example, to display  the expression
$$ \sqrt{a^2 + b^2} $$
we can use the following MathML:
<msqrt>
<mrow>
<msup>
<mi>a</mi>
<mrow>
<mn>2</mn>
</mrow>
</msup>
<mo>+</mo>
<msup>
<mi>b</mi>
<mrow>
<mn>2</mn>
</mrow>
</msup>
</mrow>
</msqrt>
Can you write Python code for it?

def start_sqrt():
    outfile.write('<msqrt>\n')

def end_sqrt():
    outfile.write('</msqrt>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The functions start_sum and end_sum write code to display formulas involving sums.  For example, to display  the expression
$$ \sum\limits_{i=1}^n i^2 $$
we can use the following MathML:
```
<munderover>
<mo>∑</mo>
<mrow>
<mi>i</mi>
<mo>=</mo>
<mn>1</mn>
</mrow>
<mrow>
<mi>n</mi>
</mrow>
</munderover>
<msup>
<mi>i</mi>
<mrow>
<mn>2</mn>
</mrow>
</msup>
```
Can you write Python code for it?

def start_sum():
    outfile.write('<munderover>\n')
    outfile.write('<mo>∑</mo>\n')

def end_sum():
    outfile.write('</munderover>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The line below generates the scanner.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The line below is necessary to trick ply.lex into assuming this program is written in an ordinary python file instead of a Jupyter notebook.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The symbol $\cdot$ is created by the following MathML:
<mo>&sdot;</mo>
Can you write Python code for it?

def write_times():
    outfile.write('<mo>⋅</mo>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The symbol $\geq$ is created by the following MathML:
<mn>&ge;</mn>
Can you write Python code for it?

def write_geq():
    outfile.write('<mo>≥</mo>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The symbol $\leq$ is created by the following MathML:
<mn>&le;</mn>
Can you write Python code for it?

def write_leq():
    outfile.write('<mo>≤</mo>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
The symbol $\pi$ is created by the following MathML:
<mn>&pi;</mn>
Can you write Python code for it?

def write_pi():
    outfile.write('<mn>π</mn>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
Variable names should be enclosed in pairs of <mi> and </mi> tags.  For example, the variable $x$ is displayed by the following MathML:
<mi>x</mi>
Can you write Python code for it?

def write_var(v):
    outfile.write('<mi>' + v + '</mi>\n')


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
We will be use the library ply to translate $\LaTeX$ into
MathML.
We only use the scanner that is provided by the module ply.lex.
Hence we import the module ply.lex that contains the scanner generator from ply.
Can you write Python code for it?


I am working on the file "Ply/LaTeX2HTML.ipynb".
The first task is:
When we see a closing brace } things get difficult.  The reason is that we need to know what type of formula is being closed. Is it a square root, the subscript of a sum, the superscript of a sum, some part of a fraction, or an exponent.  My idea is to use a stack that is attached to the lexer, i.e. we have a variable lexer.stack that stores this information.  We need a stack since formulas might be nested.
Furthermore, the scanner has two different states.  Either we are inside a formula, i.e. inside something that is enclosed in dollar symbols, or we are inside text that needs to be echoed unchanged to the output file.
Can you write Python code for it?

states = [ ('formula', 'exclusive') ]

def t_HEAD(t):
    r'\\documentclass\{article\}'
    pass

def t_BEGIN_DOCUMENT(t):
    r'\\begin\{document\}'
    start_html()

def t_END_DOCUMENT(t):
    r'\\end\{document\}'
    end_html()

def t_DOLLAR_DOLLAR(t):
    r'\$\$'
    t.lexer.begin('formula')
    t.lexer.stack = []
    t.lexer.stack.append('INITIAL')
    start_math_block()

def t_DOLLAR(t):
    "your code here"

def t_ANY(t):
    r'.|\n'
    write_any(t.value)

def t_formula_DOLLAR_DOLLAR(t):
    r'\$\$'
    t.lexer.begin('INITIAL')
    end_math()

def t_formula_DOLLAR(t):
    "your code here"


I am working on the file "Ply/Look-Ahead-Solved.ipynb".
The first task is:
Dealing with Conflicts
This notebook discusses conflicts that have their origin in insufficient looakahead.
We will discuss the following grammar:
```
a : b 'V'
  | c 'W'
b : 'X' 'U'
c : 'X' 'U'
```
Specification of the Scanner
We implement a minimal scanner.  Below we need to declare.  The reason is that Ply only works when the list
tokens is defined and contains at least one token.
Can you write Python code for it?

import ply.lex as lex

tokens   = [ 'USELESS' ]
literals = ['U', 'V', 'W', 'X']

def t_USELESS(t):
    r'This will never be used.'

__file__ = 'main'

lexer = lex.lex()


I am working on the file "Ply/Look-Ahead-Solved.ipynb".
The first task is:
Specification of the Parser
Can you write Python code for it?


I am working on the file "Ply/Look-Ahead-Solved.ipynb".
The first task is:
The start variable of our grammar is expr, but we dont't have to specify that.  The default
start variable is the first vvariable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Look-Ahead.ipynb".
The first task is:
Dealing with Lookahead Conflicts
This notebook discusses conflicts that have their origin in insufficient looakahead.
We will discuss the following grammar:
```
a : b 'U' 'V'
  | c 'U' 'W'
b : 'X'
c : 'X'
```
Specification of the Scanner
We implement a minimal scanner.  Below we need to declare.  The reason is that Ply only works when the list
tokens is defined and contains at least one token.
Can you write Python code for it?


I am working on the file "Ply/Look-Ahead.ipynb".
The first task is:
Let's look at the action table that is generated.  Conflicts are always resolved in favour of shifting.
Can you write Python code for it?


I am working on the file "Ply/Look-Ahead.ipynb".
The first task is:
Specification of the Parser
Can you write Python code for it?


I am working on the file "Ply/Look-Ahead.ipynb".
The first task is:
The start variable of our grammar is expr, but we dont't have to specify that.  The default
start variable is the first vvariable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Mysterious-Conflicts.ipynb".
The first task is:
Dealing with Mysterious Conflicts
This file shows reduce/reduce conflicts that result from the fact that the set of LR states is compressed into the set of LALR states.
Specification of the Scanner
We implement a minimal scanner for arithmetic expressions.
Can you write Python code for it?

import ply.lex as lex

tokens = [ 'X' ]

def t_X(t):
    r'x'
    return t

literals = ['v', 'w', 'y', 'z']

t_ignore  = ' \t'

def t_newline(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')

def t_error(t):
    print(f"Illegal character '{t.value[0]}'")
    t.lexer.skip(1)

__file__ = 'main'

lexer = lex.lex()


I am working on the file "Ply/Mysterious-Conflicts.ipynb".
The first task is:
Interestingly, there are no conflicts. If Ply had indeed calculated LALR tables, there would have been conflicts. However, Ply is smart enough to not merge states
if this merger would result in a reduce/reduce conflict.  Below, state 6 and state 9
should have been merged since they have the same core, but as the merger would produce
a reduce/reduce conflict, Ply does not merge these states.
Can you write Python code for it?


I am working on the file "Ply/Mysterious-Conflicts.ipynb".
The first task is:
The start variable of our grammar is expr, but we dont't have to specify that.  The default
start variable is the first vvariable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Mysterious-Conflicts.ipynb".
The first task is:
We can specify multiple expressions in a single rule.  In this case, we have used the pass statement as we just want to check whether there are reduce/reduce conflicts.
Can you write Python code for it?

def p_s(p):
    """
    s : 'v' a 'y'
      | 'w' b 'y'
      | 'v' b 'z'
      | 'w' a 'z'

    a : X

    b : X
    """
    pass

def p_error(p):
    if p:
        print(f'Syntax error at {p.value}.')
    else:
        print('Syntax error at end of input.')


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
All characters not recognized by any of the defined tokens are handled by the function t_error.
The function t.lexer.skip(1) skips the character that has not been recognized. Scanning resumes
after this character has been discarded.
Can you write Python code for it?

def t_error(t):
    print(f"Illegal character {t.value[0]} at line {t.lexer.lineno}.")
    t.lexer.skip(1)


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
Below the function lex.lex() creates the lexer specified above.  Since this code is expected to be part
of some python file but really isn't since it is placed in a Jupyter notebook we have to set the variable
__file__ manually to fool the system into believing that the code given above is located in a file
called hugo.py.  Of course, the name hugo is totally irrelevant and could be replaced by any other name.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
If we need to transform a token, we can define the token via a function.  In that case, the first line of the function
has to be a string that is a regular expression.  This regular expression then defines the token.  After that,
we can add code to transform the token.  The string that makes up the token is stored in t.value.  Below, this string
is transformed into an integer via the predefined function int.
Can you write Python code for it?

def t_NUMBER(t):
    r'0|[1-9][0-9]*'
    t.value = int(t.value)
    return t


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
If we need to transform a token, we can define the token via a function.  In that case, the first line of the function
has to be a string that is a regular expression.  This regular expression then defines the token.  After that,
we can add code to transform the token.  The string that makes up the token is stored in t.value.  Below, this string
is transformed into an integer.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
Let us feed the scanner with the string data.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
Let us feed the scanner with the string data.  This is done by calling the method input of the generated scanner.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
Lets test the generated scanner, that is stored in lexer, with the following string:
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
Next, we define regular expressions that define the tokens that are to be recognized.
Note that some operators have to be prefixed with a backslash since these operators are
also used as operators for regular expressions.  Note also that the token names have to be prefixed with
the string t_.
Can you write Python code for it?

t_PLUS    = r'\+'
t_MINUS   = r'-'
t_TIMES   = r'\*'
t_DIVIDE  = r'/'
t_LPAREN  = r'\('
t_RPAREN  = r'\)'


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
The keyword t_ignore specifies those characters that should be discarded.
In this case, spaces and tabs are ignored.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
The rule below is used to keep track of line numbers. We use the function length since there might be
more than one newline.  The member variable lexer.lineno keeps track ofthe current line number.  This
is needed for error messages.
Can you write Python code for it?

def t_newline(t):
    r'\n+'
    t.lexer.lineno += len(t.value)


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
There are two ways to define these tokens:
 - immediate token definitions define the token by assigning a regular expression to a variable of the form t_name,
   where name is the name of the token that is defined.
 - functional token definitions define the token via a function.  The regular expression that defines the token
   is the string appearing in the first line of the function body.
We see examples below.  We start with the immediate token definitions.  Note that we have to use raw strings here to prevent
the expansion of backslash sequences.  Furthermore, operator symbols have to be escaped with a backslash character.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
This example has been extracted from the official documentation of Ply.
A Tokenizer for Numbers and the Arithmetical Operators
The module ply.lex contains the code that is necessary to create a scanner.
Can you write Python code for it?


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
We start with a definition of the token names.  Note that all token names have to start with
a capital letter.
Can you write Python code for it?

tokens = [
   'NUMBER',
   'PLUS',
   'MINUS',
   'TIMES',
   'DIVIDE',
   'LPAREN',
   'RPAREN'
]


I am working on the file "Ply/Ply-Scanning-Example.ipynb".
The first task is:
We start with a definition of the token names.  Note that all token names have to start with
a capital letter.  We have to define these token names as a list with the name tokens:
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
A Simple Symbolic Calculator
This file shows how a simple symbolic calculator can be implemented using Ply.  The grammar for the language implemented by this parser is as follows:
$$
\begin{array}{lcl}
  \texttt{stmnt}   & \rightarrow & \;\texttt{IDENTIFIER} \;\texttt{':='}\; \texttt{expr}\; \texttt{';'}\
                   & \mid        & \;\texttt{expr}\; \texttt{';'}                      \[0.2cm]
  \texttt{expr}    & \rightarrow & \;\texttt{expr}\; \texttt{'+'} \; \texttt{product}  \
                   & \mid        & \;\texttt{expr}\; \texttt{'-'} \; \texttt{product}  \
                   & \mid        & \;\texttt{product}                                  \[0.2cm]
  \texttt{product} & \rightarrow & \;\texttt{product}\; \texttt{'*'} \;\texttt{factor} \
                   & \mid        & \;\texttt{product}\; \texttt{'/'} \;\texttt{factor} \
                   & \mid        & \;\texttt{factor}                                   \[0.2cm]
  \texttt{factor}  & \rightarrow &   \texttt{'('} \; \texttt{expr} \;\texttt{')'}        \
                   & \mid        & \;\texttt{NUMBER}                                   \
                   & \mid        & \;\texttt{IDENTIFIER}
\end{array}
$$
Specification of the Scanner
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
A Simple Symbolic Calculator
This file shows how a simply symbolic calculator can be implemented using Ply.
Specification of the Scanner
Can you write Python code for it?

import ply.lex as lex

tokens = [ 'NUMBER', 'IDENTIFIER', 'ASSIGN_OP' ]


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
A factor can is either an expression in parenthesis, a number, or an identifier.
factor : '(' expr ')'
          | NUMBER
          | IDENTIFIER
          ;
Can you write Python code for it?

def p_factor_group(p):
    "factor : '(' expr ')'"
    p[0] = p[2]

def p_factor_number(p):
    "factor : NUMBER"
    p[0] = p[1]

def p_factor_id(p):
    "factor : IDENTIFIER"
    p[0] = Names2Values.get(p[1], float('NaN'))


def p_factor_group(p):
    "factor : '(' expr ')'"
    p[0] = p[2]

def p_factor_number(p):
    "factor : NUMBER"
    p[0] = p[1]

def p_factor_id(p):
    "factor : IDENTIFIER"
    p[0] = Names2Values.get(p[1], float('nan'))

def p_error(p):
    if p:
        print(f'Syntax error at {p.value} in line {p.lexer.lineno}.')
    else:
        print('Syntax error at end of input.')


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
A prod is a sequence of factors that are combined with the operators * and /.
The corresponding grammar rules are:
prod : prod '*' factor
         | prod '/' factor
         | factor
         ;
Can you write Python code for it?

def p_prod_mult(p):
    "prod : prod '*' factor"
    p[0] = p[1] * p[3]

def p_prod_div(p):
    "prod : prod '/' factor"
    p[0] = p[1] / p[3]

def p_prod_factor(p):
    "prod : factor"
    p[0] = p[1]


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
An expr is a sequence of prods that are combined with the operators + and -.
The corresponding grammar rules are:
expr : expr '+' prod
         | expr '-' prod
         | prod
         ;
Can you write Python code for it?

def p_expr_plus(p):
    "expr : expr '+' prod"
    p[0] = p[1] + p[3]

def p_expr_minus(p):
    "expr : expr '-' prod"
    p[0] = p[1] - p[3]

def p_expr_prod(p):
    "expr : prod"
    p[0] = p[1]


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
Blanks and tabulators are ignored.
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
Let's look at the action table that is generated.
Can you write Python code for it?


!cat parser.out

Names2Values = {}

def main():
    while True:
        s = input('calc > ')
        if s == '':
            break
        yacc.parse(s)

main()


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
Names2Values is the dictionary that maps variable names to their values.  Initially the dictionary is empty as no variables has yet been defined.
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
Newlines are counted in order to give precise error messages.  Otherwise they are ignored.
Can you write Python code for it?

def t_newline(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
The method p_error is called if a syntax error occurs.  The argument p is the token that could not be read.  If p is None then there is a syntax error at the end of input.
Can you write Python code for it?

def p_error(p):
    if p:
        print(f"Syntax error at character number {p.lexer.lexpos} at token '{p.value}' in line {p.lexer.lineno}.")
    else:
        print('Syntax error at end of input.')


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
The start variable of our grammar is statement.
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
The token ASSIGN_OP specifies the assignment operator.
Can you write Python code for it?

def t_ASSIGN_OP(t):
    r':='
    return t

literals = ['+', '-', '*', '/', '(', ')', ';']

t_ignore  = ' \t'

def t_newline(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')

def t_error(t):
    print(f"Illegal character '{t.value[0]}'")
    t.lexer.skip(1)

__file__ = 'main'

lexer = lex.lex()


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
The token IDENTIFIER specifies the name of a variable.
Can you write Python code for it?

def t_IDENTIFIER(t):
    r'[a-zA-Z][a-zA-Z0-9_]*'
    return t


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
The token NUMBER specifies a fully featured floating point number.
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
There are only three tokens that need to be defined via regular expressions.  The other tokens consist only of a single character and can therefore be defined as literals.
Can you write Python code for it?

tokens = [ 'NUMBER', 'IDENTIFIER', 'ASSIGN_OP' ]


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
There are two grammar rules for stmnts:
stmnt : IDENTIFIER ":=" expr ";"
          | expr ';'
          ;
- If a stmnt is an assignment, the expression on the right hand side of the assignment operator is
  evaluated and the value is stored in the dictionary Names2Values.  The key used in this dictionary
  is the name of the variable on the left hand side ofthe assignment operator.
- If a stmnt is an expression, the expression is evaluated and the result of this evaluation is printed.
It is <b>very important</b> that in the grammar rules below the : is surrounded by space characters, for otherwise Ply will throw mysterious error messages at us!
Below, Names2Values is a dictionary mapping variable names to their values.  It will be defined later.
Can you write Python code for it?

def p_stmnt_assign(p):
    "stmnt : IDENTIFIER ASSIGN_OP expr ';'"
    Names2Values[p[1]] = p[3]

def p_stmnt_expr(p):
    "stmnt : expr ';'"
    print(p[1])


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
Unkown characters are reported as lexical errors.
Can you write Python code for it?

def t_error(t):
    print(f"Illegal character '{t.value[0]}' at character number {t.lexer.lexpos} in line {t.lexer.lineno}.")
    t.lexer.skip(1)

__file__ = 'main'


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
We generate the lexer.
Can you write Python code for it?


I am working on the file "Ply/Symbolic-Calculator.ipynb".
The first task is:
literals is a list operator symbols that consist of a single character.
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
And note we use pmag.dodirot_V to do the rotation.
Can you write Python code for it?

help(pmag.dodirot_V)

rot_block=pmag.dodirot_V(di_block,princ['dec'],princ['inc'])
rot_block


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
And now we can plot them using pmagplotlib.plot_map()
Can you write Python code for it?

Opts={}
Opts['sym']='wo' # sets the symbol
Opts['symsize']=10
Opts['proj']='ortho'
Opts['edge']='black'
Opts['lat_0']=90
Opts['details']={}
Opts['details']['fancy']=True # warning : this option takes a few minutes
if has_cartopy:
    plt.figure(1,(6,6)) # optional - make a map
    pmagplotlib.plot_map(1, RLats, RLons, Opts)
elif has_basemap:
    plt.figure(1,(6,6)) # optional - make a map
    pmagplotlib.plot_map_basemap(1, RLats, RLons, Opts)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
And of course look at what we have done!
Can you write Python code for it?

ipmag.plot_net(1) # make the plot
ipmag.plot_di(di_block=rot_block,color='red',title='rotated',edge='black')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
And, to do relative to the spin axis, set spin to True:
Can you write Python code for it?

N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,spin=True)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Functions demonstrated within this notebook:

Functions in PmagPy_calculations.ipynb:
aarm_magic : calculate AARM tensors
atrm_magic : calculate ATRM tensors
angle : calculates the angle between two vectors
apwp : returns predicted paleolatitudes, directions and pole latitude/longitude from apparent polar wander paths of Besse and Courtillot (2002).
b_vdm : converts B (in microT) and (magnetic) latitude to V(A)DM (see vdm_b)
bootams : calculates bootstrap statistics for tensor data
cart_dir : converts cartesian coordinates (x,y,z) to declination, inclination, intensity (see dir_cart)
di_eq : maps declination, inclinatitions to X,Y for plotting in equal area projections
di_geo : rotates declination, inclination in specimen coordinates to geographic coordinates
di_rot : rotates directions to a  coordinate system with D,I as center
di_tilt : rotates directions to stratigraphic coordinates
di_vgp : converts direction to Virtual Geomagnetic Pole (see vgp_di)
dia_vgp : converts direction and $\alpha_{95}$ to Virtual Geomagnetic Pole and dp,dm
dipole_pinc : calculates inclination given latitude assuming geocentric axial dipole
dipole_plat : calculates latitude given inclination assuming geocentric axial dipole
dir_cart : converts declination, inclination, intensity to cartesian coordinates (see cart_dir)
eigs_s : converts eigenparameters to equivalent 6 element tensor (see s_eigs)
eq_di : takes X,Y from equal area projection (e.g., from digitized coordinates) and  converts to declination, inclination
fcalc : returns the value from an F table, given the degrees of freedom.
fisher : generates sets of directions drawn from Fisher distributions with vertical true mean
fishrot : generates sets of directions drawn from Fisher distributions with arbitrary true mean
flip : flips a second mode (reverse directions) to their antipodes
gaussian : generates data drawn from a normal distribution
gobing : calculates Bingham statistics from a set of directions
gofish : calculates Fisher statistics from a set of directions
gokent :...
Can you write Python code for it?

convert.sio('arm_magic_example.dat',dir_path='data_files/aarm_magic/',specnum=3,
           location='Bushveld',codelist='AF:ANI',samp_con='3',
           meas_file='aarm_measurements.txt',peakfield=180,labfield=50, phi=-1, theta=-1)

help(ipmag.aarm_magic)

ipmag.aarm_magic('aarm_measurements.txt',dir_path='data_files/aarm_magic/')

# plot the data generated by aarm_magic:
ipmag.aniso_magic_nb(infile='data_files/aarm_magic/specimens.txt', save_plots=False)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Here is the other (equally valid) way using np.loadtext().
Can you write Python code for it?

# Numpy way:
di=np.loadtxt('data_files/angle/angle.dat').transpose() # read in file
D1=di[0:2].transpose() # assign to first array
D2=di[2:].transpose() # assign to second array
pmag.angle(D1,D2) # call pmag.angle


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Here's the sort of data  in the file:
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Let's take a look at these data in equal area projection: (see eqarea for details)
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Now let's check out the version that takes many data points at once.
Can you write Python code for it?

help(pmag.dogeo_V)

indata=np.loadtxt('data_files/di_geo/di_geo_example.dat')
print (indata)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Now we can rotate them
Can you write Python code for it?

Dt,It=pmag.dotilt_V(data) # rotate them
ipmag.plot_net(1) # and take another look
ipmag.plot_di(dec=Dt,inc=It)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Now we will use pmag.angle() to calculate the angles.
Can you write Python code for it?

# call pmag.angle
pmag.angle(di[['Dec1','Inc1']].values,di[['Dec2','Inc2']].values)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
Or as an array
Can you write Python code for it?

lats=range(-90,100,10)
incs=pmag.pinc(lats)
plt.plot(incs,lats)
plt.ylim(100,-100)
plt.xlabel('Latitude')
plt.ylabel('Inclination')
plt.axhline(0,color='black')
plt.axvline(0,color='black');


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
The data are almost in the correct format, but there is no a95 field, so that will have to be inserted (as zeros).
Can you write Python code for it?

a95=np.zeros(len(data))
a95

DIs=data.transpose()[0:2].transpose() # get the DIs
LatLons=data.transpose()[2:].transpose() # get the Lat Lons
newdata=np.column_stack((DIs,a95,LatLons)) # stitch them back together
print (newdata)

vgps=np.array(pmag.dia_vgp(newdata)) # get a tuple with lat,lon,dp,dm, convert to array
print (vgps.transpose()) #  print out the vgps


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
The file atrm_magic_example.dat in the data_files/atrm_magic directory is an SIO formatted data file containing ATRM measurement data done in a temperature of 520∘C. Note the special format of these files - the treatment column (column 2) has the temperature in centigrade followed by either a “00” for the obligatory zero field baseline step or a “10” for the first postion, and so on. These could also be ‘0‘ and ‘1’, etc..
Follow the instructions for  sio_magic to import the ATRM data into the MagIC format. The DC field was 40 μT. The sample/site naming convention used option # 1 (see help menu) and the specimen and sample name are the same (specnum=0).
We will use ipmag.atrm_magic() to calculate the best-fit tensor and write out the MagIC tables which can be used to correct remanence data  for the effects of remanent anisotropy.
Can you write Python code for it?

convert.sio('atrm_magic_example.dat',dir_path='data_files/atrm_magic/',specnum=0,
           location='unknown',codelist='T:ANI',samp_con='1',
           meas_file='measurements.txt',labfield=40, phi=-1, theta=-1)

help(ipmag.atrm_magic)

ipmag.atrm_magic('measurements.txt',dir_path='data_files/atrm_magic')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
To apply a cutoff for the Fisher k value, we just filter the DataFrame prior to calculating S_b. Let's filter for kappa>50
Can you write Python code for it?

N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,kappa=50)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
To apply the Vandamme (1994) approach, we set v to True
Can you write Python code for it?

N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,v=True)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
To flip the "reverse" directions, we set anti to 1
Can you write Python code for it?

N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,anti=True)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
To just calculate the value of S (without the within site scatter) we read in a data file and attach the correct headers to it depending on what is in it.
Can you write Python code for it?

vgp_df=pd.read_csv('data_files/scalc/scalc_example.txt',delim_whitespace=True,header=None)
if len(list(vgp_df.columns))==2:
    vgp_df.columns=['vgp_lon','vgp_lat']
    vgp_df['dir_k'],vgp_df['dir_n'],vgp_df['lat']=0,0,0
else:
    vgp_df.columns=['vgp_lon','vgp_lat','dir_k','dir_n_samples','lat']
pmag.scalc_vgp_df
N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
We are working with data that have no sigmas attached to them and want to average all the values in the file together.  Let's look at the rotated data from the s_geo example.
Can you write Python code for it?

# read in the data
Ss=np.loadtxt('data_files/s_geo/s_geo_example.dat')
# make a container for the rotated S values
SGeos=[]
for s in Ss:
    SGeos.append(pmag.dosgeo(s[0:6],s[6],s[7]))
nf,sigma,avs=pmag.sbar(SGeos)  # get the average over all the data
hpars=pmag.dohext(nf,sigma,avs)
print(hpars)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
We will calculate the field for San Diego from 3000 BCE to 1950 in 50 year increments using the hfm.OL1.A1 model of Constable et al. (2016, doi: 10.1016/j.epsl.2016.08.015).
Can you write Python code for it?

# make a list of desired dates
dates=range(-3000,1950,50) # list of dates in +/- Common Era
mod = 'hfm10k' # choose the desired model
lat,lon,alt=33,-117,0 # desired latitude, longitude and alitude
Vecs=[] # list for Dec,Inc,Int outputs
for date in dates: # step through the dates
    Vecs.append(ipmag.igrf([date,alt,lat,lon],mod=mod)) # append to list
vector_df = pd.DataFrame(Vecs)   # make it into a Pandas dataframe
vector_df.columns=['dec','inc','int']
vector_df['vadms']=pmag.b_vdm(vector_df.int.values*1e-9, lat) # calculate the VADMs
vector_df['dec_adj']=vector_df['dec']
vector_df.loc[vector_df.dec>180,['dec_adj']]=vector_df.dec-360 # adjust declinations to be -180 => 180
fig=plt.figure(1,figsize=(7,9)) # set up the figure
fig.add_subplot(411) # make 4 rows of plots, this is the first
plt.plot(dates,vector_df.dec_adj) # plot the adjusted declinations
plt.ylabel('Declination ($^{\circ}$)')
plt.title('Geomagnetic field evaluated at Lat: '+str(lat)+' / Lon: '+str(lon))
fig.add_subplot(412) # this is the second
plt.plot(dates,vector_df.inc) # plot  the inclinations
plt.ylabel('Inclination ($^{\circ}$)')
fig.add_subplot(413)
plt.plot(dates,vector_df.int*1e-3) # plot the intensites (in uT instead of nT)
plt.ylabel('Intensity ($\mu$T)')
fig.add_subplot(414) # plot the VADMs
plt.plot(dates,vector_df.vadms*1e-21) # plot as ZAm^2
plt.ylabel('VADM (ZAm$^2$)')
plt.xlabel('Dates (CE)');


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
angle
[Essentials Appendix A.3.4] [command line version]
angle calculates the angle $\alpha$ between two declination,inclination pairs.
There are several ways to use this function from the notebook - one loading the data into a Pandas dataframe, then converting to the desired arrays, or load directly into a Numpy array of desired shape.
Can you write Python code for it?

help(pmag.angle)

# Pandas way:
di=pd.read_csv('data_files/angle/angle.dat',delim_whitespace=True,header=None)
#rename column headers
di.columns=['Dec1','Inc1','Dec2','Inc2']


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
apwp
[Essentials Chapter 16]  [command line version]
The program apwp calculates paleolatitude, declination, inclination from a pole latitude and longitude based on the paper Besse and Courtillot (2002; see Essentials Chapter 16   for complete discussion). Here we will  calculate the expected direction for 100 million year old rocks at a locality in La Jolla Cove (Latitude: 33$^{\circ}$N, Longitude 117$^{\circ}$W). Assume that we are on the North American Plate! (Note that there IS no option for the Pacific plate in the program apwp, and that La Jolla was on the North American plate until a few million years ago (6?).
Within the notebook we will call pmag.apwp.
Can you write Python code for it?

help(pmag.apwp)

# here are the desired plate, latitude, longitude and age:
data=['NA',33,-117,100] # North American plate, lat and lon of San Diego at 100 Ma
pmag.apwp(data,print_results=True)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
atrm_magic
[command line version]
Anisotropy of thermal remanence (ATRM) is similar to anisotropy of anhysteretic remanence (AARM) and the procedure for obtaining the tensor is also similar. Therefore, the atrm_magic is quite similar to aarm_magic. However, the SIO lab procedures for the two experiments are somewhat different. In the ATRM experiment, there is a single, zero field step at the chosen temperature which is used as a baseline. We use only six positions (as opposed to nine for AARM) because of the additional risk of alteration at each temperature step. The positions are also different:
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
b_vdm
[Essentials Chapter 2]  [command line version]
b_vdm converts geomagnetic field intensity observed at the earth's surface at a particular (paleo)latitude and calculates the Virtual [Axial] Dipole Moment (vdm or vadm).  We will call pmag.b_vdm() directly from within the notebook. [See also vdm_b.]
Here we use the function  pmag.b_vdm() to convert an estimated paleofield value of 33 $\mu$T obtained from a lava flow at 22$^{\circ}$ N latitude to the equivalent Virtual Dipole Moment (VDM) in Am$^2$.
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
bootams
[Essentials Chapter 13]  [command line version]
bootams calculates bootstrap statistics for anisotropy tensor data in the form of:
x11 x22 x33 x12 x23 x13
It does this by selecting para-data sets and calculating the Hext average eigenparameters. It has an optional parametric bootstrap whereby the $\sigma$ for the data set as a whole is used to draw new para data sets. The bootstrapped eigenparameters are assumed to be Kent distributed and the program calculates Kent error ellipses for each set of eigenvectors. It also estimates the standard deviations of the bootstrapped eigenvalues.
bootams   reads in a file with data for the six tensor elements (x11 x22 x33 x12 x23 x13) for specimens, calls pmag.s_boot()  using a parametric or non-parametric bootstrap as desired. If all that is desired is the bootstrapped eigenparameters, pmag.s_boot() has all we need, but if the Kent ellipses are required, and we can call pmag.sbootpars() to calculated these more derived products and  print them out.
Note that every time the bootstrap program gets called, the output will be slightly different because this depends on calls to random number generators. If the answers are different by a lot, then the number of bootstrap calculations is too low. The number of bootstraps can be changed with the nb option below.
We can do all this from within the notebook as follows:
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
cart_dir
[Essentials Chapter 2]  [command line version]
cart_dir converts cartesian coordinates (X,Y,Z) to polar coordinates (Declination, Inclination, Intensity).  We will call pmag.cart2dir().
Can you write Python code for it?

help(pmag.cart2dir)

# read in data file from example file
cart=np.loadtxt('data_files/cart_dir/cart_dir_example.dat')
print ('Input: \n',cart) # print out the cartesian coordinates
# print out the  results
dirs = pmag.cart2dir(cart)
print ("Output: ")
for d in dirs:
    print ('%7.1f %7.1f %8.3e'%(d[0],d[1],d[2]))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
di_rot
[Essentials Chapter 11]
[command line version]
di_rot   rotates dec inc pairs to a new origin.  We can call pmag.dodirot() for single [dec,inc,Dbar,Ibar] data or pmag.dodirot_V() for an array of Dec, Inc pairs.  We can use the data from the di_geo example and rotate the geographic coordinate data such that the center of the distribution is the principal direction.
We do it like this:
- read in a data set with dec inc pairs
- make an equal area projection of the data to remind us what they look like
- calculate the principal component with pmag.doprinc())
- rotate the data to the principal direction
- plot the rotated data in an equal area projection.
Can you write Python code for it?

di_block=np.loadtxt('data_files/di_rot/di_rot_example.txt') # read in some data
ipmag.plot_net(1) # make the plot
ipmag.plot_di(di_block=di_block,title='geographic',color='red',edge='black')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
di_tilt
[Essentials Chapter 9] [Changing coordinate systems]
[command line version]
di_tilt can rotate a direction of Declination = 5.3 and Inclination = 71.6 to “stratigraphic” coordinates, assuming the  strike was 135 and the dip was 21. The convention in this program is to use the dip direction, which is to the “right” of this strike.
We can perform this calculation by calling  pmag.dotilt or pmag.dotilt_V() depending on if we have a single point or an array to rotate.
Can you write Python code for it?

help(pmag.dotilt)

help(pmag.dotilt_V)

# read in some data
data=np.loadtxt('data_files/di_tilt/di_tilt_example.dat') # load up the data
di_block=data[:,[0,1]] # let's look at the data first!
ipmag.plot_net(1)
ipmag.plot_di(di_block=di_block)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
di_vgp
[Essentials Chapter 2]
[command line version]
di_vgp converts directions (declination,inclination) to Virtual Geomagnetic Pole positions.   This is the inverse of vgp_di.
To do so, we will call pmag.dia_vgp() from within the notebook.
Can you write Python code for it?

help(pmag.dia_vgp)

data=np.loadtxt('data_files/di_vgp/di_vgp_example.dat') # read in some data
print (data)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
dipole_pinc
[Essentials Chapter 2]
[command line version]
If we assume a geocentric axial dipole, we can calculate an expected inclination at a given latitude and that is what dipole_pinc does.   It calls pmag.pinc() and so will we to find the expected inclination at a paleolatitude of 24$^{\circ}$S!
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
dir_cart
[Essentials Chapter 2]  [command line version]
pmag.dir2cart() converts directions (Declination, Inclination, Intensity) to cartesian coordinates (X,Y,Z).
Can you write Python code for it?

help(pmag.dir2cart)

# read in data file from example file
dirs=np.loadtxt('data_files/dir_cart/dir_cart_example.dat')
print ('Input: \n',dirs) # print out the cartesian coordinates
# print out the  results
carts = pmag.dir2cart(dirs)
print ("Output: ")
for c in carts:
    print ('%8.4e %8.4e %8.4e'%(c[0],c[1],c[2]))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
fisher mean on pandas DataFrames
There is also a function pmag.dir_df_fisher_mean() that calculates Fisher statistics on a Pandas DataFrame with directional data
Can you write Python code for it?

help(pmag.dir_df_fisher_mean)

# make the data frame
dir_df=pd.read_csv('data_files/gofish/fishrot.out',delim_whitespace=True, header=None)
dir_df.columns=['dir_dec','dir_inc']
pmag.dir_df_fisher_mean(dir_df)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
fishrot
[Essentials Chapter 11]
[command line version]
This program is similar to fisher, but allows you to specify the mean direction.
This has been implemented as ipmag.fishrot().
Can you write Python code for it?

help(ipmag.fishrot)

rotdi=ipmag.fishrot(k=50,n=5,dec=33,inc=41)
for di in rotdi:
    print ('%7.1f %7.1f'%(di[0],di[1]))

ipmag.plot_net(1)
ipmag.plot_di(di_block=rotdi)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
gaussian
[Essentials Chapter 11]
[command line version]
This program generates sets of data drawn from a normal distribution with a given mean and standard deviation.  It is just a wrapper for a call to  pmag.gaussdev() which just calls numpy.random.normal() which we could do, but we would have to import it, so it is easiest just to call the pmag version which we have already imported.
Can you write Python code for it?

help(pmag.gaussdev)

N=1000
bins=100
norm=pmag.gaussdev(10,3,N)
plt.hist(norm,bins=bins,color='black',histtype='step',density=True)
plt.xlabel('Gaussian Deviates')
plt.ylabel('Frequency');

# alternatively we can plot with ipmag.histplot:
ipmag.histplot(data=norm, xlab='Gaussian Deviates', save_plots=False, norm=-1)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
gofish
[Essentials Chapter 11]
[command line version]
gofish calculates Fisher statistics for sets of directional data.  (see the section for eqarea_ell in the PmagPy_plots_analysis documentation for nice examples).
This can be done with ipmag.fisher_mean().
Can you write Python code for it?

help(ipmag.fisher_mean)

di_block=np.loadtxt('data_files/gofish/fishrot.out')
ipmag.fisher_mean(di_block=di_block)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
goprinc
[Essentials Chapter 12]
[command line version]
goprinc calculates the principal directions (and their eigenvalues) for sets of paleomagnetic vectors.  It doesn't do any statistics on them, unlike the other programs.
We will call pmag.doprinc():
Can you write Python code for it?

help(pmag.doprinc)

di_block=np.loadtxt('data_files/goprinc/goprinc_example.txt')
pmag.doprinc(di_block)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
igrf
[Essentials Chapter 2]
[command line version]
This program gives geomagnetic field vector data for a specified place at a specified time.  It has many built in models including  IGRFs, GUFM and several archeomagnetic models.   It calls the  function ipmag.igrf() for this so that is what we will do.
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
incfish
[Essentials Chapter 11]
[command line version]
You can't get a meaningful average inclination from inclination only data because of the exponential relationship between inclinations and the true mean inclination for Fisher distributions (except exactly at the pole and the equator).  So, McFadden and Reid (1982, doi:  10.1111/j.1365-246X.1982.tb04950.x) developed a maximum liklihood estimate for getting an estimate for true mean absent declination.   pmag.doincfish() is an implementation of that concept.
Can you write Python code for it?

help(pmag.doincfish)

incs=np.loadtxt('data_files/incfish/incfish_example_inc.dat')
pmag.doincfish(incs)


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
pca
[Essentials Chapter 11] [command line version]
pca  calculates best-fit lines, planes or Fisher means through selected treatment steps along with Kirschvink (1980, doi:  10.1111/j.1365-246X.1980.tb02601.x) MAD values. The file format is a simple space delimited file with specimen name, treatment step, intensity, declination and inclination. pca.py  calls pmag.domean(), so that is what we will do here.
Can you write Python code for it?

help(pmag.domean)

# read in data as space delimited file
data=pd.read_csv('data_files/pca/pca_example.txt',\
                 delim_whitespace=True,header=None)
# we need to add a column for quality
data['quality']='g'
# strip off the specimen name and reorder records
#  from:  int,dec,inc to: dec,inc,int
data=data[[1,3,4,2,'quality']].values.tolist()
pmag.domean(data,1,10,'DE-BFL')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
s_hext
[Essentials Chapter 13] [command line version]
s_hext calculates Hext (1963, doi: 10.2307/2333905) statistics for anisotropy data in the six tensor element format.
It  calls pmag.dohext().
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
scalc
[Essentials Chapter 14] [command line version]
This program reads in data files with vgp_lon, vgp_lat and optional kappa, N, and site latitude.
It allows some filtering based on the requirements of the study, such as:
- Fisher k cutoff
- VGP latitudinal cutoff
- Vandamme (1994, doi: 10.1016/0031-9201(94)90012-4) iterative cutoff
- flipping the reverse mode to antipodes
- rotating principle direction to the spin axis
- bootstrap confidence bounds
- optionally calculates the scatter (Sp or Sf of McElhinny & McFadden, 1997) of VGPs
with  correction for within site scatter.
The filtering is just what Pandas was designed for, so we can  calls pmag.scalc_vgp_df() which works on a suitably constructed  Pandas DataFrame.
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
scalc_magic
[Essentials Chapter 14] [command line version]
This program does the same thing as scalc, but reads in a MagIC formatted file. So, we can do that easy-peasy.
Can you write Python code for it?

vgp_df=pd.read_csv('data_files/scalc_magic/sites.txt',sep='\t',header=1)
N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,anti=True)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))

vgp_df=pd.read_csv('data_files/scalc_magic/sites.txt',sep='\t',header=1)
N,S_B,low,high,cutoff=pmag.scalc_vgp_df(vgp_df,anti=True,spin=True)
print(N, '%7.1f  %7.1f ' % (S_B, cutoff))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
separate_directions
Like pmag.flip( ), pmag.separate_directions divides a directional data set into two modes.  Unlike pmag.flip( ), it returns the two separate modes (e.g., normal and reverse)
Can you write Python code for it?

help(pmag.separate_directions)

#read in the data into an array
vectors=np.loadtxt('data_files/eqarea_ell/tk03.out').transpose()
di_block=vectors[0:2].transpose() # decs are di_block[0], incs are di_block[1]
# flip the reverse directions to their normal antipodes
normal,reverse=pmag.separate_directions(di_block)
# and plot them up
ipmag.plot_net(1)
ipmag.plot_di(di_block=normal,color='red')
ipmag.plot_di(di_block=reverse,color='b')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
tk03
[Essentials Chapter 16]
[command line version]
Sometimes it is useful to generate a distribution of synthetic geomagnetic field vectors that you might expect to find from paleosecular variation of the geomagnetic field. The program tk03 generates distributions of field vectors from the PSV model of Tauxe and Kent (2004, doi: 10.1029/145GM08).  This program was implemented for notebook use as ipmag.tk03().  [See also find_ei].
Can you write Python code for it?

help(ipmag.tk03)

di_block=ipmag.tk03(lat=30)
ipmag.plot_net(1)
ipmag.plot_di(di_block=di_block,color='red',edge='black')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
uniform
[command line version]
It is at times handy to be able to generate a uniformly distributed set of directions (or geographic locations). This is done using a technique described by Fisher et al. (Fisher, N. I., Lewis, T., & Embleton, B. J. J. (1987). Statistical Analysis of Spherical Data. Cambridge: Cambridge University Press).   We do this by calling  pmag.get_unf().
Can you write Python code for it?

help(pmag.get_unf)

di_block=pmag.get_unf()
ipmag.plot_net(1)
ipmag.plot_di(di_block=di_block,color='red',edge='black')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
unsquish
[Essentials Chapter 7]
[Essentials Chapter 16]
[command line version]
This program is just the inverse of squish in that it takes "squished" data and "unsquishes" them, assuming a King (1955, doi: 10.1111/j.1365-246X.1955.tb06558.x)  relationship:  $\tan(I_o)=flat \tan(I_f)$. So, $\tan(I_f) = \tan(I_o)/flat$.
It calls pmag.unquish().
Can you write Python code for it?

help(pmag.unsquish)

di_block=np.loadtxt('data_files/unsquish/unsquish_example.dat').transpose()
decs=di_block[0]
incs=di_block[1]
flat=.4
fincs=pmag.unsquish(incs,flat)

ipmag.plot_net(1)
ipmag.plot_di(dec=decs,inc=incs,title='Squished',color='red')
ipmag.plot_net(2)
ipmag.plot_di(dec=decs,inc=fincs,title='Unsquished',color='blue')


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
vdm_b
[Essentials Chapter 2]  [command line version]
vdm_b is the inverse of b_vdm in that it converts a virtual [axial] dipole moment (vdm or vadm) to a predicted geomagnetic field intensity observed at the earth's surface at a particular (paleo)latitude. This  program calls pmag.vdm_b().
Can you write Python code for it?

help(pmag.vdm_b)

print ('%7.1f microtesla'%(pmag.vdm_b(7.159e22,22)*1e6))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
vector_mean
[Essentials Chapter 2]  [command line version]
vector_mean calculates the vector mean for a set of vectors in polar coordinates (e.g., declination, inclination, intensity).  This is similar to the Fisher mean (gofish) but uses vector length instead of unit vectors.  It calls
calls pmag.vector_mean().
Can you write Python code for it?

help(pmag.vector_mean)

data=np.loadtxt('data_files/vector_mean/vector_mean_example.dat')
Dir,R=pmag.vector_mean(data)
print (('%i %7.1f %7.1f %f')%(data.shape[0],Dir[0],Dir[1],R))


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
vgp_di
[Essentials Chapter 2]
[command line version]
We use vgp_di to convert virtual geomagnetic pole positions to predicted directions at a given location.  [See also di_vgp].
This program uses the function pmag.vgp_di().
Can you write Python code for it?


I am working on the file "PmagPy_calculations.ipynb".
The first task is:
watsons_f
[Essentials Chapter 11]
[command line version]
There are several different ways of testing whether two sets of directional data share a common mean.  One popular (although perhaps not the best) way is to use Watson's F test (Watson, 1956, doi: 10.1111/j.1365-246X.1956.tb05560.x).   [See also watsons_v or  Lisa Tauxe's bootstrap way: common_mean].
If you still want to use Waston's F, then try
 pmag.watsons_f() for this.
Can you write Python code for it?

help(pmag.watsons_f)

DI1=np.loadtxt('data_files/watsons_f/watsons_f_example_file1.dat')
DI2=np.loadtxt('data_files/watsons_f/watsons_f_example_file2.dat')
F,Fcrit=pmag.watsons_f(DI1,DI2)
print ('%7.2f %7.2f'%(F,Fcrit))


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:

AMS data
Convert AMS data to MagIC

Download KAPPABRIDGE expanded magnetic susceptibility data from the Lims Online Repository
place the downloaded .csv file in the KLY4S_data in the HOLE directory.
change kly4s_file to the correct file name in the preliminaries cell
Can you write Python code for it?

if kly4s_file:
    convert.iodp_kly4s_lore(kly4s_file, meas_out='kly4s_measurements.txt',
           spec_infile='lims_specimens.txt', spec_out='kly4s_specimens.txt',
           dir_path=magic_dir, input_dir_path=kly4s_dir,actual_volume=7)
    if 'kly4s_measurements.txt' not in meas_files:meas_files.append('kly4s_measurements.txt')
    if 'kly4s_specimens.txt' not in meas_files:meas_files.append('kly4s_specimens.txt')


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:

Downhole Plots

Fill in the section summary file in the preliminaries folder and run  the cells below
Can you write Python code for it?

if dscr_file: # checks for discrete measurements, adds depths, etc.
    srm_dscr_df=pd.read_csv(magic_dir+'/'+dscr_file,sep='\t',header=1)
    dscr_df=srm_dscr_df.copy(deep=True)
    dscr_df=dscr_df[srm_dscr_df['treat_ac_field']==demag_step]
    depth_data=pd.read_csv(magic_dir+'/lims_sites.txt',sep='\t',header=1)
    depth_data['specimen']=depth_data['site']
    depth_data=depth_data[['specimen','core_depth']]
    depth_data.sort_values(by='specimen')
    dscr_df=pd.merge(dscr_df,depth_data,on='specimen')
else:
    dscr_df="" # if no discrete samples.

if section_summary_file:
    arch_demag_step=pd.read_csv(hole+'/'+hole+'_arch_demag_step.csv')
    adj_dec_df=pd.read_csv(hole+'/'+hole+'_dec_adjusted.csv')
#Let's get the section boundaries.
# edit the file name for the Section Summary table downloaded from LIMS

    summary_df=pd.read_csv(summary_file)
    summary_df.dropna(subset=['Sect'],inplace=True)
    if type(summary_df.Sect)!='str':
        summary_df.Sect=summary_df.Sect.astype('int64')
        summary_df.Sect=summary_df.Sect.astype('str')

    summary_df=summary_df[summary_df['Sect'].str.contains('CC')==False]
    max_depth=arch_demag_step['core_depth'].max()

    summary_df=summary_df[summary_df['Top depth CSF-A (m)']<max_depth]
    sect_depths=summary_df['Top depth CSF-A (m)'].values
    summary_df['Core']=summary_df['Core'].astype('int')
    labels=summary_df['Core'].astype('str')+summary_df['Type']+'-'+summary_df['Sect'].astype('str')

    arch_demag_step=pd.read_csv(hole+'/'+hole+'_arch_demag_step.csv')
    interval=100 # how to divide up the plots
    depth_min,depth_max=0,interval
    fignum=1
    while depth_min<arch_demag_step.core_depth.max():
        iodp_funcs.make_plot(arch_demag_step,adj_dec_df,sect_depths,hole,\
                         gad_inc,depth_min,depth_max,labels,spec_df=dscr_df,fignum=fignum)
        depth_min+=interval
        depth_max+=interval
        fignum+=1


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:

Make the sample tables

Download the Sample Report and Section Summary tables (when available) for the HOLE from the LIMS online repository at http://web.ship.iodp.tamu.edu/LORE/ and save them as csv files.
Put the two .csv files in the HOLE directory created above.
Edit the name of the sample .csv  file in prelimaries cell and the 'secondary depth' column that you selected when downloading (the default is CSF-B as below) in the cell below.
Execute the cell to  populate the MagIC meta-data tables.  The depth information ends up in the lims_sites.txt table in the HOLE_MagIC directory, for example and gets used to create downhole plots.

Run the following cell to create the samples, sites, and locations tables for the hole.
Can you write Python code for it?

# Make sure the sample file is in your HOLE directory and the filename set in the Preliminaries
# Note:  this program will not run if the file is in use

if samp_file:

    comp_depth_key='Top depth CSF-B (m)'
# do the heavy lifting:
    convert.iodp_samples_csv(samp_file,input_dir_path=hole,spec_file='lims_specimens.txt',\
                         samp_file='lims_samples.txt',site_file='lims_sites.txt',\
                         dir_path=magic_dir,comp_depth_key=comp_depth_key,\
                exp_name=exp_name,exp_desc=exp_description,lat=hole_lat,\
                lon=hole_lon)
# this collects the file names that were created so they can be combined with others, e.g., those
# from the archive half measurements which are not in the sample table.
    if 'lims_specimens.txt' not in spec_files:spec_files.append('lims_specimens.txt')
    if 'lims_samples.txt' not in samp_files:samp_files.append('lims_samples.txt')
    if 'lims_sites.txt' not in site_files:site_files.append('lims_sites.txt')
# do it again to make copies for use with demag_gui
    convert.iodp_samples_csv(samp_file,input_dir_path=hole,\
                         dir_path=magic_dir,comp_depth_key=comp_depth_key,\
                exp_name=exp_name,exp_desc=exp_description,lat=hole_lat,\
                lon=hole_lon)


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:
If you did a bunch of full demagnetizations, set max_field to your maximum af field.  To save your plots, set save_plots to True.  To change the output format, change 'svg' to whatever you want ('pdf','eps','png').
Can you write Python code for it?

cnt=1
max_field=0 # set this to your peak field (in tesla) to execute this

if max_field:
    srm_dscr_df=pd.read_csv(magic_dir+'/'+dscr_file,sep='\t',header=1)
    srm_dmag=srm_dscr_df[srm_dscr_df.treat_ac_field>=max_field] # find all the demag specimens
    spc_list=srm_dmag.specimen.unique()

    for spc in spc_list:
        ipmag.zeq_magic(meas_file=dscr_file,specimen=spc,fignum=cnt,\
                spec_file='lims_specimens.txt',input_dir_path=magic_dir,save_plots=False,
                       fmt='svg')
        cnt+=3;


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:
Import the JR6 data.

download the JR6 data from LIMS and put it in the JR6_data folder in your working directory.
edit the file name in the preliminaries cell to reflect the actual file name.
execute the two cells in order.
Can you write Python code for it?

if jr6_file:
    convert.iodp_jr6_lore(jr6_file,meas_file='jr6_measurements.txt',dir_path=magic_dir,\
                     input_dir_path=jr6_dir,spec_file='lims_specimens.txt',noave=False)
    if 'jr6_measurements.txt' not in meas_files:meas_files.append('jr6_measurements.txt')
    dscr_file='jr6_measurements.txt'

# combine srm and jr6 data, we can combine the data here:
if jr6_file and srm_discrete_file:
    ipmag.combine_magic(['srm_dscr_measurements.txt','jr6_measurements.txt'],
                        outfile='dscr_measurements.txt',dir_path=magic_dir)
    dscr_file='dscr_measurements.txt'
max_field=0 # set this to your peak field (in tesla) to execute this
if max_field:
    cnt=1
    dscr_df=pd.read_csv(magic_dir+'/dscr_measurements.txt',sep='\t',header=1)
    dmag_df=dscr_df[dscr_df.treat_ac_field>=max_field] # find all the demag specimens
    spc_list=dmag_df.specimen.unique()

    for spc in spc_list:
        ipmag.zeq_magic(meas_file='dscr_measurements.txt',specimen=spc,fignum=cnt,\
                spec_file='lims_specimens.txt',input_dir_path=magic_dir,save_plots=False)
        cnt+=3;


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:
This way makes equal area plots in core coordinates.... To run it, set the False to True.
To save the plots, sset save_plots to True.  for different options, try (help(ipmag.aniso_magic_nb)).
Can you write Python code for it?

if False:
    ipmag.aniso_magic_nb(infile=magic_dir+'/kly4s_specimens.txt',\
                     verbose=False,save_plots=False,ihext=False,iboot=True,ivec=True)


I am working on the file "PmagPy_iodp_HOLE_template.ipynb".
The first task is:
To make a depth plot of your AMS data, change the False to True in the cell below and run it.
Can you write Python code for it?

if False:
    ipmag.ani_depthplot(spec_file='kly4s_specimens.txt', dir_path=magic_dir,
                    samp_file='lims_samples.txt',site_file='lims_sites.txt',
                   dmin=-1,dmax=-1,meas_file='kly4s_measurements.txt',
                   sum_file=core_summary_file)
    plt.savefig('Figures/'+hole+'_anisotropy_xmastree.pdf')


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Calculating directions from the IGRF model using ipmag.igrf
Let's use another ipmag function to determine the field predicted by the IGRF model here in La Jolla over the past 100 hundred years.
Before we do so, let's all open up ipmag.py in a text editor. Find the function ipmag.igrf and report back what functions it uses (hint they are from the pmag.py module).
To determine the IGRF field at a bunch of different years we will first create a list of years using the numpy function np.linspace. We can then take these years along with the igrf function and the location of La Jolla to calculate the local direction through time using a for loop.
Can you write Python code for it?

pmag.doigrf??

years = np.arange(1900,2020,1)

years

field = ipmag.igrf([year,0,La_Jolla_lat,La_Jolla_lon])
field

local_field = []
local_field_intensity = []
La_Jolla_lat = 32.8328
La_Jolla_lon = -117.2713

for year in years:
    field = ipmag.igrf([year,0,La_Jolla_lat,La_Jolla_lon])
    local_field.append(field)
    local_field_intensity.append(field[2])

fignum = 1
plt.figure(num=fignum,figsize=(8,8),dpi=160)
ipmag.plot_net(fignum)
ipmag.plot_di(di_block=local_field,markersize=10,label='field since 1900 in La Jolla')
ipmag.plot_di(di_block=local_field[-2:],color='red',label='field today in La Jolla')
plt.legend()
plt.show()

plt.plot(years,local_field_intensity)


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Following the instructions in the documentation string, let's make a plot and plot the directions.
Can you write Python code for it?

fignum = 1
plt.figure(num=fignum,figsize=(4,4),dpi=160)
ipmag.plot_net(fignum)
ipmag.plot_di(di_block=directions)


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Let's plot both the directions (using ipmag.plot_di) and the mean (using ipmag.plot_di_mean).
Can you write Python code for it?

fignum = 1
plt.figure(num=fignum,figsize=(4,4),dpi=160)
ipmag.plot_net(fignum)
ipmag.plot_di(di_block=lots_of_directions)
ipmag.plot_di_mean(mean['dec'],mean['inc'],mean['alpha95'],color='r')
plt.show()


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Plotting directions
Now let's say we want to plot these directions. Let's use the ipmag.plot_di function to do so. We can learn about that function by putting questions marks after it. Shift+tab also displays such information about a function.
Can you write Python code for it?


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Putting a question mark after a function results in the docstring being displayed.
Can you write Python code for it?


I am working on the file "PmagPy_structure_notebook.ipynb".
The first task is:
Using functions in the notebook
To use functions in the notebook we can import PmagPy function modules as in the code cell below. These function modules are:  pmag, a module with functions for analyzing paleomagnetic and rock magnetic data that are used within the command lines programs, the GUIs and within ipmag, a module with functions that combine and extend pmag functions and exposes pmagplotlib functions in order to generate output that works well within the Jupyter notebook environment.
Can you write Python code for it?

import pmagpy.pmag as pmag
import pmagpy.ipmag as ipmag

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
After a lot of memory-errors and heart-ache I able to transcribe 150 podcasts from 11 different series. In the future I will be adding much more. I was able to take the transcriptions and write them to a no-sql database where I have the filepath, name of the series and the transcription. The dataframe is below:
Can you write Python code for it?

import sqlite3

def connect_db():
	return sqlite3.connect('/Users/sheldon/podcasts/test.db')

def create_df_object():
	conn = sqlite3.connect('/Users/sheldon/podcasts/test.db')
	df = pd.read_sql("select * from podcast",conn)
	return df

df = create_df_object()

df.info()


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
As mentioned before I originally implemented this model into the flask app, but noticed that for each query we had to recompute the cosine similarities, which wasn't sustainable. Thus I discovered Gensim, Topic Modeling for Humans.
Part IV: Unsupervised Learning Models: Gensim & LSI
For anyone looking to play around with Gensim, I highly recommend the well documented API and the tutorials. Gensim, unlike scikit, actually has models that you can use to write to disk, which means the webserver could write the dictionary, corpus, and models all to disk and only need to read them once. Gensim also has a fantastic server module that the creator has just put out.
For gensim, it was a similar process to setup:

Clean the data (remove stopwords, capitalization, apostrophes)
Create the Corpus (a collection of text documents, in this case the transcriptions)
Create the TF-IDF Model
Create the LSI Model
Apply TFIDF on the corpus
Apply LSI on the TFIDF Matrix

LSI (Also known as LSA) does not only find similarities based on the cosine of the angle, but additionally can extract out topics of any two documents using a method called Singular Value Decomposition, to determine the pattern of relationships between teh terms and the concepts in an unstructured collection of text (https://en.wikipedia.org/wiki/Latent_semantic_analysis).
In translation we could feed in a list of podcasts transcriptions and be able to get the top related podcasts but also the themes which relate these podcasts.
We are also able to overcome a big issue of boolean keyword queries, that have similar meanings and words that have more than one meaning (IE "python internet" might return a coding podcast but "python jungle" might return a travel podcast.)
Can you write Python code for it?

import gensim
os.chdir('/Users/sheldon/git/springboard_capstone/apps/')
dictionary = gensim.corpora.Dictionary.load('models/words.dict')
corpus = gensim.corpora.MmCorpus('models/corpus.mm')
tfidf = gensim.models.tfidfmodel.TfidfModel.load('models/tfidf_model')
lsi = gensim.models.lsimodel.LsiModel.load('models/model.lsi')
index = gensim.similarities.MatrixSimilarity.load('models/corpus.index')
corpus_tfidf = tfidf[corpus]
corpus_lsi = lsi[corpus_tfidf]


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
As you'd expect the speed is much faster without having to continuously do the cosine similarity computation. But also the results from an anecdotal perspective look much more accurate. MLAR is a popualr political podcast so it makes sense that it's been clustered in that politics topic. We can explore this further:
Can you write Python code for it?

#get list of related podcasts
related_podcasts = list(get_related_podcasts(1)['index'])

def get_topics_per_podcast(podcast_index):
    def getKey(item):
        return item[1]
    topic_ids = [i for i in sorted(corpus_lsi[podcast_index], key=getKey, reverse=True) if i[1] > 0.10]
    def get_topic_arrays(topic_ids):
        x = []
        for id in topic_ids:
            list_of_words = sorted(lsi.show_topic(id[0], topn=5),key=getKey, reverse=True)
            z = []
            for word in list_of_words:
                if word[1] > .05:
                    z.append(word)
            x.append(z)
        return x
    topic_arrays = get_topic_arrays(topic_ids)
    return topic_arrays
related_podcasts_topics_words = [[related_podcasts[i],get_topics_per_podcast(related_podcasts[i])] for i in range(0, len(related_podcasts))]
episode_podcasts = list(get_related_podcasts(1)['episode'])
series_podcasts = list(get_related_podcasts(1)['series'])

for i,k in enumerate(related_podcasts_topics_words):
    print "Podcast: {}, ID: {}".format(i+1, k[0])
    print "Podcast Series: {}".format(series_podcasts[i])
    print "Episode Title: {}".format(episode_podcasts[i])
    for num, topic in enumerate(k[1]):
        print "topic: {}".format(num)
        for word in topic:
            print "word: {}, score:{}".format(word[0], word[1])


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
I'm creating two functions here:

Retrieve the related podcasts based on the cosine similarity of that partciluar podcast compared to other podcasts.

Retrieve the related podcasts based on the cosine similarity of a particular phrase.

For 1, we are computing the cosine similarities, pulling the podcast based on the index, creating a dictionary of each index and ranking, joining that to original dataframe, sorting it by the rank, and returning the data frame.
For 2, we are taking the query and doing some simple cleaning (lower and splitting into an array, transforming that array into a tfidf matrix, computing cosine similarity, and then copying the same thing as 1 with the the returned value being a dataframe. You can see the functions with their returned values below:
Can you write Python code for it?

get_related_podcasts_query_scikit('trump clinton obama guns',5)

get_related_podcasts_scikit(22,5)


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
If you look above you can see that we are accessing a csv file that has the locations of the podcasts, taking the html data and parsing it, to look for any words that end with ".mp3" and then enacting our download_mp3 function to download each podcast into it's own folder. I orginally was downlaoding from the top 30 podcasts and transcrbing 11 episodes.
Part II: Transcription
Doing some research it appears that there is a fantastic plugin called pyAudio. PyAudio uses C based dependencies and can be linked to private API's or can be used with an open-source technology called CMU Sphinx. The detailed explanation of how CMU Sphinx is able to transcribe is not covered in this report as digital signal processing and audio processing is anything but trivial, but for a quick version and extremely simple explanation. We are basically taking the mp3 files and transforming them to wav files. From the wav files we are able to split the file up by looking for silences, and then break down each ofm those words into 10 millisecond partitions. We use a Hidden Markov Model to do the recognition of the words. I highly recommend checking out http://cmusphinx.sourceforge.net/wiki/tutorialconcepts .
Thankfully some wonderful researchers were able to place this into the library of speechRecognition, the code looks like the following (note this requires dependencies for install pockesphinx. You can do so with homebrew (brew install portaudio) pip install pyaudio, pip install speechRecognition:
Can you write Python code for it?

from IPython.display import Audio
import speech_recognition as sr
r = sr.Recognizer()

with sr.Microphone() as source:
    print("Say something!")
    audio = r.listen(source)

%%time
print(r.recognize_sphinx(audio))


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
If you ran into problems running the code above, you will need to look into installing your dependencies: https://pypi.python.org/pypi/SpeechRecognition/
The next step was comparison of the different options for transcription. I compared the open-source CMU vs Waston API. Since I decided to stick with CMU I will not be including the code to transcribe using Watson, but I did compare two samples to see what the drop off would be:
Can you write Python code for it?

from nltk.corpus import stopwords
from collections import Counter
import pandas as pd
import numpy as np
import nltk.data
from __future__ import division  # Python 2 users only
import nltk, re, pprint
from nltk import word_tokenize

os.chdir("/Users/sheldon/git/springboard_capstone/")
cmu_trans = open('report_assets/transcription_cmu.txt','rU').read()
wat_trans = open('report_assets/transcription_watson_2.txt','rU').read()
stop = set(stopwords.words('english'))

## Tokenize and Lower the Words

def tokenize_and_lower(textfile):
    tokens = word_tokenize(textfile)
    lower = [w.lower() for w in tokens]
    filtered_words = [word for word in lower if word not in stop]
    series = pd.Series(filtered_words)
    return series

## Compare results with value counts, presuming that Watson is more accurate than CMU

cmu = tokenize_and_lower(cmu_trans)
wat = tokenize_and_lower(wat_trans)
cmu = pd.Series.to_frame(cmu)
wat = pd.Series.to_frame(wat)
cmu.columns = [['words']]
wat.columns = [['words']]
cmu = cmu.groupby('words').size().reset_index()
wat = wat.groupby('words').size().reset_index()
df = pd.merge(cmu, wat, on='words')
df.columns = [['words','cmu','wat']]
df['cmu_diff_wat'] = df.cmu - df.wat

%matplotlib inline
df.cmu_diff_wat.value_counts().plot(kind='bar')

df.cmu_diff_wat.value_counts()


I am working on the file "Podcast Recommendation Engine Report.ipynb".
The first task is:
Looking through these topics you can immdiately see that there are some related themes, like 2 which has "republican, conservative, and trump, constitution". In fact, LSI has been tested by having a the machine cluster documents and having humans sort documents, and the results have been very similar.
Once we have our topics we need to create similar functions from before:
Can you write Python code for it?

def get_related_podcasts(index):
    def getKey(item):
        return item[1]
    corpus = corpus_lsi[index]
    corpus = sorted(corpus, key=getKey, reverse=True)[0:10]
    related_df = pd.DataFrame(corpus,columns=['index','score'])
    final_df = pd.merge(related_df, df, on='index')[['index','episode','score','series']]
    return final_df

def get_related_podcasts_query(query):
    query = query.lower()
    vec_box = dictionary.doc2bow(query.split())
    vec_lsi = lsi[vec_box]
    sims = index[vec_lsi]
    sims = sorted(enumerate(sims), key=lambda item: -item[1])[0:10]
    related_df = pd.DataFrame(sims,columns=['index','score'])
    final_df = pd.merge(related_df, df, on='index')[['index','episode','score','series']]
    return final_df

get_related_podcasts(1)

get_related_podcasts_query("trump clinton obama guns")

%timeit get_related_podcasts(1)

%timeit get_related_podcasts_query('economics math statistics')