I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query above to look at the sticker_price of cars from the car_table in descending order:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(order_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query above to return gross margin instead of gross profit. Rename the alias as well. Limit it to 5 results
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(operator_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query above to return the average gross profit (revenue - cogs) per make (remember that "make" is in the car_table)
Extra things to try:
- Round average revenue to two decimal points
- Order the results by gross profit in descending order
- Rename the make column as "Car_Maker" and use the alias in the GROUP BY clause
- Rename gross profit column as "Avg_Gross_Profit" and use the alias in the ORDER BY clause
- Join the salesman_table and filter results to only look at revenue where first_name is Michael
 - After you've gotten the query to run with all of these adjustments, think about the risks involved with adding something in the WHERE clause that doesn't show up in the SELECT clause. Think about a potential solution to these risks.
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(group_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query from above, but instead of selecting all columns, specify just the customer gender and the revenue:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(join_cheat2)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query from above, but this time select the customer_id, gender, and revenue:
- You'll probably hit an error at first. Try to use what you've learned about this structure "table_x.column_a" to fix the issue. Why do you think you need to use this?
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(join_cheat3)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Rewrite the query to look at the other tables:
Can you write Python code for it?

run('''
    PRAGMA TABLE_INFO(sales_table)
    ''')
#print(describe_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
SQL really lets you dig.
Some other quick examples - we could do a gender breakdown of customers per car model and add a total at the bottom:
Can you write Python code for it?

run('''
    SELECT
        C.model as Car_Model,
        ROUND(SUM(CASE WHEN CUST.gender = 'female' THEN 1 END)/(COUNT(S.id)*1.0), 2) AS '% Female Customers',
        ROUND(SUM(CASE WHEN CUST.gender = 'male' THEN 1 END)/(COUNT(S.id)*1.0), 2) AS '% Male Customers'
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
        JOIN cust_table CUST on S.customer_id = CUST.customer_id
    GROUP BY
        Car_Model

    UNION ALL

    SELECT
        'Total:',
        ROUND(SUM(CASE WHEN CUST.gender = 'female' THEN 1 END)/(COUNT(S.id)*1.0), 2) AS '% Female Customers',
        ROUND(SUM(CASE WHEN CUST.gender = 'male' THEN 1 END)/(COUNT(S.id)*1.0), 2) AS '% Male Customers'
    FROM
        sales_table S
        JOIN cust_table CUST on S.customer_id = CUST.customer_id
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Say you can't remember the model of the car you're trying to look up. You know it's "out"...something. Outcast? Outstanding?   Write a query to return the model_id and model from the car_table and use LIKE to help you search:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(where_cheat3)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
So far, we've just done a simple join, also called an "inner join". To illustrate different types of joins, we're going to use a different "database" for the following lesson.  First, let's take a look at each one:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        Dog_Table
    ''')

run('''
    SELECT
        *
    FROM
        Cat_Table
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Starting with a simple example, here we'll use CASE WHEN to create a new column on the sales_table:
Can you write Python code for it?

run('''
    SELECT
        revenue,
        CASE WHEN revenue > 20000 THEN 'Revenue is more than 20,000'
            END Conditional_Column
    FROM
        sales_table
    LIMIT 10
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Structure and Formatting Query Basics:

Indentations and Returns:

Mostly arbitrary in SQL
Usually for readability

Capitalization:

Convention to put keywords (functions, clauses) in CAPS
Consistency is best

Order of Clauses:

Very strict
Not all clauses need to be present in a query, but when they are present, then they must be in the correct order
Below are the major clauses that we are going to cover. Use this list as reference if you are getting errors with your queries - there's a chance you just have the clauses in the wrong order:
        SELECT
        FROM
        JOIN...ON
        WHERE
        GROUP BY
        UNION
        ORDER BY
        LIMIT

Reading a table's structure:
PRAGMA TABLE_INFO(table_name)

Running this will let you see the column heads and data types of any table.
The SQL query above only works for SQLite, which is what we're using here. If you're interested in knowing the equivalent versions for other RDBMS options, see the table below.
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Ta-DA! Useful customer segments!
Try to break up the "Customers" column into 2 columns - one for male and one for female. Keep the age segments intact.
- Note that COUNT(***) cannot be wrapped around a CASE WHEN expression the way that other functions can. Try to think of a different way to get a count.
- Extra challenge: try to express male and female customers as a percentage of the total for each group, rounded to 2 decimal points
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(case_cheat2)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
The query below limits the number of rows to 5 results. Change it to 10 to get a quick sense of what we're doing here:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
    LIMIT 5
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
There's a column named "id" in the salesman_table too. However, it doesn't look like those IDs correspond to the sales_table IDs. In fact, it's the salesman_id column in the sales_table that corresponds to the id column in the salesman_table. More often than not, your tables will use different names for corresponding columns, and will have columns with identical names that don't correspond at all.
Write a query to join the salesman_table with the sales_table (select all columns using an asterisk)
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(join_cheat4)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
These are the names of the tables in our mini SQLite database:
sales_table
car_table
salesman_table
cust_table

Start by looking at the columns and their data types  in the sales_table.
Can you write Python code for it?

run('''
    PRAGMA TABLE_INFO(sales_table)
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
These tables are related. There's a column named "model_id" in the sales_table and a "model_id" in the car_table - but the column names don't need to be the same, what's important is that the values in the sales_table's model_id column correspond to the values in the car_table's model_id column.
You can join these tables by using these columns as keys.
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
        JOIN car_table ON sales_table.model_id = car_table.model_id
    LIMIT 10
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
This time, you're seeing everything from the Dog_Table, but only results from the Cat_Table IF the owner also has a dog.

OUTER JOINS or FULL OUTER JOINS:
SELECT
    *
FROM
    table_x X
    OUTER JOIN table_y Y ON X.column_a = Y.column_a  # Returns all rows, regardless of whether values match

Outer joins include ALL rows from both tables, even if the values on the key columns don't match up.
SQLite doesn't support this, so the query below is a workaround to show you the visual effect of an outer join
This provides a great workaround for MySQL: http://stackoverflow.com/questions/4796872/full-outer-join-in-mysql

For now, this query won't totally make sense, just pay attention to the results so you can visualize an outer join:
Can you write Python code for it?

run('''
    SELECT
        C.Owner_Name,
        Cat_Name,
        Dog_Name
    FROM
        Cat_Table C
        LEFT JOIN Dog_Table D ON D.Owner_Name = C.Owner_Name

    UNION ALL

    SELECT
      D.Owner_Name,
      ' ',
      Dog_Name
    FROM
        Dog_Table D
    WHERE
        Owner_Name NOT IN (SELECT Owner_Name from Cat_Table)
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
WHERE column LIKE:
SELECT
    column_a
FROM
    table_name
WHERE
    column_a LIKE '%text or number%'      # Filters the result_set to rows where that text or value can be found, with % standing in as a wildcard

LIKE lets you avoid issues with capitalization in quotes, and you can use % as a wildcard to stand in for any character
Useful if you have an idea of what text you're looking for, but you are not sure of the spelling or you want all results that contatin those letters
More on LIKE: http://www.w3schools.com/sql/sql_like.asp
More on wildcards: http://www.w3schools.com/sql/sql_wildcards.asp

Note that you don't have to use the whole word "cash" when you use LIKE, and that the capital "C" now doesn't cause a problem:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
    WHERE
        payment_type LIKE 'Cas%'
    ''').head()


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
When the query is longer, this method is messy. Suddenly it's harder to parse out which parts of the "WHERE" clause are actual filters, and which parts are just facilitating the join.
Note that we've covered all of these clauses and expressions by now, try to parse out what's going on:
Can you write Python code for it?

run('''
    SELECT
        C.make,
        C.model,
        S.revenue,
        CUST.gender,
        SM.first_name
    FROM
        sales_table S
        JOIN car_table C
        JOIN salesman_table SM
        JOIN cust_table CUST
    WHERE
        S.customer_id = CUST.customer_id
        AND S.model_id = C.model_id
        AND S.salesman_id = SM.id
        AND (C.model in ('Tundra', 'Camry', 'Corolla') OR C.make = 'Subaru')
        AND S.revenue between 17000 and 22000
        AND CUST.gender = 'female'
        AND SM.first_name NOT IN ('Kathleen', 'Samantha')
    LIMIT 5
    ''')


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Write a query to look at the model name and COGs for each car in car_table, then use a nested query to also look at the average COGs off all car models in a third column
- Extra Challenge: add a fourth colum using another nested query to return the difference between each car model's COGs and the average COGs
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(nest_cheat1)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Write a query to select all columns from the car_table:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(select_cheat1)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Write a query to show the average difference between the sticker_price (in car_table) and the revenue.

If you want a challenge, try to join cust_table and limit the query to only look at transactions where the customer's age is over 35
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(avg_cheat2)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
Write a query with the following criteria:
- SELECT clause:
 - salesman's last name and average revenue, rounded to the nearest cent
- FROM clause:
 - sales_table joined with the salesman_table and the cust_table
- WHERE clause:
 - only female customers
- GROUP BY clause:
 - only salespeople whose average revenue was greater than 20,000
So, in plain English, we want to see salespeople whose average revenue for female customers is greater than 20,000
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(having_where_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
You can use an alias in the ORDER BY and WHERE clauses now. Write a query to:
- pull the model_id and revenue for each transaction
- give model_id the alias "Model"
- give revenue the alias "Rev"
- limit the results to only include rows where the model_id id 36, use the alias in the WHERE clause
- order the results by revenue in descending order, use the alias in the ORDER BY clause
- Run the query
THEN:
- Try giving model_id the alias "ID" and use it in the WHERE clause, then rerun the query. What do you think is causing the error?
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(alias_cheat)


I am working on the file "Code/SQL/SQL_Intro.ipynb".
The first task is:
You can use values from another column as well.  Remember this query from the GROUP BY lesson? It's often helpful to look at information broken out by multiple groups, but it's not especially easy to digest:
Can you write Python code for it?

run('''
    SELECT
        C.Make as car_maker,
        payment_type,
        ROUND(AVG(S.revenue)) as avg_revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        C.Make,
        payment_type
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
"HAVING" in GROUP BY statements:
SELECT
    column_a,
    SUM(column_b) AS alias_b
FROM
    table_name
GROUP BY
    column_a HAVING alias_b > x   # only includes groups in column_a when the sum of column_b is greater than x

If you've applied a function to a column and want to filter to only show results meeting a particular criteria, use HAVING in your GROUP BY clause.
More on HAVING: http://www.w3schools.com/sql/sql_having.asp

The query below will sum up all the revenue for each car maker, but it will only show you results for car maker's whose total revenue is greater than 50,000:
Can you write Python code for it?

run('''
    SELECT
        C.Make as Car_Maker,
        SUM(revenue) as Total_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        Car_Maker HAVING Total_Revenue > 500000
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
ALIASES
SELECT
    T.column_a AS alias_a    # creates a nickname for column_a, and states that it's from table_name (whose alias is T)
FROM
    table_name AS T          # creates a nickname for table_name
WHERE
    alias_a = z           # refer to an alias in the WHERE clause
ORDER BY
    alias_a                  # refer to an alias in the ORDER BY clause

Aliases are optional, but save you time and make column headers cleaner
AS isn't necessary to create and alias, but commonly used
The convention is to use "AS" in the "SELECT" clause, but not in the "FROM" clause.
More on Aliases: http://www.w3schools.com/sql/sql_alias.asp

Change the aiases for model_id and revenue, or add extra columns to see how they work:
Can you write Python code for it?

run('''
    SELECT
        model_id AS Model_of_car,
        revenue AS Rev_per_car
    FROM
        sales_table
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Above, there's a column called "id".
Now look at the salesman_table again:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        salesman_table
    LIMIT 5
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Add on the average amount of revenue made per sale:
Can you write Python code for it?

run('''
    SELECT
        first_name ||  ' ' || last_name as Salesperson,
        COUNT(*) as Cars_Sold,
        ROUND(AVG(revenue)) as Revenue_per_Sale
    FROM
        sales_table S
        JOIN salesman_table M ON S.salesman_id = M.id
    GROUP BY
        Salesperson
    ORDER BY
        Cars_Sold DESC
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
All model_ids are returned, but the averages are all much lower than they should be. That's because the query first drops all rows that have revenue greater than 18000, and then averages the remaining rows.
When you use HAVING, SQL follows these steps instead (this query should look like the one you wrote in the last challenge):
SELECT
    C.model as Car_Model,
    AVG(S.revenue) as Avg_Revenue
FROM
    sales_table S
    JOIN car_table C on S.model_id = C.model_id
GROUP BY
    Car_Model HAVING Avg_Revenue < 18000

Find the sales_table and join it to the car_table (same as before)
Pull the data from the 'model' column in car_table and 'revenue' column in sales_table (same as before)
Average the rows for each Car_Model
Return only the Car_Models whose averages are less than 18,000

And as you can see, there's a big difference in these results and the results of the query that used "WHERE" instead of HAVING:
Can you write Python code for it?

run('''
    SELECT
        C.model as Car_Model,
        AVG(S.revenue) as Avg_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        Car_Model HAVING Avg_Revenue < 18000
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Because SQLite doesn't support ROLLUP, the query below is just intended to illustrate how ROLLUP would work. Don't worry about understanding the query itself, just get familiar with what's going on in the result-set:
Can you write Python code for it?

run('''
    SELECT
        C.model AS Car_Model,
        SUM(S.revenue) as Sum_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY C.model

    UNION ALL

    SELECT
        'NULL',
        SUM(S.revenue)
    FROM
        sales_table S
''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
CASE WHEN also lets you create new groups.  Start by looking at the cust_table grouped by age - remember that COUNT(***) tells you how many rows are in each group (which is the same as telling you the number of customers in each group):
Can you write Python code for it?

run('''
    SELECT
        age,
        COUNT(*) customers
    FROM
        cust_table
    GROUP BY
        age
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
CASE WHEN gives you the value "Revenue is more MORE 20,000" when revenue in that same row is greater than 20,000. Otherwise, it has no value.
Now let's add a level:
Can you write Python code for it?

run('''
    SELECT
        revenue,
        CASE WHEN revenue > 20000 THEN 'Revenue is MORE than 20,000'
             WHEN revenue < 15000 THEN 'Revenue is LESS than 15,000'
             END Conditional_Column
    FROM
        sales_table
    LIMIT 10
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
CASE WHEN makes this same information a lot easier to read by letting you pivot the result set a little.
Write a query using CASE WHEN to look at total revenue per gender, grouped by each car model
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(case_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
CONCATENATING:
Concatenating varies by RDBMS:
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Conditional Expressions: IF & CASE WHEN
SELECT
    CASE WHEN column_a = x THEN some_value
         WHEN column_a = y THEN some_value2
         ELSE some_other_value
         END some_alias                         # alias optional after END
FROM
    table_name

Conditional expressions let you use IF/THEN logic in SQL
In SQLite, you have to use CASE WHEN, but in other RDBMS you may prefer to use IF, depending on your needs
More on CASE WHEN: http://www.dotnet-tricks.com/Tutorial/sqlserver/1MS1120313-Understanding-Case-Expression-in-SQL-Server-with-Example.html
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Consider the issue we had before, where SQLite didn't support WITH ROUNDUP. We used this query as a workaround. Does it make sense now?
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
DO use LIKE on small tables and in simple queries:
LIKE is helpful if you know where to find something but you can't quite remember what it's called. Try to use a wildcard sparingly - don't use 2 when 1 will suffice:
Can you write Python code for it?

run('''
    SELECT
        model_id, model
    FROM
        car_table
    WHERE
        model LIKE '%undra'
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
DON'T use LIKE on large tables or when using JOINs:
Can you write Python code for it?

run('''
    SELECT
        C.model,
        AVG(revenue)
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    WHERE
        C.model LIKE '%undra'
''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
DON'T use an asterisk unless you absolutely have to:
This can put a lot of strain on servers. Only use if you know for certain that your using a small table
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
    ''').head()


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
DON'T use negative filters when a positive filter is possible
Let's say you want to look at cars made by Toyato and Honda, but you don't care about Subaru. It might be tempting to use a negative filter:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        car_table
    WHERE
        make != 'Subaru'
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Different RDBMS have different datatypes available:
- Oracle: http://docs.oracle.com/cd/B10501_01/appdev.920/a96624/03_types.htm
- MySQL:
 - Numeric: http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html
 - Date/time: http://dev.mysql.com/doc/refman/5.0/en/date-and-time-type-overview.html
 - String/text: http://dev.mysql.com/doc/refman/5.0/en/string-type-overview.html
- SQLite: https://www.sqlite.org/datatype3.html
- Microsoft: http://msdn.microsoft.com/en-us/library/ms187752.aspx

SELECT & FROM:

Basically every "read" query will contain a SELECT and FROM clause
In the SELECT clause, you tell SQL which columns you want to see
In the FROM clause, you tell SQL the table where those columns are located
More on SELECT: http://www.w3schools.com/sql/sql_select.asp

SELECT * (ALL COLUMNS)
SELECT          # specifies which columns you want to see
    *           # asterisk returns all columns
FROM            # specifies the table or tables where these columns can be found
    table_name

Use an asterisk to tell SQL to return all columns from the table:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Different Types of Joins
There are different types of joins you can do according to your needs. Here's a helpful way to visualize your options: http://www.codeproject.com/Articles/33052/Visual-Representation-of-SQL-Joins
However, not all types of joins are compatible with SQLite and MySQL. The table below breaks down compatibility:
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Easily create age groups and see how aggressively each group negotiates (judged by the difference between the actual sale amount and the sticker price):
Can you write Python code for it?

run('''
    SELECT
        CASE WHEN age BETWEEN 18 AND 24 THEN '18-24 years'
             WHEN age BETWEEN 25 AND 34 THEN '25-34 years'
             WHEN age BETWEEN 35 AND 44 THEN '35-44 years'
             WHEN age BETWEEN 45 AND 54 THEN '45-54 years'
             WHEN age BETWEEN 55 AND 64 THEN '55-64 years'
             END Age_Group,
        ROUND((SUM(S.revenue)-SUM(C.sticker_price))/SUM(C.sticker_price), 2) as '% Paid Below Sticker Price'
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
        JOIN cust_table CUST on S.customer_id = CUST.customer_id
    GROUP BY
        Age_Group
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Essentially, in Venn Diagram terms, and outer join lets you see all contents of both circles. This join will let you see all pet owners, regardless of whether the own only a cat or only a dog

Using the "WHERE" Clause to Join Tables
SELECT
    *
FROM
    table_x X
    JOIN table y Y
WHERE
    X.column_a = Y.column_a         # tells SQL the key for the join

Some people prefer to use the WHERE clause to specify the key for a join
Fine if the query is short, but SUPER messy when the query is complex
We won't use this moving forward, but it's good to see it in case you run across someone else's code and you need to make sense of it

When it's simple, it's not so bad:
Can you write Python code for it?

run('''
    SELECT
        C.model,
        S.revenue
    FROM
        sales_table S, car_table C
    WHERE
        S.model_id = C.model_id
    LIMIT 5
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
FUNCTIONS:
SELECT
    SUM(column_a),             # sums up the values in column_a
    AVG(column_a),             # averages the values in column_a
    ROUND(AVG(column_a), 2),   # rounds the averaged values in column_a to 2 digits
    COUNT(column_a),           # counts the number of rows in column_a
    MAX(column_a),             # returns the maximum value in column_a
    MIN(column_a),             # returns the minimum value in column_a
    GROUP_CONCAT(column_a)     # returns a comma separated list of all values in column_a
FROM
    table_name

Functions can be applied to columns to help analyze data
You can find more than just these basic few in the link below, or just Google what you're looking to do - there's a lot of help available on forums
More on functions: http://www.w3schools.com/sql/sql_functions.asp

The function below will sum up everything in the revenue column. Note that now we only get one row:
Can you write Python code for it?

run('''
    SELECT
        SUM(revenue) AS Total_Revenue
    FROM
        sales_table
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
HAVING & WHERE in the same query:

Sometimes, you will want to use WHERE and HAVING in the same query
Just be aware of the order of the steps that SQL takes
Rule of thumb: if you're applying a function to a column, you probably don't want that column in there WHERE clause

This query is only looking at Toyotas whose revenue is less than 18,000,  using WHERE to limit the results to Toyotas, and HAVING to limit the results by revenue:
Can you write Python code for it?

run('''
    SELECT
        C.model as Car_Model,
        AVG(S.revenue) as Avg_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    WHERE
        C.make = 'Toyota'
    GROUP BY
        Car_Model HAVING Avg_Revenue < 18000
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Here we'll use SQLite and use the concatenating operator || to combine words/values in different columns:
Can you write Python code for it?

run('''
    SELECT
        last_name,
        first_name,
        last_name || ', ' || first_name AS full_name
    FROM
        salesman_table
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
If you want to look at average revenue for car models that are like "%undra", run the LIKE query on the small table (car_table) first to figure out exacly what you're looking for, then use that information to search for the data you need from the sales_table
DO dip your toe in by starting with a small data set
Use WHERE to only view a few days of data at first. If the query runs quickly, add a few days at a time. If it starts to run slowly, run just a few days at a time and paste results into excel to combine results (or use Python...ask me later!!!).
The query below won't work because SQLite doesn't recognize dates, but remember these concepts when working with other RDBMS
Can you write Python code for it?

run('''
    SELECT
        revenue,
        date
    FROM
        sales_table
    WHERE
        date = '1/1/14'
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Make it easier to compare the average revenue of Jared's sales to the average revenue of per sale overall by adding a column to see by what percent each salesperson's sales are more or less than average:
Can you write Python code for it?

run('''
    SELECT
        first_name ||  ' ' || last_name as Salesperson,
        COUNT(*) as Cars_Sold,
        ROUND(AVG(revenue), 2) as Rev_per_Sale,
        ROUND((((AVG(revenue)
            - (SELECT AVG(revenue) from sales_table))
            /(SELECT AVG(revenue) from sales_table))*100), 1) || ' %'
            as RPS_Compared_to_Avg
    FROM
        sales_table S
        JOIN salesman_table M ON S.salesman_id = M.id
    GROUP BY
        Salesperson
    ORDER BY
        Cars_Sold DESC
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
NESTING

Nested queries allow you to put a query within a query
Depending on your needs, you might put a nested query in the SELECT clause, the FROM clause, or the WHERE clause

Consider the following query. We're using a nested query in the SELECT clause to see the sum of all revenue in the sales_table, and then using it again to what percentage of total revenue can be attributed to each Car_Model.
Can you write Python code for it?

run('''
    SELECT
        C.model AS Car_Model,
        SUM(S.revenue) AS Revenue_Per_Model,
        (SELECT SUM(revenue) FROM sales_table) AS Total_Revenue,
        SUM(S.revenue) / (SELECT SUM(revenue) FROM sales_table) AS Contribution_to_Revenue
    FROM
        sales_table S
        JOIN car_table C ON C.model_id = S.model_id
    GROUP BY
        Car_Model
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Notice that the Owner_Name columns on each table have some corresponding values (Michael, Gilbert, May and Elizabeth and Donna are in both tables), but they both also have values that don't overlap.
JOINS or INNER JOINS
SELECT
    *
FROM
    table_x X
    JOIN table_y Y ON X.column_a = Y.column_a      # Returns rows when values match on both tables.

This is what we used in the initial example.  Simple joins, (also called Inner Joins) will combine tables only where there are corresponding values on both tables.
Write a query below to join the Cat_Table and Dog_Table using the same method we've used before:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(inner_join_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Now let's add back the ROUND() and AVG() functions and the GROUP BY statement:
Can you write Python code for it?

run('''
    SELECT
        C.Make as Car_Maker,
        ROUND(AVG(CASE WHEN payment_type = 'cash' THEN S.revenue END)) AS Avg_Cash_Revenue,
        ROUND(AVG(CASE WHEN payment_type = 'finance' THEN S.revenue END)) AS Avg_Finance_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        C.Make
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
On a big table, this will run much more slowly than if you use a positive filter. Try this instead - it might require a little extra typing, but it will run much faster:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        car_table
    WHERE
        make in ('Toyota', 'Honda')
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
One more quick note on the basics of SELECT - technically you can SELECT a value without using FROM to specify a table. You could just tell the query exactly what you want to see in the result-set. If it's a number, you can write the exact number. If you are using various characters, put them in quotes.
See the query below as an example:
Can you write Python code for it?

run('''
    SELECT
    4,
    5,
    7,
    'various characters or text'
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Optimization:
Non-optimized queries can cause a lot of problems because tables frequently have thousands or millions of rows:
If you haven't optimized your query, it might:

Take several minutes (or even hours) to return the information you're requesting
Crash your computer
Muck up the server's processes, and you'll face the wrath of your company's system administrators once they figure out that you are the reason why the whole system has slowed down and everyone is sending them angry emails (this will probably happen to you no matter what. It's a rite of passage).

Find a few more useful optimization tips here: http://hungred.com/useful-information/ways-optimize-sql-queries/
Some of these seem strange, because we're going ling you NOT to do a bunch of things that you've learned how to do. Stick to this principal: if you're dealing with a small table, you can break a few of these rules. The larger the table, the fewer rules you can break.
DO name specific columns in the SELECT CLAUSE:
Can you write Python code for it?

run('''
    SELECT
        date,
        revenue
    FROM
        sales_table
    ''').head()


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Practice applying this "table_x.column_a" format to all columns in the SELECT clause when you are joining multiple tables, since multiple tables frequenty use the same column names even when they don't correspond.
It's common to use single-letter aliases for tables to make queries shorter. Take a look at the query below and make sure you understand what's going on with the table aliases. It's the same query that you wrote earlier, but with aliases to help identify the columns
Can you write Python code for it?

run('''
    SELECT
        S.customer_id,
        C.gender,
        S.revenue
    FROM
        sales_table AS S
        JOIN cust_table AS C on S.customer_id = C.customer_id
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query above to look at average revenue per model, and using HAVING to filter your result-set to only include models whose average revenue is less than 18,000:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(having_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query above to return the average gross profit (revenue - cogs) per make (remember that "make" is in the car_table)
Extra things to try:
- Round average revenue to two decimal points
- Order the results by gross profit in descending order
- Rename the make column as "Car_Maker" and use the alias in the GROUP BY clause
- Rename gross profit column as "Avg_Gross_Profit" and use the alias in the ORDER BY clause
- Join the salesman_table and filter results to only look at revenue where first_name is Michael
 - After you've gotten the query to run with all of these adjustments, think about the risks involved with adding something in the WHERE clause that doesn't show up in the SELECT clause. Think about a potential solution to these risks.
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(group_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query from above, but instead of selecting all columns, specify just the customer gender and the revenue:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(join_cheat2)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query from above, but this time select the customer_id, gender, and revenue:
- You'll probably hit an error at first. Try to use what you've learned about this structure "table_x.column_a" to fix the issue. Why do you think you need to use this?
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(join_cheat3)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query to find the total revenue grouped by each salesperson's first_name and by the customer's gender (gender column in cust_table)
- For an extra challenge, use the concatenating operator to use the salesperson's full name instead
- Add COUNT(S.id) to the SELECT clause to see the number of transactions in each group
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(group_cheat2)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Rewrite the query to look at the other tables:
Can you write Python code for it?

run('''
    PRAGMA TABLE_INFO(sales_table)
    ''')
#print(describe_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
So far, we've just done a simple join, also called an "inner join". To illustrate different types of joins, we're going to use a different "database" for the following lesson.  First, let's take a look at each one:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        Dog_Table
    ''')

run('''
    SELECT
        *
    FROM
        Cat_Table
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
So maybe Jared is just selling cheaper cars.
Let's go further and compare the sale price of each car against the sticker price to see how low Jared was willing to negotiate with customers. Sticker price is in anther table, but again, that's no problem with SQL:
Can you write Python code for it?

run('''
    SELECT
        first_name ||  ' ' || last_name as Salesperson,
        COUNT(*) as Cars_Sold,
        '$ ' || ROUND(AVG(revenue), 2)  as Rev_per_Sale,
        ROUND((((AVG(revenue)
            - (SELECT AVG(revenue) from sales_table where salesman_id != 215))
            /(SELECT AVG(revenue) from sales_table where salesman_id != 215))*100), 1) || ' %'
            AS RPS_Compared_to_Avg,
        ROUND((1-(SUM(revenue) / SUM(sticker_price)))*100, 1) || ' %' as Avg_Customer_Discount
    FROM
        sales_table S
        JOIN salesman_table M ON S.salesman_id = M.id
        JOIN car_table C ON S.model_id = C.model_id
    GROUP BY
        Salesperson
    ORDER BY
        Cars_Sold DESC
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Some things to note:
- Although these queries and their results are unrelated, the column header is dictated by the query that appears first
- Even though the query for "Tundra" is first, "Tundra" is second in the results. UNION will sort all results according to the normal default rules, acending order.
- Replace UNION with UNION ALL and run the query again. What changes?
Use UNION to join two queries. The first should have two columns: car model and COGs per car. The second query should show you to average COGs for all the car models, rounded to cents. You want the the average COGs to appear in the last row.
- Remember that united queries need to have the same number of columns.
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(union_cheat1)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Structure and Formatting Query Basics:

Indentations and Returns:

Mostly arbitrary in SQL
Usually for readability

Capitalization:

Convention to put keywords (functions, clauses) in CAPS
Consistency is best

Order of Clauses:

Very strict
Not all clauses need to be present in a query, but when they are present, then they must be in the correct order
Below are the major clauses that we are going to cover. Use this list as reference if you are getting errors with your queries - there's a chance you just have the clauses in the wrong order:
        SELECT
        FROM
        JOIN...ON
        WHERE
        GROUP BY
        UNION
        ORDER BY
        LIMIT

Reading a table's structure:
PRAGMA TABLE_INFO(table_name)

Running this will let you see the column heads and data types of any table.
The SQL query above only works for SQLite, which is what we're using here. If you're interested in knowing the equivalent versions for other RDBMS options, see the table below.
Can you write Python code for it?


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
The result-set above is essentially what SQL is working with right before it separates the rows into groups and averages the revenue within those groups.
Now, we're going to use some CASE WHEN statements to change this a little:
Can you write Python code for it?

run('''
    SELECT
        C.Make as Car_Maker,
        payment_type,
        CASE WHEN payment_type = 'cash' THEN S.revenue END Cash_Revenue,
        CASE WHEN payment_type = 'finance' THEN S.revenue END Finance_Revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
These are the names of the tables in our mini SQLite database:
sales_table
car_table
salesman_table
cust_table

Start by looking at the columns and their data types  in the sales_table.
Can you write Python code for it?

run('''
    PRAGMA TABLE_INFO(sales_table)
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
These tables are related. There's a column named "model_id" in the sales_table and a "model_id" in the car_table - but the column names don't need to be the same, what's important is that the values in the sales_table's model_id column correspond to the values in the car_table's model_id column.
You can join these tables by using these columns as keys.
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
        JOIN car_table ON sales_table.model_id = car_table.model_id
    LIMIT 10
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
This time, you're seeing everything from the Dog_Table, but only results from the Cat_Table IF the owner also has a dog.

OUTER JOINS or FULL OUTER JOINS:
SELECT
    *
FROM
    table_x X
    OUTER JOIN table_y Y ON X.column_a = Y.column_a  # Returns all rows, regardless of whether values match

Outer joins include ALL rows from both tables, even if the values on the key columns don't match up.
SQLite doesn't support this, so the query below is a workaround to show you the visual effect of an outer join
This provides a great workaround for MySQL: http://stackoverflow.com/questions/4796872/full-outer-join-in-mysql

For now, this query won't totally make sense, just pay attention to the results so you can visualize an outer join:
Can you write Python code for it?

run('''
    SELECT
        C.Owner_Name,
        Cat_Name,
        Dog_Name
    FROM
        Cat_Table C
        LEFT JOIN Dog_Table D ON D.Owner_Name = C.Owner_Name

    UNION ALL

    SELECT
      D.Owner_Name,
      ' ',
      Dog_Name
    FROM
        Dog_Table D
    WHERE
        Owner_Name NOT IN (SELECT Owner_Name from Cat_Table)
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Use DISTINCT to select unqiue values from the salesman_id column in sales_table. Delete DISTINCT and rerun to see the effect.
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(select_cheat3)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Use GROUP_CONCAT to return a comma-separated list of last names from the salesman_table:
Can you write Python code for it?

run('''
    SELECT NULL
''')
#print(concat_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Use || to pull the make and model from the car_table and make it appear in this format: "Model (Make)"
- give it an alias to clean up the column header, otherwise it'll look pretty messy
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(concat_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Using BETWEEN, rewrite the query to return rows where the  revenue was between 24,000 and 25,000:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(where_cheat2)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
WHERE
SELECT
    column_a
FROM
    table_name
WHERE
    column_a = x      # filters the result-set to rows where column_a's value is exactly x

A few more options for the where clause:
WHERE column_a = 'some_text'      # put text in quotations. CAPITALIZATION IS IMPORTANT

WHERE column_a != x      # filters the result-set to rows where column_a's value DOES NOT EQUAL x

WHERE column_a < x       # filters the result-set to rows where column_a's value is less than x

WHERE columna_a <= x      # filters the result-set to rows where column_a's value is less than or equal to x

WHERE column_a IN (x, y)       # column_a's value can be EITHER x OR y

WHERE column_a NOT IN (x, y)     #  column_a's value can be NEITHER x NOR y

WHERE column_a BETWEEN x AND y      # BETWEEN lets you specify a range

WHERE column_a = x AND column_b = y    # AND lets you add more filters

WHERE column_a = x OR column_b = y     # OR will include results that fulfill either criteria

WHERE (column_a = x AND column_b = y) OR (column_c = z)   # use parentheses to create complex AND/OR statements

WHERE allows you to filter the result-set to only include rows matching specific values/criteria. If the value/criteria is text, remember to put it in single or double quotation marks
More on WHERE: http://www.w3schools.com/sql/sql_where.asp

Below, WHERE filters out any rows that don't match the criteria. The result-set will only contain rows where the payment type is cash AND where the model_id is 46:
Can you write Python code for it?

run('''
    SELECT
        *
    FROM
        sales_table
    WHERE
        payment_type = 'cash'
        AND model_id = 46
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
When you want to segment your results, but there are too many different values for GROUP BY to be helpful, use CASE WHEN to make your own groups. GROUP BY the column you created with CASE WHEN to look at your newly created segments.
Can you write Python code for it?

run('''
    SELECT
        CASE WHEN age BETWEEN 18 AND 24 THEN '18-24 years'
             WHEN age BETWEEN 25 AND 34 THEN '25-34 years'
             WHEN age BETWEEN 35 AND 44 THEN '35-45 years'
             WHEN age BETWEEN 45 AND 54 THEN '45-54 years'
             WHEN age BETWEEN 55 AND 64 THEN '55-64 years'
             END Age_Group,
        COUNT(*) as Customers
    FROM
        cust_table
    GROUP BY
        Age_Group
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Wrapping Up:
Debugging:
If you run into errors when you start writing your own queries, here are some things to make sure your query has:
- The right names for columns in the SELECT clause
- Columns that can be found in the tables in the FROM clause
- Consistent use of aliases throughout (if using aliases)
- Joined tables on the corresponding column and proper aliases to indicate each table
- The correct order of clauses:
          SELECT
          FROM
          JOIN...ON
          WHERE
          GROUP BY
          UNION
          ORDER BY
          LIMIT
- Consistent use of capitalization for variables in quotes
- Fuctions and opperators for real numbers, not integers
- The same number of columns/expressions in their SELECT clauses of your queries when using UNION
Gain a deeper understanding:
http://tech.pro/tutorial/1555/10-easy-steps-to-a-complete-understanding-of-sql
Practice on other databases:
http://sqlzoo.net/wiki/SELECT_.._WHERE
Sample Queries for Business Analysis:
Let's say you recently opened a car dealership, and you now have one month's worth of sales data. You want to know how your sales team is doing.
Start by looking at the number of cars each person sold last month. The names of the sales team and the list of transactions are on different tables in your database, but SQL can help you with that:
Can you write Python code for it?

run('''
    SELECT
        first_name ||  ' ' || last_name as Salesperson,
        COUNT(*) as Cars_Sold
    FROM
        sales_table S
        JOIN salesman_table M ON S.salesman_id = M.id
    GROUP BY
        Salesperson
    ORDER BY
        Cars_Sold DESC
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Write a query to make a comma-separated list of models for each car maker:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(group_cheat1)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Write a query to select model_id and model from the car_table:
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(select_cheat2)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Write a query to show the average difference between the sticker_price (in car_table) and the revenue.

If you want a challenge, try to join cust_table and limit the query to only look at transactions where the customer's age is over 35
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(avg_cheat2)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
Write a query with the following criteria:
- SELECT clause:
 - salesman's last name and average revenue, rounded to the nearest cent
- FROM clause:
 - sales_table joined with the salesman_table and the cust_table
- WHERE clause:
 - only female customers
- GROUP BY clause:
 - only salespeople whose average revenue was greater than 20,000
So, in plain English, we want to see salespeople whose average revenue for female customers is greater than 20,000
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(having_where_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
You can also use GROUP BY with multiple columns to segment out the results further:
Can you write Python code for it?

run('''
    SELECT
        C.make AS car_caker,
        payment_type,
        ROUND(AVG(revenue)) as avg_revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        C.Make,
        payment_type
    ''')


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
You can use an alias in the ORDER BY and WHERE clauses now. Write a query to:
- pull the model_id and revenue for each transaction
- give model_id the alias "Model"
- give revenue the alias "Rev"
- limit the results to only include rows where the model_id id 36, use the alias in the WHERE clause
- order the results by revenue in descending order, use the alias in the ORDER BY clause
- Run the query
THEN:
- Try giving model_id the alias "ID" and use it in the WHERE clause, then rerun the query. What do you think is causing the error?
Can you write Python code for it?

run('''
    SELECT NULL
    ''')
#print(alias_cheat)


I am working on the file "Code/SQL/SQL_Intro_DBcopy.ipynb".
The first task is:
You can use values from another column as well.  Remember this query from the GROUP BY lesson? It's often helpful to look at information broken out by multiple groups, but it's not especially easy to digest:
Can you write Python code for it?

run('''
    SELECT
        C.Make as car_maker,
        payment_type,
        ROUND(AVG(S.revenue)) as avg_revenue
    FROM
        sales_table S
        JOIN car_table C on S.model_id = C.model_id
    GROUP BY
        C.Make,
        payment_type
    ''')


I am working on the file "Code/SSRC_LCA_evelynegroen.ipynb".
The first task is:
Step 3: calculate the squared standardized regression coefficients
Can you write Python code for it?

import statistics as stats

var_g = stats.variance(g_list[:,0])
var_x = [stats.variance(Ps_list[:,k]) for k in range(1,6)]

SSRC =  (var_x/var_g) * (RC[1:6,0]**2)
print("squared standardized regression coefficients:", SSRC)


I am working on the file "Code/SSRC_LCA_evelynegroen.ipynb".
The first task is:
Visualize
Can you write Python code for it?

import matplotlib.pyplot as plt
import matplotlib.pyplot as plt

SSRC_procent = SSRC * 100

x_label=[ 'A(1,1)', 'A(2,1)', 'A(2,2)', 'B(1,1)', 'B(1,2)']
x_pos = range(5)
plt.bar(x_pos, SSRC_procent, align='center')
plt.xticks(x_pos, x_label)
plt.title('Global sensitivity analysis: squared standardized regression coefficients')
plt.ylabel('SSRC (%)')
plt.xlabel('Parameter')
plt.show()


I am working on the file "Code/Vehicle Classification Exercise.ipynb".
The first task is:
Load the processed sensor data for bus trip (see "Process Smartphone Sensor Data" jupyter notebook).  On this trip, I took the 47 bus for about 10 minutes.
Can you write Python code for it?

dfbus = pd.read_csv('../Data/shanebus20150827_processed.csv', index_col='DateTime')

# combine into a single dataframe
df = pd.concat([dfcar, dfbus])

# Use only userAcceleration and gyroscope data, since these features are expected to generalize well.
xyz = ['X', 'Y', 'Z']
measures = ['userAcceleration', 'gyroscope']
basefeatures = [i + j for i in measures for j in xyz]
features = [i + j for i in measures for j in xyz]

# Add Gaussian smoothed features
smoothfeatures = []
for i in features:
    df[i + 'sm'] = gaussian_filter(df[i], 3)
    df[i + '2sm'] = gaussian_filter(df[i], 100)
    smoothfeatures.append(i + 'sm')
    smoothfeatures.append(i + '2sm')
features.extend(smoothfeatures)

# Generate Jerk signal
jerkfeatures = []
for i in features:
    diffsignal = np.diff(df[i])
    df[i + 'jerk'] = np.append(0, diffsignal)
    jerkfeatures.append(i + 'jerk')
features.extend(jerkfeatures)

# assign class labels
car0 = (df.index > '2015-08-25 14:35:00') & \
        (df.index <= '2015-08-25 14:42:00')

car1 = (df.index > '2015-08-25 14:43:00') & \
        (df.index <= '2015-08-25 14:48:00')

bus0 = (df.index > '2015-08-27 10:10:00') & \
        (df.index <= '2015-08-27 10:15:00')
bus1 = (df.index > '2015-08-27 10:15:00') & \
        (df.index <= '2015-08-27 10:20:00')

nc = len(df)
df['class'] = np.zeros(nc) - 1
df['class'][car0] = np.zeros(nc)
df['class'][car1] = np.zeros(nc)
df['class'][bus0] = np.ones(nc)
df['class'][bus1] = np.ones(nc)

# separate into quarters for train and validation
q1 = df[car0]
q2 = df[car1]
q3 = df[bus0]
q4 = df[bus1]
traindf = pd.concat([q2, q4])
validationdf = pd.concat([q1, q3])

# check for NaNs in the dataframes
print(traindf.isnull().sum().sum())
print(validationdf.isnull().sum().sum())

# drop NaNs
traindf = traindf.dropna()
validationdf = validationdf.dropna()

# Make the training and validation sets
X_train = traindf[features].values
y_train = traindf['class'].values
X_test = validationdf[features].values
y_test = validationdf['class'].values

# train a random forest
clf =...


I am working on the file "Code/Vehicle Classification Exercise.ipynb".
The first task is:
Load the processed sensor data for car trip (see "Process Smartphone Sensor Data" jupyter notebook).  On this trip, I drove my car from home to Censio and back and used SensorLog on my iPhone to track the trip.  The total time for the trip was about 15 minutes.
Can you write Python code for it?

dfcar = pd.read_csv('../Data/shaneiphone_exp2_processed.csv', index_col='DateTime')


I am working on the file "Code/Vehicle Classification Exercise.ipynb".
The first task is:
The smoothed gyroscopeZ data is the most useful feature.
Can you write Python code for it?

# compare bus gyroscopeZ2sm and car gyroscopeZ2sm
q1['gyroscopeZ2sm'].plot(color='blue', figsize=(12,6), kind='hist', bins=40, alpha=0.4) # car
q3['gyroscopeZ2sm'].plot(color='green', kind='hist', bins=40, alpha=0.4) # bus


I am working on the file "Code/Vehicle Classification Exercise.ipynb".
The first task is:
We're not overfitting the data, but we're also not really predicting the vehicle class very well, since we're only right about 65-70% of the time with any prediction we make.
Can you write Python code for it?

# Inspect feature importances
for i, ifeature in enumerate(features):
    print(ifeature + ': %6.4f' % clf.feature_importances_[i])


I am working on the file "Code/all_analysis.ipynb".
The first task is:
List of blocked reactions in the model
The following reactions were blocked in the original (Nagarajan,2013) model and still blocked in our model, FYI.
Can you write Python code for it?

model = M.copy()
d = {}
for rxn in model.reactions:
    if rxn.lower_bound == 0 and rxn.upper_bound == 0:
        d[rxn.id] = [rxn.name,rxn.reaction]

df = pd.DataFrame.from_dict(d)
df = df.transpose()
df.columns = ['Name','Reaction']
df


I am working on the file "Code/all_analysis.ipynb".
The first task is:
Reproduce all 6 situations from Fig. 3 of (Schuchmann, 2014)
Here we consider a similar analysis (but performed with FBA on the metabolic map instead of on paper on a small reaction network) as in Figure 3 in the publication by Schuchmann and Müller.
We extended the considered scenarios of Figure 3 by also considering alternatives in the MTHFD and MTHFR reactions since Schuchmann and Müller consider different versions of these enzymes than the model by Nagarajan et al.
In brackets we show the flux through the Nfn complex to show its contribution to each flux pattern. The set of Nfn fluxes shows that first of all it is not essential and second it may function at different flux levels and in different directions.
Can you write Python code for it?

results = [[],[],[]] # this will be filled with real results
for MTHFR in range(2):
    for MTHFD in range(2):
        for FDH in range(3):
            for HYD in range(2):
                pfbaSol = []
                model = M.copy()

                if FDH == 0:
                    pass # keep the default FDH7
                elif FDH == 1:
                    model.reactions.FDH7.upper_bound = 0; model.reactions.FDH7.lower_bound = 0;
                    model.reactions.FDHH2.lower_bound = 0; model.reactions.FDHH2.upper_bound = 0
                    model.reactions.FDHFDNADPH.lower_bound = -1000; model.reactions.FDHFDNADPH.upper_bound = 1000
                else:
                    model.reactions.FDH7.upper_bound = 0; model.reactions.FDH7.lower_bound = 0;
                    model.reactions.FDHH2.lower_bound = -1000; model.reactions.FDHH2.upper_bound = 1000
                    model.reactions.FDHFDNADPH.lower_bound = 0; model.reactions.FDHFDNADPH.upper_bound = 0

                if HYD == 0:
                    model.reactions.HYDFDN.lower_bound = -1000; model.reactions.HYDFDN.upper_bound = 0 # reversed flux! The Fd + NADH hydrogenase
                    model.reactions.HYDFDN2r.lower_bound = 0; model.reactions.HYDFDN2r.upper_bound = 0 # the Fd + NADPH hydrogenase
                else:
                    model.reactions.HYDFDN.lower_bound = 0; model.reactions.HYDFDN.upper_bound = 0 # reversed flux! The Fd + NADH hydrogenase
                    model.reactions.HYDFDN2r.lower_bound = -1000; model.reactions.HYDFDN2r.upper_bound = 1000 # the Fd + NADPH hydrogenase

                if MTHFD == 0:
                    model.reactions.MTHFD.upper_bound = 0; model.reactions.MTHFD_alt.upper_bound = 1000
                else:
                    model.reactions.MTHFD.upper_bound = 1000; model.reactions.MTHFD_alt.upper_bound = 0

                if MTHFR == 0:
                    model.reactions.MTHFR5.upper_bound = 0; model.reactions.MTHFR5_alt.upper_bound = 1000
 ...


I am working on the file "Code/all_analysis.ipynb".
The first task is:
Show that the entire Wood-Ljungdahlii pathway and the hydrogenases are needed to produce acetate
To show that the entire Wood-Ljungdahl pathway needs to be present we sequentially knock out one step in the pathway and ask the model to produce acetate from CO2/H2.
If the result is zero, acetate cannot be produced. If it is non-zero acetate may still be synthesized and the considered reaction is not predicted to be essential.
From the resulting table we see that almost all reactions are essential: the exception is the formate dehydrogenase since it may be generated from pyruvate (see the network drawing)
Can you write Python code for it?

list_of_knockouts = ['FDH7','FTHFLi','MTHFC',\
                    'MTHFD','MTHFR5','METR','CODH_ACS']
results = {}
for rxn in list_of_knockouts:
    model = M.copy()
    model.reactions.EX_ac_e.lower_bound = 0
    model.objective = model.reactions.EX_ac_e
    model.reactions.get_by_id(rxn).lower_bound = 0; model.reactions.get_by_id(rxn).upper_bound = 0
    results[rxn] = round(abs(model.optimize().objective_value),4)

pd.DataFrame(results.values(),index=results.keys(),columns=['Acetate flux'])


I am working on the file "Code/classificationANDregression_simulation_AL.ipynb".
The first task is:
Steps 4 & 5: Sample data from setting similar to data and record classification accuracy
Can you write Python code for it?

np.random.seed(12345678)  # for reproducibility, set random seed
r = 20  # define number of rois
N = 100 # number of samples at each iteration
p0 = 0.10
p1 = 0.15
# define number of subjects per class
S = np.array((8, 16, 20, 32, 40, 64, 80, 100, 120, 200, 320,
              400, 600))
S = np.array((200,300))

names = ["Linear Regression",
         "Support Vector Regression",
         "Nearest Neighbors",
         "Random Forest"] #, "Polynomial Regression"]

regressors = [
    LinearRegression(),
    SVR(kernel="linear", C=0.5, epsilon=0.01),
    KNeighborsRegressor(6, weights="distance"),
    RandomForestRegressor(max_depth=5, n_estimators=10, max_features=1)
    ]

errors = np.zeros((len(S), len(regressors), 2), dtype=np.dtype('float64'))

# sample data accordingly for each # of simulations
for idx1, s in enumerate(S):
    # null regression y = a + bx + episilon
    X = np.arange(0, s)
#     epsilon = np.random.normal(0, 0.5, s)
    eps_x = np.random.normal(0, 0.5, s)
    a = np.random.rand(s,)
    b = np.random.normal(s,)*5
    y = a + b*(X+eps_x)
    X, y, coef = datasets.make_regression(n_samples=s, n_features=1,
                                      n_informative=1, noise=10,
                                      coef=True, random_state=0)

    # reshape array to make it work with linear regression
#     y = y.reshape(-1,1)
    X = X.reshape(-1,1)

    for idx2, regr in enumerate(regressors):
        X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.4, random_state=0)

        # Train the model using the training sets
        reg = regr.fit(X_train, y_train)

        # leave one out analysis
#         loo = LeaveOneOut(len(X))
        loo = cross_validation.KFold(n=len(X), n_folds=10, shuffle=False, random_state=None)

        # compute scores for running this regressor
        scores = cross_validation.cross_val_score(reg, X, y, scoring='mean_squared_error', cv=loo)

        errors[idx1, idx2,] = [scores.mean(), scores.std()]
        print("MSE...


I am working on the file "Code/exploratory_analysis_AL.ipynb".
The first task is:
Histograms of each Column (X,Y,Z)
To determine the distributions of x,y,z coordinates, I want to plot the histograms and possibly violin plots to visualize.
Another good way of determining distributions of the features would be to plot a histogram of each 24 protein expressions in the 4 different feature metrics.
Not sure what a good way of looking at this data is...
Can you write Python code for it?

# load in volume data
list_of_locations = []
with open('../data_normalized/locations_normalized.txt') as file:
    for line in file:
        inner_list = [float(elt.strip()) for elt in line.split(',')]

        # create list of features
        list_of_locations.append(inner_list)

# conver to a numpy matrix
list_of_locations = np.array(list_of_locations)
print list_of_locations.shape

# 01: normalize
x_locations = list_of_locations[:,0]
y_locations = list_of_locations[:,1]
z_locations = list_of_locations[:,2]

fig = plt.figure(figsize=(10,20))
fig.add_subplot(311)
plt.hist(x_locations, facecolor='black', alpha=0.75)
plt.subplots_adjust(hspace=0.35)
plt.title("Locations of Synapse In Image Volume Along X Axis", fontsize=20)
plt.grid(False)
plt.xlabel("Normalized Image Location", fontsize=20)
plt.ylabel("Frequency of Occurrence", fontsize=20)
plt.xticks(fontsize=16)
plt.yticks(fontsize=16)
plt.ticklabel_format(style='sci', scilimits=(0,0))

fig.add_subplot(312)
plt.hist(y_locations, facecolor='black', alpha=0.75)
plt.title("Locations of Synapse In Image Volume Along Y Axis", fontsize=20)
plt.grid(False)
plt.xlabel("Normalized Image Location", fontsize=20)
plt.ylabel("Frequency of Occurrence", fontsize=20)
plt.xticks(fontsize=16)
plt.yticks(fontsize=16)
plt.ticklabel_format(style='sci', scilimits=(0,0))

fig.add_subplot(313)
plt.hist(z_locations, facecolor='black', alpha=0.75)
plt.title("Locations of Synapse In Image Volume Along Z Axis", fontsize=20)
plt.grid(False)
plt.xlabel("Normalized Image Location", fontsize=20)
plt.ylabel("Frequency of Occurrence", fontsize=20)
plt.xticks(fontsize=16)
plt.yticks(fontsize=16)
plt.ticklabel_format(style='sci', scilimits=(0,0))


I am working on the file "Code/exploratory_analysis_AL.ipynb".
The first task is:
Histograms of each Column For F0
To determine the distributions of x,y,z coordinates, I want to plot the histograms and possibly violin plots to visualize.
Data description from:
https://github.com/Upward-Spiral-Science/data/tree/master/syn-diversity
F0 is the feature metric of integrated brightness
Can you write Python code for it?

# create figure
fig = plt.figure(figsize=(20, 100))

width = f0_features.shape[1]
# loop through each 24 features for this specific metric
for i in range(0, width):
    feature = f0_features[:, i]
    fig.add_subplot(width/2, 2, i+1)
    plt.hist(feature)
#     plt.title("X Locations Histogram")
    plt.xlabel("Integrated Brightness")
    plt.ylabel("Frequency")

plt.tight_layout()


I am working on the file "Code/exploratory_analysis_AL.ipynb".
The first task is:
Missing Values?

There are no missing values based on our csv read function
These are all numbers
Can you write Python code for it?

def find_first(item, vec):
    """return the index of the first occurence of item in vec"""
    for i in xrange(len(vec)):
        if item == vec[i]:
            return i
    return -1

min_location = []
max_location = []

x_range = list_of_locations[:, 0]
y_range = list_of_locations[:, 1]
z_range = list_of_locations[:, 2]

mins = [min(x_range), min(y_range), min(z_range)]
maxs = [max(x_range), max(y_range), max(z_range)]

# find min/max values and where they occur
min_location = [
    {min(x_range): find_first(min(x_range), x_range)},
    {min(y_range): find_first(min(y_range), y_range)},
    {min(z_range): find_first(min(z_range), z_range)}
]
max_location = [
    {max(x_range): find_first(max(x_range), x_range)},
    {max(y_range): find_first(max(y_range), y_range)},
    {max(z_range): find_first(max(z_range), z_range)}
]
# max_location = [max(x_range), max(y_range), max(z_range)]

# print some basic stats about the list of features and locations
print "The size of the feature matrix is: ", list_of_features.shape
print "The size of the location matrix is: ", list_of_locations.shape

print "min and max are:"
print mins
print maxs

# for i in range(0, len(min_location)):
#     print json.dumps(min_location[i], indent=4, separators=(',', ': '))
#     print json.dumps(max_location[i], indent=4, separators=(',', ': '))


I am working on the file "Code/exploratory_analysis_round2.ipynb".
The first task is:
Correlation Matrix of All Protein Expressions For Each Measurement
Can you write Python code for it?

fig = plt.figure(figsize=(20,30))

index = 1
for key in features.keys():
    # Compute feature correlation matrix
    R = np.corrcoef(features[key],rowvar=0)
    # R_normalize = np.corrcoef(normalize_features, rowvar=0)
#     fig = plt.figure(figsize=(10,10))
    plt.subplot(len(features.keys())/2, 2, index)
    plt.imshow(R, cmap=plt.get_cmap('jet'), interpolation='none')
    plt.title("Correlation plot of all features f0, f1, f2, f3")
    plt.colorbar()

    plt.xticks(np.arange(0,4, 1), ['f0', 'f1', 'f2', 'f3'])
    plt.yticks(np.arange(0,4, 1), ['f0', 'f1', 'f2', 'f3'])
    plt.title(key)
    ax = plt.gca()
    ax.grid(False)
    xmin = ax.get_xlim

    index += 1
plt.tight_layout()


I am working on the file "Code/exploratory_analysis_round2.ipynb".
The first task is:
Kernel Density Estimation
? How to run kernel density estimation ?
Can you write Python code for it?

fig = plt.figure(figsize=(10,10))
for i in range(0, new_list_of_features.shape[1]):
    kde = KernelDensity().fit(new_list_of_features[:,i])
    log_dens = kde.score_samples(np.linspace(-5, 5, new_list_of_features.shape[0]))
    plt.plot(log_dens, label="kernel = gaussian")

from scipy.stats import norm
# Plot a 1D density example
N = 100
np.random.seed(1)
X = np.concatenate((np.random.normal(0, 1, 0.3 * N),
                    np.random.normal(5, 1, 0.7 * N)))[:, np.newaxis]

print X.shape

X_plot = np.linspace(-5, 10, 1000)[:, np.newaxis]

print X_plot.shape
true_dens = (0.3 * norm(0, 1).pdf(X_plot[:, 0])
             + 0.7 * norm(5, 1).pdf(X_plot[:, 0]))

fig, ax = plt.subplots()
ax.fill(X_plot[:, 0], true_dens, fc='black', alpha=0.2,
        label='input distribution')

for kernel in ['gaussian']:
    kde = KernelDensity(kernel=kernel, bandwidth=0.5).fit(X)
    log_dens = kde.score_samples(X_plot)
    ax.plot(X_plot[:, 0], np.exp(log_dens), '-',
            label="kernel = '{0}'".format(kernel))

ax.text(6, 0.38, "N={0} points".format(N))

ax.legend(loc='upper left')
ax.plot(X[:, 0], -0.005 - 0.01 * np.random.random(X.shape[0]), '+k')

ax.set_xlim(-4, 9)
ax.set_ylim(-0.02, 0.4)
plt.show()


I am working on the file "Code/inference_tyler.ipynb".
The first task is:
Load data, downsample, and keep only first 96 features
Can you write Python code for it?

with open('/Users/Tyler/Google Drive/DataScience/synapse_features_z-score.csv','r') as csvfile:
    reader = csv.reader(csvfile)
    X = np.array([[float(e) for e in r] for r in reader])

# Keep only first 96 features
X = X[np.random.choice(range(X.shape[0]),size=100000,replace=False),0:24*4]
#X = X[:,0:24*4]
print 'data loaded'


I am working on the file "Code/inference_tyler.ipynb".
The first task is:
Perform model selection using BIC to find the most likely number of mixture components $\hat{k}$
Can you write Python code for it?

lowest_bic = np.infty
bic = []
n_components_range = range(1, 21)
cv_types = ['spherical', 'tied', 'diag', 'full']
for cv_type in cv_types:
    for n_components in n_components_range:
        # Fit a mixture of Gaussians with EM
        gmm = mixture.GMM(n_components=n_components, covariance_type=cv_type)
        gmm.fit(X)
        bic.append(gmm.bic(X))
        if bic[-1] < lowest_bic:
            lowest_bic = bic[-1]
            best_gmm = gmm

bic = np.array(bic)
color_iter = itertools.cycle(['k', 'r', 'g', 'b', 'c', 'm', 'y'])
clf = best_gmm
bars = []

# Plot the BIC scores
spl = plt.subplot(1, 1, 1)
for i, (cv_type, color) in enumerate(zip(cv_types, color_iter)):
    xpos = np.array(n_components_range) + .2 * (i - 2)
    bars.append(plt.bar(xpos, bic[i * len(n_components_range):
                                  (i + 1) * len(n_components_range)],
                        width=.2, color=color))
plt.xticks(n_components_range)
plt.ylim([bic.min() * 1.01 - .01 * bic.max(), bic.max()])
plt.title('BIC score per model')
xpos = np.mod(bic.argmin(), len(n_components_range)) + .65 +\
    .2 * np.floor(bic.argmin() / len(n_components_range))
plt.text(xpos, bic.min() * 0.97 + .03 * bic.max(), '*', fontsize=14)
spl.set_xlabel('Number of components')
spl.legend([b[0] for b in bars], cv_types)
plt.show()


I am working on the file "Code/inference_tyler.ipynb".
The first task is:
Statistical inference
Step 1: Define model and assumptions
$\vec{X} ~ f_{\vec{X}} \in {F_{\vec{X}}(\cdot;\theta): \theta \in \Theta}$
We assume $f$ is a GMM and $\theta = [\bf{\mu}, \bf{\Sigma}, \vec{\pi}, k]$, where k is the number of mixture components and $\vec{\pi}$ are the mixing weights of each mixture component.
Step 2: Formalize test
$H_0: k = k_0$
$H_1: k = k_1$
Step 3: Describe the test statistic
$\Lambda = \frac{L(\theta_1;X)}{L(\theta_0;X)}$
Can you write Python code for it?

def gmm_test(X,k0,k1,nboot):
    nsample = X.shape[0]

    gmm0 = mixture.GMM(n_components=k0, covariance_type='full')
    gmm0.fit(X)
    L0 = sum(gmm0.score(X))
    gmm1 = mixture.GMM(n_components=k1, covariance_type='full')
    gmm1.fit(X)
    L1 = sum(gmm1.score(X))
    LRstat = -2*(L1 - L0)

    LRstat0 = []

    for i in range(nboot):
        Xboot = gmm0.sample(n_samples=nsample)
        gmm0_boot = mixture.GMM(n_components=k0, covariance_type = 'full')
        gmm0_boot.fit(Xboot)
        L0_boot = sum(gmm0_boot.score(Xboot))
        gmm1_boot = mixture.GMM(n_components=k1, covariance_type = 'full')
        gmm1_boot.fit(Xboot)
        L1_boot = sum(gmm1_boot.score(Xboot))
        LRstat0.append(-2*(L1_boot - L0_boot))

    ecdf, lowlim, binsize, extrapoints = cumfreq(LRstat0)
    ecdf = ecdf/len(LRstat0)

    bin = np.mean([lowlim,lowlim+binsize])
    bins = []

    for i in range(len(ecdf)):
        bins.append(bin)
        bin = bin + binsize

    if min(bins) > LRstat:
        p = 0
    else:
        p = max(ecdf[bins<=LRstat])

    return p


I am working on the file "Code/inference_tyler.ipynb".
The first task is:
Step 4b: Sample from the alternative
Can you write Python code for it?

pow_alt = np.array((), dtype=np.dtype('float64'))

gmm1 = mixture.GMM(n_components=k1, covariance_type='full')
gmm1.means_ = np.array([[-2],[0],[2]])
gmm1.covars_ = np.array([[[1]],[[1]],[[1]]])
gmm1.weights_ = np.array([.4, .2, .4])

for n in n_samples:
    p = np.array((), dtype=np.dtype('float64'))

    for i in range(n_iterations):
        X1 = gmm1.sample(n)
        p = np.append(p,gmm_test(X1,k0,k1,nboot))

    pow_alt = np.append(pow_alt, np.sum(1.0*(p < alpha))/n_iterations)

print 'finished sampling from alternative'


I am working on the file "Code/inference_tyler.ipynb".
The first task is:
Step 6: Apply test to actual data
Can you write Python code for it?

k0 = 1
k1 = 17
nboot = 100

p = gmm_test(X,k0,k1,nboot)

print p


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
After running our request, we get a Response object. This object has a status_code property, which indicates if the page was downloaded successfully.

A status_code of 200 means that the page downloaded successfully.
   * status code starting with a 2 generally indicates success
   * a code starting with a 4 or a 5 indicates an error.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
Check tab completion
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
Define a dictionary for the parameters we want to pass to the GET request for NWS server.
Can you write Python code for it?

keys = {}
for location, coordinate in zip(locations, coordinates):
    keys[location] = {'lat' : coordinate[0], 'lon' : coordinate[1]}

keys


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
Extracting information from a web page
Now that we have the content of the web page we want to extraxt certain information. BeautifulSoup is a Python package which helps us in doing that. See the documentation for more information.
We first have to import the library, and create an instance of the BeautifulSoup class to parse our document:
Can you write Python code for it?

from bs4 import BeautifulSoup

BeautifulSoup?

cl_soup = BeautifulSoup(cl.content, 'html.parser')


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
If it does not find the price, it returns a NoneType. We might exploit this fact to select only the valid links.
Can you write Python code for it?

import bs4

data = [[ad.find('a', class_='result-title hdrlnk').get_text(),
         ad.find('a', class_='result-title hdrlnk')['data-id'],
         ad.find('span', class_='result-price').get_text()] for ad in ads
            if type(ad.find('span', class_='result-price'))==bs4.element.Tag]

data[:10]

df = pd.DataFrame(data)

df.head(10)

df.shape


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
Print this out in a prettier way.
Can you write Python code for it?

#print(cl_soup.prettify())

print('Type:', type(cl_soup))

# we can access a tag
print('Title: ', cl_soup.title)

# or only the text content
print('Title: ', cl_soup.title.text)          # or
print('Title: ', cl_soup.title.get_text())


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
Recall the format of the url associated with a particular location
http://forecast.weather.gov/MapClick.php?lat=32.38092788700044&lon=-106.4794398029997
Can you write Python code for it?

keys[locations[0]]

url = 'http://forecast.weather.gov/MapClick.php'
nws = requests.get(url, params=keys[locations[0]])

nws.status_code

nws.url

nws.content[:300]

nws_soup = BeautifulSoup(nws.content, 'html.parser')

seven = nws_soup.find('div', id='seven-day-forecast-container')

seven.find(text='Friday')

seven.find(text='Friday').parent

seven.find(text='Friday').parent.parent

seven.find(text='Friday').parent.parent.find('p', class_='temp temp-high').get_text()

data = []

for location in locations:

    nws = requests.get(url, params=keys[location])
    nws_soup = BeautifulSoup(nws.content, 'html.parser')

    seven = nws_soup.find('div', id='seven-day-forecast-container')
    temp = seven.find(text='Friday').parent.parent.find('p', class_='temp temp-high').get_text()
    data.append([location, temp])

df_weather = pd.DataFrame(data, columns=['Location', 'Friday weather'])

df_weather

df_weather['high_temp'] = df_weather['Friday weather'].str.rsplit().str.get(1).astype(float)

df_weather['high_temp'].std()


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
This is a lot of information and we want to extract some part of it. Use the text or get_text() method to get the text content.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
To print out the content of the html file, use the content or text properties
This is going to be ugly and unreadable
Can you write Python code for it?

cl.text[:300]

cl.content[:500] # this works also for information which is not purely text


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We can check whether it's working fine at http://www.latlong.net/
Can you write Python code for it?

locations = ['Bozeman, Montana', 'White Sands National Monument', 'Stanford University, California']

coordinates = [Geocoder.geocode(location).coordinates for location in locations]

coordinates

for location, coordinate in zip(locations, coordinates):
    print('The coordinates of {} are:'.format(location), coordinate)


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We can find all tags of certain type with the find_all method. This returns a list.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We can see if we start clicking on the 2nd nd 3rd pages of the results that there is a structure in how they are defined
First page:
https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0
Second page:
https://newyork.craigslist.org/search/roo?s=120&availabilityMode=0&query=east%20village
Third page:
https://newyork.craigslist.org/search/roo?s=240&availabilityMode=0&query=east%20village
The number after roo?s= in the domain specifies where the listings are starting from (not inclusive). In fact, if we modify it ourselves we can fine-tune the page starting from the corresponding listing and then showing 120 listings. Try it!
We can also define the first page by puttig s=0& after roo? like this:
https://newyork.craigslist.org/search/roo?s=0&availabilityMode=0&query=east%20village
Can you write Python code for it?

# First we get the total number of listings in real time
url = 'https://newyork.craigslist.org/search/roo?query=east+village&availabilityMode=0'
cl = requests.get(url)

cl_soup = BeautifulSoup(cl.content, 'html.parser')
total_count = int(cl_soup.find('span', class_='totalcount').get_text())
print(total_count)


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We could do text anaylsis and see what words are common in ads which has a relatively higher price.
This approach is not really efficient because it only gets the first page of the search results. We see on the top of the CL page the total number of listings. In the Inspection mode we can pick an element from the page and check how it is defined in the html -- this is useful to get tags and classes efficiently.
For example, the total number of ads is a span tag with a 'totalcount' class.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We have the total number of listings with the given search specification. Breaking down the steps:
1) Specify the url of each page we want to scrape
2) For each page scrape the data -- we will reuse the code what we already have for one page
3) Save the data into one dataframe -- we can use the append method for DataFrames or the extend method for lists
Can you write Python code for it?

# 1) Specify the url
for page in range(0, total_count, 120):
    print('https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page))

# Next we write a loop to scrape all pages

df = pd.DataFrame({'Title' : [], 'ID' : [], 'Price' : []})

for page in range(0, total_count, 120):
    url = 'https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page)

    cl = requests.get(url)
    cl_soup = BeautifulSoup(cl.content, 'html.parser')

    ads = cl_soup.find_all('li', class_='result-row')
    data = pd.DataFrame([[ad.find('a', class_='result-title hdrlnk').get_text(),
                          ad.find('a', class_='result-title hdrlnk')['data-id'],
                          ad.find('span', class_='result-price').get_text()] for ad in ads
                                 if type(ad.find('span', class_='result-price'))==bs4.element.Tag],
                        columns=['Title', 'ID', 'Price'])

    df = df.append(data, ignore_index=True)

df.head()

# Do the same using the `extend` method

data = []
for page in range(0, total_count, 120):
    url = 'https://newyork.craigslist.org/search/roo?s={}&availabilityMode=0&query=east%20village'.format(page)
    cl = requests.get(url)
    cl_soup = BeautifulSoup(cl.content, 'html.parser')
    ads = cl_soup.find_all('li', class_='result-row')
    data_page = [[ad.find('a', class_='result-title hdrlnk').get_text(),
         ad.find('a', class_='result-title hdrlnk')['data-id'],
         ad.find('span', class_='result-price').get_text()] for ad in ads
        if type(ad.find('span', class_='result-price'))==bs4.element.Tag]
    data.extend(data_page)

df = pd.DataFrame(data, columns=['Title', 'ID', 'Price'])

df.head()

df.shape

df.tail()


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
We only have 118 listing because 2 listings did not have a price.
Can you write Python code for it?

df.columns = ['Title', 'ID', 'Price']

df.head()


I am working on the file "Code/notebooks/bootcamp_adv_scraping.ipynb".
The first task is:
What's going wrong? Some ads don't have a price listed, so we can't retrieve it.
Can you write Python code for it?

# if it exists then the type is
type(ads[0].find('span', class_='result-price'))


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
As per usual, we begin by listing the versions of each package that is used in this notebook.
Can you write Python code for it?

# check versions (overkill, but why not?)
print('Python version:', sys.version)
print('Pandas version: ', pd.__version__)
print('Matplotlib version: ', mpl.__version__)
print('Seaborn version: ', sns.__version__)


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Exercise: Find a style you like and recreate the plot above using that style.
The Juicy Stuff
While having seaborn set sensible defaults is convenient, it isn't a particularly large innovation. We could choose sensible defaults and set them to be our default. The main benefit of seaborn is the types of graphs that it gives you access to -- All of which could be done in matplotlib, but, instead of 5 lines of code, it would require possibly hundreds of lines of code. Trust us... This is a good thing.
We don't have time to cover everything that can be done in seaborn, but we suggest having a look at the gallery of examples.
We will cover:

kdeplot
jointplot
violinplot
pairplot
...
Can you write Python code for it?

# Move back to seaborn defaults
sns.set()


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
Exercise: We might also want to look at the distribution of prices across ticket classes. Make a violin plot of the prices over the different ticket classes.
Pairplot
Pair plots show us two things. They show us the histograms of the variables along the diagonal and then the scatter plot of each pair of variables on the off diagonal pictures.
Why might this be useful? It allows us to look get an idea of the correlations across each pair of variables and gives us an idea of their relationships across the variables.
Can you write Python code for it?

sns.pairplot(tips[["tip", "total_bill", "size"]], size=3.5)


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
We could do the same for a different style (like ggplot)
Can you write Python code for it?

plt.style.use("ggplot")

fig, ax = plt.subplots(2)

tips[tips["sex"] == "Male"].plot(x="total_bill", y="tip", ax=ax[0], kind="scatter",
                                 color="blue")
tips[tips["sex"] == "Female"].plot(x="total_bill", y="tip", ax=ax[1], kind="scatter",
                                   color="#F52887")

ax[0].set_xlim(0, 60)
ax[1].set_xlim(0, 60)
ax[0].set_ylim(0, 15)
ax[1].set_ylim(0, 15)

ax[0].set_title("Male Tips")
ax[1].set_title("Female Tips")

fig.tight_layout()


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
What did you notice about the differences in the settings of the plot?
Which do you like better? We like the second better.
Investigate other styles and create the same plot as above using a style you like. You can choose from the list in the code below.
If you have additional time, visit the seaborn docs and try changing other default settings.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
What's different about this plot?
Different colors for each company.
Can you write Python code for it?

tips.head()

sns.pairplot(tips[["tip", "total_bill", "size", "time"]], size=3.5, diag_kind="kde",
             hue="time")


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
kdeplot
What does kde stand for?
kde stands for "kernel density estimation." This is (far far far) beyond the scope of this class, but the basic idea is that this is a smoothed histogram. When we are trying to get information about distributions it sometimes looks nicer than a histogram does.
Can you write Python code for it?

fig, ax = plt.subplots()

ax.hist(tips["tip"], bins=25)

ax.set_title("Histogram of tips")
plt.show()

fig, ax = plt.subplots()

sns.kdeplot(tips["tip"], ax=ax)
ax.hist(tips["tip"], bins=25, alpha=0.25, normed=True, label="tip")
ax.legend()

fig.suptitle("Kernel Density with Histogram")

plt.show()


I am working on the file "Code/notebooks/bootcamp_advgraphics_seaborn.ipynb".
The first task is:
swarmplot
Sometimes we simply have too much data. One approach to visualizing all the data is to adjust features like the point size or transparency.
An alternative is to use a swarm plot. This is best understood by example, so let's dive in!
Can you write Python code for it?

fig, ax = plt.subplots()
sns.swarmplot(data=tips, x="day", y="total_bill")


I am working on the file "Code/notebooks/bootcamp_exam_practice.ipynb".
The first task is:
Question 3. These get progressively more difficult:

What type is dollars = '$1,234.5'?
Find and apply a method that eliminates the dollar sign from dollars.
Find and apply a method that eliminates the comma from dollars.
Eliminate both the dollar sign and comma from dollars and covert the result to a float.
Do part 4, but combined in just one line.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Question 4.
For this problem we set letters = 'abcd' as in problem 2.

Find and apply a method that converts the lower case letter 'a' to the upper case letter 'A'.
Write a loop that goes through the elements of letters and prints their upper case versions.
Bonus: Write a loop that goes through the elements of letters.  On each interation, print a string consisting of the upper and lower case versions together; eg, 'Aa'.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Question 5.
For this problem xyz is the same as defined in problem 2 (parts 1, 2, 3, and 7)

Write a loop that goes through the elements of xyz and prints them.
Modify the loop to print both the elements of xyz and their type.
Modify the loop to print only those elements that are not strings.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Data input with Pandas
We explore the public indebtedness of Argentina (country code ARG), Germany (DEU), and Greece (GRC).  For each one, we provide the ratio of government debt to GDP for every second year starting in 2002.  The data come from the IMF's World Economic Outlook.
Question 6. Write code in the cell below that reads the csv file we posted at
http://pages.stern.nyu.edu/~dbackus/Data/debt.csv
Assign the contents of the file to the object debt.
The rest of the questions in this notebook will refer to the object debt you create below.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
IPython review
We review some of the basics of IPython.  You won't be asked about IPython on the exam, but since the exam is an IPython notebook, it's essential for you to be able to work with one.
Question 1.

How do you set/choose the current cell?
How do you edit the current cell?
How do you add a new cell below the current cell?
How do you specify the current cell as code or text?
How do you delete the current cell?
How do you move the current cell up or down?
How do you run the current cell?
Add your name in bold to the bottom of the first cell in this notebook.  Bonus: Add a link to your LinkedIn or Facebook page.
How do you save the contents of your notebook?

Answers.  Enter your answers below:

click on it
click again
click on the plus (+) at the top
choose the appropriate one in the menu below Help at the top
choose the cell and click on the scirrors at the top
choose the cell and click on the up or down arrow at the top
two ways:  click on the run cell icon at the top, or shift-enter
**name**
two ways:  File, Save and Checkpoint, or cntl-S.

Python fundamentals
Question 2. Describe the type and content of these expressions:

x = 2
y = 3.0
z = "3.0"
x/y
letters = 'abcd'
letters[-1]
xyz = [x, y, z]
xyz[1]
abcd = list(letters)
abcd[-2]
case = {'a': 'A', 'b': 'B', 'c': 'C'}
case['c']
2 >= 1
x == 2

Answers. Enter your answers below:
By content and type we mean the content of the variable or expression and its type as return by the type() function.

content 2, type int
content 3.0, type float
content '3.0', type str
content 0.66666, type float
content 'abcd', type str
content 'd', type str
content [x, y, z], type list
content y=3.0, type float
content ['a', 'b', 'c', 'd'], type list
content 'c', type str
content as stated, type dictionary or dict
content 'C', type str
content True, type bool
content True, type bool
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Optional challenging questions
Good practice, but more than you'll see on the exam.
Question 10.  In the figure of the previous question:

Add a title, 14-point font, right-justified.
Put the legend in the lower left corner.
Change the line style to dashed.  (This will take some Googling, or a good guess.)
Eliminate the top and right "spines," the lines that outline the figure. [This doesn't make sense with the 538 style, which eliminates all the spines.]
Save the figure as a pdf file.
Change the style to 538.
Can you write Python code for it?

plt.style.use('fivethirtyeight')
fig, ax = plt.subplots()
debt.plot(ax=ax,
         linewidth=2,
        linestyle='dashed',
         color=['red', 'green', 'blue'])
ax.set_title('Public debt', fontsize=14, loc='right')
ax.set_ylabel('Public Debt (% of GDP)')
ax.legend(loc='lower left')
fig.savefig('debt.pdf')


I am working on the file "Code/notebooks/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Python graphics with Matplotlib
We'll continue to use the data in debt.  Make sure the index is the year.
Question 9.

Create figure and axis objects with plt.subplots().
Graph public indebtedness over time using our debt data and the axis object we just created.
Change the line width to 2.
Change the colors to ['red', 'green', 'blue'].
Change the lower limit on the y axis to zero.
Add a title to the graph.
Add a label to the y axis -- something like "Public Debt (% of GDP)".
Bonus: Make the line for Argentina thicker than the others.  Hint: Do this by plotting a separate line applied to the same axis object.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Can you write Python code for it?

fig, ax = plt.subplots()
debt.plot(ax=ax,
         linewidth=2,
         color=['red', 'green', 'blue'])
ax.set_ylim(0)
ax.set_title('Public debt')
ax.set_ylabel('Public Debt (% of GDP)')
debt['Argentina'].plot(ax=ax, linewidth=4, color='red')


I am working on the file "Code/notebooks/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Question 11.  We ran across this one in the OECD healthcare data.  The country names had numbers appended, which served as footnotes in the original spreadsheet but looked dumb when we used them as index labels.  The question is how to eliminate them.  A short version of the country names is
names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']
Do each of these in a separate code cell:

Apply the rsplit() method to us = names[-1].  What do you get?
Consult the documentation for rsplit to split us into two pieces, the country name and the number 1.  How would you extract just the country name?
Use a loop to strip the numbers from all of the elements of names.
Use a list comprehension to strip the numbers from all of the elements of names.

Hints. rsplit means split from the right.  One input is the number of splits.
Can you write Python code for it?

names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']

names[-1].rsplit()

names[-1].rsplit(maxsplit=1)

# apologies, this is harder than we thought
for n in range(len(names)):
    item = names[n]
    names[n] = item.rsplit(maxsplit=1)[0]

print(names)

# this one's easier
names = ['Australia 1', 'Canada 2', 'Chile 3', 'United States 1']
[item.rsplit(maxsplit=1)[0] for item in names]


I am working on the file "Code/notebooks/bootcamp_exam_practice_answerkey.ipynb".
The first task is:
Question 3. These get progressively more difficult:

What type is dollars = '$1,234.5'?
Find and apply a method that eliminates the dollar sign from dollars.
Find and apply a method that eliminates the comma from dollars.
Eliminate both the dollar sign and comma from dollars and covert the result to a float.
Combine the last three steps in one line.

In each case, create a code cell that delivers the answer. Please write the question number in a comment in each cell.
Can you write Python code for it?

dollars = '$1,234.5'
type(dollars)

dollars = dollars.replace('$','')
dollars

dollars = dollars.replace(',','')
dollars

dollars = float(dollars)
dollars

# we can glue the pieces together
dollars = '$1,234.5'
dollars = float(dollars.replace('$','').replace(',',''))
dollars


I am working on the file "Code/notebooks/bootcamp_examples.ipynb".
The first task is:
Comment. Chase likes heatmaps.   The one that follows needs work, but it's an attempt to reproduce the previous graph as a heatmap, where the value of the variable is represented by the color and its intensity.  Dark red:  large postive value.  Dark blue:  large negative value.  Light colors:  small values.
Can you write Python code for it?

import seaborn as sns      # graphics packages
fig, ax = plt.subplots()
sns.heatmap(g_std, linewidths=.5, ax=ax)
ax.set_yticklabels(g_std.index, visible=False);